Fresh Silverblue 41 install entering emergency mode during first boot in Silverblue after entering first luks password

Ask Fedora
, , , ,
1

Hi, I’m new here and I’d like to make a question, but first, I should explain what exactly happened:

As a workstation user, I’m learning how to use Silverblue so I can switch to it in the future. This of course had me installing and reinstalling the OS a couple times in order to see which partitioning would better suit me, Ended up coming with a partitioning that at the time, I had no issues with it. looked similar to something like this:
USB drive: 50MiB (Mebibytes) /boot/efi | 1024MiB Ext4 /boot
SSD: 50GB Btrfs root (luks2 password 1) | Remaining.GB Btrfs Home (luks2 password 2)

Then I thought to myself: 'Why not use ext4 instead of btrfs? The only feature of Btrfs I really used was snapshots to rollback, but ostree already does that much more easily. Also, I’ve once read that Btrfs copy on write (CoW) negatively impacts performance of Virtual Machines, and Ext4, unlike Btrfs, supports case folding fs/folders, which is something I want to experiment since it’s useful for windows programs on wine that need it.

So back to the installer, I reinstalled Fedora with the exact same except using Ext4 on root and /home instead of Btrfs. This is where problems started, as upon entering the passwords, instead of launching as normal, system enters emergency mode all the time, sometimes the password prompt (forgot the name of it) would only ask for one of the partition’s password and then enter emergency mode.
I found this strange, as I didn’t have this issue with Workstation before. So I tried changing the same partitioning again, this time root as Btrfs and /home as Ext4. Same issue. Then I gave up on it for the time being and went back to the original partitioning I mentioned that worked before, except that partitioning too had the same error unlike before, which made me even more confused.
I then grabbed another SSD that I knew were working correctly just so I could test if it was a hardware issue (only change being that /boot/efi and /boot are also on this new SSD), but I got the same issue, which is unfortunate (or maybe not since this means my previous SSD is fine, I guess).
I then decided to experiment doing a different experimental partitioning, with the 50GB partition for both /root /home and /var using Btrfs subvolumes, while ignoring the remaining.GB partition, and to my surprise, this worked.

I have absolutely no idea why any of this happened, and I’d like to make a few more tests in the near to see what I can do, and if necessary, find an alternative to make use of the remaining.GB partition, perhaps keeping the entire system (root, /home, /var) on the 50GB partition while mounting the remaining.GB partition during installation somewhere like /home/NAME (not sure what to name nor where to put it to be honest) but I’d like to hear some thoughts on what could I be doing wrong here, as well as maybe hearing some suggestions for what I’m trying to do.

2

Uhm have you tried to just use the automatic partitioning? It works well

Atomic desktops benefit from BTRFS afaik for snapshotting, especially root. But not sure.

There are no big reasons to separate home and root, are there?

3

As mentioned above, automatic partitioning with LUKS might suit your setup well enough.

If it doesn’t, and you’d like to troubleshoot, please post the output from cat /etc/fstab, sudo cat /etc/crypttab[1] and lsblk.

  1. Maybe a mismatch between fstab and crypttab. ↩︎

4

Unfortunately automatic partitioning would not suit me since I specifically need /boot/efi and /boot to be in a separate removable partition like a USB device (the other SSD where everything was in the same device was just for troubleshooting)

I’d like to use Ext4 for /home due to the mentioned concerns (VMs and case folding for wine applications), which automatically requires separating root and /home partitions if I want root to be Btrfs. Another reason it’s also good to have root and /home in separate partitions is because whenever I need to reinstall the system (for whatever reason), I’d need to format root, thus, having /home in a separate partition prevents it from being erased in the process as well.

I will try to check that when I get the time. Also, just for clarification, am I to do this in a separate live USB? Since on the Silverblue system I can’t get past the Luks encryption and thus I can’t check it from there I think.

5

Yes, that would be achieved by booting into a live session, mounting/decrypting the volume(s), chroot-ing, and running the commands.

6

As far as I read about it

Most steam games do not have an actual linux ports, so Windows games run via Proton, and since not every developer structures paths and file/folder names case-strictly, it makes sense, that ext4 that supports “case-folding”, would be the sensible / closest option to a “windows FS”.

So devs write paths incorrectly and this should just crash games. No “experimenting” here.

Can you in short explain the thing with the removable drives and USB? you want a removable boot partition??

7

Make sure you are not hitting the LUKS password issue described in Installing Fedora Silverblue :: Fedora Docs.

8

Alright. Maybe it’s because I never used chroot before and I’m not used to chroot-ing, but I kept getting errors about /bin/bash not existing when I tried to do in a live F41 Workstation session, and I couldn’t find any answers about it online.

Yes, specially with game mods, although I haven’t heard of games straight up crashing due to the lack of case-folding. Also, the reason I say experiment is because I’d like to test it myself even if just for the purpose of learning.

In short, I do it for personal convenience and peace of mind.

I only do this for installations on internal SSDs, since external ones connected via USB since can simply be unplugged from the USB port. But internally connected ones (especially Nvme ones) are too inconvenient to do the same, and thus, I mount the /boot/efi and /boot on an unique external device for each internal SSD to achieve the same result.

As for the actual reasons to do this in the first place:
I use multiple installations in different SSDs and want my Mainboard to only see one boot option during boot, so I don’t accidentally try to boot into the wrong installation and only notice it minutes later.
I don’t want an installation’s unecrypted partition to be physically visible to other installations, and I can’t encrypt those partitions since the former is necessary in order for the boot selection screen to even see it, and while the latter is technically possible, you can’t do it in Anaconda and it’s too inconvenient for me to do for each installation I have, and makes potential re-installations more inconvenient too.

Good to know about this. But this doesn’t seem to be the issue since I can pass the luks unlock screen with the same password if I installed the system using the default partitioning or a similar one as long as I don’t make a separate /home partition.

1 Like
9

Actually, for sharing the contents of /etc/fstab and /etc/crypttab, chroot-ing is not needed.

10

Okay, so I mounted and unlocked the root partition of the SSD in a live session, and I found it had a /root directory, which I suppose it’s the Btrfs subvolume.
What surprised me however was that inside what I assume was the Btrfs subvolume, all directories and subdirectories (except /ostree) were completely empty . There was no /etc directory in /root either.

Aren’t those directories supposed to be populated, or at the very least be symlinked to a populated directory? Because if yes, I’d imagine that would explain why the system would enter emergency mode, but why would the installation media do that? The ISO checksum matches, and it seems to pass the verification test on boot.

11

Actually this is normal for atomic variants, only that I forgot that /etc is also stored in the deployment tree.

An automatic partitioning would reveal the following directories on the root partition:

/home00
/root00
/var

This might be different with custom partitioning though.

You could search for your /etc folders in either of the deployments, following the path:

/<your-mount-point>/<your-partition-name>/root00/ostree/deploy/fedora/deploy/<your-deployment-name>/etc/

12

I tried to reinstall again, using the same names for the subvolumes as you said the automatic partitioning does (root00, home00, var), withh the Ext4 partition unchanged, but it still didn’t work.

In this same failed installation, I checked the fstaband crypttab files in the deployment’s /etc directory, and this is what I found:

> fstab

# [Some commented stuff]
UUID=55888595-6525-461b-bd29-2da5288b92b9 /                       btrfs   subvol=root00,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=e6c62f38-4a76-445b-bacb-bf983fef80ea /boot                   ext4    defaults        1 2
UUID=E916-8EA3          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=55888595-6525-461b-bd29-2da5288b92b9 /home                   btrfs   subvol=home00,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=55888595-6525-461b-bd29-2da5288b92b9 /var                    btrfs   subvol=var,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=9a8f604d-c6a0-40d0-a135-45c7921dfdc1 /var/mnt/Persistent     ext4    defaults,x-systemd.device-timeout=0 1 2

> crypttab

luks-b919822c-b1fe-40be-bca1-f4a9eb1527a5 UUID=b919822c-b1fe-40be-bca1-f4a9eb1527a5 none 
luks-8e6d3448-f4a5-4a19-8cce-120b4d2d40df UUID=8e6d3448-f4a5-4a19-8cce-120b4d2d40df none
13

Did you do a custom partitioning, but kept the default subvolumes? At which point did you add the /var/mnt/Persistent mount point to /etc/fstab?

I did test a new install of F40 using automatic partitioning and encryption, and the only difference I notice is that in /etc/crypttab I have the none discard flags instead of none only.

With the encrypted volume decrypted, could you also post the output of lsblk -f?

14

I added the /var/mnt/Persistent right during installation in the custom Anaconda installer (not the advanced one, the one in the middle between automatic and advanced partitioning) by manually typing /var/mnt/Persistent as the mount point for that partition.

I tried changing my /etc/crypttab to change the none flags to include discard as well, but still didn’t work. In another drive (an internal one), I also tried installing using F40 instead of F41, but it still didn’t work. In another attempt still using the F40 media installer, I decided to not even try mounting the Ext4 partition (the one I previously mounted as /var/mnt/Persistent) but even that also failed. What I cannot understand is that before this issue began, I was capable of installing Silverblue with no issues at all, but when the problems began, the same issue would appear even if I used the same partitioning methods I did when it was working, which I cannot understand at all.

Notes:
(Rest) = Remaining space of the drive
Only including the drive with the failed install (it includes /boot/efi and /boot since it is an externally mounted one, and thus, does not for my needs to have separate USB device for those partitions)
The names inside the square brackets are not the actual partition names

> lsblk -f

sdc                                                                                                           
├─sdc1                                        vfat        FAT16                                   E916-8EA3                              37.3M    25% /run/media/liveuser/E916-8EA3
├─sdc2                                        crypto_LUKS 2                                       b919822c-b1fe-40be-bca1-f4a9eb1527a5                
│ └─luks-b919822c-b1fe-40be-bca1-f4a9eb1527a5 btrfs                        Btrfs                  55888595-6525-461b-bd29-2da5288b92b9   39.7G    12% /run/media/liveuser/[The Btrfs partition with the subvolumes]
├─sdc3                                        ext4        1.0                                     e6c62f38-4a76-445b-bacb-bf983fef80ea  750.4M    16% /run/media/liveuser/e6c62f38-4a76-445b-bacb-bf983fef80ea
└─sdc4                                        crypto_LUKS 2                                       8e6d3448-f4a5-4a19-8cce-120b4d2d40df                
  └─luks-8e6d3448-f4a5-4a19-8cce-120b4d2d40df ext4        1.0              Ext4                   9a8f604d-c6a0-40d0-a135-45c7921dfdc1  (Rest)     0% /run/media/liveuser/[The Ext4 partition]
15

If you are still willing to troubleshoot, you could try fresh installing Silverblue with automatic install method (plus encryption) on a spare external disk that you would be willing to fully reformat. If that works, then step by step making one modification at a time with every new installation might reveal the change causing the issue. It’s time consuming though.

One more thing:
I notice the size of the vfat-formatted sdc1 partition (probably the EFI partition) has a rather small size. Was your disk formatted by any chance as MBR instead of GPT? Are you booting in legacy-BIOS mode instead of UEFI?

16

I’ve been trying to troubleshoot this issue for a few days, and after fully reformatting the external SSD I was using for these tests, I managed to get it working.
I’m not 100% sure, I think it was because I pre-formatted the partitions that would be used for the installation in another system using gnome disks, notably resizing the one intended for root to 50000000000 bytes (since it was just slightly smaller than 50000000000 bytes even though it showed as 50 GBs), and that’s where the issue most likely began.
By deleting all of the partitions created with gnome-disks and letting Anaconda create them this time, the issues appear to be gone for me. Hopefully it stays that way.

Legacy-BIOS was already disabled for me, so I believe it’s UEFI.
For /boot/efi, Anaconda let’s me set it as low as 50 MiB, and while it had always worked super well for me in Fedora Workstation, I raised it slightly to 64 MiB in the extremely unlikely case it somehow had anything to do with the issue.

1 Like