Silverblue updates after 8-16-26 result in emergency mode and failure to mount home when using ext4

Ask Fedora
,
1

I have an issue that started last week, beginning with every update after 8-16-26.

I run Fedora Silverblue with ext4 (personal preference). This has worked fine for me, but starting with the most recent updates, you get dumped into emergency mode.

"You are in emergency mode. After logging in, type “journalctl -xb” to view system logs, “systemctl reboot” to reboot, or “exit” to continue bootup.

Loading any pinned deployment from 8-16-26 or earlier, system boots fine.

Out of an abundance of precaution, checked all the hardware I could, ran overnight memtest twice, and I’ve tried different SSDs and even installing to a USB.

The results are the same. If your system is ext4, the updates after 8-16-26 fail to boot.

After a fresh (completely reformatting the entire disk, letting the Fedora installer automatically create the partitions) install you can boot, because the version on the install ISO predates 8-16-26, but then after upgrading, same issue, fail to boot, dumped to emergency mode.

This is what the journalctl errors look like for the failed boots:

Aug 21 07:33:57 fedora kernel: virt/tdx: TDX not supported by the host platform Aug 21 12:34:05 fedora systemd-fsck[930]: fsck failed with exit status 8. Aug 21 12:34:05 fedora systemd[1]: home.mount: Mount path /home is not canonical (contains a symlink). Aug 21 12:34:05 fedora systemd[1]: Failed to mount home.mount - Home Partition. Aug 21 12:34:05 fedora systemd[1]: home.mount: Mount path /home is not canonical (contains a symlink). Aug 21 12:34:05 fedora systemd[1]: Failed to mount home.mount - Home Partition. Aug 21 12:34:06 fedora kernel: UBSAN: array-index-out-of-bounds in drivers/net/wireless/mediatek/mt76/mt7921/mcu.c:474:15 Aug 21 12:34:06 fedora kernel: index 3 is out of range for type 'void *[3]' Aug 21 12:34:06 fedora systemd-tmpfiles[1045]: "/home" already exists and is not a directory. Aug 21 12:34:06 fedora systemd-tmpfiles[1045]: "/srv" already exists and is not a directory. Aug 21 12:34:06 fedora systemd-tmpfiles[1045]: "/root" already exists and is not a directory. Aug 21 12:34:06 fedora systemd[1]: home.mount: Mount path /home is not canonical (contains a symlink). Aug 21 12:34:06 fedora systemd[1]: Failed to mount home.mount - Home Partition. Aug 21 12:35:12 fedora kernel: watchdog: watchdog0: watchdog did not stop!

If I fresh install and this time choose to format everything in btrfs, then this does not happen. You can update/upgrade as expected, no issues.

I also tested Kinoite out of curiosity, and the same thing happens. It kind of seems like if you want to use ext4, every update in the past week won’t boot.

If you manually install the 8-16-26 update or earlier, but nothing after that, everything works fine and it boots as expected.

Does anyone have any ideas how to fix that?

2

I have the same problem, though I am using XFS for root and home partition.
It is systemd-gpt-auto-generator that creates the home.mount unit, and when run it fails as you saw.
I am not sure where the problem actually is but I got around it by running fdisk and changing the partition type of my home partition from type 42 (Linux home) to type 20 (Linux filesystem). The systemd-gpt-auto-generator reacts on the Type UUID of type 42 but does not do anything with type 20.

Another way to work around the problem is to add kernel parameter systemd.gpt_auto=0.
This will prevent systemd-gpt-auto-generator from running.

Note: These are not a real solutions to the problem and only ways to get things working.

3

This may also cause problems when dual-booting with another Linux distro on the same disk. Even if Silverblue / Kinoite is a standard install with everything on one BTRFS partition, systemd-gpt-auto-generator could still grab the other distro’s home partition and try to mount it.

4

Well, that is correct, though it only happens if the other distribution has a home partition with Type GUID “933ac7e1-2eb4-4f13-b844-0e14e2aef915”.
More info here: https://man.archlinux.org/man/systemd-gpt-auto-generator.8

As far as I understand systemd-gpt-auto-generator should not even try to generate a home.mount unit because it is already specified in the fstab.

5

Yes, agreed.

Another workaround (instead of changing the partition type) is to use gdisk to set bit 63 on the other-distro-home-partition’s attributes (i.e. set SD_GPT_FLAG_NO_AUTO), so systemd-gpt-auto-generator will ignore it.

Of course, that only works if the other distro doesn’t rely on systemd-gpt-auto-generator!
I’m not sure if there is a robust way for two distros to both use systemd-gpt-auto-generator and coexist on the same disk?

6

On Silverblue the fstab has a mount for /var/home I think, not for /home itself?

/home is a symlink to /var/home, which is why the error in the journal takes the form it does:

Aug 21 12:34:05 fedora systemd[1]: home.mount: Mount path /home is not canonical (contains a symlink).
Aug 21 12:34:05 fedora systemd[1]: Failed to mount home.mount - Home Partition.

If the fstab had a mount for /home then indeed systemd-gpt-auto-generator wouldn’t try to mount it.

7

In the Silverblue installation I have, the mount point in fstab is actually “/home” though this is really just a symlink to “/var/home”.

The funny thing is that it was working without problems until recently, as the OP also mentioned. So, some change must have introduced the problem.

8

Ah I see, interesting!

9

Is my report and your reply enough confirmation to open a bug report? I was reading https://docs.fedoraproject.org/en-US/quick-docs/bugzilla-file-a-bug/ and it said to post in the Community first, but this still seems to be broken.

10

I really don’t know, but do make a bug report with the information you have. You can include a link to this topic.

12

Adding the atomic-desktops tag for more visibility for the Atomic Desktop SIG

I would raise this over at the Atomic Desktop issue tracker which can be found over at https://forge.fedoraproject.org/atomic-desktops/tracker