System bricked after adding the rootflags to root in grubby

Ask Fedora
, , , ,
1

Hello, so for context my system is current booting on a snapper snapshot entry and I want to delete this entry to free up some space, to do that I need to change the boot back to root
btrfs subvolume list:

ID 256 gen 449029 top level 5 path home
ID 257 gen 448679 top level 5 path root
ID 258 gen 448675 top level 257 path root/var/lib/machines
ID 260 gen 448727 top level 5 path snapshots
ID 573 gen 449029 top level 260 path snapshots/157/snapshot

and so I run this command sudo grubby --update-kernel=ALL --args="rootflags=subvol=root" and sudo btrfs subvolume set-default 257 / to change the default btrfs subvolume to root.

Turns out this bricked my system and cannot boot with bunch of errors in the image:
this text captured with Google Lens. See the image for the accurate one

Reloading system manager configuration.

Starting default.target

[FAILED] Failed to mount boot-efi.mount /boot/efi.

Dependency falled for local-fs.target Local File Systems.

[FAILED] Dependency failed for selinux-autorelabel-mark.service - Mark the need to relabel after reboot. ]

[Failed] Failed to mount var-lib-nfs-rpc_pipefs.mount RPC Pipe File System.

1 Dependency failed for rpc-gssd.service - RPC security service for NFS client and server.

1 Dependency failed for rpc_pipefs.target.

[FAILED] Failed to mount var-11b-snapd-snap-bare-5.mount Mount unit for bare, revision 5. ) Failed to mount var-lib-snapd-snap-core22-1621.mount Mount unit for core22, revision 1621.

[Failed] Failed to mount var-lib-snapd-snap-cups-1858.mount Mount unit for cups, revision 1858.

[FAILED] Failed to mount var-lib-snapd-snap-gnome\x2d42x242284-176.mount - Mount unit for gnome-42-2284, revision 176. ] Failed to mount car-lib-snapd-snap-gtk2dcommon\2dthemes-1535.mount - Mount unit for gtk-common-themes, revision 1535.

[FAILED] Failed to mount var-lib-snapd-snap-simplex2dscan-798.Mount Mount unit for simple-scan, revision 798 ] Failed to mount var-lib-snapd-snap-snapd-21759.mount Mount unit for snapd, revision 21759.

[FAILED] Failed to mount proc-sys-fs-binfmt misc.mount - Arbitrary Executable File Formats File System.

[FAILED] Failed to start systemd-binfmt.service Set Up Additional Binary Formats. are in emergency mode. After logging in, type "journalct1 -xb" to view

You system logs, "systemctl

Failed boot error messages
Failed boot error messages978Ă—678 203 KB

I managed to get my system boot again after reverting the changes I made with live USB chroot by removing args in grubby sudo grubby --update-kernel=ALL --remove-args="rootflags=subvol=root" and btrfs default subvolume back to snapshot.

However I noticed the error popping back up after a kernel update and those rootflags=subvol=root args is back on the grubby default one and I have to go back to live USB and remove it again. Now if I have to do this every single time after kernel update it would be painful and since Im still on Fedora 41, Im afraid I will have a problem upgrading to Fedora 42 after 41 EOS. So how do I resolve the root do not want to boot or at the very least make grubby not putting the args back every kernel update. Also I have to inform that this system is halftop (means the laptop main display is dead only external one) so I cannot see GRUB menu or BIOS menu during bootup.

.snapshots/157/etc/fstab entries

#
# /etc/fstab
# Created by anaconda on Sun Oct  6 01:49:30 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
UUID=a1b0772d-58ee-4479-8bf5-8c1ffd54045a /                       btrfs   subvol=root,compress=zstd:1 0 0
UUID=23873106-1d0f-4073-a229-825f81e65493 /boot                   ext4    defaults        1 2
UUID=3D9A-8671          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=a1b0772d-58ee-4479-8bf5-8c1ffd54045a /home                   btrfs   subvol=home,compress=zstd:1 0 0
UUID=a1b0772d-58ee-4479-8bf5-8c1ffd54045a /.snapshots             btrfs   subvol=snapshots,compress=zstd:1 0 0

/etc/fstab entries


#
# /etc/fstab
# Created by anaconda on Sun Oct  6 01:49:30 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
UUID=a1b0772d-58ee-4479-8bf5-8c1ffd54045a /                       btrfs   subvol=root,compress=zstd:1 0 0
UUID=23873106-1d0f-4073-a229-825f81e65493 /boot                   ext4    defaults        1 2
UUID=3D9A-8671          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=a1b0772d-58ee-4479-8bf5-8c1ffd54045a /home                   btrfs   subvol=home,compress=zstd:1 0 0
UUID=a1b0772d-58ee-4479-8bf5-8c1ffd54045a /.snapshots             btrfs   subvol=snapshots,compress=zstd:1 0 0

/etc/default/grub. I think this contains the problem why grub keep putting those args back

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rhgb quiet rd.driver.blacklist=nouveau,nova_core modprobe.blacklist=nouveau,nova_core rootflags=subvol=root"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
2

Your mistake is in this command.

You are setting subvolume ID 257, which is the nested subvolume
ID 258 gen 448675 top level 257 path root/var/lib/machines, as the default subvolume to be mounted. As a result, only the contents of subvolume ID 257 are shown.

With Fedora’s subvolume layout you shouldn’t modify these settings; to restore them you need to reassign subvolume ID 5 as the default.

sudo btrfs subvolume set-default 5 /

Edit: If you’re trying to configure snapshots, you can use Btrfs Assistant without changing the subvolume layout.

3

Thanks for the reply, now it is clearer that is my issue. So to sum up what I need to do is

  • First I need to run sudo btrfs subvolume set-default 5 /

  • Delete the rootflags=subvol=root args from /etc/default/grub to make sure Grubby did not put rootflags every single kernel update

  • After that I suppose I could delete the snapper snapshot 157 since I have now back at regular root?

I also wondering if the 257 cannot but why does the snapshot subvolume could boot up? Is it because they contain all the necessary build up?

4

I also want to ask, if I change my default to root like you said does it also lose all of data since basically I have been booting to snapshot the whole time?

6

RE: btrfs sub list - the number in the 2nd column is the subvolid. The number after “top level” is the containing subvolume.

e.g.

ID 258 gen 448675 top level 257 path root/var/lib/machines

The subvolume machines is subvolid 258, located in subvolid 257, which is:

ID 257 gen 448679 top level 5 path root

If using snapper, I think rollback should be done through snapper, otherwise I think it’ll get confused if you start using some other tool to undo/change the state of the file system.

Edit: i’m not familiar enough with the interactions between btrfs assistant and snapper.

7

By default, btrfs sub get-default should return 5, meaning the "top-level of the file system, a.k.a. subvolume ID 5, a.k.a. the invisible, not deletable, and unnamed subvolume created at mkfs time, is what’s mounted by default when neither subvolid nor subvol mount options are used.

The use of rootflags=subvol=root is the same as -o subvol=root and therefore will override the use of subvolume set-default. I’m not sure why that doesn’t work, unless something else depends on being able to mount the file system without subvol mount option. I don’t know what that something else would be - it would take some diagnosis, maybe with systemd.log-level=debug to see where the confusion first begins. Seems like it’s not able to mount other file systems but I can’t tell why.

8

By changing the toplevel and mounting the root subvolume (ID 257) as the default at mount time, I assume there’s no way to access the home subvolume, which resides on the toplevel of the default subvolume (ID 5).

Btrfs Assistant uses Snapper only to create snapshots; for rollback it doesn’t use Snapper, but instead a custom code that moves subvolumes:
mv root root.old
mv .snapshots/1091/snapshot root

To take advantage of Snapper for snapshots and rollbacks, one would need to use the SUSE/openSUSE-style subvolume layout, which is exactly what the user was trying to set up.

@dalto Can you confirm my assumption about rollback with Btrfs Assistant?

9

That is pretty close.

It is more like this:

mv root root.old
btrfs subvolume snapshot .snapshots/1091/snapshot root

After that, move all the nested subvolumes from inside root.old to root.

We take snapshot of the snapshot because we never want to lose the snapshot inadvertently.

Yes, if you want to use snapper for root rollbacks you need a special layout. You don’t need to follow the full layout from Suse but you need to at the very least have a non-nested /.snapshots

That is one of the reason Btrfs Assistant exists. To make it possible to use snapshots without requiring a specific layout.

1 Like
10

Let’s hope this patch gets accepted, I think it would make your work easier :slight_smile:
Great job with Btrfs Assistant!

11

Thanks!

We actually don’t use btrfs progs to take snapshots. We are integrated directly with the library. I amm not sure there is an obvious use case for recursive snapshots in Btrfs Assistance though.

12

It’s still possible to mount it using option subvol=home which overrides the default subvolume. The default subvolume only gets used when neither subvol nor subvolid mount options are used.

You can also mount the top-level of the file system anytime with mount option subvol=/ or subvolid=5.

(For history reasons I forget, ID 5 was used for this special subvolume that has no name, can’t be deleted or moved, but can be snapshot. Later, ID 0 was made an alias to it. That was a long time ago, so it’s reliable to use subvolid=0 if that’s easier to remember than 5.)

1 Like
13

I want to ask previously do i still lose all the current data on the snapshot root since it is writable copy if i change to default root? Also does it will pose problems when i upgrade to Fedora 42?

14

I can’t answer specific questions about btrfs-assistant or snapper because I’m not familiar with how they work.

But there is no per se reason why you lose anything by merely changing what subvolume is used for the root file system. In effect these subvolumes are separate, they don’t affect each other. Changes to one subvolume will not change the other.

And there’s no reason why there’d be a problem upgrading so long as the system boots now. The upgrade process is not doing anything differently based on what file system is being used.

15

what im saying is that since im on snapshot subvolume root the whole time is there a chance that my default root is not up to date with the snapshot root?

16

A 100% chance. They’re separate subvolumes. A snapshot is a subvolume. More specifically it’s a pre-filled subvolume. Modification of one subvolume doesn’t affect the other.