Rollback Btrfs root snapshot

Hello,

I am playing with btrfs and especially its snapshot capability for my root (/) volume.

I have created one volume with one subvolume mounted at /.

I have created a snapshot with:

[root@k1 ~]# btrfs subvolume list /
ID 256 gen 2302 top level 5 path root
ID 259 gen 1703 top level 256 path var/lib/portables
ID 272 gen 2289 top level 256 path .snapshots/root/20201223

(by the way if someone know what is the purpose oif var/lib/portables and if there is a way to disable it)

I have set the default subvolume to be my snapshot:

[root@k1 ~]# btrfs subvolume get-default /
ID 272 gen 2289 top level 256 path root/.snapshots/root/20201224

After a reboot it seems my change has not been taken in consideration (original subvolume is being mounted):

[root@k1 ~]# btrfs subvolume get-default /
ID 272 gen 2289 top level 256 path root/.snapshots/root/20201223

I have tried to re-generate my GRUB config with grub2-mkconfig but it doesn’t change anything.

The subvol option is not set in hte grub config, so it should be the default one.

Any ideas?

(PS: I know I should have put /boot into a subvolume to be consistent in case of kernel update, this is just for learning purpose)

Thanks in advance.

Regards,

Jérémy

3 Likes

To permanently change your root sub volume etc/fstab has to be adjusted accordingly.

To ease your efforts you may consider using snapper which provides a convenient high-level way of managing snapshots.

If you are using snapper

Snapper assumes complete control over what is the active root, via btrfs subvolume set-default.

You will need to remove the subvol=root option from fstab. And also modify /etc/default/grub adding the line:
SUSE_BTRFS_SNAPSHOT_BOOTING=true

And then run grub2-mkconfig -o /boot/... filling out the rest of the path based on the location to grub.cfg for your firmware type.

Hmmm, when I do this, it does work but there’s an unwanted $(extra_cmdline) item at the end of the boot parameters. I’m not sure where it’s coming from.

/etc/grub.d/00_header:383:if [ -n "\$extra_cmdline" ]; then
/etc/grub.d/10_linux:70:	GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} \${extra_cmdline}"
/etc/grub.d/20_linux_xen:77:	GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} \${extra_cmdline}"

OK it’s related to finding read-only snapshots. I think. None of that is wired up with BootLoaderSpec stuff yet. Also, while booting read-only snapshots works up to a point, because /etc and /var are in the read-only root subvolume snapshot, they’re read-only too which will cause startup to fail.

So doing a rollback on boot is not exactly ready, you’re pretty much in adventure land if you’re playing with this (which is why none of it’s installed or enabled by default yet).

If you can boot, snapper rollback should work since it’ll just change the default subvolume.

If you aren’t using snapper

I’ll describe what I do since I’m also not using snapper. I assume that you haven’t modified fstab or any of the grub stuff and that you haven’t changed the default subvolume.

Mount the top level of the file system, make a read-only snap shot

mount /dev/sdXY /mnt
cd /mnt
btrfs sub snap -r root root.20201224

I make read-only snapshots out of habit, mostly. Ok let’s say I mess up something and want to do a rollback.

mount /dev/sdXY /mnt
cd /mnt
mv root rootmessedup
btrfs sub snap root.20201224 root
reboot

In this case I’m renaming the current root subvolume to something descriptive. Renaming it while it’s in use is not a problem (I do it all the time; and behind the scenes, btrfs tracks subvolumes by ID not name). Next I make a read-write snapshot of the snapshot I want to rollback to. And then reboot.

An optimization probably is to remove subvol=root from fstab, that way you could instead directly choose to rollback to read-write snapshots from within GRUB if you know the name. Just change rootflags=subvol=root to rootflags=subvol=root.20201224. Of course, you’d need to be making read-write snapshots in this case.

The reason I prefer snapshots “hidden” away in the normally not mounted top-level, is so they don’t clutter up other commands like find, du, backup programs, and so on. Also there is a (perhaps small) security concern with keeping stale binaries in a readily available search path, even if it is a hidden .directory. So I just prefer to keep them totally out of site.

5 Likes

I’d prefer that either and already do that for my /home file system which is backed up with btrbk. But I think snapper can’t handle this, does ist?

Not yet.

1 Like

Thank you very much for your explanation. Following your instructions i am now able to “rollback”.

The only part I don’t fully understand is:

Since the default subolume is “root”, the top level (subvolid 5) of the filesystem should not be mounted here, or I am missing something?

I had to force the top level manually:

mount -t btrfs subvolid=5 /dev/sdaXY /mnt/tmp

If you aren’t using snapper

I assume that … you haven’t changed the default subvolume.

The mkfs time default subvolume is ID 5 at the top level of the file system. Fedora’s installer doesn’t change this.

Also, I actually tested fstab conflicts after writing the earlier comment …

The fstab / entry’s use of subvol=root is superfluous and ignored. The equivalent CLI command is mount -o remount,subvol=root and it would be wrong behavior if this command could switch subvolumes on a remount. i.e. the default subvolume or rootflags defined. I’ve applied strike though to the earlier suggestion.

1 Like

The default subvolume is used only when a btrfs filesystem is mounted without subvol or subvolid options supplied.

For example, when /dev/sda3 is a btrfs,
mount /dev/sda3 /mnt, then the default subvolume will be mounted to /mnt

when subvol or subvolid is used, the default subvolume has no effect the the mount, like
mount -o subvol=rootfs /dev/sda3 /mnt
mount -o subvolid=277 /dev/sda3 /mnt

So for a rootfs roll back, you need to adjust so that the kernel root parameter (as specified in the bootloader), the fstab entry (as in /etc/boot/fstab) and the filesystem all need to match.

If your fstab/kernel cmdline use /dev/sda3, or part-UUID without subvol or subvolid options, then you need to change the default subvolume option in btrfs.

Or you need to change kernel parameter and fstab to match your desired root in the btrfs.

Hello,

Sorry if I break a rule and post it in an old topic. This is my first post here.

I do not use snapper.
I am able to rollback like Chris Murphy said.
But this will change my btrfs layout, the subvolumes /var/lib/portables and /var/lib/machines will be no more. Instead I have two directories.
I understand that this two subvolumes are made by systemd.
Is it a problem if the two of them are changed from subvolumes to directories?

If I made a flat layout with @root, @home and @var subvolumes, the /var/lib/portables and /var/lib/machines will be inside (nested) @var subvolume (instead of child of @root subvolume, if I have only @home and @root).
It is better this way? And made @var with noCOW - I see that in openSUSE @var is setting with noCOW.
Thank you.

If the subvols are outside of the current subvol tree, you can modify fstab to mount them.

Personally, when I am switching another subvol as the new rootfs, I will move these two subvols to the top level, and mount them back to /var/lib/portables /var/lib/machines using fstab.

1 Like

Hello,

Thanks for the answer.

1.If I install Fedora with @home and @root subvolumes (default install), I will have a flat layout (@home and @root subvolumes) and two nested (/var/lib/portables and /var/lib/machines) subvolumes. The /var/lib/machines and /var/lib/portables subvolumes are children of @root. In fstab will be only / and /home.
2.If I make a @var subvolume I will have a flat layout with @home, @root and @var and /var/lib/portables and /var/lib/machines will be children of @var subvolume. In fstab will be /home, / and /var.
3.You said to make a flat layout with @home, @root and another two subvolumes (/var/lib/portables and /var/lib/machines) instead of @var subvolume. In fstab will be /home, /, /var/lib/machines and /var/lib/portables. In this case, when I will switch another subvolume as new rootfs, I change @root with @new_root, /var/lib/portables and /var/lib/machines subvolumes will be the same.

So, if I want to switch another subvolume as new rootfs, in the first case, the /var/lib/portables and /var/lib/machines subvolumes will change to directories, in the second case the @var subvolume (with the nested /var/lib/portables and /var/lib/machines subvolumes) will remain the same and in the third case, only the /var/lib/portables and /var/lib/machines subvolumes will be the same.

I don’t know which is the better layout and why.

Good question. Short answer: if you aren’t using systemd to manage VMs, containers, and portable images, then you don’t need to worry about these items. Longer answer: there’s multiple snapshot/rollback regime strategies; they either involve some complexity upfront (e.g. using fstab to assemble multiple subvolumes and/or snapshots during startup) or complexity upon rollback (i…e. having to move subvolumes or snapshots to reassemble it). No matter what there are tradeoffs between a simple approach and a complex one.

Not really. /var/ contains the RPM database. We can’t do a rollback of /usr/ in the root subvolume independently of the RPM database located in /var/ or dnf/rpm will get confused. So they’d both have to be snapshot and rolled back. Or we separate out /var/lib/rpm as a subvolume, tying its snapshot/rollback with that of /usr/ and allow the rest of /var/ to continue always tracking forward (it never rolls back).

The FHS doesn’t help organize things based on a snapshot/rollback regime. We could carve things up to deal with this, which is what (open)SUSE has done for years now. But they may be re-evaluating for a simpler approach.

The logic for /var/ being nodatacow is it tends to contain things that aren’t COW friendly, like the systemd-journald log, virual machine raw/qcow2 images, and databases. Since the 'C' file attribute (using chattr +C) works by inheritance, just by setting it on /var/ everything inside inherits nodatacow. But nodatacow also means no checksumming, and no compression. Instead on Fedora we depend on systemd-journald setting nodatacow by default on /var/log/journal/ and libvirt sets it on /var/lib/libvirt/images/ and the RPM database now uses sqlite and WAL is enabled so it is copy-on-write friendly without needing nodatacow.

Anyway, all of these issues are a big part of why Fedora doesn’t yet ship with an automatic snapshot and rollback regime. And that’s why the current layout is pretty simple.

3 Likes

Thanks a lot for the great details to consider regarding btrfs subvol schemes.

Hello,

Thanks a lot for the answer.

I did not use VMs, containers, and portable images.
After your answer I understood that a @var subvolume it is not a good idea.
So I will stick with default install (a flat @root and @home layout).

I tried a lot to find the way Fedora see the btrfs layout (schemes). Is there any link where I could read about these “hidden gems”?. Thank you.

If you are in a jam, and need to use a rollback to get back to working, it’s not likely related to machines or portables. So I’d say you ignore them for now, and just do a rollback and boot. If that works, then you can mount the top-level (i.e. if you aren’t using something that changes the default subvolue such as snapper does, you just mount normally; otherwise you mount using -o subvol=/ to get the top-level) to something like /mnt/ and now you can do a full path move, e.g.

sudo mv /mnt/root.original/var/lib/portables /mnt/root.rollback/var/lib/

And that will move the subvolume, replacing the directory portables. Does systemd need a daemon-reload to see that change? :smiley: I don’t know. But again, it depends if you have a use case for systemd portables and machines, if they’re always empty then you don’t need to worry about them.

An automated snapshot/rollback service or daemon, should know how to handle this properly. But I can also argue that maybe each service that use subvolumes and snapshots perhaps one day need their own dedicated namespace in the top-level of the file system (the part not normally mounted) and they manage, name, and mount, their subvolumes to the proper location automatically. Snapshots are pretty cheap but not infinitely cheap. Subvolumes themselves are effectively infinitely cheap, so each service getting their own pile of subvolumes is not a problem, they don’t necessarily need to be snapshot, just always consistently put into position (via fstab or a native systemd mount unit) at startup time.

There can be 2^64 subvolumes (snapshots are also subvolumes, they are just pre-filled subvolumes) per Btrfs file system. And each subvolume can have 2^64 files.

Thanks for the answer(s).
It was a great help.
BTW, with the help of mr. Google I found that “systemd-journald detects btrfs and automatically does nodatacow on its journals”. But I was helped with the right direction to search.
I am not in a jam. I am in a permanent jam. I tried different btrfs layouts, rollbacks.
I only try to understand the way btrfs works. Anyway, for me, the no-snapper rollback from post #3, is the best.

Just as an example of what I do, my fstab contains:

UUID=<uuid>  /var/log               btrfs   subvol=varlog    0 0

That really is the only thing in a snapshot/rollback regime you might regularly not want to rollback, is your logs. So I never snapshot the varlog subvolume, and the systemd fstab generator will always mount this subvolume at /var/log regardless of rollback or not, so logs are always moving forward.

  • This will mount the top-level of the file system, even if you’re using something like snapper which changes the default subvolume. Adjust for your device…
    mount -o subvol=/ /dev/vda3 /mnt

  • This is fairly self explanatory and here you’ll see the subvolumes created at installation time, root and home, which look like directories. But you’ll see they have inode 256 which is the reserved inode for all subvolumes.

sudo -i
cd /mnt
ls -li
  • Create the subvolume and copy everything from old to new locations:
btrfs subvolume create varlog
chcon "system_u:object_r:var_log_t:s0" varlog
cp -a root/var/log/* varlog/
  • Update fstab (see above at top)
  • Reboot

df now shows an extra mountpoint, same size as others

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda3      103192576 4854740  96753788   5% /
/dev/vda3      103192576 4854740  96753788   5% /home
/dev/vda3      103192576 4854740  96753788   5% /var/log

Optionally go clean up the old directory that’s no longer being used, note you do not want to delete this var/log/ directory, just its contents. You still need the directory as the mount point the subvolume will be mounted onto.

sudo -i
mount -o subvol=/ /devsdXY /mnt
rm -rf /mnt/root/var/log/*

Extra info. This is why I delete the contents of the old var/log/. It’s really just an info message to ask you if you’re really sure you mean to be mounting something on top of a directory that isn’t empty. It doesn’t hurt anything to just ignore it and not do the clean up.

journalctl | grep empty
Feb 25 13:42:31 fedora systemd[1]: var-log.mount: Directory /var/log to mount over is not empty, mounting anyway.

Perhaps the same technique could be used for /var/swap/, but I haven’t tested if a varswap subvolume is mounted soon enough during startup. This would be useful for those wanting to use swapfiles on btrfs.

2 Likes

Hello,

Thanks for your help @chrismurphy.
It works great. Because I forgot about SELinux I was not able to a change a subvolume.

I have another question because I wouldn’t want to open a new thread. And I think that question has a bond with this thread.

By default Anaconda install /boot on a ext3 partition.
I want to make a /boot subvolume and keep only a /boot/efi partition with vfat. A partition layout like this: /boot/efi - vfat and /boot - subvolume; / - subvolume; /home - subvolume; /var/log - subvolume.
It is a reason to keep a /boot partition with ext3 instead a /boot subvolume?
openSuse have a /boot subvolume so I think that is possible.
Is there a difference between openSuse and Fedora who prohibit the latter to make a /boot subvolume?

Thank you

I use /boot as btrfs subvol also for Fedora 33 workstation without issues.

But for Silverblue, /boot must be a separate partition.

Thanks for the answer.
I don’t use Silverblue.
I will make a /boot subvolume for a fresh Fedora 34 beta install.

1 Like