RFC for a Fedora 36 install, with a flat BTRFS subvolume layout and a working Snapper based snapshot management setup

I am new to Fedora and I have been working on setting up a reproducible installation of Fedora 36 which allows for a full system rollback using the Snapper snapshot management tool.

Purpose of this topic: I think I have a working setup and was looking for comments regarding any mistakes or potential future problems I might encounter with this setup. I am keen to find out if any pending BLS related work will break this setup.

I created a kickstart configuration file for a reproducible install of Fedora 36 in a VM. It assumes the VM has a 15gb disk in it. (NOTE: Due to the presence of the clearpart --drives=sda --all command, only use this script to test in a VM as it will clear all partitions on the /dev/sda drive without any confirmation.)

Kickstart configuration file: test.cfg

# Generated by Anaconda v36.16.5 / pykickstart v3.36
# Documentation: https://docs.fedoraproject.org/en-US/fedora/f36/install-guide/appendixes/Kickstart_Syntax_Reference/

# Use graphical install
graphical

# System language
lang en_GB.UTF-8

# Keyboard layout
keyboard --vckeymap=gb --xlayouts='gb'

# System timezone
timezone Europe/London

# Firewall configuration
firewall --enabled --ssh

# Ensure wanted services are started on boot
services --enabled=sshd,NetworkManager,chronyd
services --disabled=geoclue,avahi-daemon

# Network information
network  --hostname=fedoravm --bootproto=dhcp --device=link --activate

# Root and testuser's password is 'Passw0rd'
rootpw --plaintext Passw0rd
user --groups=wheel --name=testuser --password=Passw0rd --plaintext --gecos="Test User"

# Automatically accept Red Hat's EULA
eula --agreed

# SELinux configuration
selinux --enforcing

# Shutdown after installation
shutdown

# Use network installation instead of CDROM installation media
url --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch

# Enable the "updates" repo before installation starts
repo --name=updates

# Setup bootloader configuration
bootloader --location=mbr --boot-drive=sda --driveorder=sda --timeout=15

# Remove all partitions from the specified drive
clearpart --drives=sda --all

# EFI Partition (600 MB)
part /boot/efi --fstype="efi" --ondisk=sda --size=600 --fsoptions="umask=0077,shortname=winnt" --label=EFI

# Fedora OS Partition (10 GB)
part btrfs.fedora --fstype="btrfs" --ondisk=sda --size=10240 --fsoptions="noatime,compress-force=zstd:2,space_cache=v2"

# - BTRFS Volume (FS_TREE)
btrfs none --label=fedora_btrfs_vol btrfs.fedora

# - BTRFS Subvolumes
btrfs / --subvol --name=btrfs_sv_root fedora_btrfs_vol
btrfs /.snapshots --subvol --name=btrfs_sv_snapshots fedora_btrfs_vol
btrfs /tmp --subvol --name=btrfs_sv_tmp fedora_btrfs_vol
btrfs /var/cache --subvol --name=btrfs_sv_var_cache fedora_btrfs_vol
btrfs /var/lib/flatpak --subvol --name=btrfs_sv_var_lib_flatpak fedora_btrfs_vol
btrfs /var/lib/libvirt --subvol --name=btrfs_sv_var_lib_libvirt fedora_btrfs_vol
btrfs /var/log --subvol --name=btrfs_sv_var_log fedora_btrfs_vol
btrfs /var/spool --subvol --name=btrfs_sv_var_spool fedora_btrfs_vol
btrfs /var/tmp --subvol --name=btrfs_sv_var_tmp fedora_btrfs_vol

# Home Data Partition (variable size)
part /home --fstype="xfs" --ondisk=sda --size=1024 --grow --label=HOME


# Package Selection
%packages
@core
which
%end


# Commands to run on the system immediately after the ks.cfg has been parsed
# and the lang, keyboard, and url options have been processed. 
%pre --log=/tmp/pre.log
echo "INFO: Running commands in %pre section."
%end


# Commands tp run after the system has been partitioned, filesystems created,
# and everything is mounted under /mnt/sysimage.
%pre-install --log=/tmp/pre-install.log
echo "INFO: Running commands in %pre-install section"
%end


# Commands to run once the installation is complete.
%post --log=/root/post.log

echo "INFO: Running commands in post-install section."

# Set up snapper (need to do this last)
echo 'SUSE_BTRFS_SNAPSHOT_BOOTING="true"' >> /etc/default/grub
grub2-editenv - set blsdir=/boot/loader/entries
#grubby --update-kernel=ALL --remove-args="rootflags=subvol=btrfs_sv_root"
btrfs -v subvolume set-default $(btrfs inspect-internal rootid /) /
grub2-mkconfig -o /boot/grub2/grub.cfg
sed -i 's/\/btrfs_sv_root//' /boot/loader/entries/*.conf
sed -i 's/subvol=btrfs_sv_root,//' /etc/fstab
rm -f /boot/efi/EFI/fedora/grub.cfg.rpmsave
dnf --assumeyes install python-dnf-plugin-snapper snapper
umount -v /.snapshots
rmdir -v /.snapshots
snapper -v --no-dbus -c root create-config /
btrfs -v subvolume delete /.snapshots
mkdir -v /.snapshots
mount -v /.snapshots
snapper -v --no-dbus -c root set-config BACKGROUND_COMPARISON="no"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_DAILY="7"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_HOURLY="0"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_MONTHLY="12"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_WEEKLY="4"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_YEARLY="2"
systemctl enable snapper-boot.timer
systemctl enable snapper-cleanup.timer
systemctl enable snapper-timeline.timer

%end


# Commands to run once the installation is complete.
%post --nochroot --log=/mnt/sysimage/root/post-nochroot.log

echo "INFO: Running commands in post-install (nochroot) section."

#
# Copy over the pre installation logs
#
cp -v /tmp/pre.log /mnt/sysimage/root
cp -v /tmp/pre-install.log /mnt/sysimage/root

%end

When Fedora 36 is installed onto a VM using the above script, the resulting disk and filesystem layout looks as follows:

[root@fedoravm ~]# lsblk -af
NAME   FSTYPE  FSVER        LABEL                  UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 vfat    FAT32        EFI                    1859-1F85                             592.6M     1% /boot/efi
├─sda2 btrfs                fedora_btrfs_vol       6cfd3a0a-2971-4b9f-a3fc-876da09fa840    8.7G     9% /var/log
│                                                                                                      /var/tmp
│                                                                                                      /var/spool
│                                                                                                      /var/lib/libvirt
│                                                                                                      /var/lib/flatpak
│                                                                                                      /var/cache
│                                                                                                      /tmp
│                                                                                                      /.snapshots
│                                                                                                      /
└─sda3 xfs                  HOME                   d57d0483-9ad4-4f99-9478-bac8ac1071fe    4.3G     1% /home
sr0    iso9660 Joliet Exten Fedora-E-dvd-x86_64-36 2022-05-04-18-33-19-00
zram0                                                                                                  [SWAP]


[root@fedoravm ~]# btrfs filesystem show /
Label: 'fedora_btrfs_vol'  uuid: 6cfd3a0a-2971-4b9f-a3fc-876da09fa840
        Total devices 1 FS bytes used 858.92MiB
        devid    1 size 10.00GiB used 2.52GiB path /dev/sda2


[root@fedoravm ~]# btrfs subvolume list -ta /
ID      gen     top level       path
--      ---     ---------       ----
256     39      5               <FS_TREE>/btrfs_sv_var_tmp
257     37      5               <FS_TREE>/btrfs_sv_var_spool
258     44      5               <FS_TREE>/btrfs_sv_var_log
259     25      5               <FS_TREE>/btrfs_sv_var_lib_libvirt
260     25      5               <FS_TREE>/btrfs_sv_var_lib_flatpak
261     37      5               <FS_TREE>/btrfs_sv_var_cache
262     39      5               <FS_TREE>/btrfs_sv_tmp
263     38      5               <FS_TREE>/btrfs_sv_snapshots
264     43      5               <FS_TREE>/btrfs_sv_root
265     37      264             btrfs_sv_root/var/lib/portables
266     37      263             <FS_TREE>/btrfs_sv_snapshots/1/snapshot


[root@fedoravm ~]# btrfs subvolume get-default /
ID 264 gen 43 top level 5 path btrfs_sv_root


[root@fedoravm ~]# cat /etc/fstab
#
# /etc/fstab
#
UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 /                       btrfs   noatime,compress-force=zstd:2,space_cache=v2 0 0
UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 /.snapshots             btrfs   subvol=btrfs_sv_snapshots,noatime,compress-force=zstd:2,space_cache=v2 0 0
UUID=1859-1F85          /boot/efi               vfat    defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
UUID=d57d0483-9ad4-4f99-9478-bac8ac1071fe /home                   xfs     defaults        0 0
UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 /tmp                    btrfs   subvol=btrfs_sv_tmp,noatime,compress-force=zstd:2,space_cache=v2 0 0
UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 /var/cache              btrfs   subvol=btrfs_sv_var_cache,noatime,compress-force=zstd:2,space_cache=v2 0 0
UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 /var/lib/flatpak        btrfs   subvol=btrfs_sv_var_lib_flatpak,noatime,compress-force=zstd:2,space_cache=v2 0 0
UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 /var/lib/libvirt        btrfs   subvol=btrfs_sv_var_lib_libvirt,noatime,compress-force=zstd:2,space_cache=v2 0 0
UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 /var/log                btrfs   subvol=btrfs_sv_var_log,noatime,compress-force=zstd:2,space_cache=v2 0 0
UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 /var/spool              btrfs   subvol=btrfs_sv_var_spool,noatime,compress-force=zstd:2,space_cache=v2 0 0
UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 /var/tmp                btrfs   subvol=btrfs_sv_var_tmp,noatime,compress-force=zstd:2,space_cache=v2 0 0


[root@fedoravm ~]# ls -la /
total 16
dr-xr-xr-x.   1 root root  158 Jul  3 16:13 .
dr-xr-xr-x.   1 root root  158 Jul  3 16:13 ..
dr-xr-xr-x.   1 root root    0 Jan 20 03:04 afs
lrwxrwxrwx.   1 root root    7 Jan 20 03:04 bin -> usr/bin
dr-xr-xr-x.   1 root root  630 Jul  3 16:12 boot
drwxr-xr-x.  19 root root 3760 Jul  3 16:14 dev
drwxr-xr-x.   1 root root 2586 Jul  3 16:14 etc
drwxr-xr-x.   3 root root   22 Jul  3 16:12 home
lrwxrwxrwx.   1 root root    7 Jan 20 03:04 lib -> usr/lib
lrwxrwxrwx.   1 root root    9 Jan 20 03:04 lib64 -> usr/lib64
drwxr-xr-x.   1 root root    0 Jan 20 03:04 media
drwxr-xr-x.   1 root root    0 Jan 20 03:04 mnt
drwxr-xr-x.   1 root root    0 Jan 20 03:04 opt
dr-xr-xr-x. 194 root root    0 Jul  3 16:14 proc
dr-xr-x---.   1 root root  260 Jul  3 16:16 root
drwxr-xr-x.  29 root root  720 Jul  3 16:14 run
lrwxrwxrwx.   1 root root    8 Jan 20 03:04 sbin -> usr/sbin
drwxr-xr-x.   1 root root    2 Jul  3 16:14 .snapshots
drwxr-xr-x.   1 root root    0 Jan 20 03:04 srv
dr-xr-xr-x.  13 root root    0 Jul  3 15:14 sys
drwxrwxrwt.   1 root root  870 Jul  3 16:15 tmp
drwxr-xr-x.   1 root root  100 Jul  3 16:11 usr
drwxr-xr-x.   1 root root  170 Jul  3 16:14 var


[root@fedoravm ~]# grub2-editenv list
blsdir=/boot/loader/entries
saved_entry=3fea0dd6fdba4a00817ea34716e91544-5.18.7-200.fc36.x86_64
boot_success=1


[root@fedoravm ~]# cat /boot/loader/entries/*.conf
title Fedora Linux (0-rescue-3fea0dd6fdba4a00817ea34716e91544) 36 (Thirty Six)
version 0-rescue-3fea0dd6fdba4a00817ea34716e91544
linux /boot/vmlinuz-0-rescue-3fea0dd6fdba4a00817ea34716e91544
initrd /boot/initramfs-0-rescue-3fea0dd6fdba4a00817ea34716e91544.img
options root=UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 ro rhgb quiet ${extra_cmdline}
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
title Fedora Linux (5.18.7-200.fc36.x86_64) 36 (Thirty Six)
version 5.18.7-200.fc36.x86_64
linux /boot/vmlinuz-5.18.7-200.fc36.x86_64
initrd /boot/initramfs-5.18.7-200.fc36.x86_64.img
options root=UUID=6cfd3a0a-2971-4b9f-a3fc-876da09fa840 ro rhgb quiet ${extra_cmdline}
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora


[root@fedoravm os-probes]# snapper ls
 # | Type   | Pre # | Date                         | User | Cleanup  | Description | Userdata
---+--------+-------+------------------------------+------+----------+-------------+---------
0  | single |       |                              | root |          | current     |
1  | single |       | Sun 03 Jul 2022 16:14:16 BST | root | number   | boot        |
2  | single |       | Sun 03 Jul 2022 17:00:04 BST | root | timeline | timeline    |

The post installation script log output contained the following:

INFO: Running commands in post-install section.
Generating grub configuration file ...
mount: /var/lib/os-prober/mount: mount(2) system call failed: No such file or directory.
       dmesg(1) may have more information after failed mount system call.
Adding boot menu entry for UEFI Firmware Settings ...
done
Fedora 36 - x86_64                               31 MB/s |  81 MB     00:02
Fedora 36 openh264 (From Cisco) - x86_64        2.4 kB/s | 2.5 kB     00:01
Fedora Modular 36 - x86_64                      3.0 MB/s | 2.4 MB     00:00
Fedora 36 - x86_64 - Updates                     17 MB/s |  20 MB     00:01
Fedora Modular 36 - x86_64 - Updates            2.7 MB/s | 2.2 MB     00:00
Dependencies resolved.
================================================================================
 Package                             Arch     Version           Repo       Size
================================================================================
Installing:
 python3-dnf-plugin-snapper          noarch   4.0.16-1.fc36     fedora     13 k
 snapper                             x86_64   0.10.1-1.fc36     fedora    475 k
Installing dependencies:
 boost-system                        x86_64   1.76.0-11.fc36    updates    16 k
 boost-thread                        x86_64   1.76.0-11.fc36    updates    58 k
 libbtrfs                            x86_64   5.18-1.fc36       updates    28 k
 python3-dnf-plugins-extras-common   noarch   4.0.16-1.fc36     fedora     62 k
 snapper-libs                        x86_64   0.10.1-1.fc36     fedora    353 k

Transaction Summary
================================================================================
Install  7 Packages

Total download size: 1.0 M
Installed size: 3.1 M
Downloading Packages:
(1/7): python3-dnf-plugins-extras-common-4.0.16 142 kB/s |  62 kB     00:00
(2/7): python3-dnf-plugin-snapper-4.0.16-1.fc36  30 kB/s |  13 kB     00:00
(3/7): boost-system-1.76.0-11.fc36.x86_64.rpm   753 kB/s |  16 kB     00:00
(4/7): snapper-0.10.1-1.fc36.x86_64.rpm         974 kB/s | 475 kB     00:00
(5/7): snapper-libs-0.10.1-1.fc36.x86_64.rpm    6.5 MB/s | 353 kB     00:00
(6/7): boost-thread-1.76.0-11.fc36.x86_64.rpm   1.4 MB/s |  58 kB     00:00
(7/7): libbtrfs-5.18-1.fc36.x86_64.rpm          1.1 MB/s |  28 kB     00:00
--------------------------------------------------------------------------------
Total                                           1.0 MB/s | 1.0 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1
  Installing       : boost-system-1.76.0-11.fc36.x86_64                     1/7
  Installing       : boost-thread-1.76.0-11.fc36.x86_64                     2/7
  Installing       : libbtrfs-5.18-1.fc36.x86_64                            3/7
  Running scriptlet: snapper-libs-0.10.1-1.fc36.x86_64                      4/7
  Installing       : snapper-libs-0.10.1-1.fc36.x86_64                      4/7
  Installing       : snapper-0.10.1-1.fc36.x86_64                           5/7
  Running scriptlet: snapper-0.10.1-1.fc36.x86_64                           5/7
  Installing       : python3-dnf-plugins-extras-common-4.0.16-1.fc36.noar   6/7
  Installing       : python3-dnf-plugin-snapper-4.0.16-1.fc36.noarch        7/7
  Running scriptlet: snapper-libs-0.10.1-1.fc36.x86_64                      7/7
  Running scriptlet: python3-dnf-plugin-snapper-4.0.16-1.fc36.noarch        7/7
Running in chroot, ignoring command 'daemon-reload'
Running in chroot, ignoring command 'reload-or-restart'

  Verifying        : python3-dnf-plugin-snapper-4.0.16-1.fc36.noarch        1/7
  Verifying        : python3-dnf-plugins-extras-common-4.0.16-1.fc36.noar   2/7
  Verifying        : snapper-0.10.1-1.fc36.x86_64                           3/7
  Verifying        : snapper-libs-0.10.1-1.fc36.x86_64                      4/7
  Verifying        : boost-system-1.76.0-11.fc36.x86_64                     5/7
  Verifying        : boost-thread-1.76.0-11.fc36.x86_64                     6/7
  Verifying        : libbtrfs-5.18-1.fc36.x86_64                            7/7

Installed:
  boost-system-1.76.0-11.fc36.x86_64
  boost-thread-1.76.0-11.fc36.x86_64
  libbtrfs-5.18-1.fc36.x86_64
  python3-dnf-plugin-snapper-4.0.16-1.fc36.noarch
  python3-dnf-plugins-extras-common-4.0.16-1.fc36.noarch
  snapper-0.10.1-1.fc36.x86_64
  snapper-libs-0.10.1-1.fc36.x86_64

Complete!
umount: /.snapshots unmounted
rmdir: removing directory, '/.snapshots'
Transaction commit: none (default)
Delete subvolume (no-commit): '//.snapshots'
mkdir: created directory '/.snapshots'
mount: /dev/sda2 mounted on /.snapshots.
Created symlink /etc/systemd/system/timers.target.wants/snapper-boot.timer -> /usr/lib/systemd/system/snapper-boot.timer.
Created symlink /etc/systemd/system/timers.target.wants/snapper-cleanup.timer -> /usr/lib/systemd/system/snapper-cleanup.timer.
Created symlink /etc/systemd/system/timers.target.wants/snapper-timeline.timer -> /usr/lib/systemd/system/snapper-timeline.timer.

The important parts for a working Snapper based full system rollback + automatic snapshot management are the way the partitions are created and the way Snapper is set up in the chrooted %post section.

The way partitions are created:

  • /boot is not a separate partition or subvolume as we need the ability to be able to rollback to previous kernels / grub configuration.

  • To achieve a flat BTRFS layout, we need / to be a subvolume.

  • Snapper requires the /.snapshots subvolume.

  • /tmp is a separate subvolume since we don’t care if tmp files are not rolled back.

  • /var/cache is a separate subvolume since we don’t care if cache files are not rolled back. (I’m not sure about this one since I am assuming apps that put items in /var/cache have robust ways of detecting stale items and refreshing cache upon snapshot rollback.)

  • /var/lib/flatpak is a separate subvolume to avoid including flatpak applications in snapshots.

  • /var/lib/libvirt is a separate subvolume to avoid including virtual machines in snapshots.

  • /var/log is a separate subvolume since we need a clean log history of what’s happened.

  • /var/spool is a separate subvolume to avoid including local mail/print queues backlog in snapshots.

  • /var/tmp is a separate subvolume since we don’t care if tmp files are not rolled back. (I’m not sure about this one since FHS states apps will expect items in /var/tmp to be around for a while.)

  • The Anaconda installer refuses to create a subvolume for /root for some reason.

The way Snapper is set up:

  • The line echo 'SUSE_BTRFS_SNAPSHOT_BOOTING="true"' >> /etc/default/grub is needed to inform Grub that it should not include the rootflags=subvol=btrfs_sv_root option in the kernel boot parameters. By not including this option, Grub will now boot from the default subvolume instead of the one specified by rootflags=subvol=btrfs_sv_root.

  • The line grub2-editenv - set blsdir=/boot/loader/entries is needed to remove btrfs_sv_root from the blsdir path to ensure that future kernel installs will generate proper config in /boot/loader/entries/*.conf files.

  • The line btrfs -v subvolume set-default $(btrfs inspect-internal rootid /) / is needed to set what the default subvolume is.

  • The line grub2-mkconfig -o /boot/grub2/grub.cfg is needed to rebuild the grub config. See note (2) below regarding a harmless mount error you may encounter when running this line.

  • The line sed -i 's/\/btrfs_sv_root//' /boot/loader/entries/*.conf is need to remove mentions of btrfs_sv_root from the existing config files in /boot/loader/entries/. I was unable to find a way to regenerate the boot loader entries on demand. From googling the topic, it seems to be a done as part of /usr/bin/kernel-install and I didn’t see any alternatives. Should I raise a bug regarding this?

  • The line sed -i 's/subvol=btrfs_sv_root,//' /etc/fstab is needed to remove a mention of btrfs_sv_root from the / mount point entry. The subvol=btrfs_sv_root is not needed.

  • We install Snapper via: dnf --assumeyes install python-dnf-plugin-snapper snapper

  • We create a Snapper config via the following. Note the “snapper create-config” command will try to create a /.snapshots subvolume which is why we unmount the existing one, remove the mount folder, let Snapper do its thing and then remove the Snapper generated subvolume and re-setup our original one.

umount -v /.snapshots
rmdir -v /.snapshots
snapper -v --no-dbus -c root create-config /
btrfs -v subvolume delete /.snapshots
mkdir -v /.snapshots
mount -v /.snapshots
  • The following lines are not needed and is personal configuration preferences. Feel free to exclude or amend:
snapper -v --no-dbus -c root set-config BACKGROUND_COMPARISON="no"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_DAILY="7"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_HOURLY="0"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_MONTHLY="12"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_WEEKLY="4"
snapper -v --no-dbus -c root set-config TIMELINE_LIMIT_YEARLY="2"
systemctl enable snapper-boot.timer
systemctl enable snapper-cleanup.timer
systemctl enable snapper-timeline.timer

Notes:

  1. I am installing the which utility since it seems when doing grub2-editenv - set blsdir=/boot/loader/entries, one of the os-prober scripts needs it. Should I raise a bug for this to be added to @core?

  2. I encountered the mount: /var/lib/os-prober/mount: mount(2) system call failed: No such file or directory. error when executing the grub2-mkconfig -o /boot/grub2/grub.cfg command after using the btrfs subvolume set-default ... command. I see a bug raised about this (1919085 – mount: /var/lib/os-prober/mount: mount(2) system call failed: No such file or directory) but then closed since it was not reproducible. I investigated what was causing it by placing echo 'place holder nnn' lines in the /usr/libexec/os-probes/50mounted-tests script and it was line 149: mount -t btrfs -o subvol="$subvol" -U "$UUID" "$tmpmnt" which caused the error when it tried to mount the var/lib/portables subvolume using the uuid of the /dev/sda2 device. Not sure if I should reopen the bug since this issue doesn’t occur after a reboot.

  3. After setting up a VM using the above kickstart configuration with one line commented out (repo --name=updates), I did the following testing:

  • I ran a dnf update to do a full system upgrade and this pulled in a kernel upgrade as well. I confirmed there was a pre/post snapshot created that captured the upgrade. Rebooting the VM showed me a new kernel on the Grub menu and selecting it allowed me to boot using the new kernel without any issues.

  • With the system upgraded, I executed a snapper --ambit classic rollback __ID__ where __ID__ was the Snapper snapshot id of the pre snapshot created before the dnf update was carried out. The --ambit classic option is needed for this first rollback only and will not be needed for any subsequent rollbacks. I think the reason it is needed is that Snapper does not know what the default snapshot is and I was not able to figure out how to tell it what the default snapshot is. Rebooting the machine after executing the rollback command showed only a single kernel in the Grub menu (as expected) and booting from that entry brought me into a system where the dnf update changes were not present.

  • I checked for any mentions of btrfs_sv_root under /boot and /etc and no new ones were present.

  • I tested making further changes and rollbacks and can confirm that btrfs subvolume get-default / correctly tracked what I expected the default snapshot to be. I think Snapper is working correctly.

  1. My one concern is that there is still a mention of btrfs_sv_root in the /boot/efi/EFI/fedora/grub.cfg file:
search --no-floppy --fs-uuid --set=dev 6cfd3a0a-2971-4b9f-a3fc-876da09fa840
set prefix=($dev)/btrfs_sv_root/boot/grub2

export $prefix
configfile $prefix/grub.cfg

As far as I can tell, there is no way to remove this since Grub will not know where to source its grub.cfg file from. Is this a bug which may break this setup in the future? Currently, my testing shows its not affecting Snapper’s ability to rollback kernels and Grub config.

  1. If the above is deemed a working / stable setup, my next steps are to investigate getting grub-btrfs (GitHub - Antynea/grub-btrfs: Include btrfs snapshots at boot options. (Grub menu)) working so that I can boot snapshots directly from the Grub menu.

  2. The above setup has some negative aspects:
    a) The presence of the /.snapshots folder means utilities like du / will try to count files / folders in snapshots when run. It is also a security risk to have the old versions of libraries / binaries left around under /.snapshots. Maybe this is mitigatable via permissions. Ideally, Snapper should be modified to not require /.snapshots to be mounted. (needs testing to see if /.snapshots can remain unmounted)
    b) Currently, Gnome Software Centre and KDE Discover use PackageKit which ignores libdnf plugins like python-dnf-plugin-snapper and so when installing software using Gnome Software Centre or KDE Discover, Snapper does not perform a pre/post snapshot. There was a bug raised for this but it was closed without any activity: (1960901 – PackageKit ignores DNF plugins). It seems a Github issue for this is currently open for libdnf: (Missing integration with snapper for auto-snapshotting · Issue #313 · rpm-software-management/libdnf · GitHub) It is important that this pre/post snapshot functionality work no matter how software is installed as otherwise the auto snapshot management benefits of Snapper will be limited to the small minority of users who use dnf only for software installation.

1 Like

I guess you need to do all these acrobatics because you are doing Suse-style rollbacks where the default subvolume is being changed?

Unfortunately, I think you will find that there are some things which don’t like it when you change the default subvolume from subvolid 5. It is designed to work that way and it should work fine but plan to find things that expect mount /dev/blah to result in the root of the partition being mounted and not the default subvolume.

Yes. Snapper is what relies on being able to change what the default subvolume is.

Unfortunately, I think you will find that there are some things which don’t like it when you change the default subvolume from subvolid 5.

This was a concern which is why I raised the topic. Finding what those things are is difficult as there are only a few Google search results on people setting up Snapper on Fedora and I did review every one those I came across. Prior to Fedora, I used openSUSE and Snapper works really well there (within its limitations) but that of course does not translate over to other distributions.

Another reason for the topic is to share how it was done so that others can try and together we build experience to determine whether this is worthwhile or not.

Only if you must use “snapper rollback”.

That isn’t the only way to restore a snapper snapshot.

The other problem is that the method only works on the root subvolume. If you want to snapshot and restore other subvolumes you can’t use that method.

For example, you have setup /home on xfs but btrfs snapshots are a great tool for /home and snapper is more than capable of managing multiple sets of snapshots.

1 Like

Yes, I understand and agree.

The other problem is that the method only works on the root subvolume. If you want to snapshot and restore other subvolumes you can’t use that method.

Ah. I was not aware about this.

For example, you have setup /home on xfs but btrfs snapshots are a great tool for /home and snapper is more than capable of managing multiple sets of snapshots.

I kept /home on a seperate partition so that I can freely restructure the OS partition / filesystem in the future. I used xfs since I’ve not yet looked at how to setup and use BTRFS for /home. I currently rely on restic based backups for getting me out of trouble when I need to recover data that was under /home. I’ll pursue using BTRFS for /home once I’ve resolved the full system rollback need and I get more familiar and experienced at using BTRFS.