How to set up F35 with BTRFS and rollback?

Ask Fedora
, , ,
1

Hello fellow users of Fedora,

After using Mint 20.x with Timeshift to roll back to a previous version of /, and having two weeks to learn Fedora, I decided to go with Fedora35 with btrfs.

After installing Fedora with btrbk, I had still some problems with rolling back. To set up btrbk, I used this nice guide Fedora Workstation 35 with automatic btrfs snapshots and backups using BTRBK | Willi Mutschler (without LUKS, but with an extra internal ssd of 128GB). I succeeded in setting up btrbk, and creating several snapshots, but not to roll back to a previous snap propperly. There were no other reason for a roll back, then just learning how to roll back. Doing/learning so, I broke my system and decided to do an clean reinstall, and ask how to proper set up Fedora 35 with btrfs, btrbk, and to roll back.

In order to do my own research, I only found either articles how to set up, or how to roll back with an already read-write snaps, or how to roll back /home with /etc/fstab. Therefore I guess that I created just read-only snaps, thus Fedora I failed to do an roll back to a previous snap of /.

Either way, I have 10 days left of my vacation to proper learn Fedora35 with btrbk. Although I will follow that guide of Mutschler again, to my understanding, he does not describe how to do an read-and-write snap and how to roll back. Feel free to drop other guides, or write some command of your own.

EDIT: it seems that I would like to learn how to do this: Changes/BtrfsWithFullSystemSnapshots - Fedora Project Wiki.

1 Like
2

I tried do the snapshot and roll back by hand, like below:
(I do not test all the commands listed below. There can be typos and/or minor errors. It is provided just to illustrate the idea.)

  1. List all btrfs subvols that needs to be snap and roll back together:
  • /, /home
  1. Locate a folder or subvol that will hold snapshots (personally I will always take read-only ones)
  • BTRFS_TREE/snaps (I use BTRFS_TREE as the top level root of the btrfs volume. It can be mounted using subvolid=5 mount option)
  • mount -o subvolid=5 /dev/sda3 /snaps
  1. Create read only snapshots
  • btrfs subvolume snapshot -r / /snaps/fedora-rootfs-today-ro
  • btrfs subvolume snapshot -r /home /snaps/home-today-ro

Up to now, it completes to ā€œbackupā€ part.

  1. Locate which folder to ā€œholdā€ the rw snapshots that can be used to boot
  • BTRFS_TREE/subvols
  • mount -o subvolid=5 /dev/sda3 /subvols
  1. Create rw snaps from the ro snaps
  • btrfs subvol snap /snaps/fedora-rootfs-today-ro /subvols/fedora-rootfs-previous
  • btrfs subvol snap /snaps/home-today-ro /subvols/home-previous
    (Personally, I will try to create a new folder inside these newly created rw-snaps, to make sure they are really rw)
  1. Adjust /etc/fstab in /subvols/fedora-rootfs-previous
    (Note the changes of subvol of / and /home)
UID=a906530d-fe89-4e8a-9d9f-9aa1d9922a6f /                       btrfs   subvol=subvols/fedora-rootfs-previous,compress=zstd:1 0 0
UUID=1e6ab2fd-d285-4d5a-bcc0-93fea1796bab /boot                   ext4    defaults        1 2
UUID=a906530d-fe89-4e8a-9d9f-9aa1d9922a6f /home                   btrfs   subvol=subvols/home-previous,compress=zstd:1 0 0
  1. Adjust /boot/loader/entries
  • I will base on the last know good boot entry, copy to a new entry, and edit (I will change title and options lines)
#cat a6829ff0e1824fc58dc23aa211bd0d2a-5.15.17-200.fc35.x86_64.conf 
title Previous Fedora Linux (5.15.17-200.fc35.x86_64) 35 (Workstation Edition)
version 5.15.17-200.fc35.x86_64
linux /vmlinuz-5.15.17-200.fc35.x86_64
initrd /initramfs-5.15.17-200.fc35.x86_64.img
options root=UUID=a906530d-fe89-4e8a-9d9f-9aa1d9922a6f ro rootflags=subvol=subvols/fedora-rootfs-previous 
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
  1. When reboot, at the grub menu, select the Previous entry for the ā€œroll backā€ entry, or select the other entry to boot the ā€œlatestā€ entry.

As you have BTRBK up and running, you just need to do steps 4-8 to create a ā€œroll backā€ instance of Fedora.

3

Thanks for this how to.

This morning on a clean install of F35, I followed the Mutschler guide again. I ignored the LUKS stuff, but did the extra internal drive stuff. (For guide, see first post.)

  • Now I will look up how to (re)name a snapshots, so that they are easier to understand by humans To name a snapshot, just create create a manual snap to the backup, with btrfs subvolume snapshot -r / /btrfs_backup/[name] (see also point 3 of @sampsonf)
  • Then I will install some stuff
  • And do a roll back suggested above

Keep you all postedā€¦

Edit: Hum, I am getting stuck at step 5, maybe because I used the settings of Mutschler, and not yours. Therefore i guess I am stuck in ā€˜translatingā€™ your suggestion into Mutschlerā€™s.

When I do a sudo btrfs subvolume list /, it gives me a full list, and I would like to roll back to ID 287 gen 431 top level 257 path btrfs_backup/fedora35-fresh

In order to ā€˜translateā€™ your setup/suggestions into Mutschlerā€™s, I guess I have to do the following sudo btrfs subvolume snap /btrfs_backup/fedora35-fresh /[location of old root]. Am i here correct?

Either way, I have thus to do the following:

  1. sudo btrfs subvolume get-default / results in ID 5 (FS_TREE)
  2. [Set default]
  3. nano /etc/fstab/ to change subvol=root into subvolid=[preferred root ID]
  4. nano /boot/loader/entries to change a conf accordingly
  5. reboot
4

Okay, I did changed all in the order above, and F35 will not reboot, I get stuck in ā€œemergency modeā€. Thus I used the live medium -that I used to install this F35- to check the files that I have changed to roll back. The results are now that /run/media/[user]/[disk]/etc/fstabhas become ā€œunwritableā€, and there is nothing in ~/boot/, not even a subdirectory ~/boot/loader/entries.

Because I updated F35 before I did set up btrbk like Mutschler suggested, the kernel is 5.15.18-200. Thus using the /boot/loader/entries I could recreate the boot loader file, and only changing the kernel version and de UUID of my disks. Or I could reinstall and do it all over again, but then there is change that i just reproduce this error.

Suggestions are of course welcomeā€¦

ps. I have only problem with rolling back the /. /home is save, and I have all files back upped on three different drives.

5

:thinking: Why not using Timeshift with Fedora 35? The Timeshift current version available from Fedora repo may not work (the version too old), but you can build it from the source or install it from Fedora Copr here.

We only need to rename the subvolume of root to @ and create another subvolume for home as @home and move the contents of /home to @home then mount @home subvol to /home from fstab. Donā€™t forget to double check the fstab and kernel parameter with grubby.

If after reboot you failed to open Timeshift from terminal, most likely you need to run sudo btrfs quota enable / and make/create manual conf in /etc/timeshift/timeshift.json.

6

Okay, this is weird. TLDR: With a cold reboot I get to choose between three boot loader entries, but with a warm boot, there are no boot loader entries. Despite every boot loader entry has subvolid=287 for root, I get subvol=5 for root.

  1. This morning after a cold boot, I got page where I could choose between three boot loader entries. First for kernel 5.15.18-200, the second for kernel 5.15.14.300, and the third for Rescue. I choose the loader for kernel 5.15.18 and Fedora35 boots fine, but with the newest snapshot of /, despite the set-default is still subvolid=287. This is only true for a cold boot.
  2. However, this is not rue for a ā€˜warmā€™ boot. When I reboot and the mother board stays on- thus the Power Supply Unit is not switched off- then Fedora35 fails to load, I get the ā€œemergency modeā€ again, and there is nothing in /boot. This is only true for a warm boot.
  3. When I shut down the computer, switch off the PSU for some second (so that the mobo is not warm/is cold), and switch on the PSU and reboot, I get that ā€œpage I could choose between three boot loader files. First for kernel 5.15.18-200, the second for kernel 5.15.14.300, and the third for Rescue. Each option gives me the newest snapshot of /, but not the set-default which is in my case subvolid=287.ā€ This is only true for a cold boot.
  4. When I do a cold reboot again (see step 3: shutdown, switch off, switch on, boot), I get that ā€œpage I could choose between three boot loader files (ā€¦)ā€.This is only true for a cold boot.
  5. When I do a warm reboot again (see step 2: shutdown, reboot, but not switching off/on the PSU), I get ā€œthe ā€˜emergency modeā€™ again, and there is nothing in /boot.ā€ This is only true for a warm boot.

Thus with a cold reboot I get to choose between three boot loader entries, but with a warm boot, there are no boot loader entries. Despite every boot loader entry has subvolid=287 for root, I get subvol=5 for root.

So, what shall I do now? Still here to learnā€¦

That is indeed a fine question, but I have made my choose already. However, here are my thoughts about my choice for btrbk: I choose to learn Fedora and Linux in general too, over to mere replace Mint and keeping my old habits.

  • Primarily: As I have already mentioned in the OP, I have some time to spend for learning Fedora, of which I still have 7 days left. I thought it would be nice, to learn more about Fedora, than as a mere replacement of Mint and keeping to my old habits. Therefore I choose to learn more about Fedora, and not to mere replace Mint.
  • Secondarily: Now I have enough time to learn more about Linux too, and set up Fedora just as I like it. Therefore installing and setting up Fedora, became for me a (crash)course Linux and related applications. Again, the difference is to learn more about Linux, rather than to mere replace Mint.
  • Conclusion: The difference is thus that I choose to learn Fedora35, over to replace Mint, in order to learn more about Linux for a more personal, flexible and scale proof setting. Where I have learned more about Fedora35 and Linux too in general with btrbk, I could not have so with a mere replacement of my old habits.

Or written the same, but differently:

  • Learn Fedora & Linux > Replace Mint with Fedora & keeping old habits.
1 Like
7

Okay, this get weirder. After I ran btrbk run, despite I do a cold or warm boot, I get stuck in ā€œemergency modeā€. Thus I get now stuck as well the boot loader profiles of kernel 5.14 and kernel 5.15, but not with the boot loader profile of Rescue.

When I ask for my default snap of btrfs in Rescue with btrfs subvol get-default /, I get ID 287 gen 1009 top level 257 path root/btrfs_backup/fedora35-fresh. However, my most recent snapshot of root is ID 302 gen 1002 top level 5 path _btrbk_snap/root.20220204T1401.

Therefore I am wondering if the preferred snapshot should be in /_btrbk_snap/ instead of /btrfs_backup/. Again, btrbk is set up with that guide of Mutschler.

Either way, the only way to use Fedora35 now is using the Rescue boot loader entry, and not a normal entry. Although this works, I prefer to use the normal way. Therefore I tend to follow a suggestion in the Telegram group Fedora Linux, and to do the following for a ā€œroll backā€ by using a live media:

  1. cd /[disk]
  2. mv root root_backup
  3. btrfs subvolume snapshot /btrbk_backup/fedora35-fresh /[disk]/
  4. btrfs properly set -ts /[disk]/root ro false
9

nano /etc/fstab/ to change subvol=root into subvolid=[preferred root ID]

This is confusing, but the / mountpoint line is a remount operation, unlike any other line in fstab. The kernel itself does the initial mount based on boot parameters: ro root=UUID=$fsuuid rootflags=subvol=root The first two are commonly seen with other file systemā€™s too but Btrfs can have many ā€œrootsā€ so the way this is defined is with rootflags= which defines mount options for the root file system. The mount option is subvol=root because the name of the subvolume being used for the root file system is ā€œrootā€ (by default, in Anaconda), hence rootflags=subvol=root. The subvol= mount option in fstab doesnā€™t make sense for /because we canā€™t change subvolumes on a remount, but we can change some other mount options on a remount like remount,rw or remount,compress=zstd:1 are valid. Whatā€™s a valid remount mount option is not exactly obvious, just like what is a VFS mount option in common to all file systems is non-obvious.

Also note that the kernel command line option rootflags=subvol=root will override the default subvolume set with btrfs subvolume set-default. In general, youā€™d use one or the other. For example, use set-default in the usual case, without rootflags option; but then maybe your rescue BLS snippet does use rootflags in order to ensure you mount a specific root, overriding the set-default subvolume.

As for the cold boot vs warm boot, I donā€™t have an explanation for this. Thereā€™s nothing unique about the system cold vs warm boot once weā€™re at GRUB. I might wonder if thereā€™s a firmware anomaly that causes it to use different NVRAM entries depending on cold vs warm boot, and in fact youā€™ve got more than one /EFI/fedora directory on the ESP? That too would be weird, but less weird than non-deterministic results that we canā€™t account for.

Another possibility is if /boot is on Btrfs, and maybe the warm vs cold boots are somehow picking different btrfs roots on the /boot file system (if you have more than one subvolume on it). You definitely can quickly get into a maze with multiple btrfs subvolumes on /boot because it can mean different /boot/loader/entries depending on which subvolume is followed.

1 Like
10

Okay, due to different guidelines, manuals, etc, I have messed it up, but this is also a way to learn more about Fedora35. When I do my roll back/restore correctly, Fedora boots just fine. When I do my roll back/restore wrong, Fedora35 boots wrong and start to behave like this:

For a proper roll back/restore, see next post.

When I followed the quit technical guide of digint for btrbk GitHub - digint/btrbk: Tool for creating snapshots and remote backups of btrfs subvolumes, I end up doing the following:

  1. $ sudo dnf update
  2. $ reboot
  3. $ sudo -i
  4. # btrfs filesystem label / fedora35
  5. # mkdir /btr I prefer to have all the btrbk stuff in just one directory
  6. # mkdir /btr/pool
  7. # mkdir /btr/pool/snap
  8. # mkdir /btr/backup This will be a mount point for my secondary disk, see below
  9. # blkid For /btr/pool on primary disk, and /btr/backup on secondary disk
  10. # echo "UUID=[Primary disk] /btr/pool btrfs subvolid=5,compress=zstd:1 0 0" >> /etc/fstab
  11. # echo "UUID=[Secundary disk] /btr/backup btrfs subvolid=5,compress=zstd:1 0 0" >> /etc/fstab
  12. # cat /etc/fstab To just check everything
  13. # mount -av Ignore SELinux error
  14. # ls /btr To just check everything
  15. # ls /btr/pool To just check everything
  16. # ls /btr/backup To just check everything
  17. # sed -i 's/compress=zstd:1/ssd,compress=zstd,discard=async/' /etc/fstab
  18. # cat /etc/fstab To just check everything
  19. # reboot
  20. $ sudo -i
  21. # dnf install btrbk
  22. # nano /etc/btrbk/btrbk.conf To set up the configure file, see /etc/brbk/btrbk/btrbk.conf.example
  23. # btrbk dryrun --progress To just test everything. When there are errors, solve them appropriately
  24. # btrbk run --progress To just run btrbk for the very first time
  25. # exit
  26. $ sudo dnf install [package] To install any package. Or start an install script
  27. $ sudo btrbk run --progress To run btrbk again, but this time after all programmes are installed

I have the btrbk.conf set up as:

#Miscellaneous
transaction_log /var/log/btrbk.log
lockfile /var/lock/btrbk.lock
#timestamp_format long
stream_buffer 256m

#Snapshots
snapshot_dir snap
snapshot_create always
snapshot_preserve_min 2d
snapshot_preserve 6d

#Backups to target
target /btr/backup
target_preserve_min 2d
target_preserve 20d 10w *m

#Which (sub)volumes to snap and backup
volume /btr/pool
subvolume root
subvolume home

2 Likes
11

For a proper roll back/restore of / with the settings above, I have to do the followings:

  1. cd /mnt
  2. mv root root_bak To move the old snap of / to a new directory
  3. btrfs subvolume snapshot /mnt/.snapshots/root-$DATE /mnt/root To move the preferred snapshot to /, so that Fedora35 can boot
  4. btrfs property set -ts /mnt/root ro false
  5. cd /btr/pool/root
  6. pwd & ls -la To make sure that every directory is moved correctly
  7. reboot and donā€™t mess anything upā€¦

When Fedora can not boot anymore, use the liveusb to cd in the disk with Fedora35 installed (step 1) and proceeed.

This above guide is from @Wolfshappen in the Telegram group Fedora Linux.

2 Likes
12

I just redid this on F36, and I have just two changes to make for a proper set up of btrbk:

Step 7: mkdir /btr/snap plural if you like, change this in your conf.
Step 8: # mkdir /btr/btrbk as our mounting point for the btrbk backups

And youā€™re up and running with F36 again.

1 Like
13

Glad to see you have it working. Could you be so kind as to mark the solution in this case? This will allow the rest of the community that is facing a similar problem to readily find your solution.

2 Likes
14

Because the topic is F35-specific, I marked the F35-related rollback/restore as solution.

@a86ul feel free to change the solution if you think that another post fits better.

15

It would be indeed helpful, when Fedora Magazine could turn this chaotic thread into a nice reading article. Chaotic, as in, learning by try and error.

16

That is really up to the community member who volunteers towrite that hypothetical blog post. If you feel like contributing to such an effort please check out Propose an Article, and if you want to write it https://docs.fedoraproject.org/en-US/fedora-magazine/contributing/

17

To my surprise, I just verified the above rollback, and have corrected as follows:

18

Step 5 is not necessary. By default btrfs snapshots are read-write, with -r flag making them read-only. Bit obscure, but any ā€œreceivedā€ snapshots (using btrfs send|receive) have a received uuid field in their metadata, and this field is unset when using the property subcommand to change ro->rw (there is a warning). Itā€™s better to just make another snapshot rather than use the property subcommand. They are really that cheap.

2 Likes