Hi!
I wondered if there was already a roadmap towards the idea of integrating system snapshots in the bootloader in the next fedoras based on btrfs?
Opensuse has a clear way to have system snapshots appearing in the bootloader / grub.
That is very precious to recover the distribution when graphics failed after an update… Just reboot in a read only snapshot selected from the bootloader…
Is there any wish/plan/roadmap towards such an (optional) easy implementation in Fedora (with eventually the tweaking of timeshift or snapper?) in a simplistic way for most users?
2 Likes
You mean something like
Changes/BtrfsWithFullSystemSnapshots
?
I’ve listened to some talk where the crew were discussing how it can be set up and about plans to integrate it further and enable by default. I believe this one: Getting Buttery with Fedora
2 Likes
Yes exactly what i mean.
But there is not any direction taken officially towards that yet, right?
For what I am familiar with topic, I’d say no, but perhaps someone officially tied to the project might chime in with their road-map 
I hope they don’t put /boot on the Btrfs volume. Doing so breaks things like https://www.kernel.org/doc/Documentation/efi-stub.txt. That layout is also incompatible with ZFS because grub has a design flaw in that it tends to lose access to the ZFS file systems when they are updated with the latest features. I know ZFS isn’t appropriately licensed for inclusion with Fedora Linux right now, but it could be someday. I’d much rather they keep /boot separate and just use traditional scripting to keep /boot updated when kernels are installed or removed.
Just my two cents.
Edit to add, I am cool with merging /boot and /boot/efi into a single partition however. That actually simplifies things IMO. But grub currently has issues with that layout (systemd-boot and syslinux, however, will work fine on a merged boot+ESP; fixing grub would be easier than changing everything else IMO).
2 Likes
There’s a bunch of issues that turn this whole thing into a game of Jenga! 
Short version:
I think we’re in the design phase; on the one hand can we adapt a snapper workflow as-is (pretty much) to the Boot Loader Spec, which is what we use in Fedora? What enhancements to Boot Loader Spec and Boot Loader Interface are needed? A lot of the people involved in the interface have other demands than just snapshot/rollback regime, so a holistic approach needs to happen to make it all possible. There are a number of a11y/i18n and UI/UX implications that need to be addressed.
is it possible we have an initial implementation that just merges existing snapper-like functionality with Boot Loader Spec? And then we have a 2.0 version down the road? Maybe. Or are those resources better used going directly to version 2.0 at a cost of a rollout farther down the road?
Longer version:
I’m not sure why /boot on Btrfs would break EFI stub any differently than /boot on XFS or ext4. It is true systemd-boot workflows depend on reading from a file system the firmware can support, and for most hardware that’s FAT16/32 (and HFS+ on Macs). There is the EfiFs project, which wraps GRUB filesystem drivers into EFI file system drivers; but there’s an sd-boot bug/enhancement that needs work to leverage it.
How much logic, complexity, and UI should the bootloader have in a snapshot/rollback regime? Even keeping it simple brings up all the a11y/i18n problems we see with disk encryption (whether bootloader or later in plymouth) so I think the interest level in this is going to be low. Further, really only one bootloader has the potential sophistication to handle such a regime, GRUB. This makes it not only a file system specific requirement but a bootloader specific one and that ends up implying an architecture division too.
What if the bootloader is simple it know nothing about Btrfs or snapshots? In traditional regimes, we don’t know for sure whether the update works until the next boot. And then the user has to intervene to do some kind of rollback via a bootloader menu of options. A common example of this is booting a read-only snapshot to test that it actually still works, and then indicating to some UI on the desktop that this worked so that a read-write snapshot is created from that working read-only snapshot, and setting it as the current sysroot to boot from, then rebooting (again). So it’s all very user interactive and requires both some amount of UI in the bootloader and on the desktop, two things that (open)SUSE has put effort into and Fedora hasn’t.
Another difference on Fedora is we’re using Boot Loader Specification. We’re no long making any modifications to grub.cfg when kernels are added or removed, the grub.cfg is a static file. All the dynamic information about boot is stored in a BLS formatted file in /boot/loader/entries. We could have a tool create not only kernel boot entries, but boot entries that indicate snapshots. Difficulty is without a pretty aggressive and rigid manager of snapshots, just providing boot entries is going to result in a lot of “forks” of the installed system and it’ll quickly become impossible to differentiate the various forks.
So that brings us to what sometimes called a “transactional” paradigm for snapshots and rollbacks, i.e. the idea is that instead of taking a snapshot for safekeeping while updating the currently running system, take a snapshot and update it “out of band” and if that fails just throw it away, no rollback required. Some sophistication could be added like trying to boot the update snapshot in a container, with some minimal milestones to act as tests to see if the update actually is bootable. If it fails, throw away the snapshot. If it succeeds, make the snapshot the next boot time system root. OK but now we have the problem with what to do with user initiated configuration changes that happen after an update begins. The update splits off from the currently running system and won’t have any configuration changes. So we need some way to duplicate the changes or merge them or prevent them. All of these have consequences of one sort or another.
But among the options I’m inclined to think we should create something transactional. And move away from having user interaction with the bootloader, aside from a single rescue option. We have a rescue option now but it’s too simple. We need a rescue environment that only boots from disk in read-only mode, then uses something like overlayfs to setup a “LiveOS”-like environment, in lieu of dropping to a dracut shell. And has working internet so users with a single computer aren’t totally left hanging, and instead get a kind of degraded/limited functionality system.
So 
it’s complicated.
7 Likes
I agree that it is complicated, but I think you might be over complicating things a bit.
Again, this is just my opinion on how things should be done.
Because none of those are FAT16/32.
This is where I think you are over-complicating things. The bootloader doesn’t have to do/know anything special. A couple very simple systemd services in dracut are sufficient to both snapshot and rollback the rootfs. I have written and use such scripts for ZFS on my own Fedora Linux systems (Add bootfs.snapshot and bootfs.rollback kernel parameters by gregory-lee-bartholomew · Pull Request #10198 · openzfs/zfs · GitHub). There is no requirement to do anything special in either the bootloader or on the desktop. The snapshot or rollback operations are initiated by specifying an option on the kernel command line.
I use BLS (even on my a few of my BIOS systems with a custom patch). It is not a problem.
OK. That is not an approach that I have considered. But it seems a little overkill to me. I’m not taking snapshots until after the updates have been installed. If the updates were to fail or damage my system in some way, I would have to rollback further – to the previous successful update – and then retry all the updates from that older point. I don’t consider this requirement unreasonable, but I can see where it might be a problem. The “if that fails” condition in you suggestion could be tricky to identify. For that reason, I don’t think I would want to take that approach.
I agree with that. I just think the work should be done in dracut and the kernel+dracut should not be on the CoW file system (it should be on the ESP). Dracut is already a very powerful out-of-band rescue environment on its own. Use it. 
I’ve been doing so for a few years on my ZFS-based Fedora Linux systems and it works great.
Just my two cents.
As a user, I am expecting something very simple (which I can use relatively soon):
- In Boot Menu, there is one entry marked as “Last Known Good”, which I can fallback to manually if needed.
- After booting “Last Know Good”, I can invoke a program to change next boot default - like bootctl --set-default=current
- I might not wrong “bad” snapshots to be removed automatically - as I might want to troubleshoot it. Thus might be another program - snapshotctl can be used to manage snapshots - like auto create by event trigger, timer trigger, etc, and auto remove by timer / entries count / manually, etc. It will be helpful if each snapshot will have status like good or bad, last booted, etc.
If the system can boot into last know good entry automatically if the update failed to boot is a bonus. Even without this bonus, it is still much better than the current setup.
1 Like
If I may have my say, as a simple user, I have been using Btrfs for many years, previously on Ubuntu and now on Fedora.
I’ve always used Timeshift, it’s a simple tool and does its job when needed, you can work from a GUI or CLI.
To give Fedora users an immediate advantage with Btrfs, isn’t it better to start proposing and running Timeshift by default on fedora?
Mint post installation or from the update tool, proposes to take a snapshot with Timeshift before performing an update.
Perhaps for the moment it is the easiest and most ready way to take advantage of Btrfs snapshots?
https://linuxmint-installation-guide.readthedocs.io/en/latest/timeshift.html
1 Like
But how does someone get to Timeshift if the system is broken? Would it allow a rollback of the entire OS? Say, for example, that someone wanted to try out Fedora Workstation 34 beta and then rollback to Fedora Workstation 33. Would Timeshift allow them to do that?
Also, I think a distinction should be made between a backup system and the recovery options. What I’m interested in is a robust and convenient recovery system (“last known good”). It should only maintain a few recent snapshots of the operating system (something on the order of 3 to 6 is quite sufficient). I would discourage this from being something that keeps massive backlogs of old installations on the user’s system. It should not be like email where old ones never get deleted until the system collapses under its own weight. If people want backups, that should be a different tool that sends the snapshots to an external device of some sort.
My two cents.
1 Like
If you can’t boot the system, you can retrieve the snapshot from a fedora live + Timeshift.
If the system starts partially, you can recover it with the timeshift CLI, if the system starts but with problems, just one click to restore.
There is a project to put Timeshift snapshots on GRUB, they use it on Garuda Linux.
Timeshift is software created for system snapshots only, not for data backups.
That’s not quite good enough for my needs. As a sysadmin, I have a half dozen physical servers that I remotely administer. The servers are configured with ipmi such that I can remotely reset them if they are hung and I can remotely access the boot menu over Serial over LAN. I need to be able rollback the server’s operating system without having to physically interact with the system and I need to be able to do it from a command line interface even when the system cannot mount its root file system.
The recovery system that I currently have implemented with a ZFS root file system and dracut installed on the ESP allows for the above.
1 Like
I don’t want to be restricted to grub. The system I currently use works with any bootloader. I currently use it with systemd-boot and syslinux. I don’t use grub because it doesn’t work properly/reliably with ZFS.
1 Like
I am referring to a use case for the user desktop running Fedora.
I’m not saying it has to be the definite solution, but pre-install it and make it work with the Btrfs subvolume layout on Fedora (there’s a fork and patch to make it work).
For example, recently I was very useful, for the test on Fedora 34, to check what worked, from the Timeshift GUI with a click I created the Fedora 33 snapshot> upgrade Fedora 34> test> Fedora 33 snapshot restore.
You can create manual snapshots, by hour, day, month, boot.
I agree to keep a few snasphots, I configure max 4 snapshots.
3 Likes
Some of that functionality is baked in rpm-ostree if you use Fedora Silverblue. I 'd suggest you give it a try in a VM to check how rpm-ostree rollback works.
1 Like
All answers above were very instructive, thank you for sharing!
I am still wondering if snapshots reachable in grub (à la Silverblue, or Opensuse) are considered by the main Devs team as a short term target?
I only recently begun working with Fedora (waiting official version 34 released to migrate my workstation). My tests in virtual machine and sandbox server seems to be OK using snapper and the project mentioned by Emanuele grub-btrfs. But please note I don’t have used it yet beyond simple tests.
Difficulty is without a pretty aggressive and rigid manager of snapshots, just providing boot entries is going to result in a lot of “forks” of the installed system and it’ll quickly become impossible to differentiate the various forks.
Perhaps BLS could be modified to allow a single boot entry file to contain a list of snapshots/subvols (for filesystems like BTRFS and ZFS that support them) rather than requiring a separate entry for each snapshot? The extra semantics would allow for boot loader UI that displays only a small list of bootable devices, but these would potentially be expandable to select a non-default subvolume entry if things went south. The software writing an entry file would still need to take care to order the snapshots usefully (presumably most recent first) and give them useful descriptive text of course.
There might be one potential issue with BLS type 2 entries. I don’t think Fedora Linux officially supports the type 2 UEFI booting yet, but it might be on the roadmap or Fedora Linux might want to consider it at some future point.
A type 2 BLS entry packages the kernel and initramfs together as a single EFI executable. This is particularly useful for secure boot because it allows all of the content (including everything in the initramfs) to be signed. But because it is a EFI executable that can be loaded directly from the system firmware, it must be stored on the ESP. The ESP is a distinct file system. Because Btrfs changes too much, it is highly unlikely that Btrfs will ever be supported for the ESP.
Because the kernel+initramfs are on the ESP, but the kernel modules are on the Btrfs (or ZFS) formatted root volume, an arbitrary snapshot cannot be chosen in the boot menu. The snapshot chosen must be one that contains the kernel modules needed for the kernel version selected in the boot menu.
IMO, the best option (and the one that I’ve implemented for ZFS), is to make a one-to-one correlation between the kernel-version and the recovery snapshot of the root file system. The way this is done for ZFS is that one sets a default kernel option – bootfs.snapshot – and that will take care of automatically creating a new snapshot of the root file system for each newly-installed kernel. The snapshot is named with the kernel version number so one can easily determine which snapshot goes with which kernel. When recovery is necessary, you make a one-time alteration of the kernel parameter interactively and change bootfs.snapshot to bootfs.rollback. Both the snapshots and the rollbacks are done from the initramfs stage before the root file system is mounted. So they are guaranteed to be clean (there is no chance that in-memory processes like databases have not flushed all their changes to storage).
I my experience, the above system works extremely well and reliably. I think the difficulty people run into is when they try to combine the boot-time recovery option with a backup system. The recovery system does not need to be (and IMO shouldn’t be) a backup system. Once you have used the recovery system to restore “something” in case of catastrophic failure, you can then (optionally) apply incremental backups to the recovery point to move the system state forward (or backward) to whatever point-in-time backup you want to restore. Moving the system state to a different backup/snapshot restore point can be done within a fully functional online (even graphical) environment.
Just my two cents.
1 Like
If we are talking about snapshot recovery and not snapshots being used for other data retention purposes, the fact that Fedora keeps 3 kernel versions around by default should be enough for most common use cases. Although it wouldn’t be bullet-proof it would be far simpler.
That being said, I have never had a single situation where I have wanted/needed to boot into an older filesystem snapshot. Even though I religiously take snapshots on both zfs and btrfs.
1 Like