Btrfs push with Fedora - where does that leave Silverblue?

ETA: the following post is user error, cmurf helped me out with a bug here, read a few posts down

I blew up my system today and tried installing with Btrfs. I’m not going to lie, it didn’t work out well and I’m a little upset about the time I lost today.

I made the USB stick with Fedora Media Writer (and the x86_64 image). My computer is Lenovo ThinkPad e585; it is using UEFI and Secure Boot. I couldn’t find anywhere if Secure Boot was perhaps a problem but even if it was, I’m not budging on that.

When in the installer I deleted all the old partitions and did an automatic Btrfs population. Install went fine and I got to the point where I needed to reboot, so I did. Device gets caught in a boot loop. I reinstalled again and the same. I reinstalled again and changed the /boot to XFS, then again with Btrfs, no dice on either.

I reinstalled just now with LUKS, LVM, and XFS, and I have a working system again. This was a frustrating experience. Not sure I can endorse this move to Btrfs. It’s really a shame we can’t use ZFS considering it is such a known good, but I might give Btrfs another shot in the winter with F33 to see if the installer is improved.

ETA: the above post is user error, read a couple posts down for explanation

1 Like

My understanding is that the /boot should be etx4.

You likely ran into this bug, which was mentioned in the 3rd post of this thread.

Bug 1753485 - silverblue on btrfs, missing rootflags param causes startup failure

1 Like

cmurf I think you are right. In that case it is my own fault for not reading, my apologies. I will give this another shot next weekend.

I’d like to try some of the optimizations - /boot is Btrfs subvolume and no swap / RAM compression. Any suggestions on getting those enabled? Are there any other flags I could enable that would be useful for testing?

It’s a legit bug that ‘rootflags’ param is missing, and the bug report is pretty intense (booting is complicated). Anyway, there is a fix for that now, and in Rawhide Silverblue, but Rawhide Silverblue also has systemd-246 and it’s causing some problems right at the end of a clean install. (You won’t his this bug if you switch to a current Rawhide deployment on Silverblue.)

So for now you’ll want to use the Fedora 32 Silverblue installer - and then before you reboot, fix the bootloader snippets. (It’s a separate bug that there are two snippets per kernel per deployment. You don’t need to fix both, but it might be less confusing to fix both.) Basically you’re looking for an existing rootflags argument, and make sure it’s rootflags=subvol=root in both. My recollection is that one snippet it’s missing, and in the other snippet it’s malformed.

Backing up a bit. When doing the Fedora 32 Silverblue installation to Btrfs, using Custom partitioning:

  1. Use the Btrfs preset in the drop down menu. Click the (paraphrased) ‘create partitions automatically’ blue text “link”.
  2. Click on the /home mount point and modify the mount point so it’s /var/home.
  3. You could optionally delete the swap partition. But you might want to keep it, because it’ll be the only swap available if you’re booting a Fedora 32 deployment. If you keep it, when Fedora 33 deployments get zram-generator, you’ll have two swap devices, with the zram-based one having a higher priority.

You might just want to see how Btrfs performs as-is without additional options. A legit experiment is to use the system exactly as it is by default, and question whether in fact the things you normally do everyday are the same.

One thing you might want to set: ‘nodatacow’ using chattr +C /var/lib/libvirt/images if you use virt-manager. The GNOME Boxes location is ~/.local/share/gnome-boxes/images.
That’s going to be done by default in Fedora 33 installs, but it isn’t done now for the user. It’s best to do it on the enclosing directory because then it means VM images copied over will just inherit ‘nodatacow’. Same for the equivalent GNOME Boxes location.

2 Likes

Hey cmurf, check it out. Just installed F32 SB with the automatic Btrfs partitioning. I really appreciate your help, my apologies again for complaining about something I didn’t read through properly.

I’ll kick the tires on this and see how it holds up, thanks again! I am satisfied with my service, you can close my ticket now :wink: Next thread I’m mulling over btw is included codecs and drivers.

1 Like

Thanks too to jakfrost, dcavalca, walters, and glb!

Sorry if this is answered in the thread or elsewhere, but if I do a clean Silverblue install after F33 is out, will I still need to use a workaround or manual intervention if I want btrfs?

1 Like

Hello @evenreven,
I think the idea was the defaults for Silverblue will be better tuned in F33. So even with F32, just using the BTRFS defaults as setup in the installer should be sufficient in most cases.

1 Like

Should we do the same for ~/.local/share/containers/ ?

Should we do the same for ~/.local/share/containers/ ?

No. Those are good targets for compression, because they are mostly read only except during updates. And one of the advantages of containers on Btrfs is the overlayfs copyup is made much less expensive due to reflink copies. (Reflink copies mean the metadata is copied, i.e. the inode plus other metadata, but not the data extents. The data extents become shared among the reflinks. Also referred to as file cloning, and also “efficient copies”.)

It’s not immediately planned but there is an even more efficient option for Btrfs, there is a Btrfs specific driver that can use snapshots instead of overlayfs+reflinks. Certain workloads that don’t need to take advantage of the page cache sharing mechanism of overlayfs would benefit from this, as Btrfs snapshots are even less expensive than reflink copies. But yeah, we’ll have to prove it with some sane benchmarking.

This is the change that landed today to automatically take care of nodatacow for tools that use libvirt: virt-manager, GNOME Boxes, Cockpit, etc.

3 Likes

Yes. My advice is to use the defaults and complain about them so they get better. :smiley: We really shouldn’t expect every day users to have to make tweaks so Btrfs behaves reasonably. It needs to behave well out of the box for the common use cases we’re targeting. It’s neat that Btrfs has ways of doing optimizations on a case by case basis, often even at the file level, for special requirements. But that can’t be a burden on the vast majority of users, or a source of excuse making like - oh well you didn’t do XYZ incantation with the magic sauce command, etc.

2 Likes

Could you please advice me if NOCOW is desirable for containers with extensive compiling workload? Also, should I use overlayfs or btrfs drivers for them?

PS: why I’m asking is because I see huge CPU load with “overlayfs” process when compiling, and it feels slow overall. So I wonder if it could be improved somehow.

‘nodatacow’ is mainly applicable for VM images. I don’t use it for containers or for compiling.

The configuration files provided by containers-common used by podman (and I think mobi and docker) on Fedora defaults to overlayfs driver. For ext4 this means a traditional copy up operation, where on Btrfs it will automatically do reflink copies. I expect overlayfs will remain the default driver for Fedora 33. I have used, and know others using the btrfs driver extensively so you can use it.

But in my compile tests between ext4, btrfs, and xfs, they’re all in the same ballpark (perhaps a couple seconds variance). I would expect container operations themselves to be faster on Btrfs or XFS compared to ext4 due to reflink copies.

If you’ve discovered a performance regression, it might be a bug so I suggest filing a bug report and include the reproduce steps - sufficiently rudimentary that non-podman familiar folks can get a particular image, get the source, and run the compile the same way with the same options as you are, and in the same A vs B configurations. That way others can test it, and profile it. The bcc-tools provides pretty sophisticated ways of measuring various kinds of latency, so it should be possible to track down.

3 Likes

I can’t remember if hitting the solution button closes the thread completely or not. I rather like the discussion going on in this thread, it is quite helpful Q&A. I could change the title?

So I’m replying from a newly minted F32 Silverblue install using BTRFS on BIOS with the system setup as follows …
1 TB spinning disk with 2MB BiosBoot partition, 1.1GB ext4 partition mounted as /boot and the remainder as BTRFS volume @ 999GB with a /var subvolume.
2 240GB SSD’s as a software RAID-1 Array BTRFS filesystem with subvolume mounted as /.
It wasn’t without pitfalls; First the installer was failing at using the custom partitioning, if I selected btrfs for filesystem and auto-allocation a popup would present an error with the option to debug or quit. After quitting and rebooting, the USB wouldn’t boot, so made another installer on newer media, same result. Chose to power down then restart and select Blivet Advanced custom, and this worked as expected.
Other issues related to using BIOS with GPT, ie. you need a 1024K BiosBoot partition, btrfs makes this 2mb. Plus I originally had the spinning disk physically located in SATA slot #3 (/dev/sdc), Grub2 would boot, because I could select the system and the logo with the spinner would come up, but then it would switch back to the text boot up, and hang there. So I moved the HDD to SATA #0 (/dev/sda) and the two SSD’s as SATA #1 and #2 respectively (/dev/sdb & /dev/sdc). This allowed the installation to proceed correctly.
For the RAID-1 array, it is important to specify the software RAID setup in advance of selecting the filesystem. Once the filesystem is selected, the software RAID option disappears if it hasn’t been selected first. I used rsync to backup my home dir to external storage, then used it to restore my home. So far so good.

2 Likes

Hi jakfrost. It sounds like you managed to get a pretty slick setup working there. :slightly_smiling_face: It’s not too far off from what I think would do personally with that sort of hardware.

It sounds like most of your problems were with grub, not Fedora. The 2mb requirement for the BiosBoot partition probably just means that the grub btrfs driver is quite large. This is one of my bigger complaints about grub – it rolls its own file system drivers. I’m always worried (and have had it happen with zfs) that the grub driver will be out-of-sync with the operating system’s file system driver in such a way that grub will not be able to access the rest of the file system and complete the boot process.

Another annoyance with grub is all the partitions that are required when only one should really be necessary. You really shouldn’t need BiosBoot + ESP + XBOOTLDR. One ESP should be quite sufficient. By keeping the kernel+initramfs on the ESP, the normal operating system’s file system driver can be used from the get go. Also, the driver in the initramfs is regularly updated with the new kernel releases, so there is little chance of it not supporting a new file system feature. This may be more of a concern for zfs users than for btrfs users since zfs allows file systems to be “upgraded” (I’m not familiar enough with btrfs to know if it supports that sort of thing).

I think the reason people are hesitant to use a merged ESP+XBOOTLDR setup is because they are worried about the security of putting the kernel on the ESP. It’s really not a problem though. Permissions (both DAC and MAC) can be set on vfat file systems at mount time. Also, see the below links for some great ideas about how you could actually make a ESP+XBOOTLDR setup much more secure than a traditional split-partition setup.

  1. https://www.reddit.com/r/archlinux/comments/bq272d/disk_encryption_is_mounting_esp_to_boot_safe_or/

  2. https://tech.michaelaltfield.net/2020/01/02/buskill-laptop-kill-cord-dead-man-switch/

Just my two cents. :slightly_smiling_face:

1 Like

Hello @glb,
I thought I was the victim of the bug @cmurf mentioned above about Silverblue on btrfs missing rootflags param causing startup failure. Although, would said bug be fixed by moving the physical interface for the drives? Perhaps Grub has difficulty with just accepting the BIOS picking SATA2 for primary boot, and expects the primary boot to reside on SATA0 in the case of my board.

Hi jakfrost ,

I can’t say for sure, and it has been a while since I messed with grub, but your problem with grub detecting the file system might be to do with grub’s “probing” logic. As I recall, grub tries to detect which drive should be “hda”, “hdb”, etc. and compiles whatever it thinks they should be into the “core” image (I think this is the first-stage bootloader for grub). There used to be some sort of drivemap file for grub that would let you override the problem when it got it wrong. I think they may have since switched to using UUID’s or something though. In the end, I think all it does is set two variables – “root” and “prefix”. It is complicated. You might try to extract that core image if you really want to troubleshoot what is going on there.

Again, I’m not at all sure that this is the cause of your problem, but it is my guess based on some similar experiences that I had with grub a long time ago.

installer was failing at using the custom partitioning, if I selected btrfs for filesystem and auto-allocation a popup would present an error with the option to debug or quit.

This is definitely a bug. If you can remember the steps that trigger the error dialog I’ll try to reproduce and file a bug report. It should be fixed.

For the RAID-1 array, it is important to specify the software RAID setup in advance of selecting the filesystem.

Custom partitioning will prevent you from using Btrfs on mdadm RAID, or LVM. Advanced-Custom won’t. It’s valid to create an mdadm raid1, and then put Btrfs on top, in particular if you’re most familiar and comfortable with mdadm. But while you do get error detection still from Btrfs in this configuration, it can’t self-heal because as far as its concerned, there’s only one copy of everything.

I thought I was the victim of the bug @cmurf mentioned above about Silverblue on btrfs missing rootflags param causing startup failure. Although, would said bug be fixed by moving the physical interface for the drives?

No.

The rootflags boot parameter is translated as “mount options for the root file system”. The Fedora way of installing to Btrfs uses subvolumes for system root and home. The way they get mounted at / and /home is the Btrfs mount option subvol. If you check your /etc/fstab/ it’ll become more clear. At boot, before fstab is available, there needs to be a way of defining what is the root file system. Ordinarily this is done only with root=UUID=<uuid> boot parameter, but on Fedora Btrfs systems there’s an additional hint, which is the subvol=root mount option you see in fstab. That becomes rootflags=subvol=root as a boot parameter.

1 Like