Minimal partition layout UEFI system with btrfs volume

Ask Fedora
, ,
1

I am experimenting with making as few physical partitions as possible, but still be able to boot from it and ended up with one EFI partition as well as one btrfs partition containing a subvolume for “/” (and optionally one for “/boot”). Focusing on the system, I disregarded data subvolumes (like “/home”) for now.

In the past I ran into some issues with an EFI/LVM configuration, where out of a sudden “/boot” was no longer supported to be placed inside the logical volume manager (something, which worked before, but Anaconda simply refused). This required me to back up and erase the entire disk to accommodate “/boot” before the start of the LVM partition.

Therefore I wonder:
How stable and/or future-proof would a system with simply two partitions be? I understand that predicting the future is a little vague to ask, but maybe someone has experience on the stability or a recommendation for or against placing “/boot” into a btrfs subvolume. The disk is not encrypted.

I found a Bugzilla issue from about ten years ago being closed with no real closing note and the last comment noting that the issue is still present.
Though not fully understanding it, it may only be a concern in conjunction when using the OS prober.

2

GRUB2 has long had Btrfs read only support. This is reliable. And GRUB2 has long supported LVM. The gotcha is that the on-disk format can change in seemingly innocuous ways that a non-kernel fs driver (i.e. GRUB’s fs modules are not derived from kernel fs code). But these events get caught in testing and they get fixed.

But it is true that changing anything can have knock on effects. And eliminating separate /boot will be a puzzle. It’s an open question if all the knock on effects can be adequately addressed.

One of which, immediately, is that not all distribution kernels or GRUB core.img (which is the basis of the grub.efi OSloader) have Btrfs support. Pretty much all of them do support ext234 though.

Next difficulty, LUKS/dm-crypt for root. GRUB supports LUKS, not all instances of GRUB have it built.

If a distro GRUB or kernel lacks support for Btrfs, then that means dual-boot (distribution hopping or sharing) becomes different. And difficult. Do we care? Maybe some will care and others won’t, but it needs to be assessed.

Next difficulty, Anaconda doesn’t support configuring GRUB to support LUKS. It’s a mostly manual process to write the grub.cfg to present the interface for the user to interact with GRUB, provide the key to unlock the LUKS volume, and then for GRUB to locate and load kernel and initramfs on the encrypted volume.

Likewise if the paradigm will be to use measured boot, to unseal the LUKS key from a TPM, that’s not something Anaconda currently does.

Btrfs is getting fscrypt support. Patches are in review upstream.

fscrypt permits per directory encryption policy. So we can simply not encrypt /boot at a directory or subvolume level. So this makes things easier. Except…

This initial implementation on Btrfs does not encrypt all metadata, just file and directory names, and file data. All the other metadata is plaintext. Since certain software installations existing can be inferred from this metadata, it’s possible some use cases will continue to call for Btrfs on dm-crypt.

Therefore it’s not certain we can ever really get rid of the idea of a separate /boot file system until we’re willing to put in the effort into the installer to configure it for encrypted boot.

Configuring the bootloader for encrypted boot has its own drawbacks in UI and UX, what sorts of burdens are placed on the user - significantly altering their workflow and troubleshooting abilities.

Another gotcha with Btrfs for /boot is that it cannot be written to by GRUB for the successful boot test we use, to conditionally reveal the GRUB menu. By default the menu is hidden. If boot fails, it’s revealed. The way this works is GRUB in the pre-boot environment writes to a single disk block, changing a single value for a boot counter field. If boot succeeds, a user space program changes this field. This way GRUB knows if the prior boot was a success or failure and therefore whether to reveal the menu or not. Whew! OK so the problem is that GRUB is disallowed from writes to Btrfs because? This would be indistinguishable from corruption. The way GRUB writes to this single block is literally like dd - it reads the block, modifies one byte, writes the block back into place. On Btrfs this looks like corruption because csums weren’t updated. So it’s disallowed. Which means if /boot is on Btrfs, no more hidden GRUB menu feature.

OK, so could we move this grubenv file from /boot to the special bootloader region created for this exact purpose on Btrfs? Yes! But then we have to get upstream GRUB to agree to the change and that’s been years long disinterest. No one so far wants to do the work and get upstream to sign off on it. We should be able get upstream to agree by a simultaneous effort by (open)SUSE and Fedora.

So…therefore there’s inertia. As you can see one tiny change ends up having a lot of knock on effects.

At present I’m using 3 partitions: EFI/FAT, Boot/unencrypted Btrfs, and everything else/Btrfs on dm-crypt. And I’m find living without the GRUB hidden menu feature, but I’m not super comfortable taking that away from everyone else.

3

I envision being able to have a total of two partitions. The ESP and btrfs. Last I checked anaconda was hardcoded to require a boot partition but it really is an arbitrary requirement. With movement towards uki support and/or container native install methods that requirement should go away but I am not performing installs that way yet.

The ESP needs to be FAT and UEFI would load either a bootloader (maybe shim or sdboot) or directly load a uki. The bootloader would load a kernel and initrd with btrfs support included (from the ESP FAT filesystem). The uki would do the same without a bootloader (efistub) that effectively does the same thing. Neither case would require the bootloader to understand btrfs.

1 Like
4

Anaconda only requires a separate /boot file system if / file system is encrypted. The custom partitioner will permit /boot/ as a directory on ext4 and XFS, and as a subvolume on Btrfs - not so much because it’s necessary but because Anaconda’s Btrfs support ties mount points to subvolumes. If you create a mount point, Anaconda creates a Btrfs subvolume for it.

it’s possible to create UEFI fs drivers from either GRUB modules (efifs) or from a kernel module (I’m not offhand familiar with such a project). And then any bootloader could read whatever file system we want. This would permit a minimalist EFI system partition. And put big and growing things on a pool of bigger storage, i.e. opting out of dedicated/boot which has the same size limitations, eventually, as an ESP.

UKI makes the problem of space consumption worse. These things are big, and are growing bigger, right now mainly due to Linux firmware, but up and coming are video driver firmware blobs as well.

So I’m not entirely sold on the idea of having a 1G EFI system partition to store all of this stuff. That may not even be big enough today except for single OS usage. If it’s truly shared, it will need to be at least 1.5G per operating system. It’s a pain to future proof things outside of reprovisioning. It’s sometimes just easier to blow it all away and start over. Not easier for the end user. Easier for those building installers and updaters.

5

For me a single ESP that eliminates a /boot is a better option. The smallest new technology sdcard I could get is 64G which is what I want to use for the ESP then have an nvme that is full disk encrypted with a detatched header also stored on the ESP. After a successful boot the sdcard can be removed an only installed for booting and when it needs to be updated.

fedora packages efifs which has efi drivers for ext2 (works on ext4), xfs, btrfs and even zfs but I think things would be better having UEFI using builtin support for FAT and not add filesystem modules.

1 Like
6

Thanks for the extensive explanations! Very good aspects.

True, but encryption is not a thing in my setup.

Interesting. Do you have a link to where it is/was discussed?

This would be an acceptable loss for me as I know I can always press Esc at the right moment to reveal the GRUB menu.

What about SystemD-Boot? Would that be an alternative to GRUB for Anaconda. As it stands Anaconda will simply install GRUB and the user does not have a choice between bootloader 1, bootloader 2 or, danger danger, no bootloader (simply boot the latest kernel installed).

Thanks for that! :slight_smile:

On UEFI, Anaconda actually requires a separate EFI partition hardwired to /boot/efi (it does not allow any negotiation to that path). /boot can safely be placed as part of / (I tried it in a VM using EFI + btrfs), but likely needs a separate partition if / is encrypted.

Even a regular installation recommends 1G /boot right now (512M minimum), which I find excessive. I would rather have something small tailored to the system to start it (e.g. unlock procedure for LUKS, if needed) and the rest on the main system partition (a.k.a. /).

According to Arch Wiki when creating the ESP the recommended size is 1G even though according to the same page the “most recommended” way (since it is stated top-most) is to place all together into /boot.

When I compare the partition structure of macOS, I also see a separate 200M EFI partition followed by the rest. Using EFI + btrfs, I guess it is the closest match to it. Though I would welcome a bit more flexibility with the path since /boot/efi appears to be discouraged nowadays.

7

I would like to let UEFI builtin filesystem support be the only UEFI filesystem use I configure. Today that means FAT. Then the ESP and Everything Else are the minimum. Only one file needs to exist in the ESP if one were to direct boot a uki. Great for cloud environments where the vm is just thrown away and reinstantiated at will. That would allow for a minimally sized ESP (which combines current ESP and boot).

On baremetal having everything that is currently in /boot (and /boot/efi) change to being in the ESP is my preference. I would only mount it on the running system when it needs to be updated. I can select what to boot in UEFO or have a boot menu by using a bootloader like sdboot, the main use I want is fwupd but rollback support is also important. Booting hardware that is not UEFI would be a separate solution and I like the separation.

Combining current /boot with Everything Else is not something that appeals to me. Everything Else virtually cannot be FAT, a bootloader would be required, the bootloader would have to support the filesystem and it makes for additional complications.

1 Like
8

I can share my experience with the two-partition setup: EFI/Btrfs.
I’ve been using this partitioning layout since Fedora 33, and as of today, on Fedora 41, I haven’t encountered any issues. I’ve subjected my system to regular upgrades, version upgrades, and various rollbacks, even between different Fedora versions (e.g., F40 to F39, etc.).

(Including the reinstallation of Fedora with Anaconda’s advanced partitioning (Blivet): I recreate the EFI partition, do not format the Btrfs partition, delete the root, cache, and tmp subvolumes, keep the home subvolume and the flatpak subvolumes to retain the user home configurations and flatpak app settings.)

emanu@fedora ~> cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Mon Jul  8 16:53:57 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
UUID=9c1c4ece-ece6-4c69-8b21-ea85865d2abf /                       btrfs   noatime,compress=zstd:1,subvol=@ 0 0
UUID=1133-967D          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=9c1c4ece-ece6-4c69-8b21-ea85865d2abf /home                   btrfs   noatime,compress=zstd:1,subvol=@home    0 0
UUID=9c1c4ece-ece6-4c69-8b21-ea85865d2abf /opt                    btrfs   noatime,compress=zstd:1,subvol=opt      0 0
UUID=9c1c4ece-ece6-4c69-8b21-ea85865d2abf /var/cache              btrfs   noatime,compress=zstd:1,subvol=var_cache 0 0
UUID=9c1c4ece-ece6-4c69-8b21-ea85865d2abf /var/lib/flatpak        btrfs   noatime,compress=zstd:1,subvol=var_lib_flatpak 0 0
UUID=9c1c4ece-ece6-4c69-8b21-ea85865d2abf /var/lib/libvirt        btrfs   noatime,compress=zstd:1,subvol=var_lib_libvirt 0 0
UUID=9c1c4ece-ece6-4c69-8b21-ea85865d2abf /var/lib/machines       btrfs   noatime,compress=zstd:1,subvol=var_lib_machines 0 0
UUID=9c1c4ece-ece6-4c69-8b21-ea85865d2abf /var/log                btrfs   noatime,compress=zstd:1,subvol=var_log 0 0
UUID=9c1c4ece-ece6-4c69-8b21-ea85865d2abf /var/tmp                btrfs   noatime,compress=zstd:1,subvol=var_tmp 0 0
9

I’ve got you beat. I’ve been using the two-partition setup since Fedora Linux 28: 1693480 – systemd-bootx64.efi broken after upgrade to F29, system unbootable (but with systemd-boot).

Likewise, I haven’t had any upgrade problems (other than the F28 → F29 one in the above bug report). I mount my ESP at /boot.

3 Likes
10

Do you use kickstart?
I could not get the ESP mounted at /boot and looking at the anaconda code I concluded that it was hardcoded such that it would not work.
Can you share relevant ks.cfg snippets?

11

Yes and no. Originally, I used Anaconda, then dropped to a command line to fix things up the way I wanted, similar to what is demo’d in this video (interestingly, up to about 4 years ago, Anaconda would let you format /boot with VFAT as shown in the video):

I also have a Bash script that calls Anaconda with some extra kickstart snippets here: GitHub - gregory-lee-bartholomew/fedora-on-zfs: A script for automating the installation of Fedora Linux on a ZFS filesystem But that script is for installing Fedora Linux on ZFS (it configures the same sort of two-partition setup).

2 Likes
12

FYI, the “separate solution” for non-UEFI doesn’t have to be drastically different. The script I shared in my previous post just installs syslinux to the $BOOT partition as an extra step if legacy boot support is requested.[1] Other than that, there is no difference between the UEFI and legacy boot installs. (It even still installs sdboot when legacy boot is requested. But the BIOS won’t call that boot loader unless it is switched to UEFI mode.)

  1. ↩︎

13

This video is very useful and it appears to still be valid for Fedora 41. Tried it and it works as shown.

Small caveats:

  • When using EFI System Partition as format for /boot, it will mark the partition as EF00 as well as format it using vfat. So, the sgdisk command for the EFI partition is no longer needed.
  • Explicitly providing “/bin/bash --login” on a freshly installed system does not seem to do anything extra (unless I missed something).
  • The video depicted the Workstation Edition, but when using the Server Edition, sgdisk will either need to be installed (dnf install -y gdisk) or done from outside the chroot cage.
  • When using btrfs as filesystem for / one would need to look up the UUID of the partition and add root=UUID=_UUID_ rootflags=subvol=_root_subvolume_name_ to the kernel command line.
1 Like
14

The chroot command runs /bin/sh -i by default. Passing /bin/bash --login just gives you (maybe) a little better shell and maybe adds /usr/sbin and /usr/local/sbin to your path (more accurately, it causes /etc/profile to be sourced). It isn’t necessary.

P.S. If you didn’t append /bin/bash --login, that might be why you couldn’t find sgdisk (but it might well be that it isn’t available on that install disk as you suspect).

$ which sgdisk
/usr/sbin/sgdisk
1 Like
15

All in all very good and helpful (and/or insightful) comments everyone. It’s hard to choose one over the others as a solution. Everyone appears to agree that a 2-partition layout would be the minimum. And with a bit of elbow grease one can even install an alternative boot loader, e.g. SystemD-boot, despite using Anaconda for the package installation.

Thank you all for your respective inputs! You guys rock!

1 Like
16

Good catch. Though at first I tried it verbatim as depicted in the video, so I actually explicitly used the shell. I therefore believe that Server Edition does not have it preinstalled. It is, as remarked, a small caveat and really easy to fix, given you have Internet access while the system is in the installation environment.

1 Like