F42 Change Proposal: Unification of boot loader updates, phase 1 (system-wide)

Project Discussion Change Proposals
,
1

Unification of boot loader updates, phase 1

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Wiki
Announced

:link: Summary

Begin the process of using single tool (likely bootupd) for most boot loader updates, especially grub and shim. This decouples package installation, e.g., via rpm transaction, from actual update in /boot and /boot/efi. This change does not affect zipl and systemd-boot.

The implementation details for this change are not fully fleshed out yet but we believe that there is a general agreement that this is the right way to go, so we are submitting it to start the discussion around how we should implement it. We will update the change based on feedback from those discussions.

:link: Owner

:link: Detailed Description

Our goal is to make boot loader updates in all environments: rpm mode, image mode, etc., safer and more reliable, and to add a fallback option so that a system is always bootable. We want to use a single interface, bootupd, to perform the actual boot loader update on a system once all new packages have been installed on that system.

On package mode systems (i.e. Workstation, Server, etc.), most components are updated monolithically along with boot loader packages, which automatically install into the /boot and/or /boot/efi directories, overwriting binaries that were there. Notably, kernel installation is performed via systemd script kernel-install, which runs at the end of the rpm transaction. Each distribution has a single directory structure constituting its bootpath, and if anything goes wrong during the subsequent boot, repair can be difficult or even impossible.

On image mode systems (i.e. bootc, CoreOS, IoT, Atomic Desktops), the boot loader update is decoupled from the system update, but, bootupd, once triggered, will also overwrite entries in /boot & /boot/efi, and any problem with the boot loaders has the potential to create an unbootable and unrecoverable system.

Rather than using different, yet equally precarious, methods for updating these critical components, we want to adapt bootupd to work on all systems in order to make boot loader updates easier and safer. It would additionally be aware of dependencies between packages, such as boot loaders and kernels. bootupd would (likely) be the interface that is used on all systems to perform the actual update once all new packages are available, and it would in the future retain a fallback option in case any problem arises during the subsequent boot process.

:link: This change will be split into two phases

Phase 1 (this change, proposed for Fedora 42)

  • Install the following packages into /usr/lib/<pkg>/<arch> rather than /boot & /boot/efi
    • grub2
    • shim
  • Separate rpm actions that happen at build time vs. runtime:
    • Add a mechanism that installs the bootloader for package mode installations as part of a RPM post trans. It is not yet defined how that will happen. Options that were suggested:
      • Add a dumb Bash script that copies the bootloader to the ESP
      • Add a new action verb to kernel-install, which will run script(s) for given component
      • Adapt bootupd to cover this case
  • Teach new boot loader location (/usr/…) to bootupd (needed for rpm-ostree & Bootable Containers variants)
  • Extend systemd package-notes to support PE files

Phase 2 (will be proposed for Fedora 43)

:link: Phase 1 (this change, proposed for Fedora 42)

In phase 1 of this change, we will focus on moving the content included in the grub and shim packages from /boot and /boot/efi to /usr and adding the logic needed to install them to /boot as part of a posttrans script.

Moving the content from /boot and /boot/efi to /usr (in the RPM packages) is a pre-requisite for all the potential future improvements as it decouples the bootloader installation phase via the RPM (that happens offline for all image mode systems and image builds in general) from the actual update in the ESP and any runtime EFI variable updates.

As we move the bootloader and shim to /usr, we now need a mechanism to sync the content from /usr to /boot and the ESP (and the MBR for BIOS systems, but we already have to do that). In all cases, the current suggestion is to do this synchronization step as part of a posttrans trigger at the end of the RPM transaction.

The first suggested option is to add a β€œdumb” Bash script that copies the new bootloader to the ESP partition.

The second suggested option is to add scripts to the kernel/install.d directory. Currently, when a new kernel gets installed, kernel-install (systemd) runs and invokes a number of bash and python scripts in /usr/lib/kernel/install.d/ which make the initramfs, create BLS entries, etc. We could extend kernel-install to add a new action that will be triggered when the relevant components (shim, grub2, sd-boot) are updated.

The third option is to adapt bootupd to install the updated bootloader to cover this case and run as part of the posttrans trigger.

:link: Phase 2 (will be proposed for Fedora 43)

In phase 2, we will look at unifying in a single tool how we do bootloader and shim updates, likely using bootupd, and extend the safe of boot loader updates, mainly on EFI based systems.

We will teach bootupd how to create a full new boot chain, e.g., shim β†’ grub2 β†’ kernel, based on newly installed components while retaining the working boot chain as a fallback option. Upon reboot, the new boot chain will be attempted, and if successful, set as the new default/fallback.

We will also need to choose how to trigger the script(s) so that in the end all new content is in a single directory in /boot, and a new efi entry is created for that directory and set to BootNext.

It has not yet been decided what trigger will be used to initiate the update. There is an idea to create a new systemd target, but obviously this has to happen before the system is rebooted.

One possibility is to use the reboot/poweroff target as the trigger: in case the system is powered off accidentally, no update would occur. If the user chooses to reboot, the new boot chain will be created just before shutdown so that it can be set as BootNext.

It has been proposed to use systemd package-notes for version numbers to track dependencies between kernels and boot loaders. This would remove bootupd’s dependency on rpm to scrape package numbers.

:link: Architecture comments

Phase 1 is architecture agnostic.

Our current targets for phase 2 are UEFI based systems: x86_64 and aarch64. Additionally, we believe ppc64le will want this support and will thus make itself more compliant in the meantime. s390 systems are out of scope at this time.

:link: Feedback

Q: What about UKIs?

A: This should not impact setups with UKIs. We will not touch the bootloader configuration itself or how GRUB works as part of this change.

:link: Benefit to Fedora

Once phase 1 and 2 are complete:

  • Boot loader updates are safer and more reliable.
  • The same mechanism is used for boot loader updates across Fedora products.
  • Fallback to known working boot loader is always available.

:link: Scope

  • Proposal owners:

    • Packaging changes to move the content from /boot to /usr/lib/… for the shim and grub2 packages.
    • Update the scripts needed to copy those files to /boot as part of a post-trans scriptlet
    • Teach bootupd about these new paths

  • Other developers:

    • Other packages that install files in /boot and /boot/efi and other bootloader packages should adapt for this change as well. It should still continue to work if they don’t, but they won’t benefit from the changes.

  • Release engineering: #Releng issue number

  • Policies and guidelines: N/A (not needed for this Change)

  • Trademark approval: N/A (not needed for this Change)

  • Alignment with the Fedora Strategy: Nothing specific

:link: Upgrade/compatibility impact

Changes will be self-consistent, so upgrading from the previous release should not be affected.

:link: Early Testing (Optional)

Do you require β€˜QA Blueprint’ support? Unclear yet

We will reach out to QA as soon as we have a proof of concept ready for testing.

:link: How To Test

We will make a COPR available with updated packages. Updating an existing system with the packages from this COPR should work and result in a working system without any specific action.

:link: User Experience

If this is done properly, this should be transparent for users.

fwupd does this already, as does systemd-boot (installs itself into /usr/lib/systemd/boot/efi). Other distributions, like Arch and Debian, install boot loaders into /usr/lib/grub/ rather than into /boot directly.

:link: Dependencies

systemd:

  • Extend package-notes to support PE files
  • Add a new action to kernel-install script if this is the option that is decided

bootupd:

  • Update to handle the new filesystem paths for the bootloader and shim components
  • Add support to run as posttrans is this is the option that is decided

:link: Contingency Plan

If each component is updated consistently, then dependency on other components is minimal. Components that are dependent, will wait until dependencies have been fulfilled before making changes.

Example: i) shim changes its installation location to /usr/lib/shim/x86_64/ and then runs a simple cp to move its components to /boot in posttrans; ii) shim script is added to /usr/lib/kernel/install.d/ and shim posttrans triggers the script, etc.

  • Contingency deadline: Beta release
  • Blocks release? Yes if not completed and not reverted

:link: Documentation

Documentation will have to be written.

:link: Release Notes

Fedora shim and grub2 packages are now installing content in /usr/lib/… and not in /boot and /boot/efi directly.

Last edited by @amoloney 2025-01-10T19:47:06Z

Last edited by @amoloney 2025-01-10T19:47:06Z

1 Like
2

How do you feel about the proposal as written?

  • Strongly in favor
  • In favor, with reservations
  • Neutral
  • Opposed, but could be convinced
  • Strongly opposed
0 voters

If you are in favor but have reservations, or are opposed but something could change your mind, please explain in a reply.

We want everyone to be heard, but many posts repeating the same thing actually makes that harder. If you have something new to say, please say it. If, instead, you find someone has already covered what you’d like to express, please simply give that post a :heart: instead of reiterating. You can even do this by email, by replying with the heart emoji or just β€œ+1”. This will make long topics easier to follow.

Please note that this is an advisory β€œstraw poll” meant to gauge sentiment. It isn’t a vote or a scientific survey. See About the Change Proposals category for more about the Change Process and moderation policy.

3

Just my two cents, but it’d be great if there were a single command that could β€œregenerate” the ESP to make a system bootable. I’d like to be able to 1) partition a drive with a ESP and root partition 2) restore a Fedora Linux backup to the root partition with a Btrfs (or ZFS) β€œsend” command 3) run a single command (perhaps interactively) that would fix/regenerate the ESP to make my system bootable (this should work even if the ESP is blank/empty). IIUC, I think the idea of adding a new verb to the existing kernel-install command would be most likely to accomplish that.

Edit: I just discovered that bootctl already supports this. It is in the man page:

--image=image
Takes a path to a disk image file or block device node. If specified, all operations are applied to file system in the indicated disk image. This option is similar to --root=, but operates on file systems stored in disk images or block devices. The disk image should either contain just a file system or a set of file systems within a GPT partition table, following the Discoverable Partitions Specification[4]. For further information on supported disk images, see systemd- nspawn(1)'s switch of the same name.

I’ll test it out the next chance I get.

4

I believe even phase 1 needs to support (or at least tolerate without breaking) systemd-boot (and sdubby?), as Fedora has supported systemd-boot for at least a release or two, and any breakage to that would be a reason to reject the current plan.

1 Like
5

Note that the RPM package that includes systemd-boot already does not install it directly to /boot nor the ESP so there is β€œnothing” to do for this package for phase 1.

6

This is also what bootupctl backend install (as part of bootupd) does but I agree that it’s not well documented.

1 Like
7

Some clarifications:

  • This change is for all Fedora variants (not just the Atomic ones).
  • This change was submitted late due to a mistake on my side. It might be delayed for F43 (and phase 2 to F44).
  • For phase 1, we do not plan to make any change to the way the system boots, only how the (GRUB) binaires are installed to /boot and the ESP as part of RPM updates.
1 Like
8

As I do understand we just talk about the config files for grub for the first stage? Take them out of the /boot folder and integrate in /usr/lib/kernel/install.d ?

If not it might be easier to understand when we know exactly about which files we talk about. Below i made a tree of my /boot folder (more or less a fresh default F41 WS installation).

/boot folder 
/boot
β”œβ”€β”€ /boot/config-6.12.6-200.fc41.x86_64
β”œβ”€β”€ /boot/config-6.12.7-200.fc41.x86_64
β”œβ”€β”€ /boot/config-6.12.8-200.fc41.x86_64
β”œβ”€β”€ /boot/efi
β”‚   β”œβ”€β”€ /boot/efi/EFI
β”‚   β”‚   β”œβ”€β”€ /boot/efi/EFI/BOOT
β”‚   β”‚   β”‚   β”œβ”€β”€ /boot/efi/EFI/BOOT/BOOTIA32.EFI
β”‚   β”‚   β”‚   β”œβ”€β”€ /boot/efi/EFI/BOOT/BOOTX64.EFI
β”‚   β”‚   β”‚   β”œβ”€β”€ /boot/efi/EFI/BOOT/fbia32.efi
β”‚   β”‚   β”‚   └── /boot/efi/EFI/BOOT/fbx64.efi
β”‚   β”‚   └── /boot/efi/EFI/fedora
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/BOOTIA32.CSV
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/BOOTX64.CSV
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/gcdia32.efi
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/gcdx64.efi
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/grub.cfg
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/grubia32.efi
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/grubx64.efi
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/mmia32.efi
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/mmx64.efi
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/shim.efi
β”‚   β”‚       β”œβ”€β”€ /boot/efi/EFI/fedora/shimia32.efi
β”‚   β”‚       └── /boot/efi/EFI/fedora/shimx64.efi
β”‚   β”œβ”€β”€ /boot/efi/mach_kernel
β”‚   └── /boot/efi/System
β”‚       └── /boot/efi/System/Library
β”‚           └── /boot/efi/System/Library/CoreServices
β”‚               └── /boot/efi/System/Library/CoreServices/SystemVersion.plist
β”œβ”€β”€ /boot/grub2
β”‚   β”œβ”€β”€ /boot/grub2/fonts
β”‚   β”‚   └── /boot/grub2/fonts/unicode.pf2
β”‚   β”œβ”€β”€ /boot/grub2/grub.cfg
β”‚   └── /boot/grub2/grubenv
β”œβ”€β”€ /boot/initramfs-0-rescue-d5a3eb99a725425a968d53cdb7ab2fde.img
β”œβ”€β”€ /boot/initramfs-6.12.6-200.fc41.x86_64.img
β”œβ”€β”€ /boot/initramfs-6.12.7-200.fc41.x86_64.img
β”œβ”€β”€ /boot/initramfs-6.12.8-200.fc41.x86_64.img
β”œβ”€β”€ /boot/loader
β”‚   └── /boot/loader/entries
β”‚       β”œβ”€β”€ /boot/loader/entries/d5a3eb99a725425a968d53cdb7ab2fde-0-rescue.conf
β”‚       β”œβ”€β”€ /boot/loader/entries/d5a3eb99a725425a968d53cdb7ab2fde-6.12.6-200.fc41.x86_64.0~custom.conf
β”‚       β”œβ”€β”€ /boot/loader/entries/d5a3eb99a725425a968d53cdb7ab2fde-6.12.6-200.fc41.x86_64.conf
β”‚       β”œβ”€β”€ /boot/loader/entries/d5a3eb99a725425a968d53cdb7ab2fde-6.12.7-200.fc41.x86_64.conf
β”‚       └── /boot/loader/entries/d5a3eb99a725425a968d53cdb7ab2fde-6.12.8-200.fc41.x86_64.conf
β”œβ”€β”€ /boot/symvers-6.12.6-200.fc41.x86_64.xz
β”œβ”€β”€ /boot/symvers-6.12.7-200.fc41.x86_64.xz
β”œβ”€β”€ /boot/symvers-6.12.8-200.fc41.x86_64.xz
β”œβ”€β”€ /boot/System.map-6.12.6-200.fc41.x86_64
β”œβ”€β”€ /boot/System.map-6.12.7-200.fc41.x86_64
β”œβ”€β”€ /boot/System.map-6.12.8-200.fc41.x86_64
β”œβ”€β”€ /boot/vmlinuz-0-rescue-d5a3eb99a725425a968d53cdb7ab2fde
β”œβ”€β”€ /boot/vmlinuz-6.12.6-200.fc41.x86_64
β”œβ”€β”€ /boot/.vmlinuz-6.12.6-200.fc41.x86_64.hmac
β”œβ”€β”€ /boot/vmlinuz-6.12.7-200.fc41.x86_64
β”œβ”€β”€ /boot/.vmlinuz-6.12.7-200.fc41.x86_64.hmac
β”œβ”€β”€ /boot/vmlinuz-6.12.8-200.fc41.x86_64
└── /boot/.vmlinuz-6.12.8-200.fc41.x86_64.hmac

12 directories, 46 files
9

Let’s take an example to make this more concrete. Here are the file listing for the following RPM packages:

/boot/efi/EFI/BOOT/BOOTX64.EFI
/boot/efi/EFI/BOOT/fbx64.efi
/boot/efi/EFI/fedora/BOOTX64.CSV
/boot/efi/EFI/fedora/mmx64.efi
/boot/efi/EFI/fedora/shim.efi
/boot/efi/EFI/fedora/shimx64.efi
/etc/dnf/protected.d/shim.conf

/boot/efi/EFI/fedora/grub.cfg
/boot/efi/EFI/fedora/grubx64.efi
/boot/grub2/fonts
/boot/grub2/fonts/unicode.pf2
/boot/grub2/grub.cfg
/boot/grub2/grubenv
/boot/loader/entries
/etc/dnf/protected.d/grub2-efi-x64.conf
/etc/grub2-efi.cfg
/etc/grub2.cfg

With this change, we would change the packages to install the files to /usr instead.

Example:

  • shim-x64
/usr/lib/shim-x64/EFI/BOOT/BOOTX64.EFI
/usr/lib/shim-x64/EFI/BOOT/fbx64.efi
/usr/lib/shim-x64/EFI/fedora/BOOTX64.CSV
/usr/lib/shim-x64/EFI/fedora/mmx64.efi
/usr/lib/shim-x64/EFI/fedora/shim.efi
/usr/lib/shim-x64/EFI/fedora/shimx64.efi
/etc/dnf/protected.d/shim.conf
  • grub2-efi-x64
/usr/lib/grub2-efi-x64/EFI/fedora/grub.cfg
/usr/lib/grub2-efi-x64/EFI/fedora/grubx64.efi
/usr/lib/grub2-efi-x64/boot/grub2/fonts
/usr/lib/grub2-efi-x64/boot/grub2/fonts/unicode.pf2
/usr/lib/grub2-efi-x64/boot/grub2/grub.cfg
/usr/lib/grub2-efi-x64/boot/grub2/grubenv
/usr/lib/grub2-efi-x64/boot/loader/entries
/etc/dnf/protected.d/grub2-efi-x64.conf
/etc/grub2-efi.cfg
/etc/grub2.cfg

And then add a script as part of a posttrans trigger that copies those files to /boot & the ESP (/boot/efi). How to copy (i.e. sync) those files is what’s discussed in the three options in the proposal.

2 Likes
10

Reply from @zbyszek in F42 Change Proposal: Unification of boot loader updates, phase 1 (system-wide) - devel - Fedora mailing-lists

11

/etc/dnf/protected.d/grub2-efi-x64.conf and /etc/dnf/protected.d/shim.conf can be moved too. See PR#29: Move yum/dnf protection removal config file under /usr - rpms/sudo - src.fedoraproject.org. I think that in F42 we won’t have dnf4 or yum, so the file can actually moved without keeping the old one in /etc.

1 Like
12

I want to address Zbyszek’s concerns in the email:
You’re correct that build time and runtime are the wrong words. It could perhaps be download time vs installation time? I’m not sure if those are exactly right, although I hope the rest of the document is better about this, and the idea of what the change is about is coming across clearly.

As for the mechanism for updating the content in /boot with what’s in /usr, I would lean towards the third option, of using bootupd or a bootupd-like tool, but I would be curious to know more about the expansion of bootctl that you mention. I agree that kernel-install is not the right place to do this.

Thanks for your questions and ideas. (:

13

And thank you for this information. We can implement that right away too.

14

I would second the idea of using bootctl. Fully restoring a ESP would then require two commands – bootctl and kernel-install – but both of those commands already have an --image=... option which is what I really want so I can (easily) restore a backup.

15

This change proposal has now been submitted to FESCo with ticket #3344 for voting.

To find out more, please visit our Changes Policy documentation.