New kernels not being installed in /boot anymore

Ask Fedora
, , ,
1

Why would Fedora stop installing kernel updates in /boot, even though the packages are installed?

This is F41, recently upgraded from F40 using plasma-discover (GUI). It has 3 kernel packages installed:

# rpm -q kernel
kernel-6.8.9-100.fc38.x86_64
kernel-6.13.12-200.fc41.x86_64
kernel-6.14.5-200.fc41.x86_64

However the first one is the only one that’s actually available even though it’s labeled “fc38”:

# ls /boot/*x86*.img
/boot/initramfs-6.8.9-100.fc38.x86_64.img

# sudo grubby --info=ALL | grep title | grep x86
title="Fedora Linux (6.8.9-100.fc38.x86_64) 38 (Thirty Eight)"

I could not find anything about those newer kernels in the dnf log, although it’s difficult to say because they are huge.

# grep -Rn 'kernel-6.13.12' -c dnf.log
0

The big question is why the kernel images 6.13 and 6.14 were not installed when these packages were installed.

Also what if the next update attempt tries to install another new kernel package and thereby uninstalls the only working kernel 6.8.9 because Fedora usually keeps 3?
Whenever a release upgrade is initiated using gnome-software or plasma-discover, it usually says that it will remove “kernel”, “kernel-headers” etc. for that reason but normally what that actually means is that it would only remove the oldest version in order to install a newer one.

4

Try running sudo dracut does it complete successfully?
Do you happen to have dkms setup to build out-of-tree kernel modules?
If so maybe they no longer build?

5

Running dracut wants to overwrite the only kernel image in the system which I obviously don’t want to do:

# dracut
dracut[F]: Will not override existing initramfs (/boot/initramfs-6.8.9-100.fc38.x86_64.img) without --force

According to the man page, this syntax can be used to select a kernel version:

       A shortcut to generate the image at the default location for a specific kernel version is:

           # dracut --kver 2.6.40-1.rc5.f20

The .x86_64 suffix is missing though.

# dracut --kver 6.13.12-200.fc41
dracut[F]: Cannot find module directory /lib/modules/6.13.12-200.fc41/
# dracut --kver 6.13.12-200.fc41.x86_64

This last command was successful, but there’s still no newly generated image in /boot.

# ls /boot/*x86*.img
/boot/initramfs-6.8.9-100.fc38.x86_64.img

# ls /boot/loader/entries/
d4033fa8d6ca48a68402a365aef6f348-0-memtest86+.conf  d4033fa8d6ca48a68402a365aef6f348-6.8.9-100.fc38.x86_64.conf
d4033fa8d6ca48a68402a365aef6f348-0-rescue.conf
6

i would check, if the /boot volume is mounted:
mount | grep /boot
Output should look like this:

/dev/nvme0n1p2 on /boot type ext4 (rw,relatime,seclabel)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
user@host:~$

my guess: perhaps, the /boot volume wasn’t mounted for some reason during installation and so those kernels files landed on the wrong volume (perhaps on “/”)

Try to umount all related /boot/* Volumes followed by /boot itself and recheck the content of /boot

7

Running dracut would only recreate the initrd file. To install the kernel as well you should run

kernel-install -v add 6.14.5-200.fc41.x86_64 /lib/modules/6.14.5-200.fc41.x86_64/vmlinuz

WIth the -v option you get additional debug information which could show what, if anything, went wrong.

8

Thanks for bringing up the kernel-install command.

It runs for a while, producing output…

# kernel-install -v add 6.14.5-200.fc41.x86_64 /lib/modules/6.14.5-200.fc41.x86_64/vmlinuz
Loaded config.
...
kernel image file (/lib/modules/6.14.5-200.fc41.x86_64/vmlinuz) set via command line.
/boot/efi/d4033fa8d6ca48a68402a365aef6f348 exists, using layout=bls.
Using ENTRY_DIR=/boot/efi/d4033fa8d6ca48a68402a365aef6f348/6.14.5-200.fc41.x86_64
mkdir -p /boot/efi/d4033fa8d6ca48a68402a365aef6f348/6.14.5-200.fc41.x86_64
Using plugins: 
...
Successfully forked off '(direxec)' as PID 387465.
dracut[I]: Executing: /usr/bin/dracut -f --verbose --kernel-image /lib/modules/6.14.5-200.fc41.x86_64/vmlinuz --no-uefi --kver 6.14.5-200.fc41.x86_64 /tmp/kernel-install.staging.UzmAXD/initrd
dracut[I]: *** Including module: systemd ***
...
/usr/lib/kernel/install.d/50-dracut.install succeeded.
...
dracut[I]: *** Creating image file '/boot/efi/d4033fa8d6ca48a68402a365aef6f348/0-rescue/initrd' ***
dracut[I]: Using auto-determined compression method 'zstd'
cp: error writing '/boot/efi/d4033fa8d6ca48a68402a365aef6f348/0-rescue/initrd': No space left on device
dracut[F]: Creation of /boot/efi/d4033fa8d6ca48a68402a365aef6f348/0-rescue/initrd failed
Creating /boot/efi/loader/entries/d4033fa8d6ca48a68402a365aef6f348-0-rescue.conf
/usr/lib/kernel/install.d/51-dracut-rescue.install: line 91: /boot/efi/loader/entries/d4033fa8d6ca48a68402a365aef6f348-0-rescue.conf: No such file or directory
/usr/lib/kernel/install.d/51-dracut-rescue.install failed with exit status 2.
/usr/lib/kernel/install.d/51-dracut-rescue.install failed with exit status 2.
(sd-exec-strv) failed with exit status 2.

So apparently, the efi partition is full (after years of operation). With btrfs subvolumes instead of partitions this would not have happened (the root filesystem is btrfs).

# df -h /boot /boot/efi
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  924M  135M  725M  16% /boot
/dev/nvme0n1p1  100M   59M   42M  59% /boot/efi

This must have been why recent updates and release upgrades failed to install newer kernel images. I wonder why that failed silently without any error. Isn’t this installation process run in a post scriptlet of the RPM package?

I’m off to shrink the main filesystem before shrinking its partition to make space for the efi partition…

9

That is odd… Why is it putting the rescue kernel into /boot/efi?
It is supposed to go in /boot.

From my system:

$ ls -1 /boot
config-6.14.4-300.fc42.x86_64
config-6.14.5-300.fc42.x86_64
config-6.14.6-300.fc42.x86_64
efi/
extlinux/
grub2/
initramfs-0-rescue-1c29c17391714633b75beebc6deb0389.img
initramfs-6.14.4-300.fc42.x86_64.img
initramfs-6.14.5-300.fc42.x86_64.img
initramfs-6.14.6-300.fc42.x86_64.img
loader/
lost+found/
symvers-6.14.4-300.fc42.x86_64.xz@
symvers-6.14.5-300.fc42.x86_64.xz@
symvers-6.14.6-300.fc42.x86_64.xz@
System.map-6.14.4-300.fc42.x86_64
System.map-6.14.5-300.fc42.x86_64
System.map-6.14.6-300.fc42.x86_64
vmlinuz-0-rescue-1c29c17391714633b75beebc6deb0389*
vmlinuz-6.14.4-300.fc42.x86_64*
vmlinuz-6.14.5-300.fc42.x86_64*
vmlinuz-6.14.6-300.fc42.x86_64*

$ ls -1 /boot/efi
EFI/
mach_kernel*
System/
'System Volume Information'/

$ ls -1 /boot/efi/EFI
BOOT/
fedora/

And for comparison by partition sizes

$ df -h /boot /boot/efi
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  974M  367M  541M  41% /boot
/dev/nvme0n1p1  599M   20M  580M   4% /boot/efi
10

The *.img files are in /boot as shown in the original post. The efi partition seems to be full because a subdirectory for kernel 6.13 is using up 35M. Also, it seems like Fedora didn’t clean up those directories whenever an old kernel package was removed to install a new one.

# du -hs /boot/efi/*
51M	/boot/efi/d4033fa8d6ca48a68402a365aef6f348
7.7M	/boot/efi/EFI
2.0K	/boot/efi/mach_kernel
8.0K	/boot/efi/System
# du -hs /boot/efi/d4033fa8d6ca48a68402a365aef6f348/*
17M	/boot/efi/d4033fa8d6ca48a68402a365aef6f348/0-rescue
2.0K	/boot/efi/d4033fa8d6ca48a68402a365aef6f348/4.15.17-300.fc27.x86_64
2.0K	/boot/efi/d4033fa8d6ca48a68402a365aef6f348/4.15.9-300.fc27.x86_64
2.0K	/boot/efi/d4033fa8d6ca48a68402a365aef6f348/4.16.4-200.fc27.x86_64
35M	/boot/efi/d4033fa8d6ca48a68402a365aef6f348/6.13.12-200.fc41.x86_64
2.0K	/boot/efi/d4033fa8d6ca48a68402a365aef6f348/6.14.5-200.fc41.x86_64
2.0K	/boot/efi/d4033fa8d6ca48a68402a365aef6f348/7.20

EFI is being used on this system. Check:

# [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
UEFI
11

That is the problem. You can remove that directory and re-run the kernel-install command.

Back quite a few releases ago some bug in the kernel install procedures caused the /boot/efi/d4033fa8d6ca48a68402a365aef6f348 directory to be created and this has been a problem ever since.

If /boot/d4033fa8d6ca48a68402a365aef6f348 exists, that can also be a problem.

2 Likes
13

And that requires minimal space (on my F41 system it’s a little under 7 MiB).

1 Like
15

I ended up recreating the EFI partition, see below.

Thanks, seems like a valuable hint! So I guess when they fixed it, they did not make sure that it will be removed on systems where it’s already been created (e.g., during the kernel-install procedure)?
Do we have a link to the bug report?

If you put it like that, I do agree. At first it seemed like having /boot/efi on its own partition is the only reason why it did not have enough space. That’s why I mentioned btrfs, also with btrfs I have backup scripts set up on some systems which automatically create a readonly snapshot either during release upgrades or at certain times and /boot/* would not be included. But I agree there are other factors and reasons why it needs to be kept separate.

16

It is more than 6 years ago. An additional issue is that some people like to customize grub2 in a way that eventually causes similar problems.

17

Recreating /boot/efi

After finding out about the lack of space in /boot/efi (see above), I decided to make that partition larger. Note that there was some old directory in /boot/efi which should not have been there, pointed out in the comment above, it looked strange to me as well but at that point I had already made space to grow the partition by shrinking the main partition by 400M and moving everything inbetween to the right. Unfortunately, the fat16 filesystem within that partition could not be resized:

# fatresize -v -s 200Mi /dev/nvme0n1p1
fatresize 1.1.0 (20240717)
part(start=2048, end=1023999, length=1021952)
Resizing file system.
No Implementation: GNU Parted cannot resize this partition to this size.  We're working on it!

Unable to grow the filesystem, I decided to recreate it.

I followed this part of the Fedora documentation:

After formatting it with FAT32, mounting it and updating its UUID in /etc/fstab, as documented, I installed the suggested packages.

Re-install the necessary packages.
# dnf reinstall grub2-efi grub2-efi-modules shim-\*

Packages for argument 'grub2-efi-modules' available, but not installed.

=>

# dnf install grub2-efi grub2-efi-modules shim-\*
Package "grub2-efi-x64-1:2.12-21.fc41.x86_64" is already installed.
Package "shim-x64-15.8-3.x86_64" is already installed.

Installing:
 grub2-efi-aa64-modules                              noarch        1:2.12-21.fc41                                        updates                        5.7 MiB
 shim-ia32                                           x86_64        15.8-3                                                fedora                         2.1 MiB
 shim-unsigned-ia32                                  x86_64        15.8-2                                                fedora                         1.4 MiB
 shim-unsigned-x64                                   x86_64        15.8-2                                                fedora                         1.8 MiB
Installing dependencies:
 grub2-efi-ia32                                      x86_64        1:2.12-21.fc41                                        updates                        5.2 MiB

I did not notice that it installed GRUB files for 32-bit architecture. It seemed promising that there were now some files in /boot/efi.

That should’ve been it, according to the documentation:

If you already have a working GRUB2 EFI configuration file, you do not need to do anything else.

Following this advice, I rebooted, only to find out that the boot menu would be gone, the system did not boot anymore.

Under “More information”, there’s a hint in the doc:

Under EFI, GRUB2 looks for its configuration in /boot/efi/EFI/fedora/grub.cfg
# rm -f /boot/efi/EFI/fedora/grub.cfg
# dnf reinstall grub2-common

Although recreating it was possible with this command, it did not fix the boot manager.

Recovery

I booted a Fedora live system from USB and used the usual procedure to enter the local installation:

root@localhost-live:/home/liveuser130# mkdir -m 0 /mnt/sysimage
root@localhost-live:/home/liveuser# mount -o subvol=root /dev/nvme0n1p4 /mnt/sysimage
root@localhost-live:/home/liveuser# mount /dev/nvme0n1p2 /mnt/sysimage/boot/
root@localhost-live:/home/liveuser# mount -o umask=0077,shortname=winnt /dev/nvme0n1p1 /mnt/sysimage/boot/efi/
root@localhost-live:/home/liveuser# mount -B /dev /mnt/sysimage/dev
root@localhost-live:/home/liveuser# mount -B /proc /mnt/sysimage/proc
root@localhost-live:/home/liveuser# mount -B /sys /mnt/sysimage/sys

root@localhost-live:/home/liveuser# mount -t efivarfs efivarfs /sys/firmware/efi/efivars

root@localhost-live:/home/liveuser# chroot /mnt/sysimage

That’s where I ran dnf reinstall grub2-common but that wasn’t enough to fix it.

Being used to non-EFI setups, I initially forgot to add efivars (/sys/firmware/efi/efivars was empty within the chroot):

# mount -t efivarfs efivarfs /sys/firmware/efi/efivars

… and without noticing that there’s a section “Restoring the bootloader” with slightly different commands (including dnf reinstall grub2-efi-\* which should’ve included the x64 package), I used the good old grub2-install command (which as per documentation should not be necessary). This made me aware that the subdirectory for the x64 architecture is missing:

# grub2-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Fedora
grub2-install: error: /usr/lib/grub/x86_64-efi/modinfo.sh doesn't exist. Please specify --target or --directory.
# ls -la /usr/lib/grub/x86_64-efi
ls: cannot access '/usr/lib/grub/x86_64-efi': No such file or directory
# ls -la /usr/lib/grub/
total 0
drwxr-xr-x. 1 root root   32 Mar 25 01:00 .
dr-xr-xr-x. 1 root root 1788 May 11 22:57 ..
drwxr-xr-x. 1 root root 5540 May 20 20:21 arm64-efi
drwxr-xr-x. 1 root root 6812 May 11 22:21 i386-pc

This made me wonder why there are two subdirectories for the wrong architecture. They were there because grub2-efi-ia32 and shim-ia32 had been installed earlier (note the asterisk in the doc: dnf install ... shim-\*). So I forcefully removed these two packages and installed the x64 one:

# dnf remove grub2-efi-ia32 shim-ia32-15.8-3.x86_64
Failed to resolve the transaction:
Problem: The operation would result in removing the following protected packages: grub2-efi-ia32, shim-ia32
# rpm -e --nodeps grub2-efi-ia32 shim-ia32
# dnf install grub2-efi-x64 grub2-efi-x64-modules shim-x64
# ls -la /usr/lib/grub/
total 0
drwxr-xr-x. 1 root root   52 May 20 21:34 .
dr-xr-xr-x. 1 root root 1788 May 11 22:57 ..
drwxr-xr-x. 1 root root 5540 May 20 20:21 arm64-efi
drwxr-xr-x. 1 root root 6812 May 11 22:21 i386-pc
drwxr-xr-x. 1 root root 6608 May 20 21:34 x86_64-efi

This created the subdirectory x86_64-efi that grub2-install was expecting.

# grub2-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Fedora 
Installing for x86_64-efi platform.
grub2-install: error: This utility should not be used for EFI platforms because it does not support UEFI Secure Boot. If you really wish to proceed, invoke the --force option.
Make sure Secure Boot is disabled before proceeding.

Note I did not force it. Seems like the missing piece was grub2-efi-x64-modules.

That and the efivars mount I initially forgot to include in the chroot.

Now the system is working again.

Conclusion

After recreating the EFI partition, following the [documentation] to “Install the bootloader files” was not enough, the system would not boot anymore.
A live system had to be used to restore the boot loader.

The original error and the missing x64 package may have been unique features in this particular case, so I added some details above hoping it’ll be useful somehow.

When you have to fix a broken boot loader, don’t forget that you may be working on an EFI system. I knew that and still set up only the standard bind mounts (without efivars) and ran grub2-install.

Finally, the kernel-install command from above was now successful, it created a new boot menu entry.

# kernel-install -v add 6.14.5-200.fc41.x86_64 /lib/modules/6.14.5-200.fc41.x86_64/vmlinuz
...
About to execute /usr/lib/kernel/install.d/99-grub-mkconfig.install add 6.14.5-200.fc41.x86_64 /boot/efi/d4033fa8d6ca48a68402a365aef6f348/6.14.5-200.fc41.x86_64 /lib/modules/6.14.5-200.fc41.x86_64/vmlinuz
Successfully forked off '(direxec)' as PID 19214.
/usr/lib/kernel/install.d/99-grub-mkconfig.install succeeded.
(sd-exec-strv) succeeded.

Boot entry confirmed working:

# ls /boot/*x86*.img
/boot/initramfs-6.14.5-200.fc41.x86_64.img  /boot/initramfs-6.8.9-100.fc38.x86_64.img
# grubby --info=ALL | grep title | grep x86
title="Fedora Linux (6.14.5-200.fc41.x86_64) 41 (Forty One)"
title="Fedora Linux (6.8.9-100.fc38.x86_64) 38 (Thirty Eight)"
1 Like