Old kernels not removed from `/boot/efi`

i have fc39 installed on a server, and it seems that old kernels are not being removed from /boot/efi/EFI. i noticed this because i got an out of space error when updating grub2-efi-x64:

Error: Transaction test error:
  installing package grub2-efi-x64-1:2.06-120.fc39.x86_64 needs 4MB more space on the /boot/efi filesystem

Error Summary
-------------
Disk Requirements:
   At least 4MB more space needed on the /boot/efi filesystem.

yes, i’m out of space:

# df -h
Filesystem          Size  Used Avail Use% Mounted on
/dev/nvme0n1p2      960M  596M  365M  63% /boot
/dev/nvme0n1p1      599M  599M     0 100% /boot/efi

i removed my old kernels with dnf remove 'kernel-*x.y.z*'. i now have only:

# rpm -qa kernel
kernel-6.7.11-200.fc39.x86_64

however, i’m still out of space because the kernels are still there (even after rebooting):

# du -sh /boot/efi/EFI/Linux/*
58M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.6.14-200.fc39.x86_64+debug.efi
42M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.6.14-200.fc39.x86_64.efi
59M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.10-200.fc39.x86_64+debug.efi
42M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.10-200.fc39.x86_64.efi
50M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.11-200.fc39.x86_64+debug.efi
42M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.11-200.fc39.x86_64.efi
59M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.6-200.fc39.x86_64+debug.efi
42M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.6-200.fc39.x86_64.efi
59M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.7-200.fc39.x86_64+debug.efi
42M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.7-200.fc39.x86_64.efi
59M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.9-200.fc39.x86_64+debug.efi
42M     /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.9-200.fc39.x86_64.efi

i thought it was only supposed to save 3 kernels:

# cat /etc/dnf/dnf.conf

[main]
gpgcheck=True
installonly_limit=3
clean_requirements_on_remove=True
best=False
skip_if_unavailable=True

what’s the correct way to remove the old kernels to recover free space on /boot/efi, and why did it run out of space in the first place?

thanks

This sounds similar to: Efi partition full - #28 by glb

1 Like

hm, i did use koji one single time to install a kernel:

cd $(mktemp -d) \
  && koji download-build --arch=x86_64 --arch=noarch kernel-6.6.14-200.fc39 \
  && sudo dnf install *

does that somehow explain having 6 kernels in there (12 if you include debug)? :thinking:

I do not have a linux subfolder … mine is fedora.

Could it be that you had an other Linux installed and that this are over-lefts from them?

i don’t think so. it was a fresh nvme ssd, and i installed fedora 39 directly on there (used btrfs option)

It doesn’t really explain why you would have 12 unless there is something broken about the cleanup scripts when debug kernels are present. The script that is supposed to remove that stuff is /usr/bin/kernel-install (it gets called with β€œremove” as its first parameter when a kernel package is removed). Unfortunately, it seems that recent versions of Fedora Linux have changed that into a compiled binary so you cannot easily inspect it to try to figure out what might have gone wrong. :confused:

so i can just move on for the time being, is there a more blessed solution than rm for these unwanted kernels in /boot/efi/EFI/Linux?

As long as there is no installed package that β€œowns” those files, just using rm should be OK. Use rpm -qf <full-path-to-file> to see if a file is still registered in RPM’s database.

lol, apparently none of them is owned by any package:

# rpm -qf /boot/efi/EFI/Linux/*
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.6.14-200.fc39.x86_64+debug.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.6.14-200.fc39.x86_64.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.10-200.fc39.x86_64+debug.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.10-200.fc39.x86_64.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.11-200.fc39.x86_64+debug.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.11-200.fc39.x86_64.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.6-200.fc39.x86_64+debug.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.6-200.fc39.x86_64.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.7-200.fc39.x86_64+debug.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.7-200.fc39.x86_64.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.9-200.fc39.x86_64+debug.efi is not owned by any package
file /boot/efi/EFI/Linux/004cf4183afa4664b32dc5d82930a17e-6.7.9-200.fc39.x86_64.efi is not owned by any package

Yeah, the files under /boot are a bit of a special case. Most of them are never registered in RPM’s db. Rather, they are added/removed/updated by β€œpost” scripts in the RPM packages.

I would avoid removing anything that matches your current kernel version just to be safe. Then, after a few more kernel updates, check back to see if things are still being added there. If not, you can finish removing the remaining (now outdated) kernel files.

1 Like

These are from the uki packages kernel-uki-virt.x86_64 and kernel-debug-uki-virt.x86_64. These should not have been installed unless there are good reason to do so.

3 Likes

idk. i’m using openzfs – could that be why?

after i cleaned up:

# tree /boot
/boot
β”œβ”€β”€ config-6.7.11-200.fc39.x86_64
β”œβ”€β”€ config-6.7.11-200.fc39.x86_64+debug
β”œβ”€β”€ config-6.8.8-200.fc39.x86_64
β”œβ”€β”€ config-6.8.8-200.fc39.x86_64+debug
β”œβ”€β”€ efi
β”‚   β”œβ”€β”€ EFI
β”‚   β”‚   β”œβ”€β”€ BOOT
β”‚   β”‚   β”‚   β”œβ”€β”€ BOOTX64.EFI
β”‚   β”‚   β”‚   └── fbx64.efi
β”‚   β”‚   β”œβ”€β”€ fedora
β”‚   β”‚   β”‚   β”œβ”€β”€ BOOTX64.CSV
β”‚   β”‚   β”‚   β”œβ”€β”€ grub.cfg
β”‚   β”‚   β”‚   β”œβ”€β”€ grub.cfg.rpmsave
β”‚   β”‚   β”‚   β”œβ”€β”€ grubx64.efi
β”‚   β”‚   β”‚   β”œβ”€β”€ mmx64.efi
β”‚   β”‚   β”‚   β”œβ”€β”€ shim.efi
β”‚   β”‚   β”‚   └── shimx64.efi
β”‚   β”‚   └── Linux
β”‚   β”‚       β”œβ”€β”€ 004cf4183afa4664b32dc5d82930a17e-6.7.11-200.fc39.x86_64+debug.efi
β”‚   β”‚       β”œβ”€β”€ 004cf4183afa4664b32dc5d82930a17e-6.7.11-200.fc39.x86_64.efi
β”‚   β”‚       β”œβ”€β”€ 004cf4183afa4664b32dc5d82930a17e-6.8.8-200.fc39.x86_64+debug.efi
β”‚   β”‚       └── 004cf4183afa4664b32dc5d82930a17e-6.8.8-200.fc39.x86_64.efi
β”‚   └── System Volume Information
β”œβ”€β”€ grub2
β”‚   β”œβ”€β”€ fonts
β”‚   β”‚   └── unicode.pf2
β”‚   β”œβ”€β”€ grub.cfg
β”‚   └── grubenv
β”œβ”€β”€ initramfs-0-rescue-004cf4183afa4664b32dc5d82930a17e.img
β”œβ”€β”€ initramfs-6.7.11-200.fc39.x86_64+debug.img
β”œβ”€β”€ initramfs-6.7.11-200.fc39.x86_64.img
β”œβ”€β”€ initramfs-6.8.8-200.fc39.x86_64+debug.img
β”œβ”€β”€ initramfs-6.8.8-200.fc39.x86_64.img
β”œβ”€β”€ loader
β”‚   └── entries
β”‚       β”œβ”€β”€ 004cf4183afa4664b32dc5d82930a17e-0-rescue.conf
β”‚       β”œβ”€β”€ 004cf4183afa4664b32dc5d82930a17e-6.7.11-200.fc39.x86_64.conf
β”‚       β”œβ”€β”€ 004cf4183afa4664b32dc5d82930a17e-6.7.11-200.fc39.x86_64+debug.conf
β”‚       β”œβ”€β”€ 004cf4183afa4664b32dc5d82930a17e-6.8.8-200.fc39.x86_64.conf
β”‚       └── 004cf4183afa4664b32dc5d82930a17e-6.8.8-200.fc39.x86_64+debug.conf
β”œβ”€β”€ symvers-6.7.11-200.fc39.x86_64+debug.xz -> /lib/modules/6.7.11-200.fc39.x86_64+debug/symvers.xz
β”œβ”€β”€ symvers-6.7.11-200.fc39.x86_64.xz -> /lib/modules/6.7.11-200.fc39.x86_64/symvers.xz
β”œβ”€β”€ symvers-6.8.8-200.fc39.x86_64+debug.xz -> /lib/modules/6.8.8-200.fc39.x86_64+debug/symvers.xz
β”œβ”€β”€ symvers-6.8.8-200.fc39.x86_64.xz -> /lib/modules/6.8.8-200.fc39.x86_64/symvers.xz
β”œβ”€β”€ System.map-6.7.11-200.fc39.x86_64
β”œβ”€β”€ System.map-6.7.11-200.fc39.x86_64+debug
β”œβ”€β”€ System.map-6.8.8-200.fc39.x86_64
β”œβ”€β”€ System.map-6.8.8-200.fc39.x86_64+debug
β”œβ”€β”€ vmlinuz-0-rescue-004cf4183afa4664b32dc5d82930a17e
β”œβ”€β”€ vmlinuz-6.7.11-200.fc39.x86_64
β”œβ”€β”€ vmlinuz-6.7.11-200.fc39.x86_64+debug
β”œβ”€β”€ vmlinuz-6.8.8-200.fc39.x86_64
└── vmlinuz-6.8.8-200.fc39.x86_64+debug

so i guess i have two issues at this point

  1. why does kernel-uki-virt’s cleanup script not remove what it puts in /boot/efi/EFI/Linux/?
  2. how can i look into why i have kernel-uki-virt in the first place if it’s nonstandard? can i look at which installed packages require it or something?

I use OpenZFS, but I do not have kernel-uki-virt installed. It seems likely it was picked up by accident at some point. I think the β€œunified kernel images” (uki) are a very new technology. They probably just haven’t had all the edge cases ironed out. I usually just run dnf remove <whatever> (without the -y) to see what other packages depend on a package. It will ask for confirmation before making any changes (as long as you do not include the -y).

If you’ve already removed the package, there are other ways to query their dependencies. I think it is under the repoquery subcommand of dnf, but I’d have to check the man page (man dnf). There should also be log files – /var/log/dnf* – that you can grep to find out when a package was installed.

It is a bug.

At some point someone installed them on your system. After that the packages gets updated whenever a new version is released. The command dnf history list kernel-uki-virt may show you when.

1 Like

indeed, i guess this is all because i installed a kernel one time with koji, and once the uki-virt packages were installed, they hung around with new kernel updates. uki kernels kept accumulating in /boot/efi since i guess its cleanup script is broken?

# dnf history list kernel-uki-virt

ID     | Command line                                                                                                                                                                            | Date and time    | Action(s)      | Altered
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    69 | system-upgrade upgrade                                                                                                                                                                  | 2024-05-04 00:22 | I, O, U        | 1030 E<
    68 | remove kernel-*6.7.11*                                                                                                                                                                  | 2024-05-03 23:57 | C, E           |   30 ><
    66 | upgrade                                                                                                                                                                                 | 2024-05-03 23:31 | I, U           |   20 ><
    63 | remove kernel-*6.7.10*                                                                                                                                                                  | 2024-05-03 23:03 | C, E           |   30 ><
    61 | remove kernel-*6.7.9*                                                                                                                                                                   | 2024-05-03 21:24 | C, E           |   30 ><
    51 | upgrade                                                                                                                                                                                 | 2024-04-03 18:03 | C, E, I, U     |   60 >E
    50 | upgrade                                                                                                                                                                                 | 2024-03-30 13:40 | C, E, I, U     |   61 E<
    40 | upgrade                                                                                                                                                                                 | 2024-03-13 02:30 | C, E, I, U     |   53 ><
    37 | upgrade                                                                                                                                                                                 | 2024-03-05 23:56 | I, U           |   25 >E
    36 | upgrade                                                                                                                                                                                 | 2024-03-01 05:11 | I, U           |   19 E<
    25 | install kernel-6.6.14-200.fc39.x86_64.rpm kernel-core-6.6.14-200.fc39.x86_64.rpm kernel-debug-6.6.14-200.fc39.x86_64.rpm kernel-debug-core-6.6.14-200.fc39.x86_64.rpm kernel-debug-deve | 2024-02-09 18:13 | C, E, I        |   42 

i confirmed koji download-build --arch=x86_64 --arch=noarch kernel-6.6.14-200.fc39 does download the uki-virt packages, so that’s how they got installed.

i guess the lessons are

  1. be mindful that koji downloads kernel pkgs you may not need/want
  2. uki-virt pkgs are bugged and will eventually fill up your /boot/efi with enough upgrades :frowning:
2 Likes