Kernels are accumulating in /boot

[snowgum (John Henderson)]

I have a problem with getting rid of older kernels, and need help please.

When I run a dnf update which provides a newer kernel, the number of kernels in /boot increases.

Before I started tinkering to try to resolve this, I had instances of dnf update ending with a “fail”.

While I have been running Fedora continuously for around 25 years, my knowledge of kernels remains very limited. I’m now well into old age (77).

The Lenovo m73 series PC I’m using has been running and updating Fedora since I bought it around 20 years ago.

My latest routine kernel update resulted in dnf ending without an error at termination. But during that update, I got a warning about only having 1% disk space left in /boot

So I deleted all “6.17.10-300.fc43.x86_64” files from /boot to free up some space for the time being.

Along the way, I got this error from dnf update:

file /usr/src/kernels/6.17.7-300.fc43.x86_64: remove failed: No such file or directory

I recently used dnf and rpm to get rid of that kernel from the system. But a memory of it obviously remains, and I see it’s mentioned in a couple of files in /boot

sudo grep -r "6.17.7" /boot
/boot/System.map-6.17.12-300.fc43.x86_64:ffffffff82651787 d str__rcu__trace_system_name
/boot/System.map-6.17.11-300.fc43.x86_64:ffffffff82651787 d str__rcu__trace_system_name

Here are some other results from my system as it now stands:

rpm -qa kernel\* | sort -V
kernel-core-6.17.12-300.fc43.x86_64
kernel-core-6.18.3-200.fc43.x86_64
kernel-core-6.18.4-200.fc43.x86_64
kernel-devel-6.17.12-300.fc43.x86_64
kernel-devel-6.18.3-200.fc43.x86_64
kernel-devel-6.18.4-200.fc43.x86_64
kernel-headers-6.18.3-200.fc43.x86_64
kernel-modules-6.17.12-300.fc43.x86_64
kernel-modules-6.18.3-200.fc43.x86_64
kernel-modules-6.18.4-200.fc43.x86_64
kernel-modules-core-6.17.12-300.fc43.x86_64
kernel-modules-core-6.18.3-200.fc43.x86_64
kernel-modules-core-6.18.4-200.fc43.x86_64
kernel-modules-extra-6.17.12-300.fc43.x86_64
kernel-modules-extra-6.18.3-200.fc43.x86_64
kernel-modules-extra-6.18.4-200.fc43.x86_64
kernel-srpm-macros-1.0-27.fc43.noarch
kernel-tools-6.18.4-200.fc43.x86_64
kernel-tools-libs-6.18.4-200.fc43.x86_64

ls -l /boot/vmlinuz*
-rwxr-xr-x. 1 root root  5139320 Feb 22  2014 /boot/vmlinuz-0-rescue-03f8b18c211b41a49cd09532d25807bc
-rwxr-xr-x. 1 root root 18180136 Dec  8 11:00 /boot/vmlinuz-6.17.11-300.fc43.x86_64
-rwxr-xr-x. 1 root root 18184232 Dec 13 11:00 /boot/vmlinuz-6.17.12-300.fc43.x86_64
-rwxr-xr-x. 1 root root 18348072 Jan  2 11:00 /boot/vmlinuz-6.18.3-200.fc43.x86_64
-rwxr-xr-x. 1 root root 18343976 Jan  8 11:00 /boot/vmlinuz-6.18.4-200.fc43.x86_64

ls -l /lib/modules
total 20
drwxr-xr-x. 7 root root 4096 Dec 10 06:58 6.17.10-300.fc43.x86_64
drwxr-xr-x. 7 root root 4096 Dec 14 12:20 6.17.11-300.fc43.x86_64
drwxr-xr-x. 7 root root 4096 Jan  8 06:52 6.17.12-300.fc43.x86_64
drwxr-xr-x. 7 root root 4096 Jan  9 06:56 6.18.3-200.fc43.x86_64
drwxr-xr-x. 7 root root 4096 Jan 12 06:26 6.18.4-200.fc43.x86_64

I’m reluctant to go further without understanding what I’m doing, for fear of breaking things irrepairably. I don’t want to take a scatter-gun approach to trying powerful commands I don’t understand the consequences of.

[einer (One)]

Hi John,

I’ll try to help

1. as root (or sudo) – rpm -qa | grep kernel <== get a list of the installed kernels and their parts
2. use the package names in the following like this:
   dnf remove {kernel_version_to be removed}

Usually, dnf keeps the last 3 kernels installed plus the rescue kernel. In older partion setups, you may not have enough space in the boot partition for this. To fix, you could edit the /etc/dnf/dnf.conf and change installonly_limit=3 to installonly_limit=2 to keep only the latest 2 kernels as a backup …

[buckaroogeek (Brad Smith)]

You might also look at this discussion: Old Kernels Removal - #2 by ankursinha

[grumpey (Joe Walker)]

You can also check with:
dnf --dump-main-config | grep installonly

And you can set with:
sudo dnf config-manager setopt installonly_limit=2

The default limit is 3.

[snowgum (John Henderson)]

@einer

1. use the package names in the following like this:
   dnf remove {kernel_version_to be removed}

Usually, dnf keeps the last 3 kernels installed plus the rescue kernel. In older partion setups, you may not have enough space in the boot partition for this.

Thank you for the reply.

sudo dnf remove kernel-core-6.17.10-300.fc43.x86_64
No packages to remove for argument: kernel-core-6.17.10-300.fc43.x86_64

sudo dnf remove kernel-core-6.17.11-300.fc43.x86_64
No packages to remove for argument: kernel-core-6.17.11-300.fc43.x86_64

For the last 20 years keeping the most recent three kernels has worked fine, with plenty of space to spare.

The problem is that more than three are being kept.

I’m sure the problem is related to the:

file /usr/src/kernels/6.17.7-300.fc43.x86_64: remove failed: No such file or directory

error, which persists.

[snowgum (John Henderson)]

@grumpey

You can also check with:
dnf --dump-main-config | grep installonly

I confirm:

dnf --dump-main-config | grep installonly 
installonly_limit = 3
installonlypkgs = kernel, kernel-PAE, installonlypkg(kernel), installonlypkg(kernel-module), installonlypkg(vm), multiversion(kernel)

[grumpey (Joe Walker)]

can you post:
sudo df -h /boot{/.,/efi}
sudo ls -l /boot

Thanks

[snowgum (John Henderson)]

sudo df -h /boot{/.,/efi}
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       474M  362M   84M  82% /boot
/dev/sda5       474M  362M   84M  82% /boot

sudo ls -l /boot
total 350975
-rw-r--r--. 1 root root   292973 Dec  8 11:00 config-6.17.11-300.fc43.x86_64
-rw-r--r--. 1 root root   292973 Dec 13 11:00 config-6.17.12-300.fc43.x86_64
-rw-r--r--. 1 root root   294203 Jan  2 11:00 config-6.18.3-200.fc43.x86_64
-rw-r--r--. 1 root root   294203 Jan  8 11:00 config-6.18.4-200.fc43.x86_64
drwx------. 4 root root     1024 Nov 11 18:46 efi
drwxr-xr-x. 2 root root     5120 Nov 11 18:40 extlinux
drwx------. 6 root root     1024 Jan  2 09:24 grub2
-rw-------. 1 root root 39289419 Feb 22  2014 initramfs-0-rescue-03f8b18c211b41a49cd09532d25807bc.img
-rw-------. 1 root root 48762869 Dec 14 12:20 initramfs-6.17.11-300.fc43.x86_64.img
-rw-------. 1 root root 48762667 Jan  8 06:52 initramfs-6.17.12-300.fc43.x86_64.img
-rw-------. 1 root root 48841940 Jan  9 06:56 initramfs-6.18.3-200.fc43.x86_64.img
-rw-------. 1 root root 48841814 Jan 12 06:26 initramfs-6.18.4-200.fc43.x86_64.img
-rw-r--r--. 1 root root   560975 Nov 25  2016 initrd-plymouth.img
drwxr-xr-x. 3 root root     1024 Nov 13  2018 loader
drwx------. 2 root root    12288 Feb 22  2014 lost+found
-rw-r--r--. 1 root root   155984 Jul 24 10:00 memtest86+x64.bin
lrwxrwxrwx. 1 root root       47 Dec 14 12:20 symvers-6.17.11-300.fc43.x86_64.xz -> /lib/modules/6.17.11-300.fc43.x86_64/symvers.xz
lrwxrwxrwx. 1 root root       47 Jan  8 06:51 symvers-6.17.12-300.fc43.x86_64.xz -> /lib/modules/6.17.12-300.fc43.x86_64/symvers.xz
lrwxrwxrwx. 1 root root       46 Jan  9 06:55 symvers-6.18.3-200.fc43.x86_64.xz -> /lib/modules/6.18.3-200.fc43.x86_64/symvers.xz
lrwxrwxrwx. 1 root root       46 Jan 12 06:25 symvers-6.18.4-200.fc43.x86_64.xz -> /lib/modules/6.18.4-200.fc43.x86_64/symvers.xz
-rw-r--r--. 1 root root 11127188 Dec  8 11:00 System.map-6.17.11-300.fc43.x86_64
-rw-r--r--. 1 root root 11127277 Dec 13 11:00 System.map-6.17.12-300.fc43.x86_64
-rw-r--r--. 1 root root 11246634 Jan  2 11:00 System.map-6.18.3-200.fc43.x86_64
-rw-r--r--. 1 root root 11246030 Jan  8 11:00 System.map-6.18.4-200.fc43.x86_64
-rwxr-xr-x. 1 root root  5139320 Feb 22  2014 vmlinuz-0-rescue-03f8b18c211b41a49cd09532d25807bc
-rwxr-xr-x. 1 root root 18180136 Dec  8 11:00 vmlinuz-6.17.11-300.fc43.x86_64
-rwxr-xr-x. 1 root root 18184232 Dec 13 11:00 vmlinuz-6.17.12-300.fc43.x86_64
-rwxr-xr-x. 1 root root 18348072 Jan  2 11:00 vmlinuz-6.18.3-200.fc43.x86_64
-rwxr-xr-x. 1 root root 18343976 Jan  8 11:00 vmlinuz-6.18.4-200.fc43.x86_64

[snowgum (John Henderson)]

Should I just try running:

grub2-mkconfig -o /boot/grub2/grub.cfg

I’ve been reluctant to try it without reassurance it will not do further harm.

[tqcharm (Mike B.)]

@snowgum

I’m sure the problem is related to the:

file /usr/src/kernels/6.17.7-300.fc43.x86_64: remove failed: No such file or directory

error, which persists.

kernel-devel is the package providing that file and subsequent files/directories. The post-install script which removes old kernels is probably trying to remove that file too. Yet you don’t have kernel-devel-6.17.7-300.fc43.x86_64 on your system, which raises the question why is the script trying to remove that file/directory?

@snowgum

So I deleted all “6.17.10-300.fc43.x86_64” files from /boot to free up some space for the time being.

The RPM packages should be removed instead, and not the files as such. Did you maybe delete the files for kernel 6.17.7 from /boot in a similar way? That would explain why the system considers those kernel packages are still on the system.

Can you remove an “intermediate” version of the kernel, say 6.17.12 or 6.18.3, in oder to make some space, then reinstall kernel-devel-6.17.7-300 and its dependencies^[1], then remove them again with the dnf remove command?

---

1. You might need to add the fedora-archive repo for that (by installing the fedora-repos-archive package) and then enable it. ↩︎

[barryascott]

@snowgum

For the last 20 years keeping the most recent three kernels has worked fine, with plenty of space to spare.

A recent change to dracut around the f43 beta time increased the size of the initrd.
Because of that the default /boot size on a fresh install was raised to 2GiB.
There is more firmware packaged into the initrd, I think gpu firmware is large, for example.

[grumpey (Joe Walker)]

@snowgum

Should I just try running:

grub2-mkconfig -o /boot/grub2/grub.cfg

I’ve been reluctant to try it without reassurance it will not do further harm.

You shouldn’t need to do this.

I would recommend doing:
Verify the kernel you are running:
uname -r

Follow the steps listed in this post to clear up the versions that are no longer installed/running.

/boot nearly full - how to free space?

Compare the list you received with dnf list installed kernel* and the list you received with ls /boot. Any files in the ‘ls’ listing that begin with config-VERSION, initramfs-VERSION, symvers-VERSION, System.map-VERSION, or vmlinuz-VERSION where VERSION does not match the ones shown in the ‘dnf list’ command can be removed manually with sudo rm /boot/<filename>.

The files that I see:
config-6.17.11-300.fc43.x86_64
initramfs-6.17.11-300.fc43.x86_64.img
symvers-6.17.11-300.fc43.x86_64.xz
System.map-6.17.11-300.fc43.x86_64
vmlinuz-6.17.11-300.fc43.x86_64

If you’re concerned about space issues:
I would suggest following the instructions here to remove the rescue kernel
or
You can reduce the number of kernels installed.
sudo dnf config-manager setopt installonly_limit=2
sudo dnf upgrade

I would monitor for errors during the next upgrade to ensure that the kernels are being removed correctly.

[computersavvy (Jeff V)]

I would add also that manually removing kernel files from /boot does not remove the other kernel related files that may exist elsewhere in the system nor does it update the rpm database that keeps track of all installed packages.

Removal of kernels and related files should always be performed (as much as possible) using the dnf package manager so consistency is assured across the board and that everything touched when a kernel is installed is also cleaned up.

I seem to have not made that clear in the link posted just above.

[snowgum (John Henderson)]

@tqcharm

you don’t have kernel-devel-6.17.7-300.fc43.x86_64 on your system, which raises the question why is the script trying to remove that file/directory?

Good question. I can only assume it is an artifact from an earlier failed dnf update. There have been a couple of those in recent times (since I upgraded to FC43 on 11 Nov 25).

@tqcharm

The RPM packages should be removed instead, and not the files as such. Did you maybe delete the files for kernel 6.17.7 from /boot in a similar way?

No, the circumstances surrounding 6.17.7 are a mystery.

@barryascott

A recent change to dracut around the f43 beta time increased the size of the initrd.
Because of that the default /boot size on a fresh install was raised to 2GiB.

Thanks for the info. Being around 20 years old, my /boot is 500MB.

A larger /boot directory isn’t going to solve my problem however, it would simply delay its becoming full.

@grumpey

Verify the kernel you are running:
uname -r

As per the /boot contents:

[quote="Joe Walker, post:13, topic:179352, username:grumpey"]
I would monitor for errors during the next upgrade to ensure that the kernels are being removed correctly.
[/quote]

uname -r
6.18.4-200.fc43.x86_64

In anticipation of that, I’ve added the empty directory as an experiment:

sudo mkdir /usr/src/kernels/6.17.7-300.fc43.x86_64

I thank all responders for their suggestions to date.

[snowgum (John Henderson)]

@grumpey

You can reduce the number of kernels installed.
sudo dnf config-manager setopt installonly_limit=2
sudo dnf upgrade

I’ve done this, thanks. I’ve never had reason to use that third kernel.

[snowgum (John Henderson)]

@snowgum

In anticipation of that, I’ve added the empty directory as an experiment:

sudo mkdir /usr/src/kernels/6.17.7-300.fc43.x86_64

This seems to have done the trick. The commands in the previous post deleted that empty directory I created.

And sudo grep -r "6.17.7" /boot no longer returns any instance of “6.17.7”.

A new kernel arriving in a new dnf update will be the acid test.

I’ll post the outcome when that happens.

[anotheruser (Mark)]

@snowgum

sudo grep -r "6.17.7" /boot
/boot/System.map-6.17.12-300.fc43.x86_64:ffffffff82651787 d str__rcu__trace_system_name
/boot/System.map-6.17.11-300.fc43.x86_64:ffffffff82651787 d str__rcu__trace_system_

@snowgum

And sudo grep -r "6.17.7" /boot no longer returns any instance of “6.17.7”.

pls consult man grep.
A DOT matches any single character except a newline.

$ cat file
line1:	yyy6A17B7xxx
line2:	yyy6A18B7xxx
line3:	aaa6.17.7ccc
line4:	ffffffff82651787

$ grep  "6.17.7" file
line1:	yyy6A17B7xxx
line3:	aaa6.17.7ccc
line4:	ffffffff82651787

$ grep  "6.1..7" file
line1:	yyy6A17B7xxx
line2:	yyy6A18B7xxx
line3:	aaa6.17.7ccc
line4:	ffffffff82651787

$ grep  "6\.17\.7" file
line3:	aaa6.17.7ccc

$ grep  -F "6.17.7" file
line3:	aaa6.17.7ccc

[barryascott]

@snowgum

A larger /boot directory isn’t going to solve my problem however, it would simply delay its becoming full.

On my new server install I added a 4GiB /boot to avoid this issue in the future. My other systems are on 1GiB /boot and below you can see what
my usage and initrd sizes are for reference.

My guess is that on a 500MiB /boot if you get rid of the rescue kernel and set to keep 2 installed kernals you will be good for a long time.

KDE Plasma Desktop

+ df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1p2  974M  392M  515M  44% /boot
+ ls -lh /boot/init*
-rw-------. 1 root root 50M Dec 14 14:05 /boot/initramfs-6.17.11-300.fc43.x86_64.img
-rw-------. 1 root root 50M Dec 19 11:56 /boot/initramfs-6.17.12-300.fc43.x86_64.img
-rw-------. 1 root root 50M Nov 24 12:25 /boot/initramfs-6.17.8-300.fc43.x86_64.img
-rw-------. 1 root root 50M Dec  5 08:53 /boot/initramfs-6.17.9-300.fc43.x86_64.img
-rw-------. 1 root root 50M Jan 10 10:46 /boot/initramfs-6.18.3-200.fc43.x86_64.img

Fedora server router

+ df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  960M  419M  542M  44% /boot
+ ls -lh /boot/init*
-rw-------. 1 root root 152M Sep 12  2024 /boot/initramfs-0-rescue-3aff3b5dfe92484ebf418e417b37a2ca.img
-rw-------. 1 root root  37M Dec 14 14:01 /boot/initramfs-6.17.11-300.fc43.x86_64.img
-rw-------. 1 root root  37M Dec 19 11:52 /boot/initramfs-6.17.12-300.fc43.x86_64.img
-rw-------. 1 root root  37M Jan 10 10:44 /boot/initramfs-6.18.3-200.fc43.x86_64.img

Fedora Server fileserver

+ df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       4.0G  555M  3.4G  14% /boot
+ ls -lh /boot/init*
-rw-------. 1 root root 220M Nov 12 16:19 /boot/initramfs-0-rescue-c0cf395800f5474c8bf3c21d5313edb5.img
-rw-------. 1 root root  41M Dec 19 11:26 /boot/initramfs-6.17.12-300.fc43.x86_64.img
-rw-------. 1 root root  41M Nov 19 17:01 /boot/initramfs-6.17.8-300.fc43.x86_64.img
-rw-------. 1 root root  41M Jan 10 10:49 /boot/initramfs-6.18.3-200.fc43.x86_64.img

[snowgum (John Henderson)]

@anotheruser

A DOT matches any single character except a newline.

Thanks for picking me up on this. I should have realized that myself.

In this case the regex “.” matches a “.” character just fine (although other characters could have matched also).

[snowgum (John Henderson)]

@snowgum

A new kernel arriving in a new dnf update will be the acid test.

I’ll post the outcome when that happens.

That went well, with no errors.

However the two recent kernels that I deleted from /boot to prevent that directory from filling again remained on my boot screen. That was even after running sudo grub2-mkconfig -o /boot/grub2/grub.cfg

sudo grep -r "6\.17\.10" /boot (and 6.17.11) found the /boot entries I had to manually remove before grub2-mkconfig -o /boot/grub2/grub.cfg resulted in a clean initial boot screen.

And of course, I had to remove the abandoned directories from /usr/lib/modules and /usr/src/kernels too.

Again, thank you all for your suggestions and contributions.