My almost fresh install has mismatched kernel versions

Ask Fedora
, , ,
1

20+ year Linux user but new to Fedora & dnf. (Ubuntu + Debian in recent years)

I recently set up a new “gaming” laptop running Fedora 40 with NVIDIA drivers (560.35.03) & CUDA installed. (I don’t game. It’s for ML.) Everything is fine.

I just set up a new desktop on Fedora 41 and things initially seemed fine. I was able to RDP into Gnome. But when I tried installing the NVIDIA drivers with

sudo dnf update
# reboot
sudo dnf install akmod-nvidia
sudo dnf install xorg-x11-drv-nvidia-cuda

nvidia apparently never built (despite me waiting quite a while):

james@mach:~$ modinfo -F version nvidia
modinfo: ERROR: Module nvidia not found.

I rebooted and saw the same. I also removed and re-installed them.

I then tried force-rebuilding, but that failed with an error message about my kernel:

james@mach:~$ sudo akmods --rebuild --force
Checking kmods exist for 6.11.4-301.fc41.x86_64            [  OK  ]
Files needed for building modules against kernel
6.11.4-301.fc41.x86_64 could not be found as the following
directories are missing:
/usr/src/kernels/6.11.4-301.fc41.x86_64/
/lib/modules/6.11.4-301.fc41.x86_64/build/Is the correct ke[FAILED]el package installed?

Something seems messed up with my kernel versions. Here’s what I’m seeing:

james@mach:~$ sudo ls /usr/src/kernels/
6.11.7-300.fc41.x86_64

james@mach:~$ uname -r
6.11.4-301.fc41.x86_64

james@mach:~$ rpm -qa kernel
kernel-6.11.4-301.fc41.x86_64
kernel-6.11.7-300.fc41.x86_64

On my laptop, I see three sets of kernel packages (but with just one headers package). I’m NOT seeing that on the desktop:

james@mach:~$ sudo dnf list --installed | grep kernel
abrt-addon-kerneloops.x86_64                         2.17.6-2.fc41                       anaconda
kernel.x86_64                                        6.11.4-301.fc41                     anaconda
kernel.x86_64                                        6.11.7-300.fc41                     updates
kernel-core.x86_64                                   6.11.4-301.fc41                     anaconda
kernel-core.x86_64                                   6.11.7-300.fc41                     updates
kernel-devel.x86_64                                  6.11.7-300.fc41                     updates
kernel-devel-matched.x86_64                          6.11.7-300.fc41                     updates
kernel-headers.x86_64                                6.11.3-300.fc41                     fedora
kernel-modules.x86_64                                6.11.4-301.fc41                     anaconda
kernel-modules.x86_64                                6.11.7-300.fc41                     updates
kernel-modules-core.x86_64                           6.11.4-301.fc41                     anaconda
kernel-modules-core.x86_64                           6.11.7-300.fc41                     updates
kernel-modules-extra.x86_64                          6.11.4-301.fc41                     anaconda
kernel-modules-extra.x86_64                          6.11.7-300.fc41                     updates
kernel-srpm-macros.noarch                            1.0-24.fc41                         fedora
kernel-tools.x86_64                                  6.11.7-300.fc41                     updates
kernel-tools-libs.x86_64                             6.11.7-300.fc41                     updates
libreport-plugin-kerneloops.x86_64                   2.17.15-3.fc41                      anaconda

The kernel-headers version doesn’t match EITHER of my two installed kernel versions!

This seems messed up, but I’m unsure why/how it might have happened and how I can fix it.

Thank you in advance for any thoughts you might share with this Fedora noob!

2

Your running kernel does not match the version of its devel package.
Boot the latest kernel or install the devel package to match the running kernel.
If the issue persists, boot the latest kernel and remove the older ones.

See also: Kernel headers update policy - #3 by vekruse

3

Thank you, @vgaetera!

I just installed the missing module:

james@mach:~$ sudo dnf install kernel-devel-6.11.4-301.fc41
...
[3/3] Installing kernel-devel-0:6.11.4-301.fc41.x86_64                                                                                                                 100% |   6.8 MiB/s |  79.1 MiB |  00m12s
Complete!ng post-install scriptlet: kernel-devel-0:6.11.4-301.fc41.x86_64

I then tried rebuilding, and it seemed to work:

james@mach:~$ sudo akmods --rebuild --force
Checking kmods exist for 6.11.4-301.fc41.x86_64            [  OK  ]
Building and installing nvidia-kmod                        [  OK  ]

But, after a while, nvidia still hadn’t shown up:

james@mach:~$ modinfo -F version nvidia
modinfo: ERROR: Module nvidia not found.

So I removed and re-installed akmod-nvidia. After a while, it showed up!

james@mach:~$ modinfo -F version nvidia
560.35.03

I then installed CUDA, seemingly successfully.

Thank you. I will definitely study grubby and figure out what’s going on with my kernels. With Debian and Ubuntu, when a new kernel is installed, it automatically becomes the new default kernel at boot. That seems not to be the case with Fedora? (Or perhaps something went wrong with my machine?)

1 Like
4

It should work automatically, but you can also change the default kernel manually:

sudo grubby --info=ALL
sudo grubby --set-default=/boot/vmlinuz-6.11.7-300.fc41.x86_64
sudo grubby --default-kernel

If the problem persists, try resetting the GRUB environment:

sudo grub2-editenv - create
1 Like
5

Thank you again, @vgaetera, for these helpful commands. It’s quite late here, so I’ll try that tomorrow after work.

Will do some research/studying first because I’m scared to death to touch GRUB. I think I lost two (of maybe eight?) Ubuntu servers during GRUB → GRUB2 upgrades quite a few years ago and another Debian laptop to a corrupted GRUB2 more recently.

6

Do you have secure boot enabled?
Did you setup the secure boot signing key?

The rpmfusion docs explain how setup the secure boot key.

7

Excellent question, @barryascott. In a draft of my post I included the following before removing it to avoid posting an essay…

FWIW, I skipped the Secure Boot steps because Secure Boot is disabled, according to both my BIOS and my OS:

james@mach:~$ mokutil --sb-state
SecureBoot disabled
Platform is in Setup Mode

I think this means I can skip the secure boot signing key steps.

8

I think the mention of SecureBoot (SB) is just a convenient way to generate a signing key.

Linux Module Signing from kernel.org

/lib/modules/6.11.7-300.fc41.x86_64/config says: # CONFIG_MODULE_SIG_FORCE is not set, so Fedora 41 should load unsigned modules. Did you look for a signature (modinfo -F signature nvidia)? I’ve always used the locally generated SB key even though I don’t use SB.

I’ve had problems with grubby. I always remove the rhgb quiet and on Nvidia systems setup an alternate command line so I can quickly switch back to nouveau.
The first line of my /etc/default/grub is:

# after editing, use: doas grub2-mkconfig -o /boot/grub2/grub.cfg
9

If you mask the nvidia fallback service, then reboot, you can do the modprobe nvidia from a terminal and see error messages that should help to explain what the problem is.

10

I was worried/confused why I seemed to not have the latest kernel-headers file, but I think I just convinced myself that I actually do.

As I’m new to Fedora and used to kernel package naming systems where related packages have consistent numberings, I’ve been spooked by my various kernel-xxx packages having different version numbers. I believe I’ve just discovered that Fedora just does this differently. I see, for example, that:

  • kernel-6.11.7-300.fc41 is the latest “stable” package in “updates” (Fedora Updates System )
  • kernel-core is a subpackage of kernel, not an installable package (?), which would explain why my update hasn’t pulled in 6.11.8-300.fc41
  • kernel-headers-6.11.3-300.fc41 is the latest for fc41 at Fedora Updates System

This is obviously obvious to many of you, but it took this Debian/Ubuntu guy a little research to figure out.

11

Thank you for sharing your thoughts, @gnwiii. Just ran modinfo -F signature nvidia and I do see a signature, despite – as I said – having skipped those steps.

Thanks for the grubby caution. I don’t know enough to even understand your suggestions on that, but I’ll do my homework and figure it out before doing anything rash. (Sadly, I have little time for this project during the week, so I’m moving slowly.)

12

I’ve just changed my default kernel via grubby. Thanks again, @vgaetera. I tried to figure out the commands myself and then confirmed they matched your suggestions.

One (unimportant) question. I believe I could have achieved this with either of these commands. If there’s a reason to prefer --set-default, I’d be curious why.

       --set-default=kernel-path
              The first entry which boots the specified kernel is made the default boot entry.

       --set-default-index=entry-index
              Makes the given entry number the default boot entry.

Anyhow, I’ve set the default to my latest kernel:

james@mach:~$ sudo grubby --set-default=/boot/vmlinuz-6.11.7-300.fc41.x86_64
The default is /boot/loader/entries/5cf058c1ad62496fbb8d9914399e6db6-6.11.7-300.fc41.x86_64.conf with index 0 and kernel /boot/vmlinuz-6.11.7-300.fc41.x86_64

james@mach:~$ sudo grubby --default-kernel
/boot/vmlinuz-6.11.7-300.fc41.x86_64

Was considering rebooting, but I now understand why @gnwiii recommends removing rhgb quiet (see: What does rhgb boot option do?), and that seems like a smart change to make before rebooting. Will try that tomorrow night.

13

Figured that out:

james@mach:~$ sudo grubby --update-kernel=ALL --remove-args='quiet'
james@mach:~$ sudo grubby --update-kernel=ALL --remove-args='rhgb'

Seems to have worked.

14

Rebooted fine. Firefox and LibreOffice work fine. But I can’t open a Terminal window! It sometimes seems to try to start, but I see only a spinner, and no Terminal ever appears. This has never happened to me before. Will investigate tomorrow.

15

Settings is also failing to open. Also saw a black spinner.

16

Setting the index number (default is 0) is the preferred method. If you set it by the kernel name/version it will always remain that kernel and will not boot the newer kernel when an update occurs.

17

There’s no difference since it converts the kernel/index to the boot entry and stores it in the GRUB environment, see for yourself:

sudo grub2-editenv - list

BTW, resetting the GRUB environment makes it boot the latest kernel version, which helps if the file is corrupted for some reason.

18

I fixed my inability to open a terminal or other apps by SSH-ing in and adding GSK_RENDERER=ngl to /etc/environment, as recommended in Latest Gnome apps not working in fedora 41 using wayland - #22 by mattipulkkinen.

Thanks for the feedback on --set-default vs. --set-default-index. My index and default kernel both look fine ATM:

james@mach:~$ sudo grubby --default-title
Fedora Linux (6.11.7-300.fc41.x86_64) 41 (Workstation Edition)
james@mach:~$ sudo grubby --default-kernel
/boot/vmlinuz-6.11.7-300.fc41.x86_64
james@mach:~$ sudo grubby --info=DEFAULT
index=0
kernel="/boot/vmlinuz-6.11.7-300.fc41.x86_64"
args="ro rootflags=subvol=root $tuned_params ipv6.disable=1 rd.driver.blacklist=nouveau modprobe.blacklist=nouveau"
root="UUID=df19fd77-..."
initrd="/boot/initramfs-6.11.7-300.fc41.x86_64.img $tuned_initrd"
title="Fedora Linux (6.11.7-300.fc41.x86_64) 41 (Workstation Edition)"
id="5cf05...-6.11.7-300.fc41.x86_64"
james@mach:~$ sudo grubby --default-index
0

I think everything is working now, aside from having to use the workaround for the current/known/active Fedora/GNOME bug.

I’ll test by this weekend whether I’m able to do any ML with GPU acceleration and report back here.