Fedora 41, nvidia-open, and dkms

jtingiris · February 21, 2025, 4:35pm

I writing this here because of nvidia-open/570.86.15 issues on Fedora 41. When dnf upgrade is run, and it installs a new kernel, then dkms will automatically build nvidia-open under the running kernel. That’s expected.

To be clear, I originally installed via the official NVIDIA CUDA Installation Guide for Linux that’s located here:

I use the remote RPM repository as described there (yes, there’s an official fedora 41 repo despite what the docs say):

It basically boils down to this …

sudo dnf config-manager addrepo --from-repofile https://developer.download.nvidia.com/compute/cuda/repos/fedora41/x86_64/cuda-fedora41.repo
sudo dnf clean all
sudo dnf -y install cuda-toolkit-12-8
sudo dnf -y install nvidia-open

… and everything works great the first time.

However, by default (without explicitly setting a value for KERNEL_UNAME), the Nvidia Makefile included uses uname -r and that causes problems after a dnf upgrade installs a new kernel and dkms is immediately run. The modules that get dynamically built are for the running kernel, not the newly installed kernel.

That is evident in the /var/lib/dkms/nvidia-open/*/*/*/log/make.log file(s). Notice it enter’s the ‘wrong’ /usr/src/kernels directory, ie:

DKMS (dkms-3.1.5) make.log for nvidia-open/570.86.15 for kernel 6.12.15-200.fc41.x86_64 (x86_64)
Fri Feb 21 09:50:50 AM EST 2025
Cleaning build area
# command: 'make' clean
make -C src/nvidia clean
make[1]: Entering directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia'
rm -f -rf _out/Linux_x86_64
make[1]: Leaving directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia'
make -C src/nvidia-modeset clean
make[1]: Entering directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia-modeset'
rm -f -rf _out/Linux_x86_64
make[1]: Leaving directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia-modeset'
make -C kernel-open clean
make[1]: Entering directory '/var/lib/dkms/nvidia-open/570.86.15/build/kernel-open'
rm -f -r conftest
make[2]: Entering directory '/usr/src/kernels/6.12.10-200.fc41.x86_64'
make[2]: Leaving directory '/usr/src/kernels/6.12.10-200.fc41.x86_64'
make[1]: Leaving directory '/var/lib/dkms/nvidia-open/570.86.15/build/kernel-open'
...

Subsequent reboots fail to load the module, eg:

kernel: nvidia: version magic '6.12.10-200.fc41.x86_64 SMP preempt mod_unload ' should be '6.12.11-200.fc41.x86_64 SMP preempt mod_unload '

Here’s a short work-around (for Fedora) that leverages KERNEL_UNAME and properly build nvidia-open for the ‘latest’ kernel.

After a dnf upgrade, a variation of following will work as root.

# do this as root
sudo su -

# fyi, version sort installed kernels
rpm -qa kernel | sed -e 's/^kernel-//g' | sort -uV

# fyi, version sort installed dkms module/module-version and kernel/arch
dkms status | sort -uV

export CURRENT_KERNEL="$(uname -r)"; echo "CURRENT_KERNEL=${CURRENT_KERNEL}"
export LATEST_KERNEL="$(rpm -qa kernel | sed -e 's/^kernel-//g' | sort -uV | tail -1)"; echo LATEST_KERNEL=${LATEST_KERNEL} # this matches uname -r

# example rebuild; pick one
export KERNEL_UNAME="6.12.11-200.fc41.x86_64"
export KERNEL_UNAME=${CURRENT_KERNEL}
export KERNEL_UNAME=${LATEST_KERNEL}
echo KERNEL_UNAME=${KERNEL_UNAME}

# set proper values for dkms build, install, etc
export DKMS_ARCH="$(dkms status | grep ${KERNEL_UNAME}, | awk -F, '{print $3}' | awk '{print $1}' | awk -F: '{print $1}')"
export DKMS_KERNEL="$(dkms status | grep ${KERNEL_UNAME}, | awk -F, '{print $2}' | awk '{print $1}')" # should be the same as KERNEL_UNAME
export DKMS_MODULE_VERSION="$(dkms status | grep ${KERNEL_UNAME}, | awk -F, '{print $1}' | awk '{print $1}')"

# manually verify values
echo DKMS_ARCH=${DKMS_ARCH}
echo DKMS_KERNEL=${DKMS_KERNEL}
echo DKMS_MODULE_VERSION=${DKMS_MODULE_VERSION}

# NOTICE! Using the LATEST_KERNEL value is easiest/safest.
# NOTICE! If you're booted with the latest kernel and the modules ARE NOT loaded, then properly rebuilding may immediately load the correct signed module and will likely reset a graphical session.
# IMPORTANT! If you're booted from the kernel you want to 'fix' then do this in a tmux, screen, or from the linux console.
KERNEL_UNAME=${DKMS_KERNEL} dkms uninstall ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH}
KERNEL_UNAME=${DKMS_KERNEL} dkms build ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH} --force
KERNEL_UNAME=${DKMS_KERNEL} dkms install ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH}
KERNEL_UNAME=${DKMS_KERNEL} dkms status ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH}

# verify the build make.log enters the correct /usr/src/kernels directory ...
less /var/lib/dkms/nvidia-open/*/${DKMS_KERNEL}/${DKMS_ARCH}/log/make.log

systemctl reboot

Using uname -r in the Makefile seems a bit too convenient for a dynamic kernel module.

This has been happening for some time (I traced my own issues back for over a year, into f40) and there are SO many threads on similar issues. Still, I couldn’t find any that really detail what I’m trying to say here or any decent workarounds. I decided to spend the morning on this issue. I’m considering submitting a PR or bug report to Nvidia, too.

In the meantime, I posted this here, with a bunch of keywords and strings, hoping it would be found more easily and help someone else. Good luck.

jtingiris · February 21, 2025, 5:10pm

I opened this issue, too

github.com/NVIDIA/open-gpu-kernel-modules

dkms incorrectly builds updated kernels with the current kernel's source

opened 04:56PM - 21 Feb 25 UTC

josephtingiris

build-problem

### NVIDIA Open GPU Kernel Modules Version nvidia-open/570.86.15 ### Operating… System and Version Fedora 41 ### Kernel Release Linux d0 6.12.15-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Feb 18 15:24:05 UTC 2025 x86_64 GNU/Linux ### Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels. - [x] I am running on a stable kernel release. ### Build Command `dnf upgrade` (which invokes `dkms build` under the running kernel) ### Terminal output/Build Log Notice it enters the 'wrong' `/usr/src/kernels` directory, ie: ``` DKMS (dkms-3.1.5) make.log for nvidia-open/570.86.15 for kernel 6.12.15-200.fc41.x86_64 (x86_64) Fri Feb 21 09:50:50 AM EST 2025 Cleaning build area # command: 'make' clean make -C src/nvidia clean make[1]: Entering directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia' rm -f -rf _out/Linux_x86_64 make[1]: Leaving directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia' make -C src/nvidia-modeset clean make[1]: Entering directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia-modeset' rm -f -rf _out/Linux_x86_64 make[1]: Leaving directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia-modeset' make -C kernel-open clean make[1]: Entering directory '/var/lib/dkms/nvidia-open/570.86.15/build/kernel-open' rm -f -r conftest make[2]: Entering directory '/usr/src/kernels/6.12.10-200.fc41.x86_64' make[2]: Leaving directory '/usr/src/kernels/6.12.10-200.fc41.x86_64' make[1]: Leaving directory '/var/lib/dkms/nvidia-open/570.86.15/build/kernel-open' ... ``` ### More Info I landed here because of `nvidia-open/570.86.15` issues on Fedora 41. When `dnf upgrade` is run, and it installs a new kernel, then `dkms` will automatically build `nvidia-open` under the running kernel (without explicitly setting a value for `KERNEL_UNAME`). The currently included [Makefile](https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/kernel-open/Makefile) is being used. That is evident in the `/var/lib/dkms/nvidia-open/*/*/*/log/make.log` file(s). Subsequent reboots fail to load the module, eg: ``` kernel: nvidia: version magic '6.12.10-200.fc41.x86_64 SMP preempt mod_unload ' should be '6.12.11-200.fc41.x86_64 SMP preempt mod_unload ' ``` Here's a short work-around (for Fedora) that leverages `KERNEL_UNAME` and will properly build `nvidia-open` for the 'latest' kernel. After a `dnf upgrade`, a variation of following will work as `root`. ```sh # do this as root sudo su - # fyi, version sort installed kernels rpm -qa kernel | sed -e 's/^kernel-//g' | sort -uV # fyi, version sort installed dkms module/module-version and kernel/arch dkms status | sort -uV export CURRENT_KERNEL="$(uname -r)"; echo "CURRENT_KERNEL=${CURRENT_KERNEL}" export LATEST_KERNEL="$(rpm -qa kernel | sed -e 's/^kernel-//g' | sort -uV | tail -1)"; echo LATEST_KERNEL=${LATEST_KERNEL} # this matches uname -r # example rebuild; pick one export KERNEL_UNAME="6.12.11-200.fc41.x86_64" export KERNEL_UNAME=${CURRENT_KERNEL} export KERNEL_UNAME=${LATEST_KERNEL} echo KERNEL_UNAME=${KERNEL_UNAME} # set proper values for dkms build, install, etc export DKMS_ARCH="$(dkms status | grep ${KERNEL_UNAME}, | awk -F, '{print $3}' | awk '{print $1}' | awk -F: '{print $1}')" export DKMS_KERNEL="$(dkms status | grep ${KERNEL_UNAME}, | awk -F, '{print $2}' | awk '{print $1}')" # should be the same as KERNEL_UNAME export DKMS_MODULE_VERSION="$(dkms status | grep ${KERNEL_UNAME}, | awk -F, '{print $1}' | awk '{print $1}')" # manually verify values echo DKMS_ARCH=${DKMS_ARCH} echo DKMS_KERNEL=${DKMS_KERNEL} echo DKMS_MODULE_VERSION=${DKMS_MODULE_VERSION} # NOTICE! Using the LATEST_KERNEL value is easiest/safest. # NOTICE! If you're booted with the latest kernel and the modules ARE NOT loaded, then properly rebuilding may immediately load the correct signed module and will likely reset a graphical session. # IMPORTANT! If you're booted from the kernel you want to 'fix' then do this in a tmux, screen, or from the linux console. KERNEL_UNAME=${DKMS_KERNEL} dkms uninstall ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH} KERNEL_UNAME=${DKMS_KERNEL} dkms build ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH} --force KERNEL_UNAME=${DKMS_KERNEL} dkms install ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH} KERNEL_UNAME=${DKMS_KERNEL} dkms status ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH} # verify the build make.log enters the correct /usr/src/kernels directory ... less /var/lib/dkms/nvidia-open/*/${DKMS_KERNEL}/${DKMS_ARCH}/log/make.log systemctl reboot ``` Using `uname -r` in the `Makefile` seems a bit too convenient for a dynamic kernel module. In the meantime, I posted the workaround here hoping it would help. Thanks

barryascott · February 21, 2025, 10:38pm

The rpmfusion developers package the nvidia “open” drivers for Fedora and make sure that they work. We recommend you use there version of the driver rather the the nvidia.com drivers that many people have reported issues with.

jtingiris · February 21, 2025, 11:16pm

Thanks. Personally, I’ve had more problems with rpmfusion and dnf update than using the official Nvidia drivers. Especially the past year or so. Nvidia documentation & support has improved greatly.

glb · February 21, 2025, 11:37pm

I think if you don’t specify the kernel version, dkms is supposed to (re)build the module for both the current kernel and the newest kernel.

Excerpted from github.com – dell/dkms:

# Here we look for the most recent kernel so that we can
# build the module for it (in addition to doing it for the
# current kernel).

It looks like dkms is using a reasonably reliable method for determining what is the newest kernel:

# Get the most recent kernel in Rhel based systems.
_get_newest_kernel_rhel() {
    rpm -q --qf="%{VERSION}-%{RELEASE}.%{ARCH}\n" --whatprovides kernel | tail -n 1
}

jtingiris · February 21, 2025, 11:45pm

Thanks. Mine is working fine again. I was just trying to share a workaround here, for others faced with the same issue. There’s a bug (or two) with the official f41 CUDA rpms, where the dkms.conf that gets used doesn’t pass KERNEL_UNAME to the Makefile. By default, the Makefile uses a uname -r and that causes it to compile the module with the kernel sources of the running kernel (rather than the updated kernel after a dnf update).

I put more detail here, too: [570][Fedora41] dkms incorrectly builds updated kernels with the current kernel's source · Issue #10 · NVIDIA/yum-packaging-nvidia-driver · GitHub

Hoping they fix it one way or another. In the meantime, I’ve changed the Makefile on my filesystem and dnf update works flawlessly again. dkms automatically compiles the drivers for the new kernel as it should.

msmafra · February 26, 2025, 2:36pm

That would only get Rhel/Fedora/CentOS kernel. On my system, for example, it wouldn’t get the latest kernel.

 ❯ rpm -q --qf="%{VERSION}-%{RELEASE}.%{ARCH}\n" --whatprovides kernel
6.13.2-cachyos1.fc41.x86_64
6.12.13-200.fc41.x86_64
6.13.3-cachyos1.fc41.x86_64
6.12.15-200.fc41.x86_64

Adding sort with -k, --key helps to order and later get the latest kernel independantly if it is RHEL format or not, but is not necessarely the definitive solution since there are a few different kernels in the COPR and unfotunetly sort -V, --version-sort would only work with the Fedora versioning pattern.

❯ rpm -q --qf="%{VERSION}-%{RELEASE}.%{ARCH}\n" --whatprovides kernel | sort -t "." -k4,4 -k1,1 -k2,2 -k3,3
6.12.13-200.fc41.x86_64
6.12.15-200.fc41.x86_64
6.13.2-cachyos1.fc41.x86_64
6.13.3-cachyos1.fc41.x86_64

glb · February 26, 2025, 3:27pm

Good catch. That issue should probably be reported to the repo.

jtingiris · February 26, 2025, 3:33pm

I already reported it to Nvidia upstream, and they’ve acepted & confirmed it will be fixed in the new 570 builds. Workarounds shouldn’t be needed much longer.

github.com/NVIDIA/open-gpu-kernel-modules

allow a way to specify the kernel

opened 10:41PM - 06 Sep 23 UTC

vans163

build-problem

### NVIDIA Open GPU Kernel Modules Version 535.104.05 ### Operating System and… Version Ubuntu 23.04 ### Kernel Release Linux 6.5.1 #1 SMP PREEMPT_DYNAMIC Wed Sep 6 18:49:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ### Build Command make modules_install ### Terminal output/Build Log ``` make[1]: Leaving directory '/root/source/open-gpu-kernel-modules/src/nvidia' cd kernel-open/nvidia/ && ln -sf ../../src/nvidia/_out/Linux_x86_64/nv-kernel.o nv-kernel.o_binary make -C kernel-open modules make[1]: Entering directory '/root/source/open-gpu-kernel-modules/kernel-open' make[2]: Entering directory '/root/source/open-gpu-kernel-modules/kernel-open' make[2]: *** /lib/modules/6.2.0-26-generic/build: No such file or directory. Stop. make[2]: Leaving directory '/root/source/open-gpu-kernel-modules/kernel-open' make[1]: *** [Makefile:82: modules] Error 2 make[1]: Leaving directory '/root/source/open-gpu-kernel-modules/kernel-open' make: *** [Makefile:59: modules] Error 2 ``` ### More Info I am running the build inside a systemd-container `make[2]: *** /lib/modules/6.2.0-26-generic/build: No such file or directory. Stop.` `6.2.0-26-generic` is the output of `uname -r`. The NVIDIA driver correctly detects this and pulls the kernel from inside the systemd-container. the installer here does not give any option to specify the kernel to install the modules to or automatically detect it.

jtingiris · February 26, 2025, 3:36pm

The new 570 spec (et al) was published here a couple days ago …

github.com/NVIDIA/yum-packaging-nvidia-driver

[570][Fedora41] dkms incorrectly builds updated kernels with the current kernel's source

opened 05:51PM - 21 Feb 25 UTC

kmittman

_Originally posted by @josephtingiris in https://github.com/NVIDIA/open-gpu-kern…el-modules/issues/791_ ### NVIDIA Open GPU Kernel Modules Version nvidia-open/570.86.15 ### Operating System and Version Fedora 41 ### Kernel Release Linux d0 6.12.15-200.fc41.x86_64 SMP PREEMPT_DYNAMIC Tue Feb 18 15:24:05 UTC 2025 x86_64 GNU/Linux ### Build Command `dnf upgrade` (which invokes `dkms build` under the running kernel) ### Terminal output/Build Log Notice it enters the 'wrong' `/usr/src/kernels` directory, ie: ```bash DKMS (dkms-3.1.5) make.log for nvidia-open/570.86.15 for kernel 6.12.15-200.fc41.x86_64 (x86_64) Fri Feb 21 09:50:50 AM EST 2025 Cleaning build area # command: 'make' clean make -C src/nvidia clean make[1]: Entering directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia' rm -f -rf _out/Linux_x86_64 make[1]: Leaving directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia' make -C src/nvidia-modeset clean make[1]: Entering directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia-modeset' rm -f -rf _out/Linux_x86_64 make[1]: Leaving directory '/var/lib/dkms/nvidia-open/570.86.15/build/src/nvidia-modeset' make -C kernel-open clean make[1]: Entering directory '/var/lib/dkms/nvidia-open/570.86.15/build/kernel-open' rm -f -r conftest make[2]: Entering directory '/usr/src/kernels/6.12.10-200.fc41.x86_64' make[2]: Leaving directory '/usr/src/kernels/6.12.10-200.fc41.x86_64' make[1]: Leaving directory '/var/lib/dkms/nvidia-open/570.86.15/build/kernel-open' ... ``` ### More Info I landed here because of `nvidia-open/570.86.15` issues on Fedora 41. When `dnf upgrade` is run, and it installs a new kernel, then `dkms` will automatically build `nvidia-open` under the running kernel (without explicitly setting a value for `KERNEL_UNAME`). The currently included [Makefile](https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/kernel-open/Makefile) is being used. That is evident in the `/var/lib/dkms/nvidia-open/*/*/*/log/make.log` file(s). Subsequent reboots fail to load the module, eg: ``` kernel: nvidia: version magic '6.12.10-200.fc41.x86_64 SMP preempt mod_unload ' should be '6.12.11-200.fc41.x86_64 SMP preempt mod_unload ' ``` Here's a short work-around (for Fedora) that leverages `KERNEL_UNAME` and will properly build `nvidia-open` for the 'latest' kernel. After a `dnf upgrade`, a variation of following will work as `root`. ```bash # do this as root sudo su - # fyi, version sort installed kernels rpm -qa kernel | sed -e 's/^kernel-//g' | sort -uV # fyi, version sort installed dkms module/module-version and kernel/arch dkms status | sort -uV export CURRENT_KERNEL="$(uname -r)"; echo "CURRENT_KERNEL=${CURRENT_KERNEL}" export LATEST_KERNEL="$(rpm -qa kernel | sed -e 's/^kernel-//g' | sort -uV | tail -1)"; echo LATEST_KERNEL=${LATEST_KERNEL} # this matches uname -r # example rebuild; pick one export KERNEL_UNAME="6.12.11-200.fc41.x86_64" export KERNEL_UNAME=${CURRENT_KERNEL} export KERNEL_UNAME=${LATEST_KERNEL} echo KERNEL_UNAME=${KERNEL_UNAME} # set proper values for dkms build, install, etc export DKMS_ARCH="$(dkms status | grep ${KERNEL_UNAME}, | awk -F, '{print $3}' | awk '{print $1}' | awk -F: '{print $1}')" export DKMS_KERNEL="$(dkms status | grep ${KERNEL_UNAME}, | awk -F, '{print $2}' | awk '{print $1}')" # should be the same as KERNEL_UNAME export DKMS_MODULE_VERSION="$(dkms status | grep ${KERNEL_UNAME}, | awk -F, '{print $1}' | awk '{print $1}')" # manually verify values echo DKMS_ARCH=${DKMS_ARCH} echo DKMS_KERNEL=${DKMS_KERNEL} echo DKMS_MODULE_VERSION=${DKMS_MODULE_VERSION} # NOTICE! Using the LATEST_KERNEL value is easiest/safest. # NOTICE! If you're booted with the latest kernel and the modules ARE NOT loaded, then properly rebuilding may immediately load the correct signed module and will likely reset a graphical session. # IMPORTANT! If you're booted from the kernel you want to 'fix' then do this in a tmux, screen, or from the linux console. KERNEL_UNAME=${DKMS_KERNEL} dkms uninstall ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH} KERNEL_UNAME=${DKMS_KERNEL} dkms build ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH} --force KERNEL_UNAME=${DKMS_KERNEL} dkms install ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH} KERNEL_UNAME=${DKMS_KERNEL} dkms status ${DKMS_MODULE_VERSION} -k ${DKMS_KERNEL}/${DKMS_ARCH} # verify the build make.log enters the correct /usr/ less /var/lib/dkms/nvidia-open/*/${DKMS_KERNEL}/${DKMS_ARCH}/log/make.log systemctl reboot ``` Using `uname -r` in the `Makefile` seems a bit too convenient for a dynamic kernel module. In the meantime, I posted the workaround here hoping it would help. Thanks > Note: transferred issue from https://github.com/NVIDIA/open-gpu-kernel-modules/issues/791

Topic		Replies	Views
Can nvidia dkms drivers be rebuilt during the update and not after? Ask Fedora kde , upgrades , dnf , nvidia	13	2003	July 9, 2023
NVIDIA drivers corrupted after every kernel update Ask Fedora f38 , updates , driver , nvidia , kernel	3	624	August 23, 2023
Yet another nvidia dkms thread Ask Fedora dkms , nvidia	1	2146	January 26, 2022
My almost fresh install has mismatched kernel versions Ask Fedora nvidia , kernel , kernel-modules , f41	17	1076	November 19, 2024
Dkms NVidia on Fedora 41 not working Ask Fedora	10	221	February 24, 2025

Fedora 41, nvidia-open, and dkms

Related topics