Downgrading nvidia driver from 535

I’m trying to downgrade my nvidia driver from 535, since it is causing regular stuttering and frame drop issues that weren’t there before.

Steps I’ve taken:

  • Remove current driver: sudo dnf remove \*nvidia\* --exclude nvidia-gpu-firmware, then reboot
  • Lock out the problematic version: sudo dnf install 'dnf-command(versionlock)', then sudo dnf versionlock exclude akmod-nvidia-535.113.01-1.fc38
  • Install drivers following the normal instructions and verifying that all packages are nvidia 350.x and versionlocking anything that tries to install a different version as above
  • Reboot

I was surprised that despite everything apparently working, I’m still on nouveau after a reboot. lsmod | grep nouveau shows it loaded, and lsmod | grep nvidia shows nothing. All expected packages are installed:

 $ dnf list --installed | grep nvidia
akmod-nvidia.x86_64                                       3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
libva-nvidia-driver.x86_64                                0.0.10-3.fc38                          @updates                                         
nvidia-gpu-firmware.noarch                                20230919-1.fc38                        @updates                                         
nvidia-persistenced.x86_64                                3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
nvidia-settings.x86_64                                    3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia.x86_64                                3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-cuda.x86_64                           3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-cuda-libs.i686                        3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-cuda-libs.x86_64                      3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-kmodsrc.x86_64                        3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-libs.i686                             3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-libs.x86_64                           3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-power.x86_64                          3:530.41.03-1.fc38                     @rpmfusion-nonfree

So I tried manually triggering the akmod build, but it fails:

 $ sudo akmods
Checking kmods exist for 6.5.8-200.fc38.x86_64             [  OK  ]
Building and installing nvidia-kmod                        [FAILED]
Building rpms failed; see /var/cache/akmods/nvidia/530.41.03-1-for-6.5.8-200.fc38.x86_64.failed.log for details

Hint: Some kmods were ignored or failed to build or install.
You can try to rebuild and install them by by calling
'/usr/sbin/akmods --force' as root.

The log file for the failed build is here, the relevant error seems to be a lot of variations on this error:

2023/10/29 18:57:13 akmodsbuild: /tmp/akmodsbuild.M8wOQ9Bg/BUILD/nvidia-kmod-530.41.03/_kmod_build_6.5.8-200.fc38.x86_64/common/inc/nv-mm.h:88:16: error: too many arguments to function 'get_user_pages'
2023/10/29 18:57:13 akmodsbuild:    88 |         return get_user_pages(current, current->mm, start, nr_pages, write,
2023/10/29 18:57:13 akmodsbuild:       |                ^~~~~~~~~~~~~~
2023/10/29 18:57:13 akmodsbuild: ./include/linux/mm.h:2430:6: note: declared here
2023/10/29 18:57:13 akmodsbuild:  2430 | long get_user_pages(unsigned long start, unsigned long nr_pages,
2023/10/29 18:57:13 akmodsbuild:       |      ^~~~~~~~~~~~~~

My reading of this is that the akmod for version 530.41.03 is failing to build against my kernel version (since the function in the error seems to be from a kernel header).

Can anybody help me finish this downgrade? Is it just not possible to downgrade without also downgrading my kernel, and if so, how can I figure out which kernel version to revert to?

System info:

  • Fedora 38 Workstation, GNOME, X11
  • Kernel: 6.5.8-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Oct 20 15:53:48 UTC 2023 x86_64 GNU/Linux
  • GPU: Nvidia 3070Ti (desktop)

You will need to also use an old kernel I would guess.
Or you will need to patch the nvidia glue code to use the changed way to call get_user_pages.
You could look at the latest driver for what is required.

What’s the best way of me finding the source of what’s being compiled?

Any idea how I might find which kernel version each is targeted for?

The name of the .c file will be in the akmodsbuild log.

From your pastbin I see this:

2023/10/29 18:57:13 akmodsbuild: In file included from /tmp/akmodsbuild.M8wOQ9Bg/BUILD/nvidia-kmod-530.41.03/_kmod_build_6.5.8-200.fc38.x86_64/common/inc/nv-pgprot.h:30,
2023/10/29 18:57:13 akmodsbuild:                  from /tmp/akmodsbuild.M8wOQ9Bg/BUILD/nvidia-kmod-530.41.03/_kmod_build_6.5.8-200.fc38.x86_64/common/inc/nv-linux.h:33,
2023/10/29 18:57:13 akmodsbuild:                  from /tmp/akmodsbuild.M8wOQ9Bg/BUILD/nvidia-kmod-530.41.03/_kmod_build_6.5.8-200.fc38.x86_64/nvidia/nv-nano-timer.c:31:
2023/10/29 18:57:13 akmodsbuild: ./include/linux/mm.h:2430:35: note: expected 'long unsigned int' but argument is of type 'struct task_struct *'
2023/10/29 18:57:13 akmodsbuild:  2430 | long get_user_pages(unsigned long start, unsigned long nr_pages,
2023/10/29 18:57:13 akmodsbuild:       |                     ~~~~~~~~~~~~~~^~~~~

So it should be in nv-nano-timer.c:31

This shows the system did not build and install the kmod-nvidia package for the currently installed nvidia packages (530.41.03).

Try that as sudo akmods --force and if it still fails then try
sudo akmods --kernels $(uname -r) --force

The same suggestion was at the bottom of what you posted for the failed akmods run.

I should have said, I tried akmods --force and it failed with exactly the same compilation error. Wouldn’t the default kernel for akmods be the current one anyway?

I meant where can I find the repo that contains the source code, not which file within the code is failing.

On my system I find

# find /usr -name nvidia-kmod*
/usr/share/nvidia-kmod-535.113.01
/usr/share/nvidia-kmod-535.113.01/nvidia-kmod-535.113.01-x86_64.tar.xz
/usr/src/akmods/nvidia-kmod.latest
/usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm

and 

ls -l /usr/src/akmods/
total 240
-rw-r--r--. 1 root root 90454 Sep 21 19:00  nvidia-kmod-535.113.01-1.fc39.src.rpm
lrwxrwxrwx. 1 root root    37 Sep 21 19:00  nvidia-kmod.latest -> nvidia-kmod-535.113.01-1.fc39.src.rpm

Those files come from installing akmod-nvidia from rpmfusion.

# dnf provides /usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm
Last metadata expiration check: 0:08:46 ago on Mon 30 Oct 2023 01:16:52 PM CDT.
akmod-nvidia-3:535.113.01-1.fc39.x86_64 : Akmod package for nvidia kernel module(s)
Repo        : @System
Matched from:
Filename    : /usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm

akmod-nvidia-3:535.113.01-1.fc39.x86_64 : Akmod package for nvidia kernel module(s)
Repo        : rpmfusion-nonfree
Matched from:
Filename    : /usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm

akmod-nvidia-3:535.113.01-1.fc39.x86_64 : Akmod package for nvidia kernel module(s)
Repo        : rpmfusion-nonfree-nvidia-driver
Matched from:
Filename    : /usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm

Has this been resolved as i like to do the same. I want to downgrade from 545x to 535.86.05

I was able to downgrade from 545 back to the last 535.129.03 with 2 simple commands. My drivers were installed from rpmfusion-nonfree and I do not know if the same would work properly with drivers installed from rpmfusion-nonfree-nvidia-drivers.
sudo dnf remove \*nvidia\* --exclude nvidia-gpu-firmware
followed by
sudo dnf install akmod-nvidia-535.129.03\* xorg-x11-drv-nvidia-cuda-535.129.03\* nvidia\*535.129.03\*

2 Likes

This helped me!
545 was giving me a lot of screen tearing, so downgrading to 535 did the trick :slight_smile:

EDIT: I also have installed from rpmfusion-nonfree:

dnf info akmod-nvidia | grep “From repo”
From repo : rpmfusion-nonfree

ok this worked for me also, now how do i freeze the 535 from being upgraded to 545?

Asking @computersavvy specifically
Won’t dnf downgrade nvidia-* --exclude nvidia-gpu-firmware be enough for anyone looking to move down a driver version? It’s been a while for me, but I believe that’s how i used to do it. . .

If you are using dnf then adding the --exclude \*nvidia\* option to the dnf command line will do that.

I initially had the 545 driver which gave me a failed cuda install so I downgraded back to 535. After a couple weeks I tried again with the upgrade and then cuda worked with the 545 drivers so I have been content since.

Hmm, interesting. so dnf downgrade your-package should work like it.

Almost.

I did not try dnf downgrade in my case since I knew exactly what I wanted to reinstall and chose not to allow it to be managed automatically by dnf (and I seldom use a downgrade so had even forgotten the command was available)

I just tested and found that the command
dnf downgrade \*nvidia\* --exclude nvidia-gpu-firmware --allowerasing would do the same.

Thanks for the reminder. :grinning:

Yeh the exclude kinda works but discovery picks up all the 545 upgrades. I added

exclude=nvidia-* into sudo nano /etc/dnf/dnf.conf and it didnt stop or filter out the 545.
So i installed sudo dnf install dnf-plugin-versionlock and added

sudo dnf versionlock add akmod-nvidia-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-modprobe-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-persistenced-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-settings-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-xconfig-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-cuda-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-cuda-libs-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-kmodsrc-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-libs-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-power-3:535.129.03-1.fc39

Nothing’s working

Yes and no.

For packages provided by fedora the downgrade will only have access to the latest updated packages in updates or to the packages in the fedora repo that were available with first release of that version. Fedora updates only keeps the most recently released packages. Thus you could not revert back to the last installed version from the fedora-updates repo if there were any versions between the oldest and newest.

For other repos such as rpmfusion it seems you may be able to go back to a specific version, and in this case from the nvidia 545 drivers to the 535 drivers since they are both still available in the rpmfusion repos. (535 in rpmfusion-nonfree and 545 in rpmfusion-nonfree-updates).

Do you have any updates? It’s reaaly interesting to see and apparently many people are having a probnlem with the new Nvidia driver.

dnf install python3-dnf-plugin-versionlock

rpm -qa xorg-x11-drv-nvidia* *kmod-nvidia* nvidia-{settings,xconfig,modprobe,persistenced} >> /etc/dnf/plugins/versionlock.list

DOWNGRADE NVIDIA FROM 545x to 535x

sudo dnf remove \*nvidia\* --exclude nvidia-gpu-firmware

Uninstall the NVIDIA driver

sudo dnf remove xorg-x11-drv-nvidia\*

sudo dnf install akmod-nvidia-535.129.03\* xorg-x11-drv-nvidia-cuda-535.129.03\* nvidia\*535.129.03\*

sudo dnf install dnf-plugin-versionlock
sudo dnf versionlock add akmod-nvidia-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-modprobe-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-persistenced-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-settings-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-xconfig-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-cuda-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-cuda-libs-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-kmodsrc-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-libs-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-power-3:535.129.03-1.fc39
sudo rm /etc/yum.repos.d/nvidia-exclude.repo
sudo dnf versionlock list
sudo dnf update
sudo dnf update
sudo dnf versionlock list

DELETE LOCKS

        # sudo dnf versionlock delete akmod-nvidia-3:535.129.03-1.fc39
        #sudo dnf versionlock delete nvidia-modprobe-3:535.129.03-1.fc39
        #sudo dnf versionlock delete nvidia-persistenced-3:535.129.03-1.fc39
        #sudo dnf versionlock delete nvidia-settings-3:535.129.03-1.fc39
        #sudo dnf versionlock delete nvidia-xconfig-3:535.129.03-1.fc39

545x still shows up in the updates, not sure how to stop dnf to filter out 545x. thats the only part im currently at a lose on. Hvant tested out since last post as im on nixos kde atm with nvidia 535x

Your more then welcom to try out my installer:

bash -c "$(curl -fsSL https://raw.githubusercontent.com/tolgaerok/tolga-scripts/main/Fedora39/setup.sh)"