Downgrading nvidia driver from 535

jmaargh · October 29, 2023, 7:05pm

I’m trying to downgrade my nvidia driver from 535, since it is causing regular stuttering and frame drop issues that weren’t there before.

Steps I’ve taken:

Remove current driver: sudo dnf remove \*nvidia\* --exclude nvidia-gpu-firmware, then reboot
Lock out the problematic version: sudo dnf install 'dnf-command(versionlock)', then sudo dnf versionlock exclude akmod-nvidia-535.113.01-1.fc38
Install drivers following the normal instructions and verifying that all packages are nvidia 350.x and versionlocking anything that tries to install a different version as above
Reboot

I was surprised that despite everything apparently working, I’m still on nouveau after a reboot. lsmod | grep nouveau shows it loaded, and lsmod | grep nvidia shows nothing. All expected packages are installed:

 $ dnf list --installed | grep nvidia
akmod-nvidia.x86_64                                       3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
libva-nvidia-driver.x86_64                                0.0.10-3.fc38                          @updates                                         
nvidia-gpu-firmware.noarch                                20230919-1.fc38                        @updates                                         
nvidia-persistenced.x86_64                                3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
nvidia-settings.x86_64                                    3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia.x86_64                                3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-cuda.x86_64                           3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-cuda-libs.i686                        3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-cuda-libs.x86_64                      3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-kmodsrc.x86_64                        3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-libs.i686                             3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-libs.x86_64                           3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
xorg-x11-drv-nvidia-power.x86_64                          3:530.41.03-1.fc38                     @rpmfusion-nonfree

So I tried manually triggering the akmod build, but it fails:

 $ sudo akmods
Checking kmods exist for 6.5.8-200.fc38.x86_64             [  OK  ]
Building and installing nvidia-kmod                        [FAILED]
Building rpms failed; see /var/cache/akmods/nvidia/530.41.03-1-for-6.5.8-200.fc38.x86_64.failed.log for details

Hint: Some kmods were ignored or failed to build or install.
You can try to rebuild and install them by by calling
'/usr/sbin/akmods --force' as root.

The log file for the failed build is here, the relevant error seems to be a lot of variations on this error:

2023/10/29 18:57:13 akmodsbuild: /tmp/akmodsbuild.M8wOQ9Bg/BUILD/nvidia-kmod-530.41.03/_kmod_build_6.5.8-200.fc38.x86_64/common/inc/nv-mm.h:88:16: error: too many arguments to function 'get_user_pages'
2023/10/29 18:57:13 akmodsbuild:    88 |         return get_user_pages(current, current->mm, start, nr_pages, write,
2023/10/29 18:57:13 akmodsbuild:       |                ^~~~~~~~~~~~~~
2023/10/29 18:57:13 akmodsbuild: ./include/linux/mm.h:2430:6: note: declared here
2023/10/29 18:57:13 akmodsbuild:  2430 | long get_user_pages(unsigned long start, unsigned long nr_pages,
2023/10/29 18:57:13 akmodsbuild:       |      ^~~~~~~~~~~~~~

My reading of this is that the akmod for version 530.41.03 is failing to build against my kernel version (since the function in the error seems to be from a kernel header).

Can anybody help me finish this downgrade? Is it just not possible to downgrade without also downgrading my kernel, and if so, how can I figure out which kernel version to revert to?

System info:

Fedora 38 Workstation, GNOME, X11
Kernel: 6.5.8-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Oct 20 15:53:48 UTC 2023 x86_64 GNU/Linux
GPU: Nvidia 3070Ti (desktop)

barryascott · October 29, 2023, 10:45pm

You will need to also use an old kernel I would guess.
Or you will need to patch the nvidia glue code to use the changed way to call get_user_pages.
You could look at the latest driver for what is required.

jmaargh · October 29, 2023, 11:14pm

What’s the best way of me finding the source of what’s being compiled?

Any idea how I might find which kernel version each is targeted for?

barryascott · October 30, 2023, 8:13am

The name of the .c file will be in the akmodsbuild log.

From your pastbin I see this:

2023/10/29 18:57:13 akmodsbuild: In file included from /tmp/akmodsbuild.M8wOQ9Bg/BUILD/nvidia-kmod-530.41.03/_kmod_build_6.5.8-200.fc38.x86_64/common/inc/nv-pgprot.h:30,
2023/10/29 18:57:13 akmodsbuild:                  from /tmp/akmodsbuild.M8wOQ9Bg/BUILD/nvidia-kmod-530.41.03/_kmod_build_6.5.8-200.fc38.x86_64/common/inc/nv-linux.h:33,
2023/10/29 18:57:13 akmodsbuild:                  from /tmp/akmodsbuild.M8wOQ9Bg/BUILD/nvidia-kmod-530.41.03/_kmod_build_6.5.8-200.fc38.x86_64/nvidia/nv-nano-timer.c:31:
2023/10/29 18:57:13 akmodsbuild: ./include/linux/mm.h:2430:35: note: expected 'long unsigned int' but argument is of type 'struct task_struct *'
2023/10/29 18:57:13 akmodsbuild:  2430 | long get_user_pages(unsigned long start, unsigned long nr_pages,
2023/10/29 18:57:13 akmodsbuild:       |                     ~~~~~~~~~~~~~~^~~~~

So it should be in nv-nano-timer.c:31

computersavvy · October 30, 2023, 5:14pm

J A:

akmod-nvidia.x86_64                                       3:530.41.03-1.fc38                     @rpmfusion-nonfree                               
libva-nvidia-driver.x86_64                                0.0.10-3.fc38                          @updates                                         
nvidia-gpu-firmware.noarch                                20230919-1.fc38                        @updates                                         
nvidia-persistenced.x86_64                                3:530.41.03-1.fc38                     @rpmfusion-nonfree

This shows the system did not build and install the kmod-nvidia package for the currently installed nvidia packages (530.41.03).

Try that as sudo akmods --force and if it still fails then try
sudo akmods --kernels $(uname -r) --force

The same suggestion was at the bottom of what you posted for the failed akmods run.

jmaargh · October 30, 2023, 5:54pm

I should have said, I tried akmods --force and it failed with exactly the same compilation error. Wouldn’t the default kernel for akmods be the current one anyway?

jmaargh · October 30, 2023, 5:55pm

I meant where can I find the repo that contains the source code, not which file within the code is failing.

computersavvy · October 30, 2023, 6:21pm

On my system I find

# find /usr -name nvidia-kmod*
/usr/share/nvidia-kmod-535.113.01
/usr/share/nvidia-kmod-535.113.01/nvidia-kmod-535.113.01-x86_64.tar.xz
/usr/src/akmods/nvidia-kmod.latest
/usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm

and 

ls -l /usr/src/akmods/
total 240
-rw-r--r--. 1 root root 90454 Sep 21 19:00  nvidia-kmod-535.113.01-1.fc39.src.rpm
lrwxrwxrwx. 1 root root    37 Sep 21 19:00  nvidia-kmod.latest -> nvidia-kmod-535.113.01-1.fc39.src.rpm

Those files come from installing akmod-nvidia from rpmfusion.

# dnf provides /usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm
Last metadata expiration check: 0:08:46 ago on Mon 30 Oct 2023 01:16:52 PM CDT.
akmod-nvidia-3:535.113.01-1.fc39.x86_64 : Akmod package for nvidia kernel module(s)
Repo        : @System
Matched from:
Filename    : /usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm

akmod-nvidia-3:535.113.01-1.fc39.x86_64 : Akmod package for nvidia kernel module(s)
Repo        : rpmfusion-nonfree
Matched from:
Filename    : /usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm

akmod-nvidia-3:535.113.01-1.fc39.x86_64 : Akmod package for nvidia kernel module(s)
Repo        : rpmfusion-nonfree-nvidia-driver
Matched from:
Filename    : /usr/src/akmods/nvidia-kmod-535.113.01-1.fc39.src.rpm

tolgaerok · December 8, 2023, 11:34pm

Has this been resolved as i like to do the same. I want to downgrade from 545x to 535.86.05

computersavvy · December 9, 2023, 12:15pm

I was able to downgrade from 545 back to the last 535.129.03 with 2 simple commands. My drivers were installed from rpmfusion-nonfree and I do not know if the same would work properly with drivers installed from rpmfusion-nonfree-nvidia-drivers.
sudo dnf remove \*nvidia\* --exclude nvidia-gpu-firmware
followed by
sudo dnf install akmod-nvidia-535.129.03\* xorg-x11-drv-nvidia-cuda-535.129.03\* nvidia\*535.129.03\*

vwingqvist · December 22, 2023, 2:27pm

This helped me!
545 was giving me a lot of screen tearing, so downgrading to 535 did the trick

EDIT: I also have installed from rpmfusion-nonfree:

dnf info akmod-nvidia | grep “From repo”
From repo : rpmfusion-nonfree

tolgaerok · December 22, 2023, 3:38pm

ok this worked for me also, now how do i freeze the 535 from being upgraded to 545?

hamrheadcorvette · December 22, 2023, 4:07pm

Asking @computersavvy specifically
Won’t dnf downgrade nvidia-* --exclude nvidia-gpu-firmware be enough for anyone looking to move down a driver version? It’s been a while for me, but I believe that’s how i used to do it. . .

computersavvy · December 22, 2023, 4:10pm

If you are using dnf then adding the --exclude \*nvidia\* option to the dnf command line will do that.

I initially had the 545 driver which gave me a failed cuda install so I downgraded back to 535. After a couple weeks I tried again with the upgrade and then cuda worked with the 545 drivers so I have been content since.

hamrheadcorvette · December 22, 2023, 4:14pm

Hmm, interesting. so dnf downgrade your-package should work like it.

computersavvy · December 22, 2023, 4:17pm

Almost.

I did not try dnf downgrade in my case since I knew exactly what I wanted to reinstall and chose not to allow it to be managed automatically by dnf (and I seldom use a downgrade so had even forgotten the command was available)

I just tested and found that the command
dnf downgrade \*nvidia\* --exclude nvidia-gpu-firmware --allowerasing would do the same.

Thanks for the reminder.

tolgaerok · December 22, 2023, 4:18pm

Yeh the exclude kinda works but discovery picks up all the 545 upgrades. I added

exclude=nvidia-* into sudo nano /etc/dnf/dnf.conf and it didnt stop or filter out the 545.
So i installed sudo dnf install dnf-plugin-versionlock and added

sudo dnf versionlock add akmod-nvidia-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-modprobe-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-persistenced-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-settings-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-xconfig-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-cuda-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-cuda-libs-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-kmodsrc-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-libs-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-power-3:535.129.03-1.fc39

Nothing’s working

computersavvy · December 22, 2023, 4:30pm

Yes and no.

For packages provided by fedora the downgrade will only have access to the latest updated packages in updates or to the packages in the fedora repo that were available with first release of that version. Fedora updates only keeps the most recently released packages. Thus you could not revert back to the last installed version from the fedora-updates repo if there were any versions between the oldest and newest.

For other repos such as rpmfusion it seems you may be able to go back to a specific version, and in this case from the nvidia 545 drivers to the 535 drivers since they are both still available in the rpmfusion repos. (535 in rpmfusion-nonfree and 545 in rpmfusion-nonfree-updates).

hamrheadcorvette · December 23, 2023, 2:59pm

Do you have any updates? It’s reaaly interesting to see and apparently many people are having a probnlem with the new Nvidia driver.

tolgaerok · December 28, 2023, 5:11pm

dnf install python3-dnf-plugin-versionlock

rpm -qa xorg-x11-drv-nvidia* *kmod-nvidia* nvidia-{settings,xconfig,modprobe,persistenced} >> /etc/dnf/plugins/versionlock.list

DOWNGRADE NVIDIA FROM 545x to 535x

sudo dnf remove \*nvidia\* --exclude nvidia-gpu-firmware

Uninstall the NVIDIA driver

sudo dnf remove xorg-x11-drv-nvidia\*

sudo dnf install akmod-nvidia-535.129.03\* xorg-x11-drv-nvidia-cuda-535.129.03\* nvidia\*535.129.03\*

sudo dnf install dnf-plugin-versionlock
sudo dnf versionlock add akmod-nvidia-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-modprobe-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-persistenced-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-settings-3:535.129.03-1.fc39
sudo dnf versionlock add nvidia-xconfig-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-cuda-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-cuda-libs-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-kmodsrc-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-libs-3:535.129.03-1.fc39
sudo dnf versionlock add xorg-x11-drv-nvidia-power-3:535.129.03-1.fc39
sudo rm /etc/yum.repos.d/nvidia-exclude.repo
sudo dnf versionlock list
sudo dnf update
sudo dnf update
sudo dnf versionlock list

DELETE LOCKS

        # sudo dnf versionlock delete akmod-nvidia-3:535.129.03-1.fc39
        #sudo dnf versionlock delete nvidia-modprobe-3:535.129.03-1.fc39
        #sudo dnf versionlock delete nvidia-persistenced-3:535.129.03-1.fc39
        #sudo dnf versionlock delete nvidia-settings-3:535.129.03-1.fc39
        #sudo dnf versionlock delete nvidia-xconfig-3:535.129.03-1.fc39

545x still shows up in the updates, not sure how to stop dnf to filter out 545x. thats the only part im currently at a lose on. Hvant tested out since last post as im on nixos kde atm with nvidia 535x

Your more then welcom to try out my installer:

bash -c "$(curl -fsSL https://raw.githubusercontent.com/tolgaerok/tolga-scripts/main/Fedora39/setup.sh)"

Topic		Replies	Views
What is the proper way to downgrade the nvidia drivers? Ask Fedora f37 , nvidia	5	3851	January 14, 2024
Nvidia drivers - falling back to nouveau after upgrading to Fedora 38 Ask Fedora f38 , nvidia	13	8370	June 6, 2023
Official Nvidia drivers made things worse - how do I get back to nouveau? Ask Fedora nvidia	18	3201	June 16, 2024
Nvidia kernel missing fallback to Nouveau Fedora 35 Ask Fedora f35 , amd , nvidia	2	442	March 27, 2022
At f35: installed dnf-plugin-system-upgrade;dnf system-upgrade download --releasever=38 Ask Fedora f35 , f37 , f38 , nvidia	11	1027	July 16, 2023

Downgrading nvidia driver from 535

DOWNGRADE NVIDIA FROM 545x to 535x

Uninstall the NVIDIA driver

DELETE LOCKS

Related topics