`rpm --rebuilddb` fails in Fedora 37

,

I recently upgraded to Fedora 37 (from Fedora 35) and am following the post-upgrade steps from here.

I am attempting to rebuild the repo database using the command strace rpm --rebuilddb.

Just after rebooting I use sudo su and then enter the command strace rpm --rebuilddb. Upon which I get this error:
error: can't create transaction lock on /usr/lib/sysimage/rpm/.rpm.lock (Permission denied)

Here is a stack trace of the command as captured by strace rpm --rebuilddb 2>&1 | tail -50, again run as root.

openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "error: ", 7error: )                  = 7
write(2, "can't create transaction lock on"..., 85can't create transaction lock on /usr/lib/sysimage/rpm/.rpm.lock (Permission denied)
) = 85
ioctl(0, TCGETS, {c_iflag=ICRNL|IXON|IUTF8, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
exit_group(-1)                          = ?
+++ exited with 255 +++
[root@nathanowenlaptop nathan]# strace rpm --rebuilddb 2>&1 | tail -50
read(3, "and(\"%{?_lto_cflags}\"):gsub(\"[%-"..., 4096) = 4096
read(3, "g_hack\" = 1 ] && for i in $(find"..., 4096) = 4096
read(3, "_missing_build_ids_terminate_bui"..., 4096) = 4096
read(3, "__filter_prov_cmd} %{__grep} -v "..., 4096) = 964
read(3, "", 4096)                       = 0
close(3)                                = 0
openat(AT_FDCWD, "/etc/rpm", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
newfstatat(3, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0
brk(0x55cc04142000)                     = 0x55cc04142000
getdents64(3, 0x55cc04119710 /* 2 entries */, 32768) = 48
getdents64(3, 0x55cc04119710 /* 0 entries */, 32768) = 0
brk(0x55cc0413a000)                     = 0x55cc0413a000
close(3)                                = 0
openat(AT_FDCWD, "/etc/rpm/macros", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/rpm/x86_64-linux/macros", O_RDONLY) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/root/.rpmmacros", 0x7ffd873179e0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/rpm/init.lua", 0x7ffd87317d70, 0) = -1 ENOENT (No such file or directory)
getgid()                                = 0
getuid()                                = 0
newfstatat(AT_FDCWD, "/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/lib", {st_mode=S_IFDIR|0555, st_size=36864, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/lib/sysimage", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/lib/sysimage/rpm", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=2998, ...}, AT_EMPTY_PATH) = 0
read(3, "# Locale name alias data base.\n#"..., 4096) = 2998
read(3, "", 4096)                       = 0
close(3)                                = 0
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/rpm.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/rpm.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/rpm.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/rpm.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/rpm.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/rpm.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
umask(022)                              = 022
openat(AT_FDCWD, "/usr/lib/sysimage/rpm/.rpm.lock", O_RDWR|O_CREAT, 0644) = -1 EACCES (Permission denied)
umask(022)                              = 022
openat(AT_FDCWD, "/usr/lib/sysimage/rpm/.rpm.lock", O_RDONLY) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "error: ", 7error: )                  = 7
write(2, "can't create transaction lock on"..., 85can't create transaction lock on /usr/lib/sysimage/rpm/.rpm.lock (Permission denied)
) = 85
ioctl(0, TCGETS, {c_iflag=ICRNL|IXON|IUTF8, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
exit_group(-1)                          = ?
+++ exited with 255 +++

A full stack trace can be found here:

What led you to decide you needed to rebuild that database?

The referenced guide says in a NOTE:
Only follow these steps if you encounter problems with your upgraded system.
and that command is shown in the section following the note.

99.99% of users never have a need to even consider going that far, and it is only after almost total failure that one would reach that part of the instructions.

We may be able to assist, but more information is needed to understand the conditions that led to doing so. It may not be (and probably is not) necessary.

Rebuilding the rpm database is essentially a last resort when all other recovery methods have been tried and have failed.

In fact for 99% of all users none of the post upgrade steps are required. Only if a problem is seen should those be done, and only as needed to recover from a specific identified situation.

I have been through a fair number of upgrades on this system and have had some problems with getting my RPMFusion NVIDIA drivers to automatically update (quite possibly an RPMFusion issue rather than an issue with my database).

My other Fedora computers have the same packages and have not run into this persistent package upgrade issue and was hoping (I did not have very high hopes) this might help

Was mostly just trying to get everything to a fairly clean slate sort of doing a clean install.

So yeah, probobly not strictly necessary. That said, I would not expect the database rebuild to fail.
Then again the steps for updating GRUB also fail so it is possible this guide is simply not up to date.

If the steps for updating grub fail (there is only one, using grub2-mkconfig) then something more serious is at work.

I have been using rpmfusion for a number of years and never have an update issue with anything I have installed from there.

Yes it might be necessary to rebuild that db, but I have only done so twice in about 10 years with continuous upgrades over the same period.

If you could give us some detailed info as to what sort of issues you are seeing we may be able to work through them and find a solution.

  1. Are you having problems doing upgrades using dnf upgrade? This may include package conflicts or other problems.
  2. What happens if you do dnf distro-sync --refresh?
  3. Do you routinely use dnf for system updates or package installation or do you use gnome software (packagekit) to manage updates?
  4. What is the output of dnf list extras? This would show packages that are not from any enabled repos and that possibly should be removed.

In so far as the NVIDIA upgrade (or in the case below install as it is difficult to simulate an upgrade). This is what I am seeing:

[nathan@nathanowenlaptop ~]$ sudo dnf install akmod-nvidia
Last metadata expiration check: 1:24:22 ago on Mon 23 Jan 2023 03:51:35 PM CST.
Error: 
 Problem: conflicting requests
  - package akmod-nvidia-3:520.56.06-1.fc37.x86_64 requires nvidia-kmod-common >= 3:520.56.06, but none of the providers can be installed
  - package akmod-nvidia-3:525.78.01-1.fc37.x86_64 requires nvidia-kmod-common >= 3:525.78.01, but none of the providers can be installed
  - package xorg-x11-drv-nvidia-3:520.56.06-1.fc37.x86_64 is filtered out by modular filtering
  - package xorg-x11-drv-nvidia-3:525.78.01-1.fc37.x86_64 is filtered out by modular filtering
(try to add '--skip-broken' to skip uninstallable packages)

This is my time-tested workaround for this issue:

#!/bin/bash

if [[ -z $SUDO_USER ]]; then
    echo "must be run with sudo"
    exit
fi

FEDORA_VERSION=$(cat /etc/fedora-release | grep -o [0-9]*)
oldver="$(dnf list installed akmod-nvidia.x86_64 | grep akmod-nvidia.x86_64 | awk '{print $2}' | sed 's/.*://').x86_64"
ver=$(curl -f -L https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/a/ | grep -o akmod-nvidia-.*rpm | grep "fc${FEDORA_VERSION}" | sort | tail -1 | sed -En "s/akmod-nvidia-(.*).rpm/\1/p")

if [ -z "$ver" ] ; then
    echo "Failed to get version"
    exit
fi

if [ "$oldver" == "$ver" ] ; then
    echo "Already latest"
    exit
fi

echo "Upgrading from $oldver to $ver"

dnf remove -y xorg-x11-drv-nvidia\*
dnf upgrade -y

sudo dnf install -y \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/a/akmod-nvidia-$ver.rpm \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/k/kmod-nvidia-$ver.rpm \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/n/nvidia-settings-$ver.rpm \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/x/xorg-x11-drv-nvidia-$ver.rpm \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/x/xorg-x11-drv-nvidia-libs-$ver.rpm \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/x/xorg-x11-drv-nvidia-power-$ver.rpm \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/x/xorg-x11-drv-nvidia-kmodsrc-$ver.rpm

sudo dnf install -y \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/x/xorg-x11-drv-nvidia-cuda-$ver.rpm \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/x/xorg-x11-drv-nvidia-cuda-libs-$ver.rpm \
https://download1.rpmfusion.org/nonfree/fedora/updates/${FEDORA_VERSION}/x86_64/n/nvidia-persistenced-$ver.rpm
``

SELinux strikes again??? Try:

restorecon -v -r /usr/lib/sysimage/rpm /var/lib/rpm

That did not work, which does not surprise me given that I already did sudo fixfiles -B onboot and restarted the system. Good guess though…

sudo su isn’t the best way to elevate your privileges. You need a lone - to su to get the full environment. Besides that, you are calling all the modules in the PAM stack twice (/etc/pam.d/sudo and then /etc/pam.d/su). Instead, you should just use sudo -i to get a clean environment with all the proper vars initialized. Also, it is possible that strace is changing the SELinux context (I don’t think it would, but I don’t know for sure what all it does). Just to be sure, you might need to run setenforce 0 to put SELinux is permissive mode before using that particular incantation. (And of course don’t forget to use setenforce 1 to re-enable SELinux when you are done.)

This has 2 problems

  1. you cannot install akmod-nvidia when akmod-nvidia is already installed. You must instead either upgrade it; or remove it then reinstall. In general this applies to any package already installed.
  2. Modular filtering

My fix would be to remove all the nvidia packages then reinstall them so they are clean. On my systems I always permanently disable the 2 modular repos, fedora-modular & updates-modular, as soon as I do an install since I have seen similar issues and it can be a pain to fix those issues.

  1. sudo dnf remove '*nvidia*' --exclude=nvidia-gpu-firmware should remove all the nvidia driver packages and leave the firmware for the system.
  2. sudo dnf install akmod-nvidia xorg-x11-drv-nvidia-cuda should install all the required packaages and build the modules for the newest kernel installed.
  3. Wait at least 5 minutes after the completion of step 2 then reboot and the drivers should properly load.

The current nvidia driver version from rpmfusion is 525.78.01. During the install you should be able to see that.

If I do that I see the following on step 2:

$ sudo dnf install akmod-nvidia xorg-x11-drv-nvidia-cuda
Last metadata expiration check: 0:19:55 ago on Tue 24 Jan 2023 05:22:44 PM CST.
All matches were filtered out by modular filtering for argument: xorg-x11-drv-nvidia-cuda
Error: Unable to find a match: xorg-x11-drv-nvidia-cuda

setenforce 0, though a bit drastic, does seem to have solved the issue with sudo rpm --rebuilddb

It is a bit drastic. But hopefully it will have corrected whatever the problem was and you won’t need to do it again.

I did find someone else that reported a similar problem in the bugzilla archives, but it was for a much older version of Fedora Linux.

https://bugzilla.redhat.com/show_bug.cgi?id=1836108#c38

Try the install by adding --best --allowerasing to that install command. If it still gives the same error then remove the cuda package from the command and try with just the akmod-nvidia package.

If this still gives the modular filtering error we will need to address that next.

Would you please post the output of dnf repolist

Can you run

ls -Zl /usr/lib/sysimage/rpm/ /var/lib/rpm

and check the results

[root@newbox ~]# ls -Zl /usr/lib/sysimage/rpm/ /var/lib/rpm
lrwxrwxrwx. 1 root root system_u:object_r:rpm_var_lib_t:s0       26 May  4  2022 /var/lib/rpm -> ../../usr/lib/sysimage/rpm

/usr/lib/sysimage/rpm/:
total 110536
-rw-r--r--. 1 root root unconfined_u:object_r:rpm_var_lib_t:s0 113086464 Jan 25 09:07 rpmdb.sqlite
-rw-r--r--. 1 root root unconfined_u:object_r:rpm_var_lib_t:s0     98304 Jan 25 09:07 rpmdb.sqlite-shm
-rw-r--r--. 1 root root unconfined_u:object_r:rpm_var_lib_t:s0         0 Jan 25 09:07 rpmdb.sqlite-wal