I recently upgraded my Fedora 40 laptop (KDE spin, but I doubt this matters) to Fedora 41 using dnf system-upgrade. After the upgrade, I noticed I could no longer search for packages with my regular user:
~ ❯ dnf search fzf
Loading system state TOML file /usr/lib/sysimage/libdnf5/packages.toml failed (see dnf5-system-state(7)): toml::parse: error opening file "/usr/lib/sysimage/libdnf5/packages.toml"
Sure enough, the file is owned by root:root and has 640 permissions. Running the search as root worked just fine, of course:
~ ❯ ls -l /usr/lib/sysimage/libdnf5/packages.toml
-rw-r-----. 1 root root 100486 10. Nov 21:48 /usr/lib/sysimage/libdnf5/packages.toml
~ ❯ sudo dnf search fzf
Updating and loading repositories:
Repositories loaded.
Matched fields: name (exact)
fzf.x86_64: A command-line fuzzy finder written in Go
Matched fields: name, summary
golang-github-msprev-fzf-bibtex.x86_64: A BibTeX source for fzf"
Since I’d prefer not to search for packages with root permissions, I installed a fresh system in a VM to compare. In that VM, I could run dnf search as a regular user and permissions for files under /usr/lib/sysimage/libdnf5/ had o+r (and o+rx for directories). On my machine, everything under /usr/lib/sysimage/libdnf5/ had o-rwx permissions. I replicated the permissions from the VM on my laptop and now dnf search runs without root privileges.
Before I raise a bug for this, I was wondering a) if this is a documented issue and I was just unable to find the right search terms for it and b) if somebody else has experienced the same thing after dnf system-upgrade to Fedora 41 or if it was caused by something on my machine.
Today, I was able to take a look at my desktop computer, which has also been upgraded from F40 to F41, and it has 644 and 755 permissions for /usr/lib/sysimage/libdnf5/*, just like the test installation in the VM. I guess what I saw on my laptop really is caused by something specific on this machine.
Just to be thorough, I checked everything with rpm -Va. While it listed some more files, nothing really looked out of the ordinary and the system runs fine (apart from not being able to dnf search as a regular user originally) so I am going to let this go.
The same happened to me today that I upgraded from 40 to 41 so it’s definitely not an isolated issue. I realized because dnf history was not working anymore as a regular used but only when executed as root.
Moreover, I had lost all dnf history data from before the upgrade. But I’m not sure if this could just happen when upgrading…
$ ls /usr/lib/sysimage/libdnf5/ -al
total 732
drwxr-xr-x. 3 root root 4096 Nov 14 18:33 .
drwxr-xr-x. 4 root root 4096 Nov 14 18:32 ..
drwx------. 2 root root 4096 Nov 14 18:32 comps_groups
-rw-------. 1 root root 32 Nov 14 18:32 environments.toml
-rw-------. 1 root root 9665 Nov 14 18:32 groups.toml
-rw-------. 1 root root 188 Nov 14 18:32 modules.toml
-rw-------. 1 root root 220060 Nov 14 18:32 nevras.toml
-rw-------. 1 root root 107249 Nov 14 18:32 packages.toml
-rw-------. 1 root root 110 Nov 14 18:32 system.toml
-rw-------. 1 root root 139264 Nov 14 18:33 transaction_history.sqlite
-rw-------. 1 root root 32768 Nov 14 18:33 transaction_history.sqlite-shm
-rw-------. 1 root root 206032 Nov 14 18:32 transaction_history.sqlite-wal
In order to test what you say I first deleted (moved) /usr/lib/sysimage/libdnf5 and dnf reinstalle(d) dnf5 which completed successfully (although I was expecting it to fail) but with the “wrong” permissions again which was very weird.
The loss of the history is expected. dnf5 and dnf4 use different databases so one does not get seen by the other. After the upgrade dnf5 builds its own database and is not aware of the content of the dnf4 database.
I tried the same thing, with the same result, and also ended up changing the permissions manually.
I dug a little deeper today and found this bug from 2014, which points towards an issue with tighter than default umask. And while it is not for dnf5, it sure was an interesting hint. I actually have tightened the default umask on my system from 022 to 027, which would explain the o-rwx I am seeing:
~ ❯ umask
027
~ ❯ sudo umask
0027
Also, comment 20, points to this open GitHub issue for dnf5, which is about cache files owned by root and not accessible by normal users. Interestingly, these should be in /var/cache/libdnf5 but the error in this comment references /usr/lib/sysimage/libdnf5/packages.toml, which is the same file that started me down this rabbit hole.
If this is indeed a umask issue, it would also explain why this occurs on upgraded systems (with modified umask) and why I did not see it in a fresh installation (with the default 022 umask).
True, but I would expect system-upgrade to initiate under root after the reboot when a completely new process is spawn. Anyway, it does not matter because I see I also have 077 in /root/.bashrc.
Yes, unless sudo is configured to modify the umask.
In fact, I also checked pkexec umask because that is what I mostly use instead of sudo and I was not sure how it handles the umask. But because that is probably not very common, I chose to show the result of sudo umask in my post.
I actually dug my rabbit hole a little deeper but left that part out of the my earlier post. I looked at the spec file for dnf5 and even though I have very little knowledge about RPMs and spec files, these were my observations: line 302 (%dir %{_prefix}/lib/sysimage/libdnf5) declares the directory /usr/lib/sysimage/libdnf5 to be part of the package and the %verify directive in the next line tells rpm -V to not compare the MD5, size, and modification time for anything under that path. Looking at the RPM file itself, there is a directory /usr/lib/sysimage/libdnf5 but no files underneath, which makes sense since this is the state of dnf5 on a particular machine.
Also note that there is only a single %attrib directive in line 394, which overrides the ownership and permission of the directory %{_var}/cache/libdnf5. In particular, all the files under /usr/lib/sysimage/libdnf5 have no permissions specified. And since they are not installed by the package, just declared to be owned by it, I doubt rpm -V can meaningfully test these.
I did not check the actual code of dnf5 but from the earlier observations and the spec file, I would guess that dnf5 creates its state files in /usr/lib/sysimage/libdnf5 when it first tries to access them, with whatever umask is in effect at that time.
Are you sure about that? I would expect that dnf system-upgrade download does exactly what it says on the tin, it downloads the RPM files for the upgrade and stores them in a temporary location. Otherwise, why would the part after dnf system-upgrade reboot take so long?
The permission of the temporarily downloaded RPM files is not the issue, though, the problem is the permissions of the files after they are installed in their regular place, /usr/lib/sysimage/libdnf5 in this case.
It indeed does whate it says: download. To do that it has to create the dnf5 database in /usr/lib/sysimage/libdnf5 as well as the cache files in /var/cache/libdnf5.
This thread was about an ugrade from Fedora 40 to 41. And F40 still came with the Python-based dnf 4 by default. So, for most people, during an update from F40 to F41 dnf 4 would download all the RPMs and nothing would touch /usr/lib/sysimage/libdnf5 until after dnf system-upgrade reboot. In this scenario, the first thing that has any contact with the actual dnf5 RPM contents is the upgrade step after it has been initiated with dnf system-upgrade reboot.
Which is why I asked if you were sure that these files had already been created during the download phase.
Thank you for the technical explanation @vekruse. It provides some clarity.
However, I disagree with this statement:
For me as a regular user who does not read the spec files often, rpm tells me that these files are owned by dnf5 (rpm -q --whatprovides shows dnf5 and rpm -V dnf5 -v lists them) and like I said above I would expect the RPMs to install the files in the expected permissions regardless of the user environment!
It’s not right to have different behavior because of the user-specified umask is hardened, if, eventually, this is the root cause, is it?
I would argue that this goes against the principle of Determinism that efforts such as the Reproducible builds are trying to achieve and I would ultimately characterize it as a bug.