F40 Change Proposal: DNFConditionalFilelists (System-Wide)

[decathorpe (Fabio Valentini)]

I agree. This also matches up with how I interpret SHOULD NOT and MUST NOT definitions from RFC 2119: RFC 2119 - Key words for use in RFCs to Indicate Requirement Levels

I see only two possibilities for resolving this:

1. Packaging Guidelines are updated to use MUST NOT instead of SHOULD NOT, dnf behaviour can be changed.
2. dnf behaviour is changed to dynamically load full file lists when necessary, Packaging Guidelines stay as they are.

[zbyszek (Zbigniew Jędrzejewski-Szmek)]

this is rough list of packages which IMO stand out from current guidelines

The guidelines explicitly say that Packages can depend on virtual file Provides. Package specify virtual Provides with a file name. This Provides becomes part of the main package metadata and other packages can depend on it without the extra filelist metadata:

$ for i in /bin/ps /bin/sh /bin/bash /sbin/service /sbin/nologin /sbin/modprobe /sbin/nologin /bin/mail /bin/mount /bin/grep /bin/mailx /sbin/fsck /sbin/mount.nfs /sbin/mount.nfs4 /sbin/ip /sbin/rpc.statd /sbin/findfs /sbin/quotaon /sbin/quotacheck /usr/lib/libc.so /usr/share/fonts/google-droid-sans-fonts/DroidSansFallbackFull.ttf /usr/share/fonts/google-droid-sans-fonts/DroidSans.ttf; do echo "$i: $(rpm -qPv $(rpm -q --whatprovides $i)|grep $i)"; done 
/bin/ps: manual: /bin/ps
/bin/sh: manual: /bin/sh
/bin/bash: manual: /bin/bash
/sbin/service: manual: /sbin/service
/sbin/nologin: manual: /sbin/nologin
/sbin/modprobe: manual: /sbin/modprobe
/sbin/nologin: manual: /sbin/nologin
/bin/mail: manual: /bin/mail
/bin/mount: manual: /bin/mount
/bin/grep: manual: /bin/grep
/bin/mailx: manual: /bin/mailx
/sbin/fsck: manual: /sbin/fsck
/sbin/mount.nfs: 
/sbin/mount.nfs4: 
/sbin/ip: manual: /sbin/ip
/sbin/rpc.statd: 
/sbin/findfs: manual: /sbin/findfs
/sbin/quotaon: 
/sbin/quotacheck: 
/usr/lib/libc.so: 
/usr/share/fonts/google-droid-sans-fonts/DroidSansFallbackFull.ttf: 
/usr/share/fonts/google-droid-sans-fonts/DroidSans.ttf: 

@vondruch

This is the list:

$ grep -R '^Requires: .*/' | grep -v \( | grep -v bindir | grep -v '/usr/bin' | grep -v '/usr/sbin' | grep -v '/etc/' | grep -v '%{_sysconfdir}'
autofs.spec:Requires: bash coreutils sed gawk grep module-init-tools /bin/ps
beakerlib.spec:Requires:   /bin/bash
beakerlib.spec:Requires:   /bin/sh
...

So actually almost all the packages in that list are correct according to Guidelines, and also correct wrt. to the proposed change.

The exceptions are:

resource-agents.spec:Requires: /sbin/mount.nfs /sbin/mount.nfs4
resource-agents.spec:Requires: /sbin/rpc.statd
resource-agents.spec:Requires: /sbin/quotaon /sbin/quotacheck
systemtap.spec:Requires: /usr/lib/libc.so
rt.spec:Requires:  /usr/share/fonts/google-droid-sans-fonts/*

Actually, for the /sbin/ paths, it’d be enough to switch to /usr/sbin/ to solve the issue.

[mattdm (Matthew Miller)]

@zbyszek

Actually, for the /sbin/ paths, it’d be enough to switch to /usr/sbin/ to solve the issue

So, that just leaves systemtap and rt.

For systemtap, I think that’s a false positive, because it’s just for EOL releases:

%if 0%{?rhel} >= 8 || 0%{?fedora} >= 20
# fweimer, personal correspondence
Recommends: glibc-devel(x86-32)
%else
Requires: /usr/lib/libc.so
%endif

I’m not sure why rt requires the font path specifically rather than google-droid-sans-fonts – I don’t think there’s another provider for that. (But if there is, seems like a virtual provides could handle the issue.)

[ngompa (Neal Gompa)]

@jkolarik

As for the filelists lazy loading solution, I don’t think it could be currently easily implemented in DNF itself. Predicting whether the presence of filelists would help resolve failure cases is challenging. Additional hints from the solver would be needed, but I believe this approach would be more complicated and delicate than the proposed solution.

libsolv already supports this capability. Insofar as predicting whether the presence of filelists would help resolve failures is pretty easy: if a failed dependency starts with a / (that is, it’s a file path), attempt to download filelists, reload the solver cache with the extra data, and then try again. If it fails that time, then it’s obviously invalid.

[zbyszek (Zbigniew Jędrzejewski-Szmek)]

For resource-agents, https://src.fedoraproject.org/rpms/resource-agents/pull-request/7.

[zbyszek (Zbigniew Jędrzejewski-Szmek)]

For rt, https://bugzilla.redhat.com/show_bug.cgi?id=1731700, and a pull request from 9 months ago…

[zbyszek (Zbigniew Jędrzejewski-Szmek)]

OK, so I think we’re good: everything is either covered by virtual provides or a false positive, and for the outstanding two packages with an actual issue, we have trivial PRs to be merged.

[amoloney (Aoife Moloney)]

This change proposal has now been submitted to FESCo with ticket #3097 for voting.

To find out more, please visit our Changes Policy documentation.

[jkolarik (Jan Kolarik)]

Hello Neal, I want to assure you that I am not overlooking your comment. In fact, I have already begun the analysis of potential implementation in DNF. However, it’s turning out to be trickier than it might seem from the DNF perspective.

[tstellar (Tom Stellard)]

However, the absence of filelists might create an issue with packages that are not correctly packaged

Based on this discussion, it sounds like this change will cause issues even with correctly packaged packages. If this is true, can this line be updated to reflect that.

[jkolarik (Jan Kolarik)]

Hi Tom, thanks for your feedback. I will reword it to better reflect the SHOULD NOT and MUST NOT definitions. To be honest, I was not aware of these formal definitions before and had perceived it just from reading the text as basically forbidden things. Unfortunately, this now cast a negative light on the situation. Although, I believe the proposal remains viable, as this is a Fedora related proposal and as Zbigniew stated, on the Fedora side there is currently just a couple of packages not complying with the discussed rule and they’re not system-critical. Also, on other distributions, the state could remain as it was till now by simply configuring the behavior to continue downloading filelists.

Speaking about the dynamically loading filelists, I’d love to have such a feature in dnf. But as of now, it appears to require a substantial amount of non-trivial work, involving the refactoring of numerous existing blocks in the dnf/libdnf codebase. I don’t think handling this for F40 is realistic. Moreover, given that it impacts essential parts of the code, I view it as a considerable risk. I think it’s better to plan such a feature for DNF5, where workflows are better streamlined.

[jkolarik (Jan Kolarik)]

I’ve revised the Upgrade/compatibility impact section on the wiki page. I hope the situation is now clarified there.

[oturpe (Otto Liljalaakso)]

I think the best way forward here would be to add the following two items to proposal’s scope:

• The packages that currently need the filelist are updated to stop that (by merging the existing pull requests)
• The Packaging guidelines are updated to say MUST NOT

The analysis that has been done in this thread shows that filelists are basically unused, and the virtual file provide workaround is there for the rare cases where filelists have some use.

One thing I am thinking about are external repositories. What should be done if some of them use file dependencies, either internally or by referring to files provided by Fedora packages? Just let the user figure pass the flag to enable them in such case?

[ngompa (Neal Gompa)]

@jkolarik

Speaking about the dynamically loading filelists, I’d love to have such a feature in dnf. But as of now, it appears to require a substantial amount of non-trivial work, involving the refactoring of numerous existing blocks in the dnf/libdnf codebase. I don’t think handling this for F40 is realistic. Moreover, given that it impacts essential parts of the code, I view it as a considerable risk. I think it’s better to plan such a feature for DNF5, where workflows are better streamlined.

Keeping in mind that the idea is to propose replacing DNF4 with DNF5 in Fedora Linux 41, should we then not even bother with this change and just have dynamically loading filelists be part of the switch to DNF v5?

[jmracek (Jaroslav Mracek)]

@decathorpe

• nes are updated to use MUST NOT instead of SHOULD NOT, dnf behaviour can be changed.
• dnf behaviour is changed to dynamically load full file lists when necessary, Packaging Guidelines stay as they are.

Yes, this is true but only 9 packages from 7 sources are effected by the change. The original list for Fedora 38 was about 70 packages, therefore there is a huge progress.

The new behavior will be configurable and when a user will experience an issue then we can provide the reasonable suggestion how to resolve it.

Personally, I am supporting to use MUST NOT instead of SHOULD NOT in guidelines in solution.

Dynamic downloading and loading filelists sounds like a good idea, but it will often end up with worst performance or not nice behavior then the current implementation (always download and load filelists), because often the filelist will be gone from all mirrors. Then the solution is - fail or re-download all repository and reset everything in DNF (start from scratch).

[jmracek (Jaroslav Mracek)]

I analyzed Fedora 40 and the situation is even better. Only 7 packages from 5 sources are affected by the change.

mpibash-mpich-examples requires: /usr/lib64/mpich/bin/mpibash_mpich
mpibash-openmpi-examples requires: /usr/lib64/openmpi/bin/mpibash_openmpi

resource-agents requires: /sbin/mount.nfs /sbin/mount.nfs4 /sbin/rpc.statd

rt requires: /usr/share/fonts/google-droid-sans-fonts/DroidSans.ttf /usr/share/fonts/google-droid-sans-fonts/DroidSansFallbackFull.ttf

scalasca-mpich requires: /usr/lib64/mpich/bin/scorep-config
scalasca-openmpi requires: /usr/lib64/openmpi/bin/scorep-config

sqlninja requires: /usr/share/sqlninja/backscan.pl /usr/share/sqlninja/bruteforce.pl /usr/share/sqlninja/dirshell.pl /usr/share/sqlninja/dns.pl /usr/share/sqlninja/escalation.pl /usr/share/sqlninja/fingerprint.pl /usr/share/sqlninja/getdata.pl /usr/share/sqlninja/icmp.pl /usr/share/sqlninja/metasploit.pl /usr/share/sqlninja/resurrectxp.pl /usr/share/sqlninja/revshell.pl /usr/share/sqlninja/session.pl /usr/share/sqlninja/sqlcmd.pl /usr/share/sqlninja/test.pl /usr/share/sqlninja/upload.pl /usr/share/sqlninja/utils.pl

Details: https://bugzilla.redhat.com/show_bug.cgi?id=2180842#c9

[jmracek (Jaroslav Mracek)]

@oturpe

One thing I am thinking about are external repositories. What should be done if some of them use file dependencies, either internally or by referring to files provided by Fedora packages? Just let the user figure pass the flag to enable them in such case?

We have a plan to provide a message with a suggestion that downloading filelist might resolve the issue including a description how to do it from command line.

[jmracek (Jaroslav Mracek)]

The feature was requested time ago and it significantly improves user experience and reduce cost of using Fedora (download, CPU, RAM, storage). The feature was not deliverable in past, because to many packages depended on FILELISTS, but this is not valid anymore because only few packages are affected by the change (2023-11-22 only 7 packages, Fedora 40). The simple workaround for remaining packages is to add Provides: <file> to original file provider, but there are better ways.

[jkolarik (Jan Kolarik)]

The lazy loading functionality may still pose challenges in DNF5, and I cannot confirm right now its presence in Fedora 41 due to the ongoing workload.

On the other hand, the implementation for the current proposal is essentially ready, making the switch quick and straightforward. Also, even if DNF5 is switched as the default in Fedora 41, this would be still beneficial for earlier Fedora versions and other distributions.

[jmracek (Jaroslav Mracek)]

@ngompa

@mattdm

Could we make it into a “MUST NOT” rule? I’m pretty unconvinced by the arguments to the contrary, given the benefit of avoiding them.

I vetoed making it MUST NOT originally and I will do so again. Lazy loading file lists needs to be implemented in DNF. It was a regression from YUM that we could not do it.

Even a dead-simple way of handling this would be “if a package’s unsatisfied dependency starts with a ‘/’, turn file lists back on and retry”.

This would be a pretty nice solution if dnf always downloads file list and then they can be loaded lazily. But it means that dnf have to downloads all filelists (no change in amount od downloaded metadata).
In case that dnf will not have downloaded filelists then there is a chance that file described in repomd is not available anywhere, because metadata were updated. This might happened with Fedora updates repository. The repository is set to refresh metadata not before 6h metadata_expire=6h. When this happens DNF has only bad options - mark all repositories with not downloadable filelists as expired and fail or start loading and updating all repositories from scratch.
@ngompa May I ask you which scenario have you in mind when you mentioned lazy loading of FILELISTS?