DNF unusable ("file too short") after system crash during upgrade

Ask Fedora
,
1

I tried to upgrade a Fedora 32 Server with the casual dnf upgrade --refresh, the plan was then to upgrade it to Fedora 35. The SSH connection broke during the upgrade and reconnecting wasn’t possible. I hard rebooted the system and got “Failed to load SELinux policy” so I temporally disabled SELinux with selinux=0 in the boot options. I can finally boot. Now any dnf upgrade or dnf install ends with the following error. My guess is the crash (due to the upgrade/SELinux) during the upgrade has corrupted some files. How to fix DNF and complete my upgrade?

Traceback (most recent call last):
  File "/usr/bin/dnf", line 57, in <module>
    from dnf.cli import main
  File "/usr/lib/python3.8/site-packages/dnf/__init__.py", line 30, in <module>
    import dnf.base
  File "/usr/lib/python3.8/site-packages/dnf/base.py", line 31, in <module>
    from dnf.comps import CompsQuery
  File "/usr/lib/python3.8/site-packages/dnf/comps.py", line 27, in <module>
    from dnf.exceptions import CompsError
  File "/usr/lib/python3.8/site-packages/dnf/exceptions.py", line 22, in <module>
    import dnf.util
  File "/usr/lib/python3.8/site-packages/dnf/util.py", line 30, in <module>
    import dnf.callback
  File "/usr/lib/python3.8/site-packages/dnf/callback.py", line 22, in <module>
    import dnf.yum.rpmtrans
  File "/usr/lib/python3.8/site-packages/dnf/yum/rpmtrans.py", line 26, in <module>
    import rpm
  File "/usr/lib64/python3.8/site-packages/rpm/__init__.py", line 38, in <module>
    from rpm._rpm import *
ImportError: /lib64/librpmbuild.so.9: file too short
2

Your best bet is to use dnf clean all to clean up whatever’s wrong with your databases or cache. Also, it’s not recommended to skip more than one version, so you’d be better off going from 32 to 34, make sure it’s working OK and then upgrade to 35 if you don’t want to do a clean install. In either case, make sure that you have a recent backup before you start.

3

Thanks for the version skipping strategy. dnf clean all gives me the same error.
I understood the package rpm-build-libs is broken, it provides /lib64/librpmbuild.so.9 from the error message. So I found the RPM online, downloaded it and executed rpm -i rpm-build-libs-4.15.1.1-1.fc32.1.x86_64.rpm to reinstall it, this gave me

error: Failed to dlopen /usr/lib64/rpm-plugins/selinux.so /usr/lib64/rpm-plugins/selinux.so: file too short
error: error reading from file rpm-build-libs-4.15.1.1-1.fc32.1.x86_64.rpm

So multiple important packages seem corrupted, I think the best thing to do is probably a clean install.

2 Likes
4

The first error was that fedora 32 has been EOL for more than 6 months so the attempt to do ‘dnf upgrade’ was certain to fail since that repo is no longer available. In fact fedora 33 is also past EOL so it would not be possible to use that repo either.

With the errors you are getting it seems likely that a clean install is best. I hope you have your home and data directories where a clean install will not wipe them. If so then a clean install should not be too seriously harmful.

If you plan in the future to always have the currently installed system fully updated before it goes EOL then the upgrade to the newer version can be done without attempting the dnf update which broke your system.

The official recommendation and supported path is to upgrade one release at a time and never skip a version. Skipping one release (while not supported) usually does work though.

5

Please read this https://discussion.fedoraproject.org/t/dnf-update-fails-importerror-lib64-libdw-so-1-file-too-short/70480

Look like you have similiar problem.

6

Not solving your problem now, but advice for the future: Use something like screen or tmux when updating over ssh. That way the update will continue even when the ssh connection breaks. And you can reconnect to the session once you have reestablished your ssh connection to see if the update has finished.

3 Likes
7

Since rpm -i worked, rpm is still functional. Does sudo rpm -Va work? That will go through your packages and verify them, and should tell you which ones are not as they should be. The rpm manual tells you what you can tweak in the verify command, and what the output means (look for “Verify Options” here or on man rpm)

3 Likes
8

Unrelated to the broken files, but still want to clarify…

Despite this release being End-of-Life, this is not true. You can still fetch repo data from old Fedora releases and update to the latest of each package installed, for F32 see for example:

1 Like
9

Thank you for all your answers, I’m going with the clean install and will keep this machine updated.

@ankursinha rpm -Va works and gives me an infinite list of files, mostly flagged as S.5...... meaning “size and MD5 differ”

@computersavvy I think augenauf is right, dnf upgrade started working properly, all downloads happened correctly but then an upgrade of one of the package probably froze the system: SSH pipe broken, no response to ping, and when I connected to iKVM I saw a full screen of errors (which I didn’t save :/)

3 Likes
10

Yeh, a clean install is probably a good thing to do. It can be fixed, but it’ll take quite a bit of work.

If this is a Fedora server, you should be able to upgrade using the cockpit web interface also, which would be safer than sshing and running commands.

If you do want to use ssh, do use screen/tmux/byobu to ensure that if your connection drops, the terminal session doesn’t.

Edit: you can also set up automatic updates:

1 Like
11

It has always been my understanding that once past EOL the fedora repos were retired and the mirrors as well due to the fact that retaining old repos that were no longer supported consumes a lot of space.

Thank you for the info and updates on that.

BTW, it seems that dnf is able to still access the repos, but gnome software gives a notice about the EOL and recommends a version upgrade. Just checked that in a VM.

2 Likes