Dnf not working properly after Fedora update crash

Ask Fedora
,
1

I tried to update the system using

sudo dnf system-upgrade download --releasever=44
sudo dnf offline reboot

But the system crashed during the installation process. After the system rebooted, dnf reported error:

$ dnf --help
Cannot load dnf5 plugin: /usr/lib64/dnf5/plugins/builddep_cmd_plugin.so
 Cannot open shared object file「/usr/lib64/dnf5/plugins/builddep_cmd_plugin.so」：/usr/lib64/dnf5/plugins/builddep_cmd_plugin.so: undefined symbol: _ZN4dnf511match_specsERNS_7ContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbbbPKc

Is it possible for me to fix dnf?

2

This shows that at least one of the libraries needed by dnf is either corrupted or
outdated, probably /lib64/libstdc++.so.6

What gives? (as text please and from a bash shell):

(
  set -x
  rpm -q dnf5 dnf5-plugins
  rpm -qf /lib64/libstdc++.so.6
  rpm -qfV /lib64/libstdc++.so.6
  dnf4 --help |& head
)

If dnf4 works, try to reinstall libstdc++ with: dnf4 distro-sync libstdc++ --releasever=44

3

It may be needed instead to downgrade libstdc++ or to upgrade dnf5-plugins.
What gives: rpm -q libstdc++ dnf5 dnf5-plugins

Since rpm depends now also of libstdc++ I fear that it will be necessary to boot
from a live media to fix that.

Also, can you send the journal of the boot that crashed during the install ?

  • Identify this boot with journalctl --list-boots
  • Execute: journalctl -b THE_BOOT_ID_OF_THE_CRASH | fpaste
  • Paste here the obtained URL
4

Thanks!

dnf4 distro-sync libstdc++ --releasever=44 didn’t work. But I fixed dnf5 with dnf4 distro-sync dnf –releasever=44.

I forgot to save the output of rpm -qfV /lib64/libstdc++.so.6:person_bowing: . But I believe there were two versions of the libstdc++ package.

5

Several crashes happened after the initial crash during the update so I can’t tell which one is the original one. Can I judge based on the content of those logs?

6

The journal contains previous boots, so forgetting to save details should not prevent you from proving the requested details. Those may help find ways to avoid others encountering the same issue.

7

I think I found the log from that time.

8

Yes, this is it, running:

16 21:36:31 fedora systemd[1]: Starting dnf5-offline-transaction.service - Offline upgrades/transactions using DNF 5...

I’ll look closer at it later.

You should now finish properly the upgrade to F44 with:

sudo dnf distro-sync \
    --refresh \
    --releasever=44 \
    --allowerasing \
    --skip-broken \
    --setopt=protected_packages=

I personally would run it protected and keeping its log as follows:

nohup sudo dnf -y distro-sync \
    --refresh \
    --releasever=44 \
    --allowerasing \
    --skip-broken \
    --setopt=protected_packages= \
    &> /var/tmp/distro-sync-01 &
tail -f /var/tmp/distro-sync-01 # Hit "Ctrl c"  to terminate this tail
9

Looking at the dnf5-offline-transaction log shows that dnf5 died on SEGV:

5月 16 21:51:56 fedora dnf5[1737]: [1715/4924] 升级 podman-5:5.8.2-1.fc4 100% |  51.2 MiB/s |  49.1 MiB |  00m01s
5月 16 21:51:56 fedora audit[1737]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:rpm_t:s0 pid=1737 comm="dnf5" exe=2F7573722F62696E2F646E6635202864656C6574656429 sig=11 res=1
5月 16 21:51:56 fedora kernel: show_signal: 12 callbacks suppressed
5月 16 21:51:56 fedora kernel: traps: dnf5[1737] general protection fault ip:7f1d42fd0182 sp:7fffec6fd130 error:0 in libzstd.so.1.5.7[93182,7f1d42f3d000+b0000]
5月 16 21:51:56 fedora kernel: audit: type=1701 audit(1778939516.343:225): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:rpm_t:s0 pid=1737 comm="dnf5" exe=2F7573722F62696E2F646E6635202864656C6574656429 sig=11 res=1
5月 16 21:51:56 fedora systemd-coredump[8078]: Process 1737 (dnf5) of user 0 terminated abnormally with signal 11/SEGV, processing...
5月 16 21:51:56 fedora systemd-coredump[8078]: Failed to connect to coredump service: No such file or directory
5月 16 21:51:56 fedora systemd[1]: dnf5-offline-transaction.service: Main process exited, code=dumped, status=11/SEGV
5月 16 21:51:56 fedora systemd[1]: dnf5-offline-transaction.service: Failed with result 'core-dump'.
5月 16 21:51:56 fedora systemd[1]: Failed to start dnf5-offline-transaction.service - Offline upgrades/transactions using DNF 5.

but the system was not able to save a coredump :frowning:

Lines related to libstdc++, dnf5 and dnf3 (== dnf4):

grep -E '1737.*/.*(libstdc\+\+|dnf5|dnf3)' /var/tmp/3d73f26b
5月 16 21:40:06 fedora dnf5[1737]: [ 205/4924] 升级 libstdc++-0:16.1.1-1 100% |  20.1 MiB/s |   3.1 MiB |  00m00s
5月 16 21:51:37 fedora dnf5[1737]: [1647/4924] 升级 libdnf5-0:5.4.2.0-1. 100% |  30.4 MiB/s |   4.3 MiB |  00m00s
5月 16 21:51:39 fedora dnf5[1737]: [1648/4924] 升级 libdnf5-cli-0:5.4.2. 100% | 522.1 KiB/s |   1.0 MiB |  00m02s
5月 16 21:51:40 fedora dnf5[1737]: [1649/4924] 安装dnf5daemon-server-0:5 100% |   3.0 MiB/s | 792.3 KiB |  00m00s
5月 16 21:51:40 fedora dnf5[1737]: [1650/4924] 安装libdnf5-plugin-appstr 100% | 470.5 KiB/s |  32.9 KiB |  00m00s
5月 16 21:51:43 fedora dnf5[1737]: [1659/4924] 安装dnf5daemon-server-pol 100% |   5.6 KiB/s | 612.0   B |  00m00s
5月 16 21:51:43 fedora dnf5[1737]: [1660/4924] 升级 dnf5-0:5.4.2.0-1.fc4 100% |  10.8 MiB/s |   3.2 MiB |  00m00s
5月 16 21:51:44 fedora dnf5[1737]: [1666/4924] 升级 python3-libdnf5-0:5. 100% |  29.6 MiB/s |   9.8 MiB |  00m00s

This shows that dnf5-plugins was not updated before the crash.
I guess that your dnf4 distro-sync dnf –releasever=44 updated it.