After offline upgrade via DNF5, "/init: error while loading shared libraries: libsystemd-core-256.9-2.fc41.so: cannot open shared object file: No such file or directory" causes kernel panic on boot

Ask Fedora
, , , , ,
1

After having the undermentioned occur, I decided to reinstall my installation of Fedora 41:

This has worked for approximately 2 weeks. However, I just had a very similar error in SystemD occur that has again rendered by OS installation unusable:

  1. After running the undermentioned:

    #!/usr/bin/env sh
sudo dnf upgrade --offline -y && \
sudo dnf5 offline reboot -y

    I left for 30 minutes, and when I returned, it has again become stuck on boot.

  2. Consequently, I forced a reinitialisation of the motherboard via its onboard RESET_SW button.

  3. However, this has rendered SystemD corrupt and unbootable, again:

    IMG_20241211_135117Z
    IMG_20241211_135117Z1920Ă—1440 345 KB

    The error messages of note are:

    1. /init: error while loading shared libraries: libsystemd-core-256.9-2.fc41.so: cannot open shared object file: No such file or directory

    2. Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00

What’s going on? This was a damn near stock installation. [1]

Environment

  1. #!/usr/bin/env bash
kinfo

  2. #!/usr/bin/env xdg-open
Operating System   : Fedora Linux 41
Kernel Version     : 6.11.10-300.fc41.x86_64 (64-bit)
Graphics Platform  : Wayland
Processors         : 12 Ă— AMD Ryzen 5 7600X 6-Core Processor
Memory             : 30.4 GiB of RAM
Graphics Processor : AMD Radeon RX 5700

  1. 496876 – All windows release grab every ⪅ 5s ↩︎

1 Like
2

Well, I’ve found the fix. Run the 6.11.10 kernel, not the 6.11.11.

I’m seriously surprised that their QA is this bad.

Anyone know how to ensure that my inevitable Bugzilla report is considered to be high priority?

1 Like
3

You are part of “their QA”. Feel free to enable updates-testing, install packages from testing and report feedback on bodhi.

3 Likes
4

@Flo, isn’t that enabled by default? The undermentioned contains a lot of references to such repositories, and the output it contains is from dnf5 repoquery --enabled:

Thanks. Filed bodhi.fedoraproject.org/updates/FEDORA-2024-b06c6291d7#comment-3864129.

I’d like to provide traces, but the way that unix.stackexchange.com/revisions/60574/5 acquired them seems a little too complex for me.

1 Like
5

Filed at bugzilla.redhat.com/show_bug.cgi?id=2331678#c0.

1 Like
6

not that I know of.
updates is enabled by default, updates-testing should not be enabled by default.

dnf repolist --enabled

Good that you filed a bug on bz since the update had been pushed to stable (a day or two until it appear in the updates repo) before you submitted negative karma on bodhi.

2 Likes
7

@augenauf, that’s it - I’ve got updates, but not *.-testing.

Thanks. That’s worrisome…

8

Hey, Roke! I wanted to coment this at Bodhi, but it’s Markdown suport is terrible, so I’ve copied it here so that I can link to it:

This update initially caused a kernel panic when I ran sudo dnf5 offline reboot, but it now boots into .11 too, unlike your #comment-3864129. I’ve given it negative karma because it’s the first time I’ve had a kernel panic (and it seems like quite a bad thing) but since it works now, IDK whether that’s correct.

I have nearly identical hardware to @RokeJulianLockhart:

  1. #!/usr/bin/env sh
kinfo

  2. Operating System   : Fedora Linux 41
Kernel Version     : 6.11.11-300.fc41.x86_64 (64-bit)
Graphics Platform  : Wayland
Processors         : 12 Ă— AMD Ryzen 5 7600X 6-Core Processor
Memory             : 30.4 GiB of RAM
Graphics Processor : AMD Radeon RX 7900 XTX
Manufacturer       : ASRock
Product Name       : X670E Taichi

However, I don’t even have WayDroid installed (much less VirtualBox or QEMU/KVM, etcetera) so there’s definitely something about our hardware that causes this.

Tried to copy how you do it. :person_shrugging:

1 Like
9

Posted to FEDORA-2024-b06c6291d7 — bugfix update for kernel — Fedora Updates System too.

1 Like
10

6.11.11 marks the end of life of 6.11. Future fixes are likely to take place only in 6.12.

As far as I have seen, the tests of 6.12 have been positive since 6.12.3, it was just waited for the bpftool maintainer since then (don’t know what this was about), and it seems this has happened already:

6.12.4 is now in bodhi’s testing and the bodhi tests are so far mostly positive as well (the one with negative karma refers to two bugs of around 2006, so I am not sure if I take that report serious).

So you might try to test 6.12.4 the way it was pushed to bodhi-testing and see if your issue is still present with that one (or stick with 6.10.10 until 6.12.X has been pushed to stable if you prefer that). If this is a kernel issue, we can say more or less for sure that it will be only processed if it still occurs on 6.12:

https://bodhi.fedoraproject.org/updates/FEDORA-2024-9fb3492511

If it still occurs on 6.12, you might increase the amount of kernels that remain installed in advance to further updates, in order to ensure that you still keep 6.11.10 installed (default is 3 kernels).

2 Likes
11

@py0xc3, I’ll wait for that. Many thanks!

Do you know of an up-to-date tutorial for that?

@jaredrichardwilliam, I forgot to include mine. Thanks. Added to /1#p-366100-environment-1:

12

According to man dnf.conf (which already refers to dnf5.conf - DNF5 Configuration Reference and therefore seems to have been already updated to dnf5), this has not changed compared to the old dnf:

In the file /etc/dnf/dnf.conf you have a line with installonly_limit. By default, this should be installonly_limit=3 I presume. E.g., I have set this on my system to 5 (installonly_limit=5), which means that it retains 5 kernels, and no longer only 3.

If you increase the number, you will see in the next update if it works: you currently have 3 kernels installed (I presume), so if this is set to, e.g., 4, then it will not delete any kernel when installing the next one.

Don’t forget that on most installations, /boot is a separated partition. So ensure that /boot has sufficient space for retaining more than 3 kernels. I have 403 MB left with 5 kernels installed on a /boot with a total of 973 MB (x86_64).

1 Like
13

@py0xc3, 6.12.4 works! [1] Many thanks for the advice.

  1. FEDORA-2024-9fb3492511 — security update for bpftool, kernel, & 1 more — Fedora Updates System ↩︎

1 Like
14

Just as a quick update, 6.12.4 is now released for fedora. I installed it from updates.

2 Likes
15

Yeah, that’s how I got it. I don’t use the -testing repositories (in order to avoid this kind of thing, ironically).

16

Just as a quick note, you can also temporarily enable updates-testing just to install and test a new kernel: sudo dnf update --enable-repo=updates-testing kernel\*.
Also, for dnf5, better place your config file in the new drop-in directories such as /etc/dnf/libdnf5.conf.d/20-user-settings.conf.

1 Like
17

Thank you! I’ll do that next time.

18

The links to bodhi (e.g., the one above) contain the very DNF command (with advisory limitation) to only install the very package and its very dependencies. In the link for 6.12.4, it was sudo dnf upgrade --refresh --advisory=FEDORA-2024-9fb3492511. This is usually the safest way to ensure to only get what you want but get all you want: This command would have installed only the kernel files related to kernel-6.12.4-200.fc41 (including bpftool-7.5.0-1.fc41 and kernel-headers as it marks a major update to a new stable kernel that needs new headers and such) for your very architecture, and if you have additional kernel packages that depend on the very kernel (e.g., kernel-tools), then these would have been installed/updated as well, but if you have not installed them, then not. The advisory commands are very explicit in this respect. These commands also do NOT enable the testing repos but only install the very files of the very bodhi update from testing once. This means, at the very moment the testing kernel is pushed to stable, everything is back to normal and everything-stable-only.

2 Likes