Kernel panic on F43 KDE -> F44 upgrade

[penguinistrator (Ivan Pertov)]

I am trying to upgrade fedora 43 kde according to the official guidelines:

Upgrading Fedora Linux Using DNF System Plugin

on a lenovo laptop with i7-13700H CPU with integrated iris xe graphics, 32gb ram but running into some issues. System has 2 nvme drives, one with windows, one with fedora. Fedora drive has two partitions:
/ - btrfs ~500gb (free 150gb)
/boot/efi - vfat - 1gb (free 962mb)

The recommended approach with offline reboot successfully downloads all the packages, goes into reboot, starts upgrading but gets stuck on 94% with flashing capslock (in plymouth ui). If i press escape and look through text based UI it reaches reboot target, when the capslook started to flash. Also at this point i see some graphical artifact on top left corner of the screen, i suppose it should be ASCII penguin which also suggests i’m in kernel panic mode.

If i forcefully reboot with 10 sec press of a power button, i get into grub and see that update made a new record for 6.19.14-300.fc44.x86_64 kernel, but if i try to boot it i get an error that img file does not exist (seems like dracut failed to make it)

If i try the other option of the old kernel, same version, but fc43 instead of fc44, i get straight into kernel panic.

Good thing that i’ve got snapper snapshots to quickly rollback too and bring my system back to working state.

What i also tried - making an online upgrade with:
sudo dnf upgrade --releasever=44 --refresh

I think i’ve briefly seen the error of dracut failing some dependencies, but the system went fast into reboot nevertheless and i got into the same state i had with offline update. New grub entry for fc44 kernel which lacks img and old fc43 entries booting into kernel panic.

I think i have some unlucky hardware\software combination which somehow breaks the process rebuilding a new kernel, but i can’t figure out how to diagnose it further, because all the logs i have are inaccessable since the system is already in kernel panic mode.

Any help would be appreciated, i think i’m gonna work towards an approach of how to build fc44 kernel manually without system restart to get some human readable errors from this process and act upon them.

[francismontagnac]

Have a look perhaps to this topic:

Fedora failing to boot after attempting to update to from 42 to 43

One of your screenshot shows that dracut tried wrongly to write in /boot/efi/MACHINE_ID where MACHINE_ID is the identifier of your machine. See cat /etc/machine-id This has been reported during some updates to 43. Those MACHINE_ID directories should be suppressed and the last kernel re-installed. If /boot contains the vmlinuz fc42 files (that you see in GRUB), but no .fc43 ones, try this: sudo mv /boot/$(cat /etc/machine-id){,-OFF} # Ignore if /boot/MACHINE_ID doesn't exist sudo mv /boot/ef…

@penguinistrator

What i also tried - making an online upgrade with:
sudo dnf upgrade --releasever=44 --refresh

I think i’ve briefly seen the error of dracut failing some dependencies, but the system went fast into reboot nevertheless

This is weird: this should not have make the machine reboot.

You may also try an online update but using distro-sync

[penguinistrator (Ivan Pertov)]

Thanks, i’ll have a look at it. So far i’ve made a live usb for fedora 44, booted from it, mounted and chrooted to my broken-after-upgrade system, cd’ed to /lib/modules and run the dracut:
/usr/bin/dracut -v --force /boot/initramfs-6.19.14-300.fc44.x86_64.img 6.19.14-300.fc44.x86_644

and it stopped with errors:
dracut-install: ERROR: could not locate dependency libsystemd-shared-258.7-1.fc43.so requested by '/var/tmp/dracut.diPQOZw/initramfs/usr/lib/systemd/systemd'
there were a total of 7 errors like that, all requesting this exact dependency but requested by different modules

Gonna try to find why am i missing this dependency and how can i get it

[penguinistrator (Ivan Pertov)]

I’ve figured it out, distro-sync command pointed me the right way.

TL;DR Solution for me was:
Make sure that the file:
/usr/lib/systemd/system/service.d/10-timeout-abort.conf
is removed and does not get recreated across reboots. After that update went fine.

Long story:
Booted with live cd into broken system, tried to recreate the img with dracut and failed. What i noticed - dracut looked for dependencies of fc43 systemd libs and not fc44. What was the reason? The reason was - somehow update did not update my systemd package to fc44 version, but did update systemd-shared package. So there was no libs of fc43 to build an img, but the systemd itself was fc43.

Tried doing distro-sync - it did notice that my systemd package was outdated and gave an error:

[3/4] Upgrading systemd-0:259.5-1.fc44.x86_64
[RPM] unpacking of archive failed on file /usr/lib/systemd/system/service.d/10-timeout-abort.conf;69f2fcc5: cpio: rename failed - Directory not empty

Then it was a matter of removing this file (it’s not easy, it has immutable -i attribute, you can view it from lsattr command, i had to resort to chattr -R -i ./service.d to remove it.

But it still got recreated after reboot, so i had to use the instructions from the said file to disable it entirely:

[Service]
# See https://fedoraproject.org/wiki/Changes/Shorter_Shutdown_Timer.
#
# To facilitate debugging when a service fails to stop cleanly,
# TimeoutStopFailureMode=abort is set to "crash" services that fail to stop in
# the time allotted. This will cause the service to be terminated with SIGABRT
# and a coredump to be generated.
#
# To undo this configuration change, create a mask file:
#   sudo mkdir -p /etc/systemd/system/service.d
#   sudo ln -sv /dev/null /etc/systemd/system/service.d/10-timeout-abort.conf
TimeoutStopFailureMode=abort