Kernel 6.12 will not boot (cannot mount root)

My root and boot partition is:
UUID=xxxxxx / btrfs subvol=root,compress=zstd:1 0 0
UUID=xxxxxx /boot ext4 defaults,discard 1 2

but kernel 6.12.4-200 cannot mount it… the machine just gets a kernel panic and dies. The previous one, 6.11.11-300 boots just fine. Any ideas why?

PS: I’m reinstalling the whole 6.12 again to see if that helps… Maybe it got a /boot disk full or something.

here is how it went:

turgut@big-laptop:~$ rpm -qa|grep kernel|grep 6.12
kernel-modules-core-6.12.4-200.fc41.x86_64
kernel-core-6.12.4-200.fc41.x86_64
kernel-modules-6.12.4-200.fc41.x86_64
kernel-devel-6.12.4-200.fc41.x86_64
kernel-devel-matched-6.12.4-200.fc41.x86_64
kernel-tools-libs-6.12.4-200.fc41.x86_64
kernel-tools-6.12.4-200.fc41.x86_64
kernel-modules-extra-6.12.4-200.fc41.x86_64
kernel-headers-6.12.4-200.fc41.x86_64
kernel-6.12.4-200.fc41.x86_64
turgut@big-laptop:~$ sudo dnf reinstall rpm -qa|grep kernel|grep 6.12
Updating and loading repositories:
Repositories loaded.
Package Arch Version Repository Size
Reinstalling:
kernel x86_64 6.12.4-200.fc41 updates 0.0 B
replacing kernel x86_64 6.12.4-200.fc41 updates 0.0 B
kernel-core x86_64 6.12.4-200.fc41 updates 73.6 MiB
replacing kernel-core x86_64 6.12.4-200.fc41 updates 73.6 MiB
kernel-devel x86_64 6.12.4-200.fc41 updates 76.6 MiB
replacing kernel-devel x86_64 6.12.4-200.fc41 updates 76.6 MiB
kernel-devel-matched x86_64 6.12.4-200.fc41 updates 0.0 B
replacing kernel-devel-matched x86_64 6.12.4-200.fc41 updates 0.0 B
kernel-headers x86_64 6.12.4-200.fc41 updates 6.4 MiB
replacing kernel-headers x86_64 6.12.4-200.fc41 updates 6.4 MiB
kernel-modules x86_64 6.12.4-200.fc41 updates 62.5 MiB
replacing kernel-modules x86_64 6.12.4-200.fc41 updates 62.5 MiB
kernel-modules-core x86_64 6.12.4-200.fc41 updates 37.1 MiB
replacing kernel-modules-core x86_64 6.12.4-200.fc41 updates 37.1 MiB
kernel-modules-extra x86_64 6.12.4-200.fc41 updates 2.7 MiB
replacing kernel-modules-extra x86_64 6.12.4-200.fc41 updates 2.7 MiB
kernel-tools x86_64 6.12.4-200.fc41 updates 926.2 KiB
replacing kernel-tools x86_64 6.12.4-200.fc41 updates 926.2 KiB
kernel-tools-libs x86_64 6.12.4-200.fc41 updates 34.3 KiB
replacing kernel-tools-libs x86_64 6.12.4-200.fc41 updates 34.3 KiB

Transaction Summary:
Reinstalling: 10 packages
Replacing: 10 package

Total size of inbound packages is 148 MiB. Need to download 148 MiB.
After this operation, 0 B extra will be used (install 260 MiB, remove 260 MiB).
[ 1/10] kernel-modules-core-0:6.12.4-200.fc41.x86_64 100% | 114.7 KiB/s | 38.4 MiB | 05m43s

Curl error (28): Timeout was reached for http://linus.iyte.edu.tr/linux/fedora/linux/updates/41/Everything/x86_64/Packages/
[ 2/10] kernel-modules-0:6.12.4-200.fc41.x86_64 100% | 1.9 MiB/s | 64.2 MiB | 00m33s
[ 3/10] kernel-devel-0:6.12.4-200.fc41.x86_64 100% | 1.1 MiB/s | 21.0 MiB | 00m19s
[ 4/10] kernel-devel-matched-0:6.12.4-200.fc41.x86_64 100% | 629.6 KiB/s | 190.8 KiB | 00m00s
[ 5/10] kernel-tools-libs-0:6.12.4-200.fc41.x86_64 100% | 664.4 KiB/s | 203.3 KiB | 00m00s
[ 6/10] kernel-tools-0:6.12.4-200.fc41.x86_64 100% | 790.8 KiB/s | 474.5 KiB | 00m01s
[ 7/10] kernel-modules-extra-0:6.12.4-200.fc41.x86_64 100% | 850.2 KiB/s | 2.9 MiB | 00m04s
[ 8/10] kernel-headers-0:6.12.4-200.fc41.x86_64 100% | 928.8 KiB/s | 1.6 MiB | 00m02s
[ 9/10] kernel-0:6.12.4-200.fc41.x86_64 100% | 662.2 KiB/s | 190.7 KiB | 00m00s
[10/10] kernel-core-0:6.12.4-200.fc41.x86_64 91% [================ ] | 0.0 B/s | 16.8 MiB | -01m16s
[10/10] kernel-core-0:6.12.4-200.fc41.x86_64 91% [================ ] | 2.8 KiB/s | 16.8 MiB | -09m23s
[10/10] kernel-core-0:6.12.4-200.fc41.x86_64 100% | 36.1 KiB/s | 18.4 MiB | 08m41s


[10/10] Total 100% | 289.4 KiB/s | 147.6 MiB | 08m42s
Running transaction
[ 1/22] Verify package files 100% | 20.0 B/s | 10.0 B | 00m00s
[ 2/22] Prepare transaction 100% | 12.0 B/s | 20.0 B | 00m02s
[ 3/22] Reinstalling kernel-core-0:6.12.4-200.fc41.x86_64 100% | 40.6 MiB/s | 26.9 MiB | 00m01s
[ 4/22] Reinstalling kernel-modules-core-0:6.12.4-200.fc41.x86_64 100% | 22.2 MiB/s | 37.6 MiB | 00m02s
[ 5/22] Reinstalling kernel-modules-0:6.12.4-200.fc41.x86_64 100% | 7.8 MiB/s | 62.9 MiB | 00m08s
[ 6/22] Reinstalling kernel-tools-libs-0:6.12.4-200.fc41.x86_64 100% | 49.6 KiB/s | 34.7 KiB | 00m01s
[ 7/22] Reinstalling kernel-devel-0:6.12.4-200.fc41.x86_64 100% | 1.1 MiB/s | 80.7 MiB | 01m13s
[ 8/22] Reinstalling kernel-devel-matched-0:6.12.4-200.fc41.x86_64 100% | 8.6 KiB/s | 124.0 B | 00m00s
[ 9/22] Reinstalling kernel-tools-0:6.12.4-200.fc41.x86_64 100% | 11.2 MiB/s | 932.1 KiB | 00m00s
[10/22] Reinstalling kernel-modules-extra-0:6.12.4-200.fc41.x86_64 100% | 433.4 KiB/s | 2.7 MiB | 00m06s
[11/22] Reinstalling kernel-0:6.12.4-200.fc41.x86_64 100% | 9.3 KiB/s | 124.0 B | 00m00s
[12/22] Reinstalling kernel-headers-0:6.12.4-200.fc41.x86_64 100% | 15.6 MiB/s | 6.6 MiB | 00m00s
[13/22] Removing kernel-0:6.12.4-200.fc41.x86_64 100% | 19.5 KiB/s | 100.0 B | 00m00s
[14/22] Removing kernel-modules-extra-0:6.12.4-200.fc41.x86_64 100% | 22.0 B/s | 143.0 B | 00m06s
[15/22] Removing kernel-modules-0:6.12.4-200.fc41.x86_64 100% | 365.0 B/s | 2.3 KiB | 00m06s
[16/22] Removing kernel-devel-matched-0:6.12.4-200.fc41.x86_64 100% | 19.5 KiB/s | 100.0 B | 00m00s
[17/22] Removing kernel-tools-0:6.12.4-200.fc41.x86_64 100% | 3.6 KiB/s | 40.0 B | 00m00s
[18/22] Removing kernel-modules-core-0:6.12.4-200.fc41.x86_64 100% | 4.7 KiB/s | 3.1 KiB | 00m01s

Running pre-uninstall scriptlet: kernel-core-0:6.12.4-200.fc41.x86_64
Finished pre-uninstall scriptlet: kernel-core-0:6.12.4-200.fc41.x86_64
Scriptlet output:
Module digimend/13 is not installed for kernel 6.12.4-200.fc41.x86_64 (x86_64x86_64). Skipping…
Module digimend/13 is not built for kernel 6.12.4-200.fc41.x86_64 (x86_64x86_64). Skipping…

[19/22] Removing kernel-core-0:6.12.4-200.fc41.x86_64 100% | 160.0 B/s | 17.0 B | 00m00s
[20/22] Removing kernel-devel-0:6.12.4-200.fc41.x86_64 100% | 116.4 KiB/s | 22.9 KiB | 00m00s
[21/22] Removing kernel-headers-0:6.12.4-200.fc41.x86_64 100% | 32.0 KiB/s | 1.0 KiB | 00m00s
[22/22] Removing kernel-tools-libs-0:6.12.4-200.fc41.x86_64 100% | 0.0 B/s | 2.0 B | 00m15s

Running post-transaction scriptlet: kernel-core-0:6.12.4-200.fc41.x86_64
Non-critical error in post-transaction scriptlet: kernel-core-0:6.12.4-200.fc41.x86_64
Scriptlet output:

Sign command: /lib/modules/6.12.4-200.fc41.x86_64/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub

Cleaning build area… done.
Building module(s)…(bad exit status: 2)
Failed command:
make -j16 KERNELRELEASE=6.12.4-200.fc41.x86_64 KVERSION=6.12.4-200.fc41.x86_64

Error! Bad return status for module build on kernel: 6.12.4-200.fc41.x86_64 (x86_64)
Consult /var/lib/dkms/digimend/13/build/make.log for more information.
Autoinstall on 6.12.4-200.fc41.x86_64 failed for module(s) digimend(10).

Error! One or more modules failed to install during autoinstall.
Refer to previous errors for more information.
/usr/lib/kernel/install.d/40-dkms.install failed with exit status 11.

[RPM] %posttrans(kernel-core-6.12.4-200.fc41.x86_64) scriptlet failed, exit status 11
Complete!

What is the panic you see?

Details of your hardware and disks might be useful to know.
What is the output of these commands?

inxi -Fzxx
lsblk -f

Maybe is not related at all but looks like it not finish the kernel installation process. Failed because of dkms tablet’s module, apparently. Its been a while since I last used my tablet, but it’s maybe related to the digimend module. Build error: linux 6.12 renames asm/unaligned.h · Issue #706 · DIGImend/digimend-kernel-drivers · GitHub

Many thanks. I removed those drivers and then I did a dracut and it fixed my problem.
It now boots successfully

1 Like

I saw the same root cause for my fedora VM running under Parallels.
The parallels tools uses dkms to build and that failed.
(Parallels have not update their code to support the 6.12 kernels yet).

I was without a initrd.

To fix I booted to the previous kernal and use dracut to make the missing
initrd with:

dracut --kver 6.12.4-200.fc41.aarch64

After that I could boot with the 6.12 kernel.

2 Likes

Note that many 3rd party apps that use dkms to build modules seem unable to match the pace of fedora kernel upgrades. Apps that are provided through rpmfusion are specifically tested for function before releasing upgrades so this is seldom a problem for those apps.

Both these problems (digimend and parallels) report the same type issue.

3 Likes

Even after I removed the offending module, I still get the same problem.
A new kernel, 6.12.5-200 was introduced and the same issue. interstingly, after I boot from a previous kernel I can run dracut --kver 6.12.5-200.fc41.x86_64
with no problems. Why can’t the “dnf update” do the same?

In my case because the dkms process failed and that prevents the initrd being created. I saw the error in the output of the dnf update when it was running scripts i recall.

1 Like

Had the same issue coming from 6.11.(something) to 6.12.5-200.x86_64. Had a kernel panic on the 6.12.5-200 after reboot, threw a fit about initramfs.

Rebooted into prior kernel, ran the suggested dracut command:
dracut --kver 6.12.5-200.fc41.x86_64

Rebooted and the new kernel did throw a “dkms.service failed to start” but booted successfully after a few moments.

After login, I checked “systemctl status dkms.service”, was in failed state and wouldn’t restart…log output from status showed that “anbox-binder” and “anbox-ashmem” were the cause.

Took the following steps to fully resolve:

  1. Reinstalled the relevant to make sure nothing was missed or corrupt
    sudo dnf reinstall kernel-devel kernel-headers

  2. (to confirm the packages matched the kernel)
    uname -r
    rpm -q kernel-devel kernel-headers

  3. get bad dkms modules all cleaned up, again in this case looks like it was Anbox modules
    sudo dkms remove anbox-binder/1 --all
    sudo dkms remove anbox-ashmem/1 --all
    sudo dkms autoinstall --verbose

  4. make sure dkms was all updated
    sudo dnf update dkms

  5. restart and verify the dkms unit was successfully started
    sudo systemctl restart dkms.service
    sudo systemctl status dkms.service

voila… resolved. After reboot, dkms no longer complains when booting into 6.12.5-200.x86_64.

1 Like