Unable to boot after upgrading kernel to 6.8.9 and 6.8.10

Ask Fedora
, , ,
1

image
image1920×1440 590 KB

I installed fedora in a exteral SSD connected via USB4 / Thunderbolt port. Seems some drivers are removed in 6.8.9 and later

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0 931.5G  0 disk 
└─sda1        8:1    0 931.5G  0 part 
zram0       252:0    0     8G  0 disk [SWAP]
nvme1n1     259:0    0 953.9G  0 disk 
├─nvme1n1p1 259:1    0   300M  0 part 
├─nvme1n1p2 259:2    0   128M  0 part 
└─nvme1n1p3 259:3    0 953.4G  0 part 
nvme0n1     259:4    0   3.7T  0 disk 
├─nvme0n1p1 259:5    0     1G  0 part /boot/efi
├─nvme0n1p2 259:6    0     1G  0 part /boot
├─nvme0n1p3 259:7    0    48G  0 part /
└─nvme0n1p4 259:8    0   3.7T  0 part 

$ ls -l /boot
total 435604
-rw-r--r--  1 root root   9017236 May 17 08:00 System.map-6.8.10-300.fc40.x86_64
-rw-r--r--  1 root root   8905609 Apr 27 08:00 System.map-6.8.7-303.fsync.fc40.x86_64
-rw-r--r--  1 root root   9023224 May  4 08:00 System.map-6.8.9-301.fsync.fc40.x86_64
-rw-r--r--  1 root root    272218 May 17 08:00 config-6.8.10-300.fc40.x86_64
-rw-r--r--  1 root root    274727 Apr 27 08:00 config-6.8.7-303.fsync.fc40.x86_64
-rw-r--r--  1 root root    275069 May  4 08:00 config-6.8.9-301.fsync.fc40.x86_64
drwx------  4 root root      4096 Jan  1  1970 efi
drwx------. 3 root root      4096 May 22 19:19 grub2
-rw-------. 1 root root 116489425 Nov  9  2023 initramfs-0-rescue-1836d5fbcc0a424f9d13898fb3aae9c5.img
-rw-------  1 root root  71942001 May 22 18:34 initramfs-6.8.10-300.fc40.x86_64.img
-rw-------  1 root root  84281132 May 11 17:35 initramfs-6.8.7-303.fsync.fc40.x86_64.img
-rw-------  1 root root  84662225 May 11 17:53 initramfs-6.8.9-301.fsync.fc40.x86_64.img
drwxr-xr-x. 3 root root      4096 Nov  9  2023 loader
drwx------. 2 root root     16384 Nov  9  2023 lost+found
lrwxrwxrwx  1 root root        46 May 22 18:34 symvers-6.8.10-300.fc40.x86_64.xz -> /lib/modules/6.8.10-300.fc40.x86_64/symvers.xz
lrwxrwxrwx  1 root root        51 Apr 28 18:19 symvers-6.8.7-303.fsync.fc40.x86_64.xz -> /lib/modules/6.8.7-303.fsync.fc40.x86_64/symvers.xz
lrwxrwxrwx  1 root root        51 May 11 17:53 symvers-6.8.9-301.fsync.fc40.x86_64.xz -> /lib/modules/6.8.9-301.fsync.fc40.x86_64/symvers.xz
-rwxr-xr-x. 1 root root  14536168 Nov  9  2023 vmlinuz-0-rescue-1836d5fbcc0a424f9d13898fb3aae9c5
-rwxr-xr-x  1 root root  15712072 May 17 08:00 vmlinuz-6.8.10-300.fc40.x86_64
-rwxr-xr-x  1 root root  14909424 Apr 27 08:00 vmlinuz-6.8.7-303.fsync.fc40.x86_64
-rwxr-xr-x  1 root root  15708144 May  4 08:00 vmlinuz-6.8.9-301.fsync.fc40.x86_64

Currently I booted into 6.8.7-303.fsync.fc40.x86_64

btrfsck seems fine

$ sudo btrfsck -p /dev/nvme0n1p3 --force
[sudo] carter 的密码：
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/nvme0n1p3
UUID: 5ed97d0e-5d94-4828-806c-3fe232d9612c
[1/7] checking root items                      (0:00:00 elapsed, 505198 items checked)
[2/7] checking extents                         (0:00:01 elapsed, 48697 items checked)
[3/7] checking free space tree                 (0:00:01 elapsed, 33 items checked)
[4/7] checking fs roots                        (0:00:01 elapsed, 43595 items checked)
[5/7] checking csums (without verifying data)  (0:00:00 elapsed, 182743 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 5 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 21483831296 bytes used, no error found
total csum bytes: 19888028
total tree bytes: 797671424
total fs tree bytes: 717111296
total extent tree bytes: 49856512
btree space waste bytes: 162692554
file data blocks allocated: 32050565120
 referenced 32910327808

image
image2237×1124 374 KB

Fedora is installed in a partition of the external SSD aigo NVMe SSD P7000Z 4TB

Previous I asked in reddit and got no useful answer. I decided to wait fedora releasing a new kernel. Today I installed the latest kernel 6.8.10 the issue still exists.

1 Like
2

Where do you have those kernels from? They don’t appear to be stock Fedora kernels… (maybe except the 6.8.10 one)

1 Like
3

It is from sentry/kernel-fsync Copr

It should be irrelevant. I uninstalled all fsync kernels and installed official 6.8.5-301. I successfully booted into 6.8.5

total 316136
-rw-r--r--  1 root root   9017236 May 17 08:00 System.map-6.8.10-300.fc40.x86_64
-rw-r--r--  1 root root   8897695 Apr 11 08:00 System.map-6.8.5-301.fc40.x86_64
-rw-r--r--  1 root root    272218 May 17 08:00 config-6.8.10-300.fc40.x86_64
-rw-r--r--  1 root root    271916 Apr 11 08:00 config-6.8.5-301.fc40.x86_64
drwx------  4 root root      4096 Jan  1  1970 efi
drwx------. 3 root root      4096 May 22 21:19 grub2
-rw-------. 1 root root 116489425 Nov  9  2023 initramfs-0-rescue-1836d5fbcc0a424f9d13898fb3aae9c5.img
-rw-------  1 root root  71942001 May 22 18:34 initramfs-6.8.10-300.fc40.x86_64.img
-rw-------  1 root root  71574339 May 22 21:16 initramfs-6.8.5-301.fc40.x86_64.img
drwxr-xr-x. 3 root root      4096 Nov  9  2023 loader
drwx------. 2 root root     16384 Nov  9  2023 lost+found
lrwxrwxrwx  1 root root        46 May 22 18:34 symvers-6.8.10-300.fc40.x86_64.xz -> /lib/modules/6.8.10-300.fc40.x86_64/symvers.xz
lrwxrwxrwx  1 root root        45 May 22 21:16 symvers-6.8.5-301.fc40.x86_64.xz -> /lib/modules/6.8.5-301.fc40.x86_64/symvers.xz
-rwxr-xr-x. 1 root root  14536168 Nov  9  2023 vmlinuz-0-rescue-1836d5fbcc0a424f9d13898fb3aae9c5
-rwxr-xr-x  1 root root  15712072 May 17 08:00 vmlinuz-6.8.10-300.fc40.x86_64
-rwxr-xr-x  1 root root  14966600 Apr 11 08:00 vmlinuz-6.8.5-301.fc40.x86_64

However the nvidia driver building failed because of lacking 6.8.5 kernel headers

4

It is relevant when talking about compiling kernel modules.

sudo dnf install kernel-devel kernel-headers.
I believe the kernel headers package for the 6.8 kernels is at 6.8.3 while the devel package matches the kernel version. You may need to install the kernel-devel package for the 6.8.5 kernel then do a dnf upgrade to get the same package for the 6.8.10 kernel.

These kinds of problems can usually be avoided by using only kernels provided directly from fedora. Akmods can only use the fedora packages to build the driver modules.

1 Like
5

The problem is that I can’t boot into 6.8.10. It’s ok not to have nvidia drivers because I have intel intergrated gpu

EDIT: after installing kernel-devel-6.8.5 the nvidia driver was successfully built

6

Then boot the .5 kernel and collect logs from the failed .10 boot.

7

I’m noob. Any instructions?

8

Perform the dnf upgrade step I mentioned above, then run
sudo akmods --force --kernels 6.8.10-300.fc40.x86_64 to build the kernel modules for the 6.8.10 kernel.

9 
❯ LANG=C sudo dnf upgrade --refresh
Copr repo for PyCharm owned by phracek                                                                     2.1 kB/s | 1.8 kB     00:00    
Copr repo for switcheroo-control_discrete owned by sentry                                                  2.1 kB/s | 1.5 kB     00:00    
Copr repo for wezterm-nightly owned by wezfurlong                                                          2.0 kB/s | 1.5 kB     00:00    
Fedora 40 - x86_64                                                                                          34 kB/s | 5.4 kB     00:00    
Fedora 40 openh264 (From Cisco) - x86_64                                                                   4.3 kB/s | 989  B     00:00    
Fedora 40 - x86_64 - Updates                                                                                39 kB/s | 5.8 kB     00:00    
google-chrome                                                                                               11 kB/s | 1.3 kB     00:00    
microsoft-edge                                                                                             2.3 kB/s | 1.5 kB     00:00    
RPM Fusion for Fedora 40 - Free                                                                            9.2 kB/s |  11 kB     00:01    
RPM Fusion for Fedora 40 - Free tainted                                                                    9.2 kB/s |  11 kB     00:01    
RPM Fusion for Fedora 40 - Free - Updates                                                                   10 kB/s |  11 kB     00:01    
RPM Fusion for Fedora 40 - Nonfree                                                                          14 kB/s |  15 kB     00:01    
RPM Fusion for Fedora 40 - Nonfree - NVIDIA Driver                                                          13 kB/s |  16 kB     00:01    
RPM Fusion for Fedora 40 - Nonfree - Steam                                                                  12 kB/s |  14 kB     00:01    
RPM Fusion for Fedora 40 - Nonfree tainted                                                                  13 kB/s |  15 kB     00:01    
RPM Fusion for Fedora 40 - Nonfree - Updates                                                                13 kB/s |  15 kB     00:01    
Visual Studio Code                                                                                         7.4 kB/s | 1.5 kB     00:00    
warpdotdev                                                                                                 2.1 kB/s | 1.5 kB     00:00    
Dependencies resolved.
Nothing to do.
Complete!

~ took 14s 
❯ LANG=C sudo akmods --force
Checking kmods exist for 6.8.5-301.fc40.x86_64             [  OK  ]

$ LANG=C sudo akmods --force --kernels 6.8.10-300.fc40.x86_64
Checking kmods exist for 6.8.10-300.fc40.x86_64            [  OK  ]
Checking kmods exist for 6.8.5-301.fc40.x86_64             [  OK  ]
10

I edited the command above

11

It just prints OK. I will try to boot into 6.8.10 again

12

Can you boot to the 6.8.10 kernel now?
If not then follow the instructions from Flo above and give us the log info needed.

13

1000018007
10000180071920×1440 145 KB

I removed quiet
1000018008
10000180081920×2560 859 KB

Stuck here for about 2 minutes

Still no response. I think that kernel has crashed. Keyboard including ctrl alt delete doesnt work

14

I booted into .5 again. How can I find the logs of .10?

I tried journalctl -b but I could not find logs of .10. There were too many of them

15

Boot to the 6.8.5 kernel then run journalctl -b -1 to see the logs for the failed boot if logging had time to start

If that fails to show the failed boot attempt then run sudo dnf remove kernel*6.8.10* to remove all the 6.8.10 kernel packages followed by a reboot then reinstall the newer kernel with dnf upgrade Wait at least 5 minutes after the upgrade completes before rebooting and trying the 6.8.10 kernel again.

16

journalctl -b -1

It shows logs of Linux version 6.8.5-301.fc40.x86_64

sudo dnf remove kernel6.8.10

It will remove a lot of dependencies and I really don’t want to do that

image
image918×1294 223 KB

The .10 kernel was just installed 4 hours ago and I doubt that reinstalling it can change anything

If I can reinstall the kernel without uninstalling these deps I can have a try

17

In this situation, I would add the kernel packages exactly by name to make up the ‘sudo dnf remove …’
to only target the 6.8.10 kernel packages. Then use --exclude=< package>,… to unallow the removal of whatever other packages it’s trying to remove that pull in all those others you want to keep.

18

sudo dnf remove kernel*6.8.10* --noautoremove

This should avoid removing all the unused dependencies.

The removal of akmod-nvidia and akmods when removing the 6.8.10 kernel is very odd since I thought you claimed to have the nvidia driver working with the 6.8.5 kernel.
Please show the output of dnf list installed kernel* and dnf list installed \*nvidia\*so we can tell exactly what is installed and functional. (do this before removing anything)

19

  1. To remove specific kernel without removing all those other packages, you can also use for example (change to your specific kernel version):
    sudo dnf remove $(rpm -qa | grep ^kernel | grep 6.8.10 | grep -v headers)

  2. There seems to be an issue with thunderbolt/usb4 ssd’s/nvme’s from kernel 6.8.8 or 6.8.9 and later. Workaround is to edit /etc/default/grub and add thunderbolt.host_reset=false to GRUB_CMDLINE_LINUX= (or if you can’t boot anymore, edit kernel startup by pressing e at the kernel selection screen and add the parameter there). Credits for this one go to @andreas574 in topic: Fedora 40 returns Btrfs error with Kernels 6.8.9 and 6.8.10. Boots fine with Kernel 6.8.5

20

Added btrfs, filesystem, nvme