I encountered an issue on my Fedora installation where systemd reported that a module failed to load, and journalctl mentioned errors related to snd. While attempting to fix this, I inadvertently corrupted my kernel, causing the system to go into a kernel panic immediately upon boot.
To recover, I booted into a live environment using a USB stick and mounted my Linux root, boot, and other necessary partitions. I then chrooted into the system using the following commands:
sudo mount -o subvol=root /dev/nvme0n1p4 /mnt
sudo mount /dev/nvme0n1p3 /mnt/boot
sudo mount /dev/nvme0n1p1 /mnt/boot/efi
sudo mount -t proc /proc /mnt/proc
sudo mount --rbind /dev /mnt/dev
sudo mount --make-rslave /mnt/dev
sudo mount --rbind /sys /mnt/sys
sudo mount --make-rslave /mnt/sys
sudo mount --rbind /run /mnt/run
sudo mount --make-rslave /mnt/run
sudo chroot /mnt
After chrooting, I reinstalled the kernel, kernel modules, and kernel core, and then rebuilt initramfs and the GRUB configuration. This resolved the kernel panic, allowing the system to reach the Fedora loading screen.
However, the system now gets stuck at starting the drive for root specified by GRUB entry for Fedora. While searching for a solution, I came across this guide, but I have been unable to proceed because I cannot open a TTY or run the rpm command.
Was this happening with all of the available kernels, i.e. you couldn’t boot with either of the available GRUB entries?
Do you mean you can see the bootsplash (Plymouth), bot don’t reach the login screen? You could change the boot process to verbose, by pressing Esc while booting, or by editing the GRUB menu and removing rhgb quiet. You might see some hints there.
The grub2-mkconfig command from the guide you have followed might be (part of) the issue. AFAIK, the correct command is grub2-mkconfig -o /boot/grub2/grub.cfg.
Also, are you able to provide a few command outputs (from a live session if needed)?
lsblk
fdisk -l
cat /etc/fstab (or /mnt/etc/fstab if drives mounted in a live session w/o chroot)
inxi -Fzxx
grubby --info=ALL (might work from a live session with chrooting, not sure).
I not only corrupted the kernel but also accidentally removed the rescue boot entry. This left me with just one kernel, which was panicking every time I tried to boot. Unfortunately, I had already cleared the older kernel versions while attempting to fix the systemd module loading issue, so I had no fallback option.
Yes, I was referring to the bootsplash/Plymouth screen. As I mentioned earlier, the boot process gets stuck at starting the drive for root showing something like by-uuid/xxx-x-x-x.device/ start (*s/no limit)
In my previous experiences, both this command and grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg had worked for me. So, I executed both commands this time to be safe.
Well for ixni -Fzxx it said command not found. The rest outputs are as follows:
lsblk >
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 2G 1 loop
loop1 7:1 0 8G 1 loop
├─live-rw 253:1 0 8G 0 dm
└─live-base 253:2 0 8G 1 dm
loop2 7:2 0 32G 0 loop
└─live-rw 253:1 0 8G 0 dm
sda 8:0 0 931.5G 0 disk
└─sda1 8:1 0 931.5G 0 part
└─bitlk-471e7c38-d58c-4dc4-9f06-3e88dde40bca 253:3 0 931.5G 0 crypt /run/media/liveuser/Research
sdb 8:16 1 28.7G 0 disk
├─sdb1 8:17 1 28.6G 0 part
└─sdb2 8:18 1 32M 0 part /run/media/liveuser/VTOYEFI
zram0 252:0 0 8G 0 disk [SWAP]
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 116M 0 part /boot/efi
├─nvme0n1p2 259:2 0 674.1G 0 part
├─nvme0n1p3 259:3 0 1G 0 part /boot
├─nvme0n1p4 259:4 0 255G 0 part /home
│ /
├─nvme0n1p5 259:5 0 701M 0 part
└─nvme0n1p6 259:6 0 634.7M 0 part
fdisk -l >
Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: WDC WD10EZEX-60W
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 0CB04F23-18ED-9722-F06B-55BA8FB0633A
Device Start End Sectors Size Type
/dev/sda1 34 1953525133 1953525100 931.5G Microsoft basic data
Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: CT1000P3SSD8
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A50D7FD4-BA05-46ED-B331-4FD68B271987
Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 239615 237568 116M EFI System
/dev/nvme0n1p2 239616 1413916671 1413677056 674.1G Microsoft basic data
/dev/nvme0n1p3 1413916672 1416013823 2097152 1G Linux extended boot
/dev/nvme0n1p4 1416013824 1950789631 534775808 255G Linux filesystem
/dev/nvme0n1p5 1950789632 1952225279 1435648 701M Windows recovery environment
/dev/nvme0n1p6 1952225280 1953525103 1299824 634.7M Windows recovery environment
Disk /dev/sdb: 28.67 GiB, 30784094208 bytes, 60125184 sectors
Disk model: SanDisk 3.2Gen1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E3D5BD8A-23F4-43DF-AF10-995D3E9477C3
Device Start End Sectors Size Type
/dev/sdb1 2048 60059607 60057560 28.6G Microsoft basic data
/dev/sdb2 60059608 60125143 65536 32M Microsoft basic data
Disk /dev/dm-0: 2.14 GiB, 2295853056 bytes, 4484088 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: BD0CE27D-A6E0-4C37-BEEC-AB4B6F9B6A73
Device Start End Sectors Size Type
/dev/dm-0p1 64 4457875 4457812 2.1G Microsoft basic data
/dev/dm-0p2 4457876 4483423 25548 12.5M EFI System
/dev/dm-0p3 4483424 4484023 600 300K Microsoft basic data
Disk /dev/loop0: 1.97 GiB, 2110570496 bytes, 4122208 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop1: 8 GiB, 8592031744 bytes, 16781312 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop2: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/live-rw: 8 GiB, 8592031744 bytes, 16781312 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/live-base: 8 GiB, 8592031744 bytes, 16781312 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/zram0: 8 GiB, 8589934592 bytes, 2097152 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/mapper/bitlk-471e7c38-d58c-4dc4-9f06-3e88dde40bca: 931.51 GiB, 1000204851200 bytes, 1953525100 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Alignment offset: 3072 bytes
Disklabel type: dos
Disk identifier: 0x69205244
Device Boot Start End Sectors Size Id Type
/dev/mapper/bitlk-471e7c38-d58c-4dc4-9f06-3e88dde40bca-part1 218129509 1920119918 1701990410 811.6G 72 unknown
/dev/mapper/bitlk-471e7c38-d58c-4dc4-9f06-3e88dde40bca-part2 729050177 1273024900 543974724 259.4G 74 unknown
/dev/mapper/bitlk-471e7c38-d58c-4dc4-9f06-3e88dde40bca-part3 168653938 168653938 0 0B 65 Novell Netware 386
/dev/mapper/bitlk-471e7c38-d58c-4dc4-9f06-3e88dde40bca-part4 2692939776 2692991410 51635 25.2M 0 Empty
Partition table entries are not in disk order.
cat /etc/fstab >
#
# /etc/fstab
# Created by anaconda on Wed Jul 31 07:38:02 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
UUID=e1e8be88-90bf-4616-b644-8b62c42e4fcb / btrfs subvol=root,compress=zstd:1 0 0
UID=4ece65c0-962b-4248-b35b-1de09ead3334 /boot ext4 defaults 1 2
UUID=707F-5BD8 /boot/efi vfat umask=0077,shortname=winnt 0 2
UUID=e1e8be88-90bf-4616-b644-8b62c42e4fcb /home btrfs subvol=home,compress=zstd:1 0 0
# Bitlocker Drives
# /dev/mapper/research /mnt/Research ntfs nosuid,nodev,nofail,x-gvfs-show 0 0
# /dev/mapper/windows /mnt/569087BF9087A3D9 ntfs nosuid,nodev,nofail,x-gvfs-show 0 0
grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg is obsolete. Here:
% doas head /boot/grub2/grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
### BEGIN /etc/grub.d/00_header ###
set pager=1
F40 uses a fixed stub file that points to the above location for grub.cfg.
% doas cat /boot/efi/EFI/fedora/grub.cfg
search --no-floppy --fs-uuid --set=dev b831937b-259e-46b4-a9cd-c4fb22c1c072
set prefix=($dev)/grub2
export $prefix
configfile $prefix/grub.cfg
I don’t think this will hurt while the two grub.cfg files are the same, but only /boot/grub2/grub.cfg will change when the kernel is upgraded, at which point the mistake does matter.
It is not included in the Live Environment, but should offer to install it.
Did you try booting with all of the 3 entries, with the same result? I have noticed a difference in the id parameter between the GRUB entries with index 0 and 1.
I think this was the correct command for fedora on uefi boot systems up until about the release of fedora 34. Since that time fedora only modifies the /boot/grub2/grub.cfg file and the file in /boot/efi/EFI/fedora/grub.cfg is supposed to be a stub pointer file that redirects grub to the file in /boot/grub2 as noted.
While running the grub2-mkconfig command and altering that file does not break the system immediately, it will never be updated during system updates so the file remains static and does not reflect any boot or kernel updates from that time on without manually updating it.
To restore the automatic configuration the simplest way there are 2 steps.
sudo rm /boot/grub2/grub.cfg /boot/efi/EFI/fedora/grub.cfg to remove the misconfigured files.
sudo dnf reinstall grub2-common to recreate the files properly
reboot.
After running step 2 the result of cat /boot/efi/EFI/fedora/grub.cfg should show something like this, where the UUID at the end of the first line will be the one for your mounted /boot partition as is shown in fstab.
Well that’s because I did that while testing, trying to get this working.
I did and the problem is similar to what I am having (as in same error) but I think reasons might be different.
Because for me there is no resume=UUID=<uuid> but rather it gets stuck on starting the root partition.
One thing that might be worth mentioning, according to the screenshot it is trying to access /dev/disks/by-uuid but when in maintenance mode I did ls /dev/disks/by-uuid it said no directory or file like that exists.
That’s probably because of the typo (disks vs disk).
From what I have seen elsewhere, the boot process not finding the disk by uuid can be caused by a failing disk generating delay in the disk search sequence. This isn’t probably your case though, since the issue appeared while troubleshooting the system.
Maybe initramfs didn’t get built correctly. You might wanna try again, this time using the correct grub2-mkconfig command (including the cleanup steps detailed by Jeff above)?
I am not sure, but maybe this mount subtree operation was wrong, and sudo mount --make-rshare /mnt/devshould have been correct when chrooting? I am guessing here, so please read the documentation and make backups before trying it.
Other workarounds that might work, could be to change the root= boot flags to boot by part-uuid or similar, given that those folders are there, or simply by creating the symlinks below /dev/disk/by-uuid/ manually.
As you suggested sudo mount --make-rshare /mnt/dev, I tried it instead of --make-rslave and then dnf reinstall kernel kernel-core kernel-modules -y > rebuild initramfs > grub2-mkconfig -o /boot/grub2/grub.cfg.
But the issue persisted.
But just to test I rebooted into my live environment and I ran all the commands mentioned above again but replaced dnf reinstall kernel kernel-core kernel-modules -y with dnf reinstall kernel kernel-*
and yippee this fixed my issue. I guess the issue was other parts of kernel which manage devices were also corrupted not just kernel, kernel-core, kernel-modules
