I haven’t read through the entire thread yet. But the bug I was hitting has recently been fixed. The update is now in testing. I tried it yesterday and it solved the issue I saw.
I’ll read through this thread in a bit and comment where appropriate. My troubleshooting journey is here, in case it has not been mentioned yet.
Thanks for your input @gui1ty . I’ll test and post the results in this thread as soon as I get my hands on the update.
Having read through the thread now, I’d say the issue is caused by the same bug I hit when upgrading from F37 to F39. The difference for you is, that you have root on an md device, where in my situation only data partitions are on md devices. I could simply comment out the data partitions and boot into the system.
Anyway, since there’s an update with a fixed libblkid available, all you need to do is pull in that update directly from Bodhi with sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-b24d74c260. To be on the safe side I’d follow up with dracut --force to make sure that the fixed libblkid is available in initramfs. After that you should be able to boot without issues again.
I did the dnf and dracut commands. Unfortunately, it did not fix it. I still end up in the grub prompt. Is there any logs that I should be sharing/submitting to?
Maybe it’s a different bug? To confirm (or deny), what’s the output of sudo blkid /dev/sd[ab][124]?
Also, does blkid --version report the correct version? On my system it reports blkid from util-linux 2.39.3 (libblkid 2.39.3, 04-Dec-2023) and rpm -qv util-linux shows util-linux-2.39.3-5.fc39.x86_64 which fixed the issue for me.
One last thing that might be worth in shedding some light is journalctl -xb to look at the boot log. Maybe there’s something else failing.
Here are the results
$ sudo blkid /dev/sd[ab][124]
/dev/sda1: UUID="83ad994f-6eac-8253-a048-1b1d50df417c" UUID_SUB="f1a90a5c-5b55-80fd-6899-e9f76c070a69" LABEL="localhost-live:md0" TYPE="linux_raid_member" PARTUUID="4ee35beb-2cf5-45ac-92fa-07c9721d9868"
/dev/sda2: UUID="8aad2c00-0cc3-da8f-ccd9-ea4e06390a4b" UUID_SUB="cd41891a-82a2-0ab9-56bd-3d0ba4316731" LABEL="localhost-live:md1" TYPE="linux_raid_member" PARTUUID="5456ee0c-22f0-432c-bf8b-6323fa1ff777"
/dev/sda4: UUID="9ca1cb6c-7c05-e51f-19f1-8ea137fa6366" UUID_SUB="7230dd2d-d132-0add-1ddf-45cd26aa998f" LABEL="localhost-live:md2" TYPE="linux_raid_member" PARTUUID="3c57543a-add4-4f80-b8ef-ea9472123707"
/dev/sdb1: UUID="83ad994f-6eac-8253-a048-1b1d50df417c" UUID_SUB="6b823943-e204-ff27-7ca6-57c1a2e31d0b" LABEL="localhost-live:md0" TYPE="linux_raid_member" PARTUUID="6027e4e3-c911-41fe-a1c8-9b9cf933aa3d"
/dev/sdb2: UUID="8aad2c00-0cc3-da8f-ccd9-ea4e06390a4b" UUID_SUB="081fa0cf-48e6-b713-a9ad-b3da6e18a6e6" LABEL="localhost-live:md1" TYPE="linux_raid_member" PARTUUID="4613036e-7917-4acb-b44e-0b9de8bd8fa0"
/dev/sdb4: UUID="9ca1cb6c-7c05-e51f-19f1-8ea137fa6366" UUID_SUB="919fbaaa-6ef8-5b38-14c8-165dbf99fece" LABEL="localhost-live:md2" TYPE="linux_raid_member" PARTUUID="7af91d6e-160b-46eb-bb22-cbbdafa8d6be"
$ blkid --version
blkid from util-linux 2.39.3 (libblkid 2.39.3, 04-Dec-2023)
$ rpm -qv util-linux
util-linux-2.39.3-5.fc39.x86_64
The output of “journalctl -xb” is too long. I’ll paste it here later when I figure out how to do it in a easily readable way.
That was meant for doing some investigation yourself. 
But if you’d like to share it, try journalctl -xb | fpaste from a running system. It will return the URL to the pastebin.
The output of blkid looks correct. It contains the information needed by udev to assemble the array. Do you see the same output running the command in emergency mode - the shell you get upon failed boot?
Has anything changed at all in the boot process? Different error messages or the like?
I just read through the previous posts again and only now realized that you are actually describing two issues:
I only had problems with 2. Have you fixed 1 yet? Or are you still stuck with grub not doing its job?
I just did a reboot and I am still ending up in the grub prompt. After executing boot,
The message in the journalctl -xb that really caught my eyes is
Feb 09 00:17:59 abedpc dracut-initqueue[469]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
Feb 09 00:17:59 abedpc dracut-initqueue[469]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-uuid\x2fae50b88c-e7de-43a4-a573-dc2a1febae9b.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
Feb 09 00:17:59 abedpc dracut-initqueue[469]: [ -e "/dev/disk/by-uuid/ae50b88c-e7de-43a4-a573-dc2a1febae9b" ]
Feb 09 00:17:59 abedpc dracut-initqueue[469]: fi"
Feb 09 00:17:59 abedpc dracut-initqueue[469]: Warning: dracut-initqueue: starting timeout scripts
This goes on for about 500 lines.
After manually assembling the raid devices, the UUID is now found.
Feb 09 00:21:16 abedpc systemd[1]: Found device dev-disk-by\x2duuid-ae50b88c\x2de7de\x2d43a4\x2da573\x2ddc2a1febae9b.device - /dev/disk/by-uuid/ae50b88c-e7de-43a4-a573-dc2a1febae9b.
The pastebin link is UNTITLED - Pastebin Service
Have you re-run your grub2-mkconfig ... command after you got your system back online? (Make sure /boot (and /boot/efi?) is mounted before re-running grub2-mkconfig.)
I’m not entirely sure if the two issues are related or not. To start I would try to treat them separately.
I take it you are able to boot manually by passing the required arguments to grub and subsequently assembling the array manually and finish booting.
For the grub issue, once the system is booted, run:
grub2-install /dev/sda (assuming /dev/sda is your primary hard disk)
grub2-mkconfig -o /boot/grub2/grub.cfg
To further troubleshoot the RAID assembly issue, if I understand you correctly, you are running btrfs on top of md devices for /, /home and /var. In the output of /proc/mdstat, you posted earlier, I see three md devices listed. Can you tell a little bit more about the setup? I’m trying to get a better understanding of it. Actually, the output of lsblk -f should give me a pretty good idea.
Could you also post the output of sudo mdadm --examine --export /dev/sda[124] and the contents of /etc/mdadm.conf, if any?
I think what makes your case more complicated is the fact that you have the root partition on btrfs on top of md raid. The UUID used for mounting the root partition only becomes available after the md raid device containing the btrfs volumes is assembled.
That essentially means the automatic assembly still fails, making the UUID required to mount the btrfs volumes as per /etc/fstab unavailable. But with the updated util-linux that should no longer be an issue, unless the bug causing you trouble is still there. Did you run dracut --force after updating util-linux?
@glb Thank you very much! I forgot about the grub2-mkconfig. I executed
sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
now everything is fine. I can now boot normally and the md raid is being assembled automatically.
@gui1ty Thank you very much!! dracut --force was executed after updating util-linux. Just to share my setup, here are the output of the configs/commands
$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0
0 100% /var/lib/snapd/snap/bare/5
loop1
0 100% /var/lib/snapd/snap/core18/2796
loop2
0 100% /var/lib/snapd/snap/core18/2812
loop3
0 100% /var/lib/snapd/snap/core20/2015
loop4
0 100% /var/lib/snapd/snap/core20/2105
loop5
0 100% /var/lib/snapd/snap/gnome-3-28-1804/198
loop6
0 100% /var/lib/snapd/snap/gtk-common-themes/1535
loop7
0 100% /var/lib/snapd/snap/snapd/20290
loop8
0 100% /var/lib/snapd/snap/snapd/20671
loop9
0 100% /var/lib/snapd/snap/tradingview/48
loop10
0 100% /var/lib/snapd/snap/tradingview/49
sda
├─sda1
│ linux_ 1.0 localhost-live:md0 83ad994f-6eac-8253-a048-1b1d50df417c
│ └─md0
│ vfat FAT16 efi 8E0C-931D 236.6M 7% /boot/efi
├─sda2
│ linux_ 1.2 localhost-live:md1 8aad2c00-0cc3-da8f-ccd9-ea4e06390a4b
│ └─md1
│ ext4 1.0 boot b5499039-a7b5-462b-8d01-8ca15029fd56 161M 61% /boot
├─sda3
│ swap 1 swap 0be0bc6f-0798-4b43-b94c-9ea33765b25c [SWAP]
└─sda4
linux_ 1.2 localhost-live:md2 9ca1cb6c-7c05-e51f-19f1-8ea137fa6366
└─md2
btrfs ae50b88c-e7de-43a4-a573-dc2a1febae9b 241.7G 47% /home
/var
/
sdb
├─sdb1
│ linux_ 1.0 localhost-live:md0 83ad994f-6eac-8253-a048-1b1d50df417c
│ └─md0
│ vfat FAT16 efi 8E0C-931D 236.6M 7% /boot/efi
├─sdb2
│ linux_ 1.2 localhost-live:md1 8aad2c00-0cc3-da8f-ccd9-ea4e06390a4b
│ └─md1
│ ext4 1.0 boot b5499039-a7b5-462b-8d01-8ca15029fd56 161M 61% /boot
├─sdb3
│ swap 1 swap 80d4f8dd-d196-4292-b886-6a953c361310 [SWAP]
└─sdb4
linux_ 1.2 localhost-live:md2 9ca1cb6c-7c05-e51f-19f1-8ea137fa6366
└─md2
btrfs ae50b88c-e7de-43a4-a573-dc2a1febae9b 241.7G 47% /home
/var
/
$ sudo mdadm --examine --export /dev/sda[124]
MD_LEVEL=raid1
MD_DEVICES=2
MD_NAME=localhost-live:md0
MD_ARRAY_SIZE=268.37MB
MD_UUID=83ad994f:6eac8253:a0481b1d:50df417c
MD_UPDATE_TIME=1707434807
MD_DEV_UUID=f1a90a5c:5b5580fd:6899e9f7:6c070a69
MD_EVENTS=972
MD_LEVEL=raid1
MD_DEVICES=2
MD_NAME=localhost-live:md1
MD_ARRAY_SIZE=535.82MB
MD_UUID=8aad2c00:0cc3da8f:ccd9ea4e:06390a4b
MD_UPDATE_TIME=1707434976
MD_DEV_UUID=cd41891a:82a20ab9:56bd3d0b:a4316731
MD_EVENTS=1285
MD_LEVEL=raid1
MD_DEVICES=2
MD_NAME=localhost-live:md2
MD_ARRAY_SIZE=494.87GB
MD_UUID=9ca1cb6c:7c05e51f:19f18ea1:37fa6366
MD_UPDATE_TIME=1707435431
MD_DEV_UUID=7230dd2d:d1320add:1ddf45cd:26aa998f
MD_EVENTS=89093
$ cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md/md0 level=raid1 num-devices=2 UUID=83ad994f:6eac8253:a0481b1d:50df417c
ARRAY /dev/md/md1 level=raid1 num-devices=2 UUID=8aad2c00:0cc3da8f:ccd9ea4e:06390a4b
ARRAY /dev/md/md2 level=raid1 num-devices=2 UUID=9ca1cb6c:7c05e51f:19f18ea1:37fa6366
No it is not fine; (or rather it is only fine until the next kernel update).
That file is only a pointer to redirect grub to the actual grub.cfg file at /boot/grub2/grub.cfg, and has been so since about fedora 32.
After overwriting it as you have done the system will never properly update when a kernel is updated and will be stuck with the current kernel & config.
The fix is to
sudo rm /boot/grub2/grub.cfg /boot/efi/EFI/fedora/grub.cfgsudo dnf reinstall grub2-common grub2-efi-\*In the future when running grub2-mkconfig always direct the output to /boot/grub2/grub.cfg or use one of the /etc/grub2.cfg or /etc/grub2-efi.cfg links to the same file as the target.
and
both apply.
Hi @computersavvy . I did steps 1 and 2. I rebooted to test it out but it dropped me into the grub prompt. In dracut emergency shell, I saw that the md raid devices need to be assembled. After assembling the raid devices, I mounted md2 to sysroot. I exited the dracut emergency shell and booted successfully. I executed
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Unfortunately, I still end up in grub prompt after rebooting.
I checked if the 2 files were recreated by the dnf reinstall and it is verified that they were recreated.
$ sudo ls /boot/grub2/grub.cfg
/boot/grub2/grub.cfg
$ sudo ls /boot/efi/EFI/fedora/grub.cfg
/boot/efi/EFI/fedora/grub.cfg
Is there something else that I should be doing? I am tempted to execute
sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
but I would like my machine to be configured in such a way that indicated fixes in the docs will be applicable.
I’m not an expert when it comes to the modern boot chain. But I’m wondering if your current setup is making things more complicated.
Looking at the output of lsblk -f it appears your EFI partition is on md0 and your boot partition on md1:
sda
├─sda1
│ linux_ 1.0 localhost-live:md0 83ad994f-6eac-8253-a048-1b1d50df417c
│ └─md0
│ vfat FAT16 efi 8E0C-931D 236.6M 7% /boot/efi
├─sda2
│ linux_ 1.2 localhost-live:md1 8aad2c00-0cc3-da8f-ccd9-ea4e06390a4b
│ └─md1
│ ext4 1.0 boot b5499039-a7b5-462b-8d01-8ca15029fd56 161M 61% /boot
Is there a reason for that?
Even your btrfs volume being on an md raid device is unusal since btrfs volumes can span multiple disks natively.
For comparison take a look at the recommended disk layout. Moving the btrfs volume off md is probably taking it too far, but you could consider moving your boot and EFI partitions as recommended. E.g. make /dev/sda1 the EFI partition (vfat) and /dev/sda2 the boot partition (ext4). That’s basically removing the two md devices (md0 and md1) below it[1].
Just a caution: you want to think that through, before setting out on that endeavor. Best to make a plan and have someone look over it. ↩︎
I was wondering the same thing on the md{0,1}. Unfortunately, years ago when I am fresh installing my system, there were very limited documentation on efi + raid 1. I was experimenting on installing it on raid 1 and back then the results were good. I could pull out either drive and it will still be running. I also tried running the build-in raid 1 of btrfs but unfortunately it needed 3 drives. 2 will be the live and 1 is the hot spare. It was my first time using ssd and all resources that I read was pointing to btrfs as the best for ssd. I came across a thread that they too were using md raid 1 instead of the built-in raid of btrfs because md raid is more mature.
I am currently researching on how to go about moving my sd{a,b}{1,2} out of md raid.
The steps are as follows (as a starting point):
/dev/sda1 from md0 with mdadm --fail --remove /dev/md/md0 /dev/sda1/dev/sda2 from md1 with mdadm --fail --remove /dev/md/md1 /dev/sda2The arrays are now degraded, but that’s fine. You’ll need the data on it to copy them over later.
/dev/sda1 with mdadm --zero-superblock /dev/sda1/dev/sda2 with mdadm --zero-superblock /dev/sda2This is to avoid accidents mostly. Since you are going to reformat, they will be overwritten eventually anyway.
0b (W95 FAT32) (not entirely sure about the type)83 (Linux)/dev/sda1 as vfat/dev/sdb2 as ext4Now you can copy over the data from the array to the partitions.
dd if=/dev/md/md0 of=/dev/sda1dd if=/dev/md/md1 of=/dev/sda2Add the newly created partitions to /etc/fstab. You can either use the UUIDs or the partitions, whatever you prefer. Just make sure that /dev/sda2 is mounted on /boot and /dev/sda1 on /boot/efi.
Now mount the partitions and re-run the grub installation. I believe you will also need to re-run dracut --force (at least it won’t hurt doing so).
After confirming that booting works without issues, you could proceed with first commenting out the mount options for the btrfs volumes on md0 and md1, test everything keeps working and then nuke /dev/sdb1 and /dev/sdb2 by performing steps 1 to 4 for these partitions. After all that is done you may need to update your /etc/mdadm.conf.
All of the above, while taking care of avoiding errors, comes without warranties. Make sure you understand the steps. Don’t hesitate asking if unsure.
I really appreciate the instructions @gui1ty !
I just have some questions/clarifications.
1. Remove /dev/sda1 from md0 with mdadm --fail --remove /dev/md/md0 /dev/sda1
2. Remove /dev/sdb1 from md1 with mdadm --fail --remove /dev/md/md1 /dev/sdb1
Do you mean
1. Remove /dev/sda1 from md0 with mdadm --fail --remove /dev/md0 /dev/sda1
2. Remove /dev/sda2 from md1 with mdadm --fail --remove /dev/md1 /dev/sda2
I am thinking that were are removing /dev/sda{1,2} from the /dev/md{0,1} and just leaving /dev/sdb{1,2} in /dev/md{0,1}.
Then
3. Wipe the superblock from /dev/sda1 with mdadm --zero-superblock /dev/sda1
4. Wipe the superblock from /dev/sdb1 with mdadm --zero-superblock /dev/sdb1
Do you mean
3. Wipe the superblock from /dev/sda1 with mdadm --zero-superblock /dev/sda1
4. Wipe the superblock from /dev/sda2 with mdadm --zero-superblock /dev/sda2
I am thinking we are wiping the superblock of the removed /dev/sda{1,2}.
Then
5. Change partition type of sda1 to 0b (W95 FAT32) (not entirely sure about the type)
6. Change partition type of sdb1 to 83 (Linux)
7. Format /dev/sda1 as vfat
8. Format /dev/sdb2 as ext4
Should that be
5. Change partition type of sda1 to 0b (W95 FAT32) (not entirely sure about the type)
6. Change partition type of sda2 to 83 (Linux)
7. Format /dev/sda1 as vfat
8. Format /dev/sda2 as ext4
I am thinking we are changing the partition type and formatting of the removed /dev/sda{1,2}
Then
9. Copy EFI partition with dd if=/dev/md/md0 of=/dev/sda1
10. Copy boot partition with dd if=/dev/md/md1 of=/dev/sdb1
Do you mean
9. Copy EFI partition with dd if=/dev/md0 of=/dev/sda1
10. Copy boot partition with dd if=/dev/md1 of=/dev/sda2
I am thinking we are copying the data of /dev/sdb{1,2} which are still in /dev/md{0,1} to the removed /dev/sda{1,2}.
Then
Add the newly created partitions to /etc/fstab. You can either use the UUIDs or the partitions, whatever you prefer. Just make sure that /dev/sdb1 is mounted on /boot and /dev/sda1 on /boot/efi.
Do you mean
Replace the UUID values of /boot/efi and /boot in /etc/fstab with the UUID of /dev/sda1 and /dev/sda2 respectively.
I am thinking adding the new entries instead of replacing them would cause conflicts because there would be 2 entries of /boot/efi and /boot in /etc/fstab. Would that be the case? I haven’t done that so having 2 different partitions mounted on the same mount point is new to me.
Mmm, I don’t think that will work. /dev/md/md1 is 535.82MB but /dev/sdb1 is only 268.37MB.
Edit: If you have space for it somewhere, I’d use the partclone command to make a temporary backup of both md0 and md1 before doing anything (even if the only place you have to but it is on your third RAID device).
My apologies. I screwed up there. i’ll edit it my post fixing the instructions.
If that is the case (haven’t looked yet) then it will not work indeed. But aren’t those RAID1 mirrors?