Kernel initram vmlinuz and grub all out of wack

Our Neighbors Asahi Linux Ask Asahi
, ,
1

I updated to 42 today and things were really laggy, slow and lots of apps wouldnt start. I peeked at the kernel version at it was ancient 6.9.12-401.asahi.fc40.aarch64+16k. Not sure how it got that old but probably from an old attempt to add the special module for hdmi displaylink and ever since the new kernels were just failing to install after failing to compile the module. So, I deleted the evdi module, reinstalled the latest kernel 6.14.4-400.asahi.fc42.aarch64+16k and rebooted but it didnt switch and wasnt in the grub menu. So I tried dracut and grubmkconfig but alas, nothing seemed to work. What finally got me booted in to a working system was copying the initram and vmlinuz files from /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k
over the top of the two current files in /boot

So grub says im bootin 6.9.12 but its actually booting 6.14.4

its like yum install kernel, dracut and grub config arent doing their automated thing.

How do I fix this mess?

Adding an image of some /boot files

kernel-issues
kernel-issues1453×989 204 KB

2

Ive got it all cleaned up manually but it still does not explain why I had to copy the initram, and vmlinuz files to /boot and the .conf to /boot/loader/entries and edit the path in the .conf

image
image1625×877 165 KB

I guess we will see what happens when the next kernel tries to install with an update?

3

I’m not sure what triggered this but I think the omission of GRUB_ENABLE_BLSCFG=true in /etc/default/grub might be a contributing factor to things going of the rails fully. This is an inadvertent omission triggered by a bug in the image generation tool in images generated since May 2024 (I haven’t checked when we started using those images in the production installer).

If GRUB_ENABLE_BLSCFG=true is missing please do not add it yourself as we plan to deploy a script which fixes this and other issues. After PR#21: grub2 installtion and config fixups - fedora-asahi/asahi-platform-metapackage - Pagure.io is merged and the package is updated FAR_grub2_config_fixup.sh will be installed.
sudo FAR_grub2_config_fixup.sh should then perform these fixes. If you’re comfortable with it feel free to grab it from Tree - fedora-asahi/asahi-platform-metapackage - Pagure.io and test it before hand.

To test changes and to debug the kernel install issue you can run sudo kernel-install -v add 6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz. This replicates what happens during kernel updates. The verbose output might be helpful in figuring out what goes wrong. Testing and hopefully fixing it now has the advantage of not needing to waiting for the next kernel update and forgetting details of this issue.

4

Thank you Janne!

ebable bls was already in default/grub, its last write time was yesterday but wasnt me.
I created and ran the fixup

If it matters I think this install started from 38 beta.


root@makukumba:/bin# ./FAR_grub2_config_fixup.sh 
Adjust grub defaults to Fedora defaults
Regenerating "/boot/grub2/grub.cfg"
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done
Adding "menu_auto_hide=1" to "/boot/grub2/grubenv"
root@makukumba:/bin# cat /boot/grub2/grubenv 
# GRUB Environment Block
# WARNING: Do not edit this file by tools other than grub-editenv!!!
boot_success=1
boot_indeterminate=0
saved_entry=019b1f1ad4a74447933047f0280fa6f2-6.9.12-401.asahi.fc40.aarch64+16k
menu_auto_hide=1
##############################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################root@makukumba:/bin# 
root@makukumba:/bin# cat /etc/default/grub
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_DISABLE_RECOVERY=true
GRUB_DISABLE_OS_PROBER=true
GRUB_CMDLINE_LINUX_DEFAULT="rhgb quiet rootflags=subvol=root"
GRUB_DISTRIBUTOR="Fedora Linux Asahi Remix"
GRUB_ENABLE_BLSCFG=true
GRUB_TERMINAL="console"
GRUB_TIMEOUT=5
root@makukumba:/bin#

Then I tried the kernel install command and it gave some dire looking warnings at the begining but ended in success:


root@makukumba:/tmp# kernel-install -v add 6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Loaded config.
MACHINE_ID=019b1f1ad4a74447933047f0280fa6f2 set via /etc/machine-id.
Found container virtualization none.
/dev/nvme0n1p5: Partition has wrong PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4 for XBOOTLDR partition.
Couldn't find an XBOOTLDR partition.
Failed to check file system type of "/efi": No such file or directory
File system "/boot" is not a FAT EFI System Partition (ESP) file system.
Using EFI System Partition at /boot/efi as $BOOT_ROOT.
Using entry token: 019b1f1ad4a74447933047f0280fa6f2
kernel version (6.14.4-400.asahi.fc42.aarch64+16k) set via command line.
kernel image file (/lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz) set via command line.
/boot/efi/019b1f1ad4a74447933047f0280fa6f2 exists, using layout=bls.
Using ENTRY_DIR=/boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k
mkdir -p /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k
Using plugins: 
  /usr/lib/kernel/install.d/10-devicetree.install
  /usr/lib/kernel/install.d/20-grub.install
  /usr/lib/kernel/install.d/40-dkms.install
  /usr/lib/kernel/install.d/50-depmod.install
  /usr/lib/kernel/install.d/50-dracut.install
  /usr/lib/kernel/install.d/60-kdump.install
  /usr/lib/kernel/install.d/90-loaderentry.install
  /usr/lib/kernel/install.d/90-uki-copy.install
  /usr/lib/kernel/install.d/92-crashkernel.install
  /usr/lib/kernel/install.d/92-tuned.install
  /usr/lib/kernel/install.d/95-kernel-hooks.install
  /usr/lib/kernel/install.d/99-grub-mkconfig.install
Plugin environment: 
  LC_COLLATE=C.UTF-8
  KERNEL_INSTALL_VERBOSE=1
  KERNEL_INSTALL_IMAGE_TYPE=pe
  KERNEL_INSTALL_MACHINE_ID=019b1f1ad4a74447933047f0280fa6f2
  KERNEL_INSTALL_ENTRY_TOKEN=019b1f1ad4a74447933047f0280fa6f2
  KERNEL_INSTALL_BOOT_ROOT=/boot/efi
  KERNEL_INSTALL_LAYOUT=bls
  KERNEL_INSTALL_INITRD_GENERATOR=
  KERNEL_INSTALL_UKI_GENERATOR=
  KERNEL_INSTALL_STAGING_AREA=/tmp/kernel-install.staging.aav4ci
Plugin arguments: add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(sd-exec-strv)' as PID 8647.
About to execute /usr/lib/kernel/install.d/10-devicetree.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 8648.
/usr/lib/kernel/install.d/10-devicetree.install succeeded.
About to execute /usr/lib/kernel/install.d/20-grub.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 8653.
/usr/lib/kernel/install.d/20-grub.install succeeded.
About to execute /usr/lib/kernel/install.d/40-dkms.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 8654.
/usr/lib/kernel/install.d/40-dkms.install succeeded.
About to execute /usr/lib/kernel/install.d/50-depmod.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 8698.
+depmod -a 6.14.4-400.asahi.fc42.aarch64+16k
/usr/lib/kernel/install.d/50-depmod.install succeeded.
About to execute /usr/lib/kernel/install.d/50-dracut.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 8699.
dracut[I]: Executing: /usr/bin/dracut -f --verbose --kernel-image /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz --no-uefi --kver 6.14.4-400.asahi.fc42.aarch64+16k /tmp/kernel-install.staging.aav4ci/initrd
dracut[I]: *** Including module: bash ***
dracut[I]: *** Including module: shell-interpreter ***
dracut[I]: *** Including module: systemd ***
dracut[I]: *** Including module: fips ***
dracut[I]: *** Including module: fips-crypto-policies ***
dracut[I]: *** Including module: systemd-ask-password ***
dracut[I]: *** Including module: systemd-battery-check ***
dracut[I]: *** Including module: systemd-initrd ***
dracut[I]: *** Including module: systemd-journald ***
dracut[I]: *** Including module: systemd-modules-load ***
dracut[I]: *** Including module: systemd-sysctl ***
dracut[I]: *** Including module: systemd-sysusers ***
dracut[I]: *** Including module: systemd-tmpfiles ***
dracut[I]: *** Including module: systemd-udevd ***
dracut[I]: *** Including module: nss-softokn ***
dracut[I]: *** Including module: i18n ***
dracut[I]: *** Including module: drm ***
dracut[I]: *** Including module: plymouth ***
dracut[I]: *** Including module: btrfs ***
dracut[I]: *** Including module: kernel-modules ***
dracut[I]: *** Including module: kernel-modules-extra ***
dracut[I]: *** Including module: kernel-modules-asahi ***
dracut[I]: *** Including module: rootfs-block ***
dracut[I]: *** Including module: terminfo ***
dracut[I]: *** Including module: udev-rules ***
dracut[I]: *** Including module: dracut-systemd ***
dracut[I]: *** Including module: usrmount ***
dracut[I]: *** Including module: asahi-firmware ***
dracut[I]: *** Including module: base ***
dracut[I]: *** Including module: fs-lib ***
dracut[I]: *** Including module: memstrack ***
dracut[I]: *** Including module: openssl ***
dracut[I]: *** Including module: shutdown ***
dracut[I]: *** Including modules done ***
dracut[I]: *** Installing kernel module dependencies ***
dracut[I]: *** Installing kernel module dependencies done ***
dracut[I]: *** Resolving executable dependencies ***
dracut[I]: *** Resolving executable dependencies done ***
dracut[I]: *** Hardlinking files ***
dracut[I]: *** Hardlinking files done ***
dracut[I]: *** Store current command line parameters ***
dracut[I]: *** Creating image file '/tmp/kernel-install.staging.aav4ci/initrd' ***
dracut[I]: Using auto-determined compression method 'pigz'
dracut[I]: *** Creating initramfs image file '/tmp/kernel-install.staging.aav4ci/initrd' done ***
/usr/lib/kernel/install.d/50-dracut.install succeeded.
About to execute /usr/lib/kernel/install.d/60-kdump.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 18568.
/usr/lib/kernel/install.d/60-kdump.install succeeded.
About to execute /usr/lib/kernel/install.d/90-loaderentry.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 18569.
Installing /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k/initrd
Creating /boot/efi/loader/entries/019b1f1ad4a74447933047f0280fa6f2-6.14.4-400.asahi.fc42.aarch64+16k.conf
/usr/lib/kernel/install.d/90-loaderentry.install succeeded.
About to execute /usr/lib/kernel/install.d/90-uki-copy.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 18581.
/usr/lib/kernel/install.d/90-uki-copy.install succeeded.
About to execute /usr/lib/kernel/install.d/92-crashkernel.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 18582.
/usr/lib/kernel/install.d/92-crashkernel.install succeeded.
About to execute /usr/lib/kernel/install.d/92-tuned.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 18591.
/usr/lib/kernel/install.d/92-tuned.install succeeded.
About to execute /usr/lib/kernel/install.d/95-kernel-hooks.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 18592.
/usr/lib/kernel/install.d/95-kernel-hooks.install succeeded.
About to execute /usr/lib/kernel/install.d/99-grub-mkconfig.install add 6.14.4-400.asahi.fc42.aarch64+16k /boot/efi/019b1f1ad4a74447933047f0280fa6f2/6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
Successfully forked off '(exec-inner)' as PID 18593.
/usr/lib/kernel/install.d/99-grub-mkconfig.install succeeded.
(sd-exec-strv) succeeded.
root@makukumba:/tmp#

looks like it did edit the boot config just now:

root@makukumba:/boot/loader/entries# ls -l
total 4
-rwx------. 1 root root 460 May  3 10:31 019b1f1ad4a74447933047f0280fa6f2-6.14.4-400.asahi.fc42.aarch64+16k.conf
root@makukumba:/boot/loader/entries# cat 019b1f1ad4a74447933047f0280fa6f2-6.14.4-400.asahi.fc42.aarch64+16k.conf 
# Boot Loader Specification type#1 entry
# File created by /usr/lib/kernel/install.d/90-loaderentry.install (systemd 257.5-2.fc42)
title      Fedora Linux Asahi Remix 42 (Forty Two [Adams])
version    6.14.4-400.asahi.fc42.aarch64+16k
machine-id 019b1f1ad4a74447933047f0280fa6f2
sort-key   fedora-asahi-remix
options root=UUID=86ad63d7-5618-4548-8ba8-78d68efa4738 ro rootflags=subvol=root  rhgb quiet rootflags=subvol=root
linux      /linux
initrd     /initrd
root@makukumba:/boot/loader/entries#

but I cant tell what it modified, looks the same to me though its encouraging that it touched that file as it wasnt being created before. weird to note the initrd and linux are still there named vanilla like a created them.

5

It’s not intended to that /boot/efi/019b1f1ad4a74447933047f0280fa6f2 (that’s the machine-id) exists. It should be enough to remove it and then run kernel-install again. See F39 kernels fail to install when /boot/efi/$machineid is present for general fedora discussion on the topic. The reason stated there doesn’t apply. Was that maybe something stated in the displaylink installation instructions? Please not the create date of /boot/efi/019b1f1ad4a74447933047f0280fa6f2

1 Like
6

deleted /boot/efi/019b1f1ad4a74447933047f0280fa6f2
ran kernel-install -v add 6.14.4-400.asahi.fc42.aarch64+16k /lib/modules/6.14.4-400.asahi.fc42.aarch64+16k/vmlinuz
.conf now looks good and new initram and vmlinuz got created with kernel name

I bet you are right, the dkms kernel module build process for evdi probably created that disk id dir back when I was on fc38 and I never noticed that new kernels werent installing during updates.

Thank you, hopefuilly this helps anyone else, I think there were many who did the evdi build or at least tried and might have unexpected /boot configs.