Grub2 missing kernel entries on UEFI system

[grzegorzb (Grzegorz)]

Hi,

I have a strange issue with my grub2 installation, grub boot list does not show entries for installed kernels (only one kernel is available: 6.7.3-200.fc39.x86_64). It’s looks like that that entries for the new kernels were created under /boot/efi/loader/entries/ but system takes under consideration only entries from /boot/loader/entries/. I already tried reinstall grub itself using this instruction but without any effect. I will be grateful for any suggestions.

root@hp840:/home/grzegorz# grubby --info=ALL
index=0
kernel="/boot/vmlinuz-6.7.3-200.fc39.x86_64"
args="ro rd.luks.uuid=luks-736ff1d3-b55e-4f65-a892-fb3ddf84876f rd.lvm.lv=rootvg/rootlv rhgb quiet"
root="/dev/mapper/rootvg-rootlv"
initrd="/boot/initramfs-6.7.3-200.fc39.x86_64.img $tuned_initrd"
title="Fedora Linux (6.7.3-200.fc39.x86_64) 39 (Thirty Nine)"
id="fb3c1fe83e654cd69a64e0256d9580a9-6.7.3-200.fc39.x86_64"
index=1
kernel="/boot/vmlinuz-0-rescue-fb3c1fe83e654cd69a64e0256d9580a9"
args="ro rd.luks.uuid=luks-736ff1d3-b55e-4f65-a892-fb3ddf84876f rd.lvm.lv=rootvg/rootlv rhgb quiet"
root="/dev/mapper/rootvg-rootlv"
initrd="/boot/initramfs-0-rescue-fb3c1fe83e654cd69a64e0256d9580a9.img"
title="Fedora Linux (0-rescue-fb3c1fe83e654cd69a64e0256d9580a9) 35 (Workstation Edition)"
id="fb3c1fe83e654cd69a64e0256d9580a9-0-rescue"

root@hp840:/home/grzegorz# ls -al /boot/efi/loader/entries/
total 32
drwx------. 2 root root 4096 Mar 24 23:14 .
drwx------. 3 root root 4096 Mar 24 22:57 ..
-rwx------. 1 root root  453 Mar 24 22:43 fb3c1fe83e654cd69a64e0256d9580a9-0-rescue.conf
-rwx------. 1 root root  655 Mar 24 22:43 fb3c1fe83e654cd69a64e0256d9580a9-6.7.5-200.fc39.x86_64.conf
-rwx------. 1 root root  655 Mar 24 22:43 fb3c1fe83e654cd69a64e0256d9580a9-6.7.6-200.fc39.x86_64.conf
-rwx------. 1 root root  655 Mar 24 22:43 fb3c1fe83e654cd69a64e0256d9580a9-6.7.7-200.fc39.x86_64.conf
-rwx------. 1 root root  610 Mar 24 23:18 fb3c1fe83e654cd69a64e0256d9580a9-6.7.9-200.fc39.x86_64.conf
-rwx------. 1 root root  610 Mar 24 23:01 fb3c1fe83e654cd69a64e0256d9580a9-6.8.1-201.fc39.x86_64.conf

root@hp840:/home/grzegorz# ls -al /boot/loader/entries/
total 16
drwx------. 2 root root 4096 Mar 25 17:03 .
drwxr-xr-x. 3 root root 4096 Mar  5  2022 ..
-rw-r--r--. 1 root root  457 Mar 25 17:03 fb3c1fe83e654cd69a64e0256d9580a9-0-rescue.conf
-rw-r--r--. 1 root root  383 Mar 25 17:03 fb3c1fe83e654cd69a64e0256d9580a9-6.7.3-200.fc39.x86_64.conf

root@hp840:/home/grzegorz# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-736ff1d3-b55e-4f65-a892-fb3ddf84876f rd.lvm.lv=rootvg/rootlv rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

[computersavvy (Jeff V)]

please post the following.
sudo sh -c "ls -lR /boot/"

edit – command typo fixed

[grzegorzb (Grzegorz)]

@computersavvy

sudo sh -c “ls -lr /boot/”

Here you go:

root@hp840:/home/grzegorz# sudo sh -c "ls -lr /boot/"
total 165416
-rwxr-xr-x. 1 root root 14782280 Jan 31 01:00 vmlinuz-6.7.3-200.fc39.x86_64
-rwxr-xr-x. 1 root root 11036144 Mar  5  2022 vmlinuz-0-rescue-fb3c1fe83e654cd69a64e0256d9580a9
-rw-r--r--. 1 root root  8849769 Jan 31 01:00 System.map-6.7.3-200.fc39.x86_64
lrwxrwxrwx. 1 root root       45 Mar 24 22:07 symvers-6.8.1-201.fc39.x86_64.xz -> /lib/modules/6.8.1-201.fc39.x86_64/symvers.xz
lrwxrwxrwx. 1 root root       45 Mar 13 13:24 symvers-6.7.9-200.fc39.x86_64.xz -> /lib/modules/6.7.9-200.fc39.x86_64/symvers.xz
lrwxrwxrwx. 1 root root       45 Mar  4 18:05 symvers-6.7.7-200.fc39.x86_64.xz -> /lib/modules/6.7.7-200.fc39.x86_64/symvers.xz
lrwxrwxrwx. 1 root root       45 Feb 28 18:47 symvers-6.7.6-200.fc39.x86_64.xz -> /lib/modules/6.7.6-200.fc39.x86_64/symvers.xz
lrwxrwxrwx. 1 root root       45 Feb 24 11:22 symvers-6.7.5-200.fc39.x86_64.xz -> /lib/modules/6.7.5-200.fc39.x86_64/symvers.xz
lrwxrwxrwx. 1 root root       45 Feb  9 15:04 symvers-6.7.3-200.fc39.x86_64.xz -> /lib/modules/6.7.3-200.fc39.x86_64/symvers.xz
drwx------. 2 root root    16384 Mar  5  2022 lost+found
drwxr-xr-x. 3 root root     4096 Mar  5  2022 loader
-rw-------. 1 root root 42105336 Feb  9 15:04 initramfs-6.7.3-200.fc39.x86_64.img
-rw-------. 1 root root 92302035 Mar  5  2022 initramfs-0-rescue-fb3c1fe83e654cd69a64e0256d9580a9.img
drwx------. 3 root root     4096 Mar 25 17:09 grub2
drwxr-xr-x. 2 root root     4096 Sep 19  2023 extlinux
drwx------. 6 root root     4096 Jan  1  1970 efi
-rw-r--r--. 1 root root   269368 Jan 31 01:00 config-6.7.3-200.fc39.x86_64

root@hp840:/home/grzegorz# mount |grep boot
/dev/nvme0n1p2 on /boot type ext4 (rw,relatime,seclabel)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)

[computersavvy (Jeff V)]

That did not respond as I would have expected. It should have recursively parsed and reported each subdirectory of /boot as well.
What shell are you using? echo $SHELL

Mine gives

$ sudo sh -c "ls -lR /boot/"
/boot/:
total 459300
-rw-r--r--. 1 root root    269378 Feb 22 18:00 config-6.7.6-200.fc39.x86_64
-rw-r--r--. 1 root root    269378 Feb 29 18:00 config-6.7.7-200.fc39.x86_64
-rw-r--r--. 1 root root    269378 Mar  5 18:00 config-6.7.9-200.fc39.x86_64
drwx------. 4 root root     16384 Dec 31  1969 efi
drwx------. 5 root root      4096 Mar 24 18:42 grub2
-rw-------. 1 root root 157884040 Mar 24 18:22 initramfs-0-rescue-c50eedc1bcb245299bc96b1173897ac2.img
-rw-------. 1 root root  75057408 Mar  3 19:35 initramfs-6.7.6-200.fc39.x86_64.img
-rw-------. 1 root root  75259068 Mar  5 18:44 initramfs-6.7.7-200.fc39.x86_64.img
-rw-------. 1 root root  75508985 Mar 24 18:22 initramfs-6.7.9-200.fc39.x86_64.img
drwxr-xr-x. 3 root root      4096 May  9  2023 loader
drwx------. 2 root root     16384 May  9  2023 lost+found
lrwxrwxrwx. 1 root root        45 Mar  3 19:35 symvers-6.7.6-200.fc39.x86_64.xz -> /lib/modules/6.7.6-200.fc39.x86_64/symvers.xz
lrwxrwxrwx. 1 root root        45 Mar  5 18:44 symvers-6.7.7-200.fc39.x86_64.xz -> /lib/modules/6.7.7-200.fc39.x86_64/symvers.xz
lrwxrwxrwx. 1 root root        45 Mar 13 08:56 symvers-6.7.9-200.fc39.x86_64.xz -> /lib/modules/6.7.9-200.fc39.x86_64/symvers.xz
-rw-r--r--. 1 root root   8850731 Feb 22 18:00 System.map-6.7.6-200.fc39.x86_64
-rw-r--r--. 1 root root   8851783 Feb 29 18:00 System.map-6.7.7-200.fc39.x86_64
-rw-r--r--. 1 root root   8851994 Mar  5 18:00 System.map-6.7.9-200.fc39.x86_64
-rwxr-xr-x. 1 root root  14790472 Mar 24 18:22 vmlinuz-0-rescue-c50eedc1bcb245299bc96b1173897ac2
-rwxr-xr-x. 1 root root  14794568 Feb 22 18:00 vmlinuz-6.7.6-200.fc39.x86_64
-rwxr-xr-x. 1 root root  14794568 Feb 29 18:00 vmlinuz-6.7.7-200.fc39.x86_64
-rwxr-xr-x. 1 root root  14790472 Mar  5 18:00 vmlinuz-6.7.9-200.fc39.x86_64

/boot/efi:
total 12
drwx------. 4 root root 4096 Jul 18  2023 EFI
-rwx------. 1 root root   34 Jul 19  2023 mach_kernel
drwx------. 3 root root 4096 Jul 19  2023 System

/boot/efi/EFI:
total 8
drwx------. 2 root root 4096 Mar 24 18:25 BOOT
drwx------. 2 root root 4096 Mar 24 18:25 fedora

/boot/efi/EFI/BOOT:
total 1820
-rwx------. 1 root root 747681 Mar 18 19:00 BOOTIA32.EFI
-rwx------. 1 root root 949424 Mar 18 19:00 BOOTX64.EFI
-rwx------. 1 root root  70360 Mar 18 19:00 fbia32.efi
-rwx------. 1 root root  87816 Mar 18 19:00 fbx64.efi

/boot/efi/EFI/fedora:
total 17628
-rwx------. 1 root root     112 Mar 18 19:00 BOOTIA32.CSV
-rwx------. 1 root root     110 Mar 18 19:00 BOOTX64.CSV
-rwx------. 1 root root 2960704 Feb  6 18:00 gcdia32.efi
-rwx------. 1 root root 3968320 Feb  6 18:00 gcdx64.efi
-rwx------. 1 root root     144 May  9  2023 grub.cfg
-rwx------. 1 root root 2960704 Feb  6 18:00 grubia32.efi
-rwx------. 1 root root 3968320 Feb  6 18:00 grubx64.efi
-rwx------. 1 root root  673992 Mar 18 19:00 mmia32.efi
-rwx------. 1 root root  848080 Mar 18 19:00 mmx64.efi
-rwx------. 1 root root  949424 Mar 18 19:00 shim.efi
-rwx------. 1 root root  747681 Mar 18 19:00 shimia32.efi
-rwx------. 1 root root  949424 Mar 18 19:00 shimx64.efi

/boot/efi/System:
total 4
drwx------. 3 root root 4096 Jul 19  2023 Library

/boot/efi/System/Library:
total 4
drwx------. 2 root root 4096 Oct 11 09:58 CoreServices

/boot/efi/System/Library/CoreServices:
total 4
-rwx------. 1 root root 384 Jul 19  2023 SystemVersion.plist

/boot/grub2:
total 24
drwx------. 2 root root 4096 Mar 13 08:55 fonts
-rw-------. 1 root root 6602 Mar  3 19:35 grub.cfg
-rw-------. 1 root root 1024 Mar 24 18:42 grubenv
drwxr-xr-x. 2 root root 4096 May 10  2023 i386-pc
drwxr-xr-x. 2 root root 4096 May 10  2023 x86_64-efi

/boot/grub2/fonts:
total 2344
-rwx------. 1 root root 2394108 Feb  6 18:00 unicode.pf2

/boot/grub2/i386-pc:
total 48
-rw-r--r--. 1 root root 22532 Apr 11  2023 multiboot2.mod
-rw-r--r--. 1 root root 20592 Apr 11  2023 relocator.mod

/boot/grub2/x86_64-efi:
total 80
-rw-r--r--. 1 root root 38520 Apr 11  2023 multiboot2.mod
-rw-r--r--. 1 root root 40904 Apr 11  2023 relocator.mod

/boot/loader:
total 4
drwx------. 2 root root 4096 Mar 24 18:22 entries

/boot/loader/entries:
total 16
-rw-r--r--. 1 root root 512 Mar 24 18:22 c50eedc1bcb245299bc96b1173897ac2-0-rescue.conf
-rw-r--r--. 1 root root 460 Mar  3 19:35 c50eedc1bcb245299bc96b1173897ac2-6.7.6-200.fc39.x86_64.conf
-rw-r--r--. 1 root root 460 Mar  5 18:44 c50eedc1bcb245299bc96b1173897ac2-6.7.7-200.fc39.x86_64.conf
-rw-r--r--. 1 root root 460 Mar 24 18:22 c50eedc1bcb245299bc96b1173897ac2-6.7.9-200.fc39.x86_64.conf

/boot/lost+found:

[computersavvy (Jeff V)]

Also try sudo sh -c "ls -lR /boot/efi/"

edit-- command typo fixed

[grzegorzb (Grzegorz)]

Bash of course, here is the output:

grzegorz@hp840:~/kernel_test/kernel_6.8$ echo $SHELL
/bin/bash
grzegorz@hp840:~/kernel_test/kernel_6.8$ sudo sh -c "ls -lr /boot/efi/"
total 20
drwx------. 3 root root 4096 Jul 20  2023 System
-rwx------. 1 root root    0 Jul 31  2022 screen%03d.txt
-rwx------. 1 root root   34 Jul 20  2023 mach_kernel
drwx------. 3 root root 4096 Mar 24 22:57 loader
drwx------. 8 root root 4096 Mar 25 17:34 fb3c1fe83e654cd69a64e0256d9580a9
drwx------. 5 root root 4096 Mar 19 17:48 EFI

grzegorz@hp840:~/kernel_test/kernel_6.8$ sudo ls -lr /boot/efi/
total 20
drwx------. 3 root root 4096 Jul 20  2023 System
-rwx------. 1 root root    0 Jul 31  2022 screen%03d.txt
-rwx------. 1 root root   34 Jul 20  2023 mach_kernel
drwx------. 3 root root 4096 Mar 24 22:57 loader
drwx------. 8 root root 4096 Mar 25 17:34 fb3c1fe83e654cd69a64e0256d9580a9
drwx------. 5 root root 4096 Mar 19 17:48 EFI

Maybe this output would be healpful somehow:

root@hp840:/home/grzegorz# tree /boot
/boot
├── config-6.7.3-200.fc39.x86_64
├── efi
│   ├── EFI
│   │   ├── BOOT
│   │   │   ├── BOOTIA32.EFI
│   │   │   ├── BOOTX64.EFI
│   │   │   ├── fbia32.efi
│   │   │   └── fbx64.efi
│   │   ├── fedora
│   │   │   ├── BOOTIA32.CSV
│   │   │   ├── BOOTX64.CSV
│   │   │   ├── gcdia32.efi
│   │   │   ├── gcdx64.efi
│   │   │   ├── grub.cfg
│   │   │   ├── grub.cfg.rpmsave
│   │   │   ├── grubia32.efi
│   │   │   ├── grubx64.efi
│   │   │   ├── mmia32.efi
│   │   │   ├── mmx64.efi
│   │   │   ├── shim.efi
│   │   │   ├── shimia32.efi
│   │   │   └── shimx64.efi
│   │   └── HP
│   │       ├── BIOS
│   │       │   ├── Current
│   │       │   │   └── Q78_012700.bin
│   │       │   ├── New
│   │       │   └── Previous
│   │       │       └── Q78_012600.bin
│   │       └── DEVFW
│   │           ├── CCG5.bin
│   │           └── ME.bin
│   ├── fb3c1fe83e654cd69a64e0256d9580a9
│   │   ├── 0-rescue
│   │   │   ├── initrd
│   │   │   └── linux
│   │   ├── 6.7.5-200.fc39.x86_64
│   │   │   ├── initrd
│   │   │   └── linux
│   │   ├── 6.7.6-200.fc39.x86_64
│   │   │   ├── initrd
│   │   │   └── linux
│   │   ├── 6.7.7-200.fc39.x86_64
│   │   │   ├── initrd
│   │   │   └── linux
│   │   ├── 6.7.9-200.fc39.x86_64
│   │   │   ├── initrd
│   │   │   └── linux
│   │   └── 6.8.1-201.fc39.x86_64
│   │       ├── initrd
│   │       └── linux
│   ├── loader
│   │   └── entries
│   │       ├── fb3c1fe83e654cd69a64e0256d9580a9-0-rescue.conf
│   │       ├── fb3c1fe83e654cd69a64e0256d9580a9-6.7.5-200.fc39.x86_64.conf
│   │       ├── fb3c1fe83e654cd69a64e0256d9580a9-6.7.6-200.fc39.x86_64.conf
│   │       ├── fb3c1fe83e654cd69a64e0256d9580a9-6.7.7-200.fc39.x86_64.conf
│   │       ├── fb3c1fe83e654cd69a64e0256d9580a9-6.7.9-200.fc39.x86_64.conf
│   │       └── fb3c1fe83e654cd69a64e0256d9580a9-6.8.1-201.fc39.x86_64.conf
│   ├── mach_kernel
│   ├── screen%03d.txt
│   └── System
│       └── Library
│           └── CoreServices
│               └── SystemVersion.plist
├── extlinux
│   ├── cat.c32
│   ├── chain.c32
│   ├── cmd.c32
│   ├── cmenu.c32
│   ├── config.c32
│   ├── cptime.c32
│   ├── cpu.c32
│   ├── cpuid.c32
│   ├── cpuidtest.c32
│   ├── debug.c32
│   ├── dhcp.c32
│   ├── dir.c32
│   ├── disk.c32
│   ├── dmi.c32
│   ├── dmitest.c32
│   ├── elf.c32
│   ├── ethersel.c32
│   ├── gfxboot.c32
│   ├── gpxecmd.c32
│   ├── hdt.c32
│   ├── hexdump.c32
│   ├── host.c32
│   ├── ifcpu64.c32
│   ├── ifcpu.c32
│   ├── ifmemdsk.c32
│   ├── ifplop.c32
│   ├── kbdmap.c32
│   ├── kontron_wdt.c32
│   ├── ldlinux.c32
│   ├── lfs.c32
│   ├── libcom32.c32
│   ├── libgpl.c32
│   ├── liblua.c32
│   ├── libmenu.c32
│   ├── libutil.c32
│   ├── linux.c32
│   ├── ls.c32
│   ├── lua.c32
│   ├── mboot.c32
│   ├── memdisk
│   ├── meminfo.c32
│   ├── menu.c32
│   ├── pci.c32
│   ├── pcitest.c32
│   ├── pmload.c32
│   ├── poweroff.c32
│   ├── prdhcp.c32
│   ├── pwd.c32
│   ├── pxechn.c32
│   ├── reboot.c32
│   ├── rosh.c32
│   ├── sanboot.c32
│   ├── sdi.c32
│   ├── sysdump.c32
│   ├── syslinux.c32
│   ├── vesa.c32
│   ├── vesainfo.c32
│   ├── vesamenu.c32
│   ├── vpdtest.c32
│   ├── whichsys.c32
│   └── zzjson.c32
├── grub2
│   ├── fonts
│   │   └── unicode.pf2
│   ├── grub.cfg
│   └── grubenv
├── initramfs-0-rescue-fb3c1fe83e654cd69a64e0256d9580a9.img
├── initramfs-6.7.3-200.fc39.x86_64.img
├── loader
│   └── entries
│       ├── fb3c1fe83e654cd69a64e0256d9580a9-0-rescue.conf
│       └── fb3c1fe83e654cd69a64e0256d9580a9-6.7.3-200.fc39.x86_64.conf
├── lost+found
├── symvers-6.7.3-200.fc39.x86_64.xz -> /lib/modules/6.7.3-200.fc39.x86_64/symvers.xz
├── symvers-6.7.5-200.fc39.x86_64.xz -> /lib/modules/6.7.5-200.fc39.x86_64/symvers.xz
├── symvers-6.7.6-200.fc39.x86_64.xz -> /lib/modules/6.7.6-200.fc39.x86_64/symvers.xz
├── symvers-6.7.7-200.fc39.x86_64.xz -> /lib/modules/6.7.7-200.fc39.x86_64/symvers.xz
├── symvers-6.7.9-200.fc39.x86_64.xz -> /lib/modules/6.7.9-200.fc39.x86_64/symvers.xz
├── symvers-6.8.1-201.fc39.x86_64.xz -> /lib/modules/6.8.1-201.fc39.x86_64/symvers.xz
├── System.map-6.7.3-200.fc39.x86_64
├── vmlinuz-0-rescue-fb3c1fe83e654cd69a64e0256d9580a9
└── vmlinuz-6.7.3-200.fc39.x86_64

[computersavvy (Jeff V)]

@grzegorzb

drwx------. 8 root root 4096 Mar 25 17:34 fb3c1fe83e654cd69a64e0256d9580a9

This directory /boot/efi/fb3c1fe83e654cd69a64e0256d9580a9 appears to be the problem. There are several threads about this where when that directory exists the files that normally would be placed in /boot are instead placed under that directory.

F39 kernels fail to install when /boot/efi/$machineid is present

Problem On certain Fedora systems, which have been installed a long time ago (multiple years) and continuously upgraded, new Fedora 39 kernel packages may fail to install (and then fail to appear in the GRUB bootloader menu, when displayed). The error during kernel installation might look like this: Running scriptlet: kernel-core-6.5.9-300.fc39.x86_64 2/2 /usr/lib/kernel/install.d/51-dracut-rescue.install: line 81: /boot/efi/loader/entries/1604a63a532947d99d18bf17a754e8…

If you remove that directory and its content
sudo rm -r /boot/efi/fb3c1fe83e654cd69a64e0256d9580a9
then run sudo dnf reinstall kernel* it should solve the issue for the current kernel.

You may read through the entire thread if necessary to understand what happened and why.
Once that directory is removed the issue should not reappear in the future.

My error in the earlier ls command. It should have been
sudo sh -c "ls -lR /boot/"
I will fix that above.

[grzegorzb (Grzegorz)]

@computersavvy thanks for the help, it worked as a charm!
Thank you also for the link, I am one of those guys who update my fedora instance in place since many releases, I wasn’t aware of such issue. Thanks once again.