VM boots into F38 kernel after upgrading to F39, despite grub configuration being correct

Ask Fedora
, , ,
1

A few weeks ago I upgrade a VM (running in AWS) from Fedora 38 to Fedora 39. (The VM was originally installed as Fedora 32 and has been upgrade through (I think) every release since, although it’s possible I upgraded directly from 34 to 36.) But every time I reboot it (and even after a few kernel updates since the F39 upgrade), it boots into the remaining F38 kernel.

I don’t think the problem I’m seeing is F39 kernels fail to install when /boot/efi/$machineid is present because I don’t have that directory present. And the kernels do appear to be in the grub configuration:

# grubby --info=ALL
index=0
kernel="/boot/vmlinuz-6.5.12-300.fc39.x86_64"
args="ro no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8"
root="UUID=d1b37ed4-3bbb-40b2-a6ba-f377f0c90217"
initrd="/boot/initramfs-6.5.12-300.fc39.x86_64.img"
title="Fedora Linux (6.5.12-300.fc39.x86_64) 39 (Cloud Edition)"
id="d94ee5bb27e74aa0a42ac4399922f9b6-6.5.12-300.fc39.x86_64"
index=1
kernel="/boot/vmlinuz-6.5.11-300.fc39.x86_64"
args="ro no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8"
root="UUID=d1b37ed4-3bbb-40b2-a6ba-f377f0c90217"
initrd="/boot/initramfs-6.5.11-300.fc39.x86_64.img"
title="Fedora Linux (6.5.11-300.fc39.x86_64) 39 (Cloud Edition)"
id="d94ee5bb27e74aa0a42ac4399922f9b6-6.5.11-300.fc39.x86_64"
index=2
kernel="/boot/vmlinuz-6.5.6-200.fc38.x86_64"
args="ro no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8"
root="UUID=d1b37ed4-3bbb-40b2-a6ba-f377f0c90217"
initrd="/boot/initramfs-6.5.6-200.fc38.x86_64.img"
title="Fedora Linux (6.5.6-200.fc38.x86_64) 38 (Cloud Edition)"
id="d94ee5bb27e74aa0a42ac4399922f9b6-6.5.6-200.fc38.x86_64"

# grubby --info=DEFAULT
index=0
kernel="/boot/vmlinuz-6.5.12-300.fc39.x86_64"
args="ro no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8"
root="UUID=d1b37ed4-3bbb-40b2-a6ba-f377f0c90217"
initrd="/boot/initramfs-6.5.12-300.fc39.x86_64.img"
title="Fedora Linux (6.5.12-300.fc39.x86_64) 39 (Cloud Edition)"
id="d94ee5bb27e74aa0a42ac4399922f9b6-6.5.12-300.fc39.x86_64"

# uname -a
Linux ip-172-31-51-230.ec2.internal 6.5.6-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Oct  6 19:02:35 UTC 2023 x86_64 GNU/Linux

# find /boot/ -type f | sort
/boot/config-6.5.11-300.fc39.x86_64
/boot/config-6.5.12-300.fc39.x86_64
/boot/config-6.5.6-200.fc38.x86_64
/boot/efi/EFI/BOOT/BOOTIA32.EFI
/boot/efi/EFI/BOOT/BOOTX64.EFI
/boot/efi/EFI/BOOT/fbia32.efi
/boot/efi/EFI/BOOT/fbx64.efi
/boot/efi/EFI/fedora/BOOTIA32.CSV
/boot/efi/EFI/fedora/BOOTX64.CSV
/boot/efi/EFI/fedora/grubia32.efi
/boot/efi/EFI/fedora/grubx64.efi
/boot/efi/EFI/fedora/mmia32.efi
/boot/efi/EFI/fedora/mmx64.efi
/boot/efi/EFI/fedora/shim.efi
/boot/efi/EFI/fedora/shimia32.efi
/boot/efi/EFI/fedora/shimx64.efi
/boot/extlinux/cat.c32
/boot/extlinux/chain.c32
/boot/extlinux/cmd.c32
/boot/extlinux/cmenu.c32
/boot/extlinux/config.c32
/boot/extlinux/cptime.c32
/boot/extlinux/cpu.c32
/boot/extlinux/cpuid.c32
/boot/extlinux/cpuidtest.c32
/boot/extlinux/debug.c32
/boot/extlinux/dhcp.c32
/boot/extlinux/dir.c32
/boot/extlinux/disk.c32
/boot/extlinux/dmi.c32
/boot/extlinux/dmitest.c32
/boot/extlinux/elf.c32
/boot/extlinux/ethersel.c32
/boot/extlinux/gfxboot.c32
/boot/extlinux/gpxecmd.c32
/boot/extlinux/hdt.c32
/boot/extlinux/hexdump.c32
/boot/extlinux/host.c32
/boot/extlinux/ifcpu64.c32
/boot/extlinux/ifcpu.c32
/boot/extlinux/ifmemdsk.c32
/boot/extlinux/ifplop.c32
/boot/extlinux/kbdmap.c32
/boot/extlinux/kontron_wdt.c32
/boot/extlinux/ldlinux.c32
/boot/extlinux/lfs.c32
/boot/extlinux/libcom32.c32
/boot/extlinux/libgpl.c32
/boot/extlinux/liblua.c32
/boot/extlinux/libmenu.c32
/boot/extlinux/libutil.c32
/boot/extlinux/linux.c32
/boot/extlinux/ls.c32
/boot/extlinux/lua.c32
/boot/extlinux/mboot.c32
/boot/extlinux/memdisk
/boot/extlinux/meminfo.c32
/boot/extlinux/menu.c32
/boot/extlinux/pci.c32
/boot/extlinux/pcitest.c32
/boot/extlinux/pmload.c32
/boot/extlinux/poweroff.c32
/boot/extlinux/prdhcp.c32
/boot/extlinux/pwd.c32
/boot/extlinux/pxechn.c32
/boot/extlinux/reboot.c32
/boot/extlinux/rosh.c32
/boot/extlinux/sanboot.c32
/boot/extlinux/sdi.c32
/boot/extlinux/sysdump.c32
/boot/extlinux/syslinux.c32
/boot/extlinux/vesa.c32
/boot/extlinux/vesainfo.c32
/boot/extlinux/vesamenu.c32
/boot/extlinux/vpdtest.c32
/boot/extlinux/whichsys.c32
/boot/extlinux/zzjson.c32
/boot/grub2/device.map
/boot/grub2/fonts/unicode.pf2
/boot/grub2/grub.cfg
/boot/grub2/grubenv
/boot/grub2/i386-pc/acpi.mod
/boot/grub2/i386-pc/adler32.mod
/boot/grub2/i386-pc/affs.mod
/boot/grub2/i386-pc/afs.mod
/boot/grub2/i386-pc/ahci.mod
/boot/grub2/i386-pc/all_video.mod
/boot/grub2/i386-pc/aout.mod
/boot/grub2/i386-pc/archelp.mod
/boot/grub2/i386-pc/ata.mod
/boot/grub2/i386-pc/at_keyboard.mod
/boot/grub2/i386-pc/backtrace.mod
/boot/grub2/i386-pc/bfs.mod
/boot/grub2/i386-pc/biosdisk.mod
/boot/grub2/i386-pc/bitmap.mod
/boot/grub2/i386-pc/bitmap_scale.mod
/boot/grub2/i386-pc/blocklist.mod
/boot/grub2/i386-pc/blscfg.mod
/boot/grub2/i386-pc/boot.img
/boot/grub2/i386-pc/boot.mod
/boot/grub2/i386-pc/bsd.mod
/boot/grub2/i386-pc/bswap_test.mod
/boot/grub2/i386-pc/btrfs.mod
/boot/grub2/i386-pc/bufio.mod
/boot/grub2/i386-pc/cat.mod
/boot/grub2/i386-pc/cbfs.mod
/boot/grub2/i386-pc/cbls.mod
/boot/grub2/i386-pc/cbmemc.mod
/boot/grub2/i386-pc/cbtable.mod
/boot/grub2/i386-pc/cbtime.mod
/boot/grub2/i386-pc/chain.mod
/boot/grub2/i386-pc/cmdline_cat_test.mod
/boot/grub2/i386-pc/cmosdump.mod
/boot/grub2/i386-pc/cmostest.mod
/boot/grub2/i386-pc/cmp.mod
/boot/grub2/i386-pc/cmp_test.mod
/boot/grub2/i386-pc/command.lst
/boot/grub2/i386-pc/configfile.mod
/boot/grub2/i386-pc/core.img
/boot/grub2/i386-pc/cpio_be.mod
/boot/grub2/i386-pc/cpio.mod
/boot/grub2/i386-pc/cpuid.mod
/boot/grub2/i386-pc/crc64.mod
/boot/grub2/i386-pc/cryptodisk.mod
/boot/grub2/i386-pc/crypto.lst
/boot/grub2/i386-pc/crypto.mod
/boot/grub2/i386-pc/cs5536.mod
/boot/grub2/i386-pc/ctz_test.mod
/boot/grub2/i386-pc/datehook.mod
/boot/grub2/i386-pc/date.mod
/boot/grub2/i386-pc/datetime.mod
/boot/grub2/i386-pc/diskfilter.mod
/boot/grub2/i386-pc/disk.mod
/boot/grub2/i386-pc/div.mod
/boot/grub2/i386-pc/div_test.mod
/boot/grub2/i386-pc/dm_nv.mod
/boot/grub2/i386-pc/drivemap.mod
/boot/grub2/i386-pc/echo.mod
/boot/grub2/i386-pc/efiemu.mod
/boot/grub2/i386-pc/ehci.mod
/boot/grub2/i386-pc/elf.mod
/boot/grub2/i386-pc/eval.mod
/boot/grub2/i386-pc/exfat.mod
/boot/grub2/i386-pc/exfctest.mod
/boot/grub2/i386-pc/ext2.mod
/boot/grub2/i386-pc/extcmd.mod
/boot/grub2/i386-pc/f2fs.mod
/boot/grub2/i386-pc/fat.mod
/boot/grub2/i386-pc/file.mod
/boot/grub2/i386-pc/font.mod
/boot/grub2/i386-pc/freedos.mod
/boot/grub2/i386-pc/fshelp.mod
/boot/grub2/i386-pc/fs.lst
/boot/grub2/i386-pc/functional_test.mod
/boot/grub2/i386-pc/gcry_arcfour.mod
/boot/grub2/i386-pc/gcry_blowfish.mod
/boot/grub2/i386-pc/gcry_camellia.mod
/boot/grub2/i386-pc/gcry_cast5.mod
/boot/grub2/i386-pc/gcry_crc.mod
/boot/grub2/i386-pc/gcry_des.mod
/boot/grub2/i386-pc/gcry_dsa.mod
/boot/grub2/i386-pc/gcry_idea.mod
/boot/grub2/i386-pc/gcry_md4.mod
/boot/grub2/i386-pc/gcry_md5.mod
/boot/grub2/i386-pc/gcry_rfc2268.mod
/boot/grub2/i386-pc/gcry_rijndael.mod
/boot/grub2/i386-pc/gcry_rmd160.mod
/boot/grub2/i386-pc/gcry_rsa.mod
/boot/grub2/i386-pc/gcry_seed.mod
/boot/grub2/i386-pc/gcry_serpent.mod
/boot/grub2/i386-pc/gcry_sha1.mod
/boot/grub2/i386-pc/gcry_sha256.mod
/boot/grub2/i386-pc/gcry_sha512.mod
/boot/grub2/i386-pc/gcry_tiger.mod
/boot/grub2/i386-pc/gcry_twofish.mod
/boot/grub2/i386-pc/gcry_whirlpool.mod
/boot/grub2/i386-pc/gdb.mod
/boot/grub2/i386-pc/geli.mod
/boot/grub2/i386-pc/gettext.mod
/boot/grub2/i386-pc/gfxmenu.mod
/boot/grub2/i386-pc/gfxterm_background.mod
/boot/grub2/i386-pc/gfxterm_menu.mod
/boot/grub2/i386-pc/gfxterm.mod
/boot/grub2/i386-pc/gptsync.mod
/boot/grub2/i386-pc/gzio.mod
/boot/grub2/i386-pc/halt.mod
/boot/grub2/i386-pc/hashsum.mod
/boot/grub2/i386-pc/hdparm.mod
/boot/grub2/i386-pc/hello.mod
/boot/grub2/i386-pc/help.mod
/boot/grub2/i386-pc/hexdump.mod
/boot/grub2/i386-pc/hfs.mod
/boot/grub2/i386-pc/hfspluscomp.mod
/boot/grub2/i386-pc/hfsplus.mod
/boot/grub2/i386-pc/http.mod
/boot/grub2/i386-pc/increment.mod
/boot/grub2/i386-pc/iorw.mod
/boot/grub2/i386-pc/iso9660.mod
/boot/grub2/i386-pc/jfs.mod
/boot/grub2/i386-pc/jpeg.mod
/boot/grub2/i386-pc/keylayouts.mod
/boot/grub2/i386-pc/keystatus.mod
/boot/grub2/i386-pc/ldm.mod
/boot/grub2/i386-pc/legacycfg.mod
/boot/grub2/i386-pc/legacy_password_test.mod
/boot/grub2/i386-pc/linux.mod
/boot/grub2/i386-pc/loadenv.mod
/boot/grub2/i386-pc/loopback.mod
/boot/grub2/i386-pc/lsacpi.mod
/boot/grub2/i386-pc/lsapm.mod
/boot/grub2/i386-pc/lsmmap.mod
/boot/grub2/i386-pc/ls.mod
/boot/grub2/i386-pc/lspci.mod
/boot/grub2/i386-pc/luks.mod
/boot/grub2/i386-pc/lvm.mod
/boot/grub2/i386-pc/lzopio.mod
/boot/grub2/i386-pc/macbless.mod
/boot/grub2/i386-pc/macho.mod
/boot/grub2/i386-pc/mda_text.mod
/boot/grub2/i386-pc/mdraid09_be.mod
/boot/grub2/i386-pc/mdraid09.mod
/boot/grub2/i386-pc/mdraid1x.mod
/boot/grub2/i386-pc/memdisk.mod
/boot/grub2/i386-pc/memrw.mod
/boot/grub2/i386-pc/minicmd.mod
/boot/grub2/i386-pc/minix2_be.mod
/boot/grub2/i386-pc/minix2.mod
/boot/grub2/i386-pc/minix3_be.mod
/boot/grub2/i386-pc/minix3.mod
/boot/grub2/i386-pc/minix_be.mod
/boot/grub2/i386-pc/minix.mod
/boot/grub2/i386-pc/mmap.mod
/boot/grub2/i386-pc/moddep.lst
/boot/grub2/i386-pc/modinfo.sh
/boot/grub2/i386-pc/morse.mod
/boot/grub2/i386-pc/mpi.mod
/boot/grub2/i386-pc/msdospart.mod
/boot/grub2/i386-pc/mul_test.mod
/boot/grub2/i386-pc/multiboot2.mod
/boot/grub2/i386-pc/multiboot.mod
/boot/grub2/i386-pc/nativedisk.mod
/boot/grub2/i386-pc/net.mod
/boot/grub2/i386-pc/newc.mod
/boot/grub2/i386-pc/nilfs2.mod
/boot/grub2/i386-pc/normal.mod
/boot/grub2/i386-pc/ntfscomp.mod
/boot/grub2/i386-pc/ntfs.mod
/boot/grub2/i386-pc/ntldr.mod
/boot/grub2/i386-pc/odc.mod
/boot/grub2/i386-pc/offsetio.mod
/boot/grub2/i386-pc/ohci.mod
/boot/grub2/i386-pc/part_acorn.mod
/boot/grub2/i386-pc/part_amiga.mod
/boot/grub2/i386-pc/part_apple.mod
/boot/grub2/i386-pc/part_bsd.mod
/boot/grub2/i386-pc/part_dfly.mod
/boot/grub2/i386-pc/part_dvh.mod
/boot/grub2/i386-pc/part_gpt.mod
/boot/grub2/i386-pc/partmap.lst
/boot/grub2/i386-pc/part_msdos.mod
/boot/grub2/i386-pc/part_plan.mod
/boot/grub2/i386-pc/part_sun.mod
/boot/grub2/i386-pc/part_sunpc.mod
/boot/grub2/i386-pc/parttool.lst
/boot/grub2/i386-pc/parttool.mod
/boot/grub2/i386-pc/password.mod
/boot/grub2/i386-pc/password_pbkdf2.mod
/boot/grub2/i386-pc/pata.mod
/boot/grub2/i386-pc/pbkdf2.mod
/boot/grub2/i386-pc/pbkdf2_test.mod
/boot/grub2/i386-pc/pcidump.mod
/boot/grub2/i386-pc/pci.mod
/boot/grub2/i386-pc/pgp.mod
/boot/grub2/i386-pc/plan9.mod
/boot/grub2/i386-pc/play.mod
/boot/grub2/i386-pc/png.mod
/boot/grub2/i386-pc/priority_queue.mod
/boot/grub2/i386-pc/probe.mod
/boot/grub2/i386-pc/procfs.mod
/boot/grub2/i386-pc/progress.mod
/boot/grub2/i386-pc/pxechain.mod
/boot/grub2/i386-pc/pxe.mod
/boot/grub2/i386-pc/raid5rec.mod
/boot/grub2/i386-pc/raid6rec.mod
/boot/grub2/i386-pc/random.mod
/boot/grub2/i386-pc/rdmsr.mod
/boot/grub2/i386-pc/read.mod
/boot/grub2/i386-pc/reboot.mod
/boot/grub2/i386-pc/regexp.mod
/boot/grub2/i386-pc/reiserfs.mod
/boot/grub2/i386-pc/relocator.mod
/boot/grub2/i386-pc/romfs.mod
/boot/grub2/i386-pc/scsi.mod
/boot/grub2/i386-pc/search_fs_file.mod
/boot/grub2/i386-pc/search_fs_uuid.mod
/boot/grub2/i386-pc/search_label.mod
/boot/grub2/i386-pc/search.mod
/boot/grub2/i386-pc/sendkey.mod
/boot/grub2/i386-pc/serial.mod
/boot/grub2/i386-pc/setjmp.mod
/boot/grub2/i386-pc/setjmp_test.mod
/boot/grub2/i386-pc/setpci.mod
/boot/grub2/i386-pc/sfs.mod
/boot/grub2/i386-pc/shift_test.mod
/boot/grub2/i386-pc/signature_test.mod
/boot/grub2/i386-pc/sleep.mod
/boot/grub2/i386-pc/sleep_test.mod
/boot/grub2/i386-pc/spkmodem.mod
/boot/grub2/i386-pc/squash4.mod
/boot/grub2/i386-pc/strtoull_test.mod
/boot/grub2/i386-pc/syslinuxcfg.mod
/boot/grub2/i386-pc/tar.mod
/boot/grub2/i386-pc/terminal.lst
/boot/grub2/i386-pc/terminal.mod
/boot/grub2/i386-pc/terminfo.mod
/boot/grub2/i386-pc/test_blockarg.mod
/boot/grub2/i386-pc/testload.mod
/boot/grub2/i386-pc/test.mod
/boot/grub2/i386-pc/testspeed.mod
/boot/grub2/i386-pc/tftp.mod
/boot/grub2/i386-pc/tga.mod
/boot/grub2/i386-pc/time.mod
/boot/grub2/i386-pc/trig.mod
/boot/grub2/i386-pc/tr.mod
/boot/grub2/i386-pc/truecrypt.mod
/boot/grub2/i386-pc/true.mod
/boot/grub2/i386-pc/udf.mod
/boot/grub2/i386-pc/ufs1_be.mod
/boot/grub2/i386-pc/ufs1.mod
/boot/grub2/i386-pc/ufs2.mod
/boot/grub2/i386-pc/uhci.mod
/boot/grub2/i386-pc/usb_keyboard.mod
/boot/grub2/i386-pc/usb.mod
/boot/grub2/i386-pc/usbms.mod
/boot/grub2/i386-pc/usbserial_common.mod
/boot/grub2/i386-pc/usbserial_ftdi.mod
/boot/grub2/i386-pc/usbserial_pl2303.mod
/boot/grub2/i386-pc/usbserial_usbdebug.mod
/boot/grub2/i386-pc/usbtest.mod
/boot/grub2/i386-pc/vbe.mod
/boot/grub2/i386-pc/verifiers.mod
/boot/grub2/i386-pc/version.mod
/boot/grub2/i386-pc/vga.mod
/boot/grub2/i386-pc/vga_text.mod
/boot/grub2/i386-pc/video_bochs.mod
/boot/grub2/i386-pc/video_cirrus.mod
/boot/grub2/i386-pc/video_colors.mod
/boot/grub2/i386-pc/video_fb.mod
/boot/grub2/i386-pc/videoinfo.mod
/boot/grub2/i386-pc/video.lst
/boot/grub2/i386-pc/video.mod
/boot/grub2/i386-pc/videotest_checksum.mod
/boot/grub2/i386-pc/videotest.mod
/boot/grub2/i386-pc/wrmsr.mod
/boot/grub2/i386-pc/xfs.mod
/boot/grub2/i386-pc/xnu.mod
/boot/grub2/i386-pc/xnu_uuid.mod
/boot/grub2/i386-pc/xnu_uuid_test.mod
/boot/grub2/i386-pc/xzio.mod
/boot/grub2/i386-pc/zfscrypt.mod
/boot/grub2/i386-pc/zfsinfo.mod
/boot/grub2/i386-pc/zfs.mod
/boot/grub2/i386-pc/zstd.mod
/boot/initramfs-6.5.11-300.fc39.x86_64.img
/boot/initramfs-6.5.12-300.fc39.x86_64.img
/boot/initramfs-6.5.6-200.fc38.x86_64.img
/boot/loader/entries/d94ee5bb27e74aa0a42ac4399922f9b6-6.5.11-300.fc39.x86_64.conf
/boot/loader/entries/d94ee5bb27e74aa0a42ac4399922f9b6-6.5.12-300.fc39.x86_64.conf
/boot/loader/entries/d94ee5bb27e74aa0a42ac4399922f9b6-6.5.6-200.fc38.x86_64.conf
/boot/System.map-6.5.11-300.fc39.x86_64
/boot/System.map-6.5.12-300.fc39.x86_64
/boot/System.map-6.5.6-200.fc38.x86_64
/boot/vmlinuz-6.5.11-300.fc39.x86_64
/boot/.vmlinuz-6.5.11-300.fc39.x86_64.hmac
/boot/vmlinuz-6.5.12-300.fc39.x86_64
/boot/.vmlinuz-6.5.12-300.fc39.x86_64.hmac
/boot/vmlinuz-6.5.6-200.fc38.x86_64
/boot/.vmlinuz-6.5.6-200.fc38.x86_64.hmac

/boot/ is not a separate partition, nor is anything inside it.

It’s not clear to me why this is happening or how to fix it.

2

Manually boot the latest kernel and try this way:

sudo dnf remove --oldinstallonly
sudo grub2-editenv - create
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo dracut -f --regenerate-all
3

I think that would need to be sudo grub2-editenv - create

Also, it may be an selinux issue. There was another post just in the last 2 days with similar issues and the fix was to run sudo touch /.autorelabel then reboot, following which the dnf system-upgrade procedure was repeated successfully.

The kernel 6.5.6 for f38 is quite old and before updating to a newer version of fedora it is always recommended that the user upgrade the existing install to the latest version before beginning the upgrade (sudo dnf upgrade --refresh)

4

I did the upgrade from F38 to F39 a few weeks ago, back when that was the current F38 kernel. (And the upgrade was fine, except for still using the old kernel.)

5

Ok, so the issue is it refuses to actually boot to F39.
What is the content of /etc/fedora-release?
What is the output of dnf list installed kernel*?
Have you tried setting selinux to permissive then rebooting to see if it then is able to boot F39 kernel?
What is the output of ls /boot?

6

The suggestion of touch /.autorelabel and rebooting (twice) didn’t help. The output of those commands is:

# cat /etc/fedora-release 
Fedora release 39 (Thirty Nine)
# dnf list installed 'kernel*'
Installed Packages
kernel-core.x86_64                                 6.5.6-200.fc38                           @updates
kernel-core.x86_64                                 6.5.11-300.fc39                          @updates
kernel-core.x86_64                                 6.5.12-300.fc39                          @updates
kernel-headers.x86_64                              6.5.4-300.fc39                           @fedora 
kernel-modules-core.x86_64                         6.5.6-200.fc38                           @updates
kernel-modules-core.x86_64                         6.5.11-300.fc39                          @updates
kernel-modules-core.x86_64                         6.5.12-300.fc39                          @updates
# ls /boot
config-6.5.11-300.fc39.x86_64         symvers-6.5.11-300.fc39.x86_64.xz
config-6.5.12-300.fc39.x86_64         symvers-6.5.12-300.fc39.x86_64.xz
config-6.5.6-200.fc38.x86_64          symvers-6.5.6-200.fc38.x86_64.xz
efi                                   System.map-6.5.11-300.fc39.x86_64
extlinux                              System.map-6.5.12-300.fc39.x86_64
grub2                                 System.map-6.5.6-200.fc38.x86_64
initramfs-6.5.11-300.fc39.x86_64.img  vmlinuz-6.5.11-300.fc39.x86_64
initramfs-6.5.12-300.fc39.x86_64.img  vmlinuz-6.5.12-300.fc39.x86_64
initramfs-6.5.6-200.fc38.x86_64.img   vmlinuz-6.5.6-200.fc38.x86_64
loader

Will try some of the other suggestions later.

7

Is it possible that at some point the fedora 38 kernel was set as default?
Does the grub menu appear during boot so you might select a newer kernel? If not then usually pressing and holding the shift key when it starts to boot will cause the menu to appear.

The info you posted shows current F39 kernels but the F38 kernel is still there because it has been the running kernel during updates.

You also can force the grub menu to appear during boot by running
sudo grub2-editenv - unset menu_auto_hide
To hide it again sudo grub2-editenv - set menu_auto_hide=1

8

OK, the suggestion above that fixed the immediate problem was:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

/boot/grub2/grub.cfg hadn’t been regenerated since October 10, and had the 6.5.6, 6.5.5, and 6.4.15 kernels as options.

Then the question is why kernel updates aren’t regenerating it. Do they think I’m using systemd-boot?

9

To tell the truth, this should no longer be necessary:
Changes/BootLoaderSpecByDefault - Fedora Project Wiki
Probably something has gone wrong in your case.

Note that release upgrades often require to apply the new configs:
Upgrading Fedora Using DNF System Plugin :: Fedora Docs

10

I wonder if at some point the grub.cfg file in /boot/efi/EFI/fedora/grub.cfg was overwritten and is not correct.

Please post the output of cat /boot/efi/EFI/fedora/grub.cfg so we can confirm it is as expected. There should only be about 4 lines of text in that file. If it is as expected then /boot/grub2/grub.cfg should properly be used to boot and that file should be automatically updated with each kernel update.

11

There’s no such file /boot/efi/EFI/fedora/grub.cfg. (See list of all files in /boot/ above.)

12

I see that now. Which may indicate you are using legacy (MBR) boot.
Please post the output of cat /sys/firmware/efi/efivars

Are you using a 32 bit system? or has it been 32 bit in the past.?
I see a lot of files that seem unusual for a 64 bit system.
My system only has 2 files in /boot/grub2/i386-pc/ and yours has lots of files there.
The same for /boot/extlinux/ which does not exist on my system

13

There’s no /sys/firmware/efi/ directory.

14

That means that you are booting in legacy BIOS mode.

Since Fedora 39, the kernel updates only updates the bls configuration files and not longer modifies the contents of the grub.cfg file. If you have selected to turn GRUB_ENABLE_BLSCFG off, your boot configuration will not be updated, so you have to do that afterwards.

The boot configuration are found in the /boot/loader/entries directory, for example

/boot/loader/entries/b4146134cc534b2bb35b75e467b91eb4-6.5.10-300.fc39.x86_64.conf
/boot/loader/entries/b4146134cc534b2bb35b75e467b91eb4-6.5.12-300.fc39.x86_64.conf
/boot/loader/entries/b4146134cc534b2bb35b75e467b91eb4-6.5.11-300.fc39.x86_64.conf

The grub.cfg file does need to use these files for this to work.

15

I didn’t choose to turn off GRUB_ENABLE_BLSCFG – I just installed Fedora 32 (or maybe 31?) a few years ago and kept upgrading it to the next version. But I do see GRUB_ENABLE_BLSCFG=false in /etc/default/grub (which has a last-modified timestamp of 2020-10-10 19:06:19.684771880 +0000, which is about 5 minutes after the last-modified timestamps of /etc/hostname and /etc/machine-id).

I guess I should try changing that line to true?

16

Yeah, that seems to have generated a /boot/grub2/grub.cfg that doesn’t have the manual entry listings, so it seems like that works.

17

Yes, there’s actually a special tool for this:

sudo grub2-switch-to-blscfg
18

BLS configuration became standard by Fedora 30, and when updating to Fedora 30 or 31 the configuration was automatically converted.