Updated to Fedora 43 and booting gets stuck

Ask Fedora
1

Last night I upgraded my Thinkpad T470 from 39 to 41 and then was able to log in and upgrade from 41 to 43, but booting 43 is repeatedly getting stuck on the black screen with a single underscore, after [ OK ] Started gdm.service - GNOME Display Manager. I did find the other threads and help online suggesting it was nvidia drivers (I did have them on rpmfusion already), so I’ve tried adding nomodeset plymouth.enable=0 to boot, which isn’t getting me to the login screen, but sometimes helps me get to a TTY.

The TTYs are hard to use most of the time because they keep losing focus for some reason. I get usually around 2-5 seconds of cursor blinking each time I hit ctrl-alt-Fx, and sometimes it just freezes permanently and I can neither switch nor type anything (but ctrl-alt-del does force a reboot). Only occasionally I can get in.

  • sudo dnf upgrade --refresh to upgrade all packages
  • sudo dnf remove '*nvidia*' to remove all nvidia packages in case of conflict
  • sudo akmods --force --rebuild, initially failed as akmods wasn’t installed
  • sudo dnf install akmod-nvidia xorg-x11-drv-nvidia-cuda to reinstall drivers
  • sudo dnf reinstall shim\* grub2\* had some error messages the first time but on the second time was fine. Still don’t see the new kernel on boot.

Presently, booting with nomodeset seems to get stuck after the GDM service is started, and sometimes with nomodeset plymouth.enable=0 I can login on a TTY.

I’ve only had the option to boot with kernel 6.5.12-100.fc37.x86_64 during all of this (booting the rescue image just freezes), but kernel-devel is upgraded to 6.17.12-300.fc43.x86_64. sudo akmods --force --rebuild gives me an error that development files for 6.5.12-100.fc37 are missing. Confirmed I have fc41 and fc43 via dnf list installed kernel though.

What should I do? Do I need to update grub to use the new kernel, run some other specific dnf commands, or use my live USB to repair the install (couldn’t find an option for that)?

If I need to run commands by booting into the live USB instead of rolling the dice on the TTYs, how do I mount and chroot into the LUKS-encrypted hard drive?

EDIT: I was finally able to remember to check journalctl when I had a working TTY. All my boots since updating to 43 include gdm or gnome-session-i dumping core about once a second (copied manually or by OCR):

    Module /usr/libexec/gnome-session-init-worker from rpm gnome-session-49.2-2.fc43.x86_64
    Stack trace:
        g_log_structured_array
        [...]

gdm[1903]: Gdm: GdmDisplay: Session never registered, failing
gdm[1903]: Child process -1977 was already dead.

along with PAM failures in user@60579.service, like Authentication service cannot retrieve authentication info, unable to dlopen(/usr/lib64/security/pam_lastlog.so), spawning /usr/lib/systemd/systemd: Operation not permitted, etc.
gnome-session-wayland failed to start

gdm dumps core at g_logv from g_log from get_fallback_session_name.

If these are crashing because the kernel is too old, then it goes back to needing to update grub to use the new kernel. Otherwise, not sure.

2

It sounds like you may have somehow locked that kernel as the default.
It would have been best if you had solved this boot issue with the first upgrade.

You did not say how you performed the upgrade. Some use the gnome software tool to upgrade but I always recommend that users follow the guide in this doc.

Note that Fedora 39 still used dnf4 but fedora 41 and later use dnf5 which is slightly different but only matters at step 5 in the procedure above.

In any case, the first step is to solve the issue with booting using the f39 kernel.
Does the grub boot menu appear?
If not it can be forced to appear by holding down the shift key when powering on and that menu should appear.
When the grub menu appears do you have the option to select a different kernel for booting? If so then select the kernel for f43 (6.17.12) and continue booting. If not then we need to know what you actually see.

Once booted we will need additional info.
The black screen sounds like what seems common with an nvidia gpu and newer kernels but using the default nouveau (open source) driver

One thing that would be helpful as well would be to see the output of ls /boot so we can confirm the newer kernels are actually installed properly.

3

I did indeed use the gnome software tool this time, though I remember using the system-upgrade plugin in the past. Grub does always appear, and no, it only shows the f37 kernel, and /ls/boot reflects this:

$ ls /boot
520f44320ac640c68996d2e055036d31
config-6.5.12-100.fc37.x86_64
efi
extlinux
grub2
initramfs-0-rescue-d2a5a88c339344618652114345ab0320.img
initramfs-6.5.12-100.fc37.x86_64.img
loader
lost+found
memtest86+x64.efi
symvers-6.5.12-100.fc37.x86_64.xz
System.map-6.5.12-100.fc37.x86_64
vmlinuz-0-rescue-d2a5a88c339344618652114345ab0320
vmlinuz-6.5.12-100.fc37.x86_64

I also previously ran sudo grub2-mkconfig while troubleshooting this, which added nouveau to a couple of blacklists in the boot command, it didn’t change anything.

4

That entry in the ls /boot output is probably the cause. On systems that were in use some time back (years ago) a directory with the machine ID was created under /boot and it seems that upgrades never removed that directory and now kernels are placed there as long as that directory exists.

I suggest that

  1. you first remove all the kernel data related to f41 & f43. Those kernels can be identified with dnf list --installed kernel and removed by using the kernel version number using sudo dnf remove kernel\*<version number>\*.

  2. Once those kernels have been removed remove that extraneous directory and any remaining content with sudo rm -r /boot/520f44320ac640c68996d2e055036d31

  3. After that directory has been removed, reinstall the newer kernel with sudo dnf upgrade kernel\* and verify it is properly installed with ls /boot
    If it installs properly then another reboot should use that kernel for booting.

Finally you may need to reinstall the nvidia driver because it may have been removed with the kernel removal.

5

That directory has one subdirectory for fc27 (not 37!), which is itself empty, so I do not think the kernels are installed there.

6

In any case, I did as you suggested, and /boot still only has fc37. I note that dnf list --installed kernel shows fc37 in green and fc43 in blue.

Edit: actually, now I’ve actually done as you suggested. I only removed kernel at the new versions the first time, oops. Your command did indeed remove a lot of packages I’ll need to readd later. But it does fail to install kernel-core with some scriptlet errors: `error writing ‘/boot/efi/d2a5a[…]/0-rescue/initrd’: No space left on device’

7

df -h says

/dev/sda2   974M   141M   766M   16% /boot
/dev/sda1   200M    37M   163M   19% /boot/efi

which seems not that close to full to me?

8

It seems you have the same problem as Unable to install new kernels due to lack of space in /boot/efi.

Try thus to remove /boot/efi/d2a5a[…] then reinstall the F43 kernel.

1 Like
9

I guess that this is due to gdm trying to start (but failing) at the same time.

To prevent that, try to add to the kernel command line at the grub prompt:

  • systemd.unit=multi-user.target

If it works:

  • keep it until this problem is fixed with:
    • sudo systemctl set-default multi-user.target
    • undo with: sudo systemctl set-default graphical.target
  • you can test gdm with:
    • sudo systemctl isolate graphical.target
      (or: sudo systemctl start gdm)
1 Like
10

Thank you! systemd.unit=multi-user.target works very well to reliably get me a commandline now. I ran

$ sudo -s
# rm -r /boot/efi/d2a5a<tab>

and then reinstalled the f43 kernel again, and confirmed that it does appear in /boot, so this was also a success! I followed up by reinstalling akmods and rebuilding drivers and then attempted to reboot with the f43 kernel.

It defaults to showing me all the debug output instead of needing an Escape keypress, which is fine, though I have to Escape to see the luks prompt by itself. But then I get [ TIME ] Timed out waiting for device dev-gpt\xd2auto\x2droot.device - /dev/gpt/auto-root and it drops me into emergency mode with a root password prompt, suggesting I run journalctl and save /run/initramfs/rdsosreport.txt to supply to a bug report.

The only interesting thing I see there besides the timeout and dependency failure is systemd[1]: System is tainted: unmerged-bin.

11

You made perhaps a typo in your luks passphrase or typed it too late.

Ca you reboot with the rhgb parameter (as it was before I guess) ?

This can be ignored for now. F43 merges /usr/sbin and /usr/bin. This
message says I think that this merge is partial.

12

rhgb adds back the graphical startup I expected, not sure why it’s not in there, should I rerun grub2-mkconfig? The command started as just linux ($root)/vmlinuz-6.17.12-300.fc43.x86_64 rd.driver.blacklist=nouveau,nova_core modprobe.blacklist=nouveau,nova_core. (I guess quiet is what hides the debug logs initially too.) The command for the fc37 kernel includes root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.luks.uuid=luks-[uuid] rd.lvm.lv=fedora/swap.

Also, no, the logs did report “Finished cryptsetup” (and didn’t reprompt me) and then we waited on dev-gpt for about 25 more seconds. I don’t see why hanging out on the luks passphrase screen for too long should break startup instead of waiting, but I just tested it–it sure does when booting like this.

ls /dev doesn’t show any device starting with g.

13

Normally no.
rhgb may have been suppressed since you added plymouth.enable=0

Can you send:

  • the content of /etc/default/grub
  • the output of ls -1 /boot/loader/entries/
  • the output of grep kernelopts= /boot/grub2/grub.cfg

and if /boot/loader/entries exists:

  • cat /boot/loader/entries/*fc37*.conf
  • cat /boot/loader/entries/*fc43*.conf
14

Neither /etc nor /boot exist/are mounted in the emergency mode, so these are the results from using the commandline under fc37, in case that matters.

OCRed and corrected by hand:

$ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release.*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root rd.luks.uuid=luks-772a86f5-8ad4-4a87-b2fa-340bbd5d3149 rd.lvm.lv=fedora/swap rhgb quiet rd.driver.blacklist=nouveau,nova_core modprobe.blacklist=nouveau,nova_core"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
$ sudo ls -1 /boot/loader/entries
d2a5a88c339344618652114345ab0320-0-memtest86+.conf
d2a5a88c339344618652114345ab0320-0-rescue.conf
d2a5a88c339344618652114345ab0320-6.17.12-300.fc43.x86_64.conf
d2a5a88c339344618652114345ab0320-6.5.12-100.fc37.x86_64.conf
$ sudo grep kernelopts= /boot/grub/grub.cfg
  set kernelopts="root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.luks.uuid=luks-772a86f5-8ad4-4a87-b2fa-340bbd5d3149 rd.lvm.lv=fedora/swap rhgb quiet rd.driver.blacklist=nouveau,nova_core modprobe.blacklist=nouveau,nova_core

$ sudo-s
# cat /boot/loader/entries/*fc37*.conf
title Fedora Linux (6.5.12-100.fc37.x86_64) 37 (Workstation Edition)
version 6.5.12-100.fc37.x86_64
linux /vmlinuz-6.5.12-100.fc37.x86_64
initrd /initramfs-6.5.12-100.fc37.x86_64.img $tuned_initrd
options root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.luks.uuid=luks-772a86f5-8ad4-4a87-b2fa-340bbd5d3149 rd.lvm.lv=fedora/swap rhgb quiet $tuned_params rd.driver.blacklist=nouveau,nova_core modprobe.blacklist=nouveau,nova_core
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
# cat /boot/loader/entries/*fc43*.conf
title Fedora Linux (6.17.12-300.fc43.x86_64) 43 (Workstation Edition)
version 6.17.12-300.fc43.x86_64
linux /vmlinuz-6.17.12-300.fc43.x86_64
initrd /initramfs-6.17.12-300.fc43.86_64.img $tuned_initrd
options $tuned_params rd.driver.blacklist=nouveau,nova_core modprobe.blacklist=nouveau,nova_core
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
15

This post: update leads to : waiting on device dev-gpt\x2dauto\x2droot.device looks promising but ensure first that your live USB is still working, for the case one breaks the
f37 boot.

/boot/loader/entries/*fc43*.conf is missing the root= part as in this post.

Try thus:

  • grubby --update-kernel=/boot/mlinuz-6.17.12-300.fc43.x86_64 --args="root=/dev/mapper/fedora-root"

Another way is to re-run grub2-mkconfig -o /boot/grub2/grub.cfg that
regenerate also the /boot/loader/entries/, but with the risk of
breaking the fc37 boot.

16

See also: Fedora 41 - Root and arguments are not retained when the kernel is updated - #8 by vekruse

17

There are more parameters missing. Re-executing grub2-mkconfig should
thus be better than using grubby.

18

I reran grub2-mkconfig -o /boot/grub2/grub.cfg and double-checked the boot loader entries to be sure it didn’t break fc37 and rebooted. It did work to get the kernel moving past the cryptsetup step, but it did not progress past the original issue of hanging at _ on a blank screen. The TTYs had the same symptom as earlier so I rebooted with systemd.unit=multi-user.target in the fc43 kernel to get to a commandline, and consulted journalctl again to find GDM crashlooping once more.

What pops out to me:

PAM unable to dlopen(/usr/lib64/security/pam_lastlog.so): ... No such file or directory
Activation request for 'org.freedesktop.home1' failed: The systemd unit 'dbus-org.freedesktop.home1.service' could not be found.
could not obtain user info (gdm-greeter)
PAM failed: Authentication service cannot retrieve authentication info
Failed to set up PAM session: Operation not permitted
Failed at step PAM spawning /usr/lib/systemd/systemd: Operation not permitted
Failed to start user@60578.service - User Manager for UID 60578.
dbus-daemon: Activating service name='org.freedesktop.systemd1' requested by ':1.0'
dbus-daemon: Activated service 'org.freedesktop.systemd1' failed: Process org.freedesktop.systemd1 exited with status 1
Failed to upload environment to systemd: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: NAme "org.freedesktop.systemd1" does not exist
Failed to check if unit gnome-session-wayland@gnome-login.target is active: (same error)
Failed to reset failed state of units: (same error)
Failed to start unit gnome-session-wayland@gnome-login.target: (same error but in red)

gnome-session-i coredump for users 60578 - 60583

Gdm: GdmSession: no session desktop files installed, aborting...
gdm coredump for user 0

user@60582.service: Failed to spawn executor: No such file or directory
19

Try to follow: No login screen after upgrade to f43 (Solved: authselect was never set up on a system upgraded through many versions)

1 Like
20

I did this, and on reboot it froze after “Started GDM Service” so I rebooted again and watched the debug scroll by, and it eventually loaded the login screen! Thanks a ton!

To summarize what worked for what was happening:

2 Likes