F39: Not booting, blank screen after update, stuck with 'This OS version is past its end-of-support date'

Ask Fedora
1

A computer running Fedora 39 is unable to boot after installing updates using plasma-discover (KDE). It shows a warning about being eol and that’s it.

IMG_20250108_124457 2
IMG_20250108_124457 23613×628 198 KB

Note that it has not been upgraded yet because of new bugs observed on other Fedora installations that were upgraded to F40 and beyond, including:

  • Not booting anymore because initramfs image for newly installed kernel version was not created during the update: error: file ‘/initramfs-…img’ not found. 2
  • Some keyboard shortcuts not working anymore (like Guake, KDE)
  • Keyboard layouts disappearing or layout switching sometimes not working
  • Keyboard input randomly stopping to work altogether for some programs (like Leafpad) while still working in others like KWrite (similar issue)
  • KDE panels not on screen edge anymore (menu not “a mile high” anymore), context menu on windows now listing workspaces that the window should be mirrored on where “move to workspace” used to be…
  • Desktop session started with Wayland after upgrade, KDE package needs to be installed manually and selected on login screen to get X11 back (plasma-workspace-x11)
  • Tray icons disappearing (except default ones)

Those points might be off-topic but then again, I feel like they should be mentioned somewhere as it’s probably not worth it creating a topic for each one individually. Anyway, this is why this particular Fedora 39 installation has not been upgraded yet. Unfortunately, a simple update rendered it unusable for a while.

After a long search and rescue operation, it turns out that the Nvidia graphics driver somehow broke the system after this update. Since all similar questions I found suggested removing all nvidia packages which I would not recommend doing before checking if this driver is indeed the culprit (what if that was not the reason, you’d have one extra problem that might cause boot issues) - here’s a workaround:

In the Grub boot menu, hit E to change the boot options. Go to the Linux boot line which starts with initramfs and ends with:

... rhgb quiet rd.driver.blacklist=nouveau modprobe.blacklist=nouveau

Remove both blacklist instances or replace nouveau with nvidia. Also, rhgb quiet may be removed to see more information during the boot process. This actually helped, the system would boot normally, login screen appears, KDE works etc. The only difference at first sight seems to be that in some rare case, the mouse cursor would be slow to move. Also, after some minutes of inactivity, the screen is dimmed which is a KDE setting that should not be on but seems to be enabled after the update. Otherwise, everything seems to be working with the Nouveau graphics driver.

To confirm that it’s actually being used:

$ inxi -G | grep NVIDIA
  Device-2: NVIDIA GP108 [GeForce GT 1030] driver: nouveau v: kernel

Now, this is a workaround but the question is: What’s the real fix? And is it a known issue that the Nvidia driver might prevent Fedora from booting after updating?

# rpm -qa | grep nvidia | grep -v 'kmod-'
nvidia-gpu-firmware-20241110-1.fc39.noarch
xorg-x11-drv-nvidia-kmodsrc-560.35.03-5.fc39.x86_64
nvidia-modprobe-560.35.03-1.fc39.x86_64
xorg-x11-drv-nvidia-cuda-libs-560.35.03-5.fc39.x86_64
xorg-x11-drv-nvidia-libs-560.35.03-5.fc39.x86_64
nvidia-persistenced-560.35.03-1.fc39.x86_64
xorg-x11-drv-nvidia-xorg-libs-560.35.03-5.fc39.x86_64
nvidia-settings-560.35.03-1.fc39.x86_64
xorg-x11-drv-nvidia-power-560.35.03-5.fc39.x86_64
xorg-x11-drv-nvidia-560.35.03-5.fc39.x86_64
xorg-x11-drv-nvidia-cuda-560.35.03-5.fc39.x86_64

# dnf info nvidia-modprobe-560.35.03-1.fc39.x86_64 | grep repo
From repo    : rpmfusion-nonfree-updates
2

Generally, the way to deal with a driver not working with the most recent kernel is to revert to using the previous kernel (select the previous kernel from the boot menu) and, if necessary, file a bug report to get the driver updated (or the kernel fixed). There is a reason that Fedora Linux keeps the previous two kernels installed and available as a fallback option.

Any kernel driver can (potentially) prevent the system from booting. There probably isn’t much that can be done about that since they need direct access to low-level memory areas to function. Linux kernel drivers are modular though and it is (usually) possible to blacklist one that is not working and use an alternative driver (as you demonstrated).

1 Like
3

That didn’t help, wouldn’t boot with the previous kernel version either.

Indeed, I agree. In fact, this is why the installonly_limit setting was set to a higher value than 3.

I agree, but then again… We’re not talking about some obscure slide scanner from 2003 with self-compiled driver packages or something (slightly exaggerating here). It’s the graphics that stopped working, Nvidia, which probably half the world uses (again, might be slightly exaggerated but you get the point). With the official driver package. I’m arguing that in 2024, graphics should just work without expecting the user to 1) get the right idea that it might be the graphics driver, not the eol message on the screen; 2) modify the boot options; 3) depending on the situation or who you follow, remove nvidia packages, install nouveau or just make the boot change permanent.

In other words: It’s such a standard setup with a common graphics card, I would expect lots of other users being affected as well and so there should be a known bug report somewhere. So, is there something I missed?

Apart from that, I’ve noticed that there is a bunch of nvidia packages installed (plus some older kmod packages), from the rpmfusion repository. The version number seems to be 560 but on the NVidia site, 550 seems to be the latest production-ready version if I read it correctly.

4

That really shouldn’t happen. Unless the kernel and initrd files on the boot partition have been modified. I consider it a major bug if the driver updates are modifying your known-good fallback kernels. Can you look at the timestamps on your initramfs files and see if the older ones were changed? Edit: You might also check ls "/lib/modules/<kernel-version>/extra" to see if the drivers were updated there. Again, they shouldn’t be for older kernel installations. But if they have the same timestamps across several directory trees, then the driver updater is doing something it shouldn’t.

I understand. That said, the kernel devs and/or the driver devs can be a bit reckless (or over-enthusiastic about getting the latest and greatest power saving feature to work) at times.

Since you are running an EOL version, it is likely that the problem has already been reported and fixed and the answer you would eventually get if you reported the problem would be to update your system. But you’ve already indicated that you are not ready to do that, so the only other option is to go back to the previous kernel.

Linux is trying to support a lot of hardware that wasn’t designed to work with Linux. There are other operating system companies that pay the hardware manufactures to make the hardware work with their OS. To some extent, it’s the trade-off with Linux that “you get what you pay for”. Linux is often an afterthought for hardware/firmware developers.

1 Like
5

It is a known fact (or at least reported here several times) that the 550 or 560 drivers installed from nvidia do not function properly with the newer fedora kernels. Installing the nvidia driver from rpmfusion gives version 565 for the newer kernels.

Recently it was noted that rpmfusion has discontinued the repos for several versions
of fedora that are EOL and they needed storage space.

Upgrading to at least f40 seems quite reasonable. (f41 would seem a better choice to me)

Agreed 100%.
However reality is that the OS manufacturers upon which the hardware vendors rely have an arrangement where the drivers are guaranteed to work with their OS. The big kahuna tends to get priority since they have the lions share of the market and lesser entities such as linux have a much smaller share of the market so their needs are secondary to almost everyone.

Some hardware manufacturers do not support linux at all. For those cases drivers may be reverse engineered but are seldom as good as those provided by the hardware source.

2 Likes
6

Where do I find this information, is there a “known bugs” section somewhere listing (in)compatible kernel versions? Seems like a major issue but which kernel update should be avoided if using the nvidia driver version 560? I guess 6.6.11, what about 6.5.6… I checked the rpmfusion page but couldn’t find such a warning there.

Thanks, good to know. Though if they discontinue Nvidia support that early (39 was released a year ago), it makes me wonder if there’s a long-term benefit using rpmfusion to install those drivers as opposed to installing the official drivers from Nvidia directly.

As I wrote above, I delayed this upgrade due to various new bugs, one being Fedora not booting anymore and with this in mind, I do not consider it reasonable to upgrade, unless there’s extra time to fix bugs. However, previous release upgrades did not cause such fatal bugs as far as I remember and generally, I still prefer the non-interactive nature of Fedora upgrades compared to Debian etc. I digress.

About that, quote:

Is 565 the version that should be used with kernel 6.6.11? I’m not sure if upgrading to a driver which has such a warning is a reasonable choice, though.

Again, this is about graphics, i.e., standard hardware. And Nvidia supports Linux, when selecting card model and operating system, it says that 550.142, released Dec 17th,
is the recommended version and there’s a manual download option. I’m considering installing that one.

By the way, with the Nouveau driver, the slow mouse issue is getting annoying. At times, on dark pages, it’s actually very difficult to move the mouse across the screen.

7

I’m sorry, you’re right. Selecting a previous kernel version does indeed work now (btw. the timestamps have not changed). Don’t know why it didn’t work before, could be an oversight or being impatient, being under time pressure. Sorry about the confusion.

That’s what I wanted to find out when I opened this question.

8

No list I am aware of, but the issue is the driver source as far as I can determine.
The drivers from nivdia seem to be the ones having issues and the drivers installed from rpmfusion seldom seem to have issues.

The 550 and 560 drivers were replaced on rpmfusion quite some time past. If using f41 the only drivers they provide are the 565 version.

As I understand it if you keep your system updated (both drivers and kernels) there are seldom serious problems with graphics. In fact the most recent problems with not being able to boot into a newer kernel is due to the use of drivers from nvidia (or other sources) that use dkms to compile the drivers. It has not happened with those whose drivers are compiled using akmods (rpmfusion).

These are not “fatal” bugs. There have been workarounds available for everyone as I recall. It does require the user to fix the issue.

Bugs will appear continuously, some one type and some another type. Those bugs get fixed and things move forward. Refusing to upgrade and remaining on an EOL version blocks you from all future upgrades, bug fixes, etc. Nothing ever gets changed on a release version after it is designated EOL.

Your choice!

1 Like
9

I’d just like to point out that it was just that, a system update, which broke it so that it wouldn’t start anymore. By the way, I’m not someone who never updates anything, in fact I sometimes have to convince others so they would let me update their workstation. I also agree with most of the rest and that it’s rare, but that doesn’t help much, I was hoping for a specific “known issue”, something that would allow me to know which kernel version I should avoid for now and when it’s safe to upgrade without making it worse.

How is it not a fatal bug if the system won’t boot anymore after installing updates? I’m sorry, I agree with almost everything else but not this.

Also, since you’re mentioning F39 being EOL again. I’ve listed some of the bugs which made me stop and wait. You see, I’m generally not refusing to update and I would’ve upgraded long ago if there were clear statements about those bugs being fixed but without anything like that, I’d have to rely on the hope that it’s been taken care of. Again, I upgraded another Fedora system and it failed to boot for a different reason, so my typical reaction is to wait until we know that this is fixed instead of repeating it on another system breaking it in the same way. You’ve pointed me to another bug, the 565 drivers possibly failing altogether when using DVI. What if I don’t wait for a fix, upgrade and then I’m bit by this bug?

Btw. when it happened, I scrolled through the journal but did not find anything related to the graphics driver.

10

This has been fixed. I just updated my kernel and a new initramfs was generated:

# ls -al /boot/788e20240aaf43f46f617cde4b703bd1/6.12.8-200.fc41.x86_64
total 96456
drwx------. 2 root root     4096 Jan  9 18:23 .
drwx------. 6 root root     4096 Jan  9 18:21 ..
-rw-------. 1 root root 82094291 Jan  9 18:23 initrd
-rw-------. 1 root root 16664936 Jan  9 18:23 linux

# rpm -q dkms
dkms-3.1.4-3.fc41.noarch

As for the rest, I cannot say because I do not use KDE.

There should be bug trackers for each identified bug that will indicate its status. They mostly exist on github.com and gitlab.com, but there are others scattered about and it can be difficult to find the right tracker. If you know the RPM package that contains the software, you can often find the tracker listed on the URL field in the output from rpm -qi <package-name>. For example:

$ rpm -qi sway
Name        : sway
Version     : 1.10
Release     : 1.fc41
Architecture: x86_64
Install Date: Tue Jan  7 22:33:31 2025
Group       : Unspecified
Size        : 882059
License     : MIT
Signature   : RSA/SHA256, Sun Oct 27 21:35:12 2024, Key ID d0622462e99d6ad1
Source RPM  : sway-1.10-1.fc41.src.rpm
Build Date  : Sun Oct 27 21:30:18 2024
Build Host  : buildhw-x86-16.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://github.com/swaywm/sway
Bug URL     : https://bugz.fedoraproject.org/sway
Summary     : i3-compatible window manager for Wayland
Description :
Sway is a tiling window manager supporting Wayland compositor protocol and
i3-compatible configuration.

If you want to test a system update, but be sure that you can reverse the update in case a bug still exists, there is a way to do that with Btrfs snapshots: Make use of Btrfs snapshots to upgrade Fedora Linux with easy fallback - Fedora Magazine

1 Like
11

That’s great, thanks for confirming!

Now if I knew for sure that version 565 of the Nvidia driver is compatible with whichever kernel version currently ships with F40, I would consider upgrading now. But then again, I’ve just learned that this version of the Nvidia driver might break DVI support and I’m not sure if I’m ready for that.

Some of the KDE issues are ui glitches rather than fatal bugs. Those alone would not stop me from upgrading.

I know there may be other trackers (I believe Nvidia doesn’t have a public bugtracker but there’s a user forum). I came here thinking other affected users of Fedora 39+ would be most likely to see this here.

Re: BTRFS. Interesting idea, but quite a manual process. Maybe time to test that Zypper equivalent mentioned there.

1 Like