Unable to boot with latest package updates

Hi, bit of a novice here but I seem to be running into issues with latest kernel/package updates. System is a Lenovo Legion AMD 7840HS laptop.

When I try to graphical boot, the system hangs. Sometimes I cannot even enter LUKS unlock phrase before it completely freezes, and sometimes it will work and freeze afterward. I tried to follow instructions to disable graphical boot so I could see boot logs, and with graphical boot disabled each time the system would freeze and become unresponsive while I try to enter the LUKS unlock phrase.

I do have some layered packages on my current install, not sure if I should try to rebase to a clean release ? Here is my current working deployment (39.20240226.0) that i pinned:

Deployments:
  fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240228.0 (2024-02-28T00:48:13Z)
               BaseCommit: 7bded7dd4858735c04f18acb58b02f945044d1e0a50c09af5c4112cd522fcc4b
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
                     Diff: 12 upgraded, 1 added
          LayeredPackages: brave-browser brave-keyring butane coreos-installer langpacks-en librewolf mullvad-vpn pam-u2f pamu2fcfg
            LocalPackages: trezor-udev-2-1.noarch veracrypt-1.26.7-1.x86_64

  fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240228.0 (2024-02-28T00:48:13Z)
               BaseCommit: 7bded7dd4858735c04f18acb58b02f945044d1e0a50c09af5c4112cd522fcc4b
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: brave-browser brave-keyring butane coreos-installer langpacks-en librewolf mullvad-vpn pam-u2f pamu2fcfg
            LocalPackages: veracrypt-1.26.7-1.x86_64
                   Pinned: yes

● fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240226.0 (2024-02-26T00:36:30Z)
               BaseCommit: fe742a69a4c13327f9fb9b04bdda61ea10704f9945941a762d42ab1eaefbbcb6
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: brave-browser brave-keyring butane coreos-installer langpacks-en librewolf mullvad-vpn pam-u2f pamu2fcfg
            LocalPackages: veracrypt-1.26.7-1.x86_64
                   Pinned: yes

And here is the upgrade preview:

rpm-ostree upgrade --preview
1 metadata, 0 content objects fetched; 592 B transferred in 2 seconds; 0 bytes content written
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora google-chrome rpmfusion-nonfree-nvidia-driver copr:copr.fedorainfracloud.org:phracek:PyCharm rpmfusion-nonfree-steam brave-browser repository mullvad-stable updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'google-chrome'... done
Updating metadata for 'rpmfusion-nonfree-nvidia-driver'... done
Updating metadata for 'copr:copr.fedorainfracloud.org:phracek:PyCharm'... done
Updating metadata for 'rpmfusion-nonfree-steam'... done
Updating metadata for 'brave-browser'... done
Updating metadata for 'repository'... done
Updating metadata for 'mullvad-stable'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'updates'; generated: 2024-02-28T01:02:48Z solvables: 23361
rpm-md repo 'fedora'; generated: 2023-11-01T00:12:39Z solvables: 70825
rpm-md repo 'google-chrome'; generated: 2024-02-27T19:02:18Z solvables: 3
rpm-md repo 'rpmfusion-nonfree-nvidia-driver'; generated: 2024-02-24T16:32:59Z solvables: 29
rpm-md repo 'copr:copr.fedorainfracloud.org:phracek:PyCharm'; generated: 2024-02-27T09:54:29Z solvables: 17
rpm-md repo 'rpmfusion-nonfree-steam'; generated: 2024-02-24T18:17:39Z solvables: 2
rpm-md repo 'brave-browser'; generated: 2024-02-28T20:07:59Z solvables: 60
rpm-md repo 'repository'; generated: 2024-02-24T16:29:37Z solvables: 88
rpm-md repo 'mullvad-stable'; generated: 2023-12-06T13:47:05Z solvables: 1
rpm-md repo 'updates-archive'; generated: 2024-02-28T01:31:25Z solvables: 33907
Note: --check and --preview may be unreliable.  See https://github.com/coreos/rpm-ostree/issues/1579
AvailableUpdate:
        Version: 39.20240228.0 (2024-02-28T00:48:13Z)
         Commit: 7bded7dd4858735c04f18acb58b02f945044d1e0a50c09af5c4112cd522fcc4b
   GPGSignature: 1 signature
                 Signature made Tue 27 Feb 2024 07:49:02 PM EST using RSA key ID 75CF5AC418B8E74C
                 Good signature from "Fedora <fedora-39-primary@fedoraproject.org>"
  SecAdvisories: FEDORA-2024-a95bdde55b  Unknown    wpa_supplicant-1:2.10-9.fc39.x86_64
                   TRIAGE CVE-2023-52160 wpa_supplicant: potential authorization bypass [fedora-all]
                   https://bugzilla.redhat.com/show_bug.cgi?id=2264594
                 FEDORA-2024-d16d94b00d  Moderate   kernel-6.7.6-200.fc39.x86_64
                 FEDORA-2024-d16d94b00d  Moderate   kernel-core-6.7.6-200.fc39.x86_64
                 FEDORA-2024-d16d94b00d  Moderate   kernel-modules-6.7.6-200.fc39.x86_64
                 FEDORA-2024-d16d94b00d  Moderate   kernel-modules-core-6.7.6-200.fc39.x86_64
                 FEDORA-2024-d16d94b00d  Moderate   kernel-modules-extra-6.7.6-200.fc39.x86_64
                   CVE-2024-23850 kernel: btrfs_get_root_ref has an assertion failure and crash because a subvolume can be read out too soon after its root item is inserted upon subvolume creation
                   https://bugzilla.redhat.com/show_bug.cgi?id=2260044
                   CVE-2024-23851 kernel: copy_params can attempt to allocate more than INT_MAX bytes  and crash
                   https://bugzilla.redhat.com/show_bug.cgi?id=2260046
                   CVE-2023-52437 kernel: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"
                   https://bugzilla.redhat.com/show_bug.cgi?id=2265269
                   CVE-2024-26585 kernel: tls: race between tx work scheduling and socket close
                   https://bugzilla.redhat.com/show_bug.cgi?id=2265517
                   CVE-2024-26582 kernel: tls: use-after-free with partial reads and async decrypt
                   https://bugzilla.redhat.com/show_bug.cgi?id=2265518
                   CVE-2024-26584 kernel: tls: handle backlogging of crypto requests
                   https://bugzilla.redhat.com/show_bug.cgi?id=2265519
                   CVE-2024-26583 kernel: tls: race between async notify and socket close
                   https://bugzilla.redhat.com/show_bug.cgi?id=2265520
                   CVE-2024-26593 kernel: i2c: i801: Fix block process call transactions
                   https://bugzilla.redhat.com/show_bug.cgi?id=2265646
                   CVE-2024-26603 kernel: x86/fpu: Stop relying on userspace for info to fault in xsave buffer
                   https://bugzilla.redhat.com/show_bug.cgi?id=2265833
                   CVE-2024-26604 kernel: null pointer dereference in kobject
                   https://bugzilla.redhat.com/show_bug.cgi?id=2266257
                   CVE-2024-26606 kernel: signal epoll threads of self-work
                   https://bugzilla.redhat.com/show_bug.cgi?id=2266286
       Upgraded: brave-browser 1.63.162-1 -> 1.63.165-1
                 fedora-release-common 39-34 -> 39-36
                 fedora-release-identity-silverblue 39-34 -> 39-36
                 fedora-release-ostree-desktop 39-34 -> 39-36
                 fedora-release-silverblue 39-34 -> 39-36
                 git-core 2.43.2-1.fc39 -> 2.44.0-1.fc39
                 git-core-doc 2.43.2-1.fc39 -> 2.44.0-1.fc39
                 kernel 6.7.5-200.fc39 -> 6.7.6-200.fc39
                 kernel-core 6.7.5-200.fc39 -> 6.7.6-200.fc39
                 kernel-modules 6.7.5-200.fc39 -> 6.7.6-200.fc39
                 kernel-modules-core 6.7.5-200.fc39 -> 6.7.6-200.fc39
                 kernel-modules-extra 6.7.5-200.fc39 -> 6.7.6-200.fc39
                 wpa_supplicant 1:2.10-8.fc39 -> 1:2.10-9.fc39

So obviously one of these packages is causing issues, but I have no idea how to figure out which one, or what to do next. For compatibility sake I have disabled the ideapad_laptop kernel module a while ago, because it was causing issues with charging, sleep/wake and unexpected shutdowns… not sure if that could be related to this in some way?

Thanks for any assistance you folks can offer!

It seems like you’re encountering freezing and boot issues after attempting to upgrade your system. Here’s a simplified troubleshooting approach:

  1. Identify Problematic Package: Start by considering the recently upgraded packages. In your case, it’s likely one of the packages from the upgrade preview causing the issue. You can try rolling back the upgrade for each package one by one to isolate the problematic one.
  2. Check Known Issues: Review the release notes or bug reports for each upgraded package, especially the kernel and firmware-related ones, as they can significantly impact system stability.
  3. Re-enable Disabled Modules: Since you disabled the ideapad_laptop kernel module for compatibility reasons, it’s worth re-enabling it temporarily to see if it resolves the freezing issue. If the freezing persists, you can rule out the module as the cause.
  4. Consult Community: Seek assistance from the Fedora community forums or mailing lists. Others might have experienced similar issues or have insights into resolving them.
  5. Consider Clean Installation: If troubleshooting becomes too complex or time-consuming, you might consider backing up your data and performing a clean installation of Fedora, especially if you suspect layered packages might be contributing to the issues.

By following these steps, you should be able to diagnose and resolve the freezing and boot issues you’re experiencing after the latest kernel/package updates on your Lenovo Legion laptop.

Best Regard
Danish Hafeez | QA Assistant

Removed coreos

Hi,
Do you post this answer to advertise a company? (I’ve removed the link since it’s not in line with Fedora’s CoC).

Your tips make no sense at all….

Re 1: silverblue is image based, you can boot an older image but there is no way to roll back individual packages.

Re2: how?

Re4: that’s what OP just did.

Re5: completely exaggerated in my opinion, especially since older image still boots and has been pinned, and OP hasn’t even started troubleshooting.

there is option in editior to isert link then why you removed link?

Because the link you posted is not relevant to OP’s question but rather appears to be advertisement.

His response looks almost certainly like it was made by a chatbot, lol

Do you have any ideas how I might start to resolve this issue? right now it feels like my only options are to do nothing and receive no package updates, or rebase to a clean install?

I did try re enabling the disabled kernel module which didn’t help. Not sure what else I can do… Where would I look to find bug reports to see if others are having a similar issue? Or how might i begin the process of isolating which package updates are causing the problem? I’m trying to figure out how to remove pending ostree package updates so I can apply them one by one, but i can’t seem to figure out how to do that… let me know if you or anyone else has any ideas. Thanks

I edited the post by @danishhafeez to remove the obvious advertising link.

Advertising is one of the implicitly forbidden actions in the Code of Conduct and has been removed many times over the years.

You can remove quiet and rhgb parameters with

rpm-ostree kargs --delete-if-present quiet --delete-if-present rhgb

to get some debug information during system boot.

Additionally, I think you can selectively update packages from the 39.20240226.0 image using rpm-ostree override replace <bodhi link> to identify the package responsible for the issue.

Thank you for your response.

I tried booting without quiet and rhgb kernel args several times, and i’m not sure the information it was giving me was useful, since the process just randomly freezes at any given point. There were no useful error messages, and it froze in different places each time.

One time it actually got all the way to letting me enter LUKS phrase, eventually i got a blue “Welcome to Fedora” text at the bottom, and the cursor remained flashing but it didn’t proceed beyond that point into the actual OS. Hitting escape would flash the screen between text boot, and a grey background with 3 dots, which i usually see before GUI login. Not sure what that means though.

As far as selectively updating packages, how can i remove the pending ostree deployments ? My current pinned one has already attempted to automatically upgrade everything into a new deployment, so I feel like i need to get rid of all of that and start fresh by adding the new packages each individually… or can i somehow create a new deployment based on my current pinned one, and add packages one by one?

Based on the symptoms and the changed packages between 39.20240226.0 and 39.20240228.0 my hunch says that something changed in the kernel that is causing you trouble. You may want to file an issue against the kernel - Log in to Red Hat Bugzilla

(Or try searching the bugs against the kernel for something that looks similar)

As far as selectively updating packages, how can i remove the pending ostree deployments ? My current pinned one has already attempted to automatically upgrade everything into a new deployment, so I feel like i need to get rid of all of that and start fresh by adding the new packages each individually… or can i somehow create a new deployment based on my current pinned one, and add packages one by one?

You can clean up the pending deployments with rpm-ostree cleanup -p, but that won’t clean up any layered packages.

You could try the nuclear option of rpm-ostree reset and then doing an rpm-ostree upgrade. Or try removing the packages during the upgrade with rpm-ostree upgrade --uninstall <package>, booting into the new deployment and checking to see if things improved.

Thank you. i cleaned up the pending deployments, then used rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2024-88847bc77a to override the kernel update back to previous 6.7.5-200.fc39. This allowed me to upgrade all other non-kernel packages and now I can boot. So clearly something about the latest kernel update is causing problems.

I put feedback on bodhi for the newer broken package, so hopefully it can get fixed eventually. At least for now I can keep using the existing kernel and get all other updates until this gets resolved.