Silverblue fix for f41 mesa problem

fannullone · January 11, 2025, 8:23am

I think I am hitting fedora-41-not-booting-after-update-a-blank-screen-appears
The solution provided is for fc41, how can be fixed for fedora silverblue41?
In my case deployment0 is affected by the problem and doesnt boot while deployment1 is last working deploy which I have pinned:

Summary

sudo rpm-ostree -v status
State: idle
AutomaticUpdates: disabled
Deployments:
  fedora:fedora/41/x86_64/silverblue (index: 0)
                  Version: 41.20250111.0 (2025-01-11T01:06:39Z)
               BaseCommit: 1d09e0f1389d3c381a01fce4fde39b55d3023f347fa81be7d9391fea591228bf
                           ├─ repo-0 (2024-10-24T13:55:59Z)
                           ├─ repo-1 (2025-01-11T00:18:58Z)
                           └─ repo-2 (2025-01-11T00:26:24Z)
                   Commit: 1d6a4d9fc9f897379fbb49213e36c146fc13f84c7bf49ca3af97701622c72c8f
                           ├─ copr:copr.fedorainfracloud.org:phracek:PyCharm (2024-12-16T12:28:59Z)
                           ├─ fedora (2024-10-24T13:55:59Z)
                           ├─ fedora-cisco-openh264 (2024-03-11T19:22:31Z)
                           ├─ google-chrome (2025-01-10T21:41:09Z)
                           ├─ rpmfusion-nonfree-nvidia-driver (2024-12-17T11:02:42Z)
                           ├─ rpmfusion-nonfree-steam (2024-12-02T07:45:38Z)
                           ├─ updates (2025-01-10T01:25:24Z)
                           └─ updates-archive (2025-01-11T02:42:15Z)
                   Staged: no
                StateRoot: fedora
             GPGSignature: 1 signature
                           Signature made sab 11 gen 2025, 02:08:17 using RSA key ID D0622462E99D6AD1
                           Good signature from "Fedora <fedora-41-primary@fedoraproject.org>"
            SecAdvisories: FEDORA-2024-a94430d221  Low        curl-8.9.1-3.fc41.x86_64
                           FEDORA-2024-a94430d221  Low        libcurl-minimal-8.9.1-3.fc41.x86_64
                             CVE-2024-9681 curl: HSTS subdomain overwrites parent cache entry [fedora-41]
                             https://bugzilla.redhat.com/show_bug.cgi?id=2326407
                           FEDORA-2025-6fcde64d77  Critical   firefox-134.0-1.fc41.x86_64
                           FEDORA-2025-6fcde64d77  Critical   firefox-langpacks-134.0-1.fc41.x86_64
                 Upgraded: SDL2 2.30.9-1.fc41 -> 2.30.11-1.fc41
                           audit 4.0.2-1.fc41 -> 4.0.3-1.fc41
                           audit-libs 4.0.2-1.fc41 -> 4.0.3-1.fc41
                           audit-rules 4.0.2-1.fc41 -> 4.0.3-1.fc41
                           composefs 1.0.6-1.fc41 -> 1.0.8-1.fc41
                           composefs-libs 1.0.6-1.fc41 -> 1.0.8-1.fc41
                           cups 1:2.4.11-8.fc41 -> 1:2.4.11-9.fc41
                           cups-client 1:2.4.11-8.fc41 -> 1:2.4.11-9.fc41
                           cups-filesystem 1:2.4.11-8.fc41 -> 1:2.4.11-9.fc41
                           cups-ipptool 1:2.4.11-8.fc41 -> 1:2.4.11-9.fc41
                           cups-libs 1:2.4.11-8.fc41 -> 1:2.4.11-9.fc41
                           curl 8.9.1-2.fc41 -> 8.9.1-3.fc41
                           dnsmasq 2.90-3.fc41 -> 2.90-4.fc41
                           firefox 133.0.3-2.fc41 -> 134.0-1.fc41
                           firefox-langpacks 133.0.3-2.fc41 -> 134.0-1.fc41
                           gst-editing-services 1.24.10-1.fc41 -> 1.24.11-1.fc41
                           gstreamer1 1.24.10-1.fc41 -> 1.24.11-1.fc41
                           gstreamer1-plugin-libav 1.24.10-1.fc41 -> 1.24.11-1.fc41
                           gstreamer1-plugins-bad-free 1.24.10-1.fc41 -> 1.24.11-1.fc41
                           gstreamer1-plugins-bad-free-libs 1.24.10-1.fc41 -> 1.24.11-1.fc41
                           gstreamer1-plugins-base 1.24.10-1.fc41 -> 1.24.11-1.fc41
                           gstreamer1-plugins-good 1.24.10-1.fc41 -> 1.24.11-1.fc41
                           gstreamer1-plugins-good-qt 1.24.10-1.fc41 -> 1.24.11-1.fc41
                           gstreamer1-plugins-ugly-free 1.24.10-1.fc41 -> 1.24.11-1.fc41
                           hwdata 0.390-1.fc41 -> 0.391-1.fc41
                           ibus-chewing 2.1.2-1.fc41 -> 2.1.3-1.fc41
                           ibus-typing-booster 2.27.2-1.fc41 -> 2.27.7-1.fc41
                           kernel 6.12.6-200.fc41 -> 6.12.8-200.fc41
                           kernel-core 6.12.6-200.fc41 -> 6.12.8-200.fc41
                           kernel-modules 6.12.6-200.fc41 -> 6.12.8-200.fc41
                           kernel-modules-core 6.12.6-200.fc41 -> 6.12.8-200.fc41
                           kernel-modules-extra 6.12.6-200.fc41 -> 6.12.8-200.fc41
                           kernel-tools 6.12.6-200.fc41 -> 6.12.8-200.fc41
                           kernel-tools-libs 6.12.6-200.fc41 -> 6.12.8-200.fc41
                           libXrender 0.9.11-7.fc41 -> 0.9.12-1.fc41
                           libXxf86vm 1.1.5-7.fc41 -> 1.1.6-1.fc41
                           libcurl-minimal 8.9.1-2.fc41 -> 8.9.1-3.fc41
                           libdnf5 5.2.8.1-2.fc41 -> 5.2.8.1-3.fc41
                           libedit 3.1-53.20240808cvs.fc41 -> 3.1-54.20250104cvs.fc41
                           libical 3.0.18-2.fc41 -> 3.0.19-1.fc41
                           libical-glib 3.0.18-2.fc41 -> 3.0.19-1.fc41
                           libinput 1.27.0-2.fc41 -> 1.27.1-1.fc41
                           libnfsidmap 1:2.8.1-3.rc2.fc41 -> 1:2.8.1-4.rc2.fc41
                           libtirpc 1.3.6-1.fc41 -> 1.3.6-1.rc3.fc41
                           libxcrypt 4.4.36-12.fc41 -> 4.4.37-4.fc41
                           mesa-dri-drivers 24.2.8-1.fc41 -> 24.3.2-2.fc41
                           mesa-filesystem 24.2.8-1.fc41 -> 24.3.2-2.fc41
                           mesa-libEGL 24.2.8-1.fc41 -> 24.3.2-2.fc41
                           mesa-libGL 24.2.8-1.fc41 -> 24.3.2-2.fc41
                           mesa-libgbm 24.2.8-1.fc41 -> 24.3.2-2.fc41
                           mesa-libglapi 24.2.8-1.fc41 -> 24.3.2-2.fc41
                           mesa-va-drivers 24.2.8-1.fc41 -> 24.3.2-2.fc41
                           mesa-vulkan-drivers 24.2.8-1.fc41 -> 24.3.2-2.fc41
                           nfs-utils 1:2.8.1-3.rc2.fc41 -> 1:2.8.1-4.rc2.fc41
                           oniguruma 6.9.9-4.fc41 -> 6.9.10-1.fc41
                           openssl-libs 1:3.2.2-9.fc41 -> 1:3.2.2-11.fc41
                           policycoreutils 3.7-5.fc41 -> 3.7-6.fc41
                           policycoreutils-python-utils 3.7-5.fc41 -> 3.7-6.fc41
                           python3-audit 4.0.2-1.fc41 -> 4.0.3-1.fc41
                           python3-boto3 1.35.86-1.fc41 -> 1.35.92-1.fc41
                           python3-botocore 1.35.86-1.fc41 -> 1.35.92-1.fc41
                           python3-perf 6.12.6-200.fc41 -> 6.12.8-200.fc41
                           python3-policycoreutils 3.7-5.fc41 -> 3.7-6.fc41
                           selinux-policy 41.27-1.fc41 -> 41.28-1.fc41
                           selinux-policy-targeted 41.27-1.fc41 -> 41.28-1.fc41
                           systemd 256.10-1.fc41 -> 256.11-1.fc41
                           systemd-container 256.10-1.fc41 -> 256.11-1.fc41
                           systemd-libs 256.10-1.fc41 -> 256.11-1.fc41
                           systemd-networkd 256.10-1.fc41 -> 256.11-1.fc41
                           systemd-oomd-defaults 256.10-1.fc41 -> 256.11-1.fc41
                           systemd-pam 256.10-1.fc41 -> 256.11-1.fc41
                           systemd-resolved 256.10-1.fc41 -> 256.11-1.fc41
                           systemd-udev 256.10-1.fc41 -> 256.11-1.fc41
                           taglib 1.12-11.fc41 -> 1.13.1-1.fc41
                           vim-data 2:9.1.919-1.fc41 -> 2:9.1.984-1.fc41
                           vim-minimal 2:9.1.919-1.fc41 -> 2:9.1.984-1.fc41
                           xxhash-libs 0.8.2-4.fc41 -> 0.8.3-1.fc41
          LayeredPackages: dnf gnome-tweaks yq
            LocalPackages: epson-inkjet-printer-escpr-1.8.5-1.x86_64 epson-printer-utility-1.1.3-1.x86_64 iscan-2.30.4-2.x86_64 iscan-data-1.39.2-1.noarch iscan-network-nt-1.1.2-1.x86_64

● fedora:fedora/41/x86_64/silverblue (index: 1)
                  Version: 41.20241231.0 (2024-12-31T02:41:09Z)
               BaseCommit: 200cce846055ea8b606fc1008a0b36ec85f31bcf01b77ee9c34dfa7e07bd16ee
                           ├─ repo-0 (2024-10-24T13:55:59Z)
                           ├─ repo-1 (2024-12-31T02:18:47Z)
                           └─ repo-2 (2024-12-31T02:23:26Z)
                   Commit: 481274306f2fac085d714817c67adcbd85ef901a9f1c1de138a80fd164c37ab7
                StateRoot: fedora
             GPGSignature: 1 signature
                           Signature made mar 31 dic 2024, 03:41:56 using RSA key ID D0622462E99D6AD1
                           Good signature from "Fedora <fedora-41-primary@fedoraproject.org>"
          LayeredPackages: dnf gnome-tweaks yq
            LocalPackages: epson-inkjet-printer-escpr-1.8.5-1.x86_64 epson-printer-utility-1.1.3-1.x86_64 iscan-2.30.4-2.x86_64 iscan-data-1.39.2-1.noarch iscan-network-nt-1.1.2-1.x86_64
                   Pinned: yes

tqcharm · January 11, 2025, 7:54pm

The easiest solution is to roll back to the previous deployment, and stay on that deployment until the fixed mesa lands in the OSTree images:

rpm-ostree rollback

This option has the drawback that all packages remain on an earlier version.

Another option is to override the mesa packages^[1], which will allow upgrades to newer OSTree images and only keep the specific mesa packages on an earlier version:

sudo rpm-ostree override replace \
https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-dri-drivers-24.2.8-1.fc41.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-filesystem-24.2.8-1.fc41.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libgbm-24.2.8-1.fc41.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libglapi-24.2.8-1.fc41.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libEGL-24.2.8-1.fc41.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libGL-24.2.8-1.fc41.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-vulkan-drivers-24.2.8-1.fc41.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-va-drivers-24.2.8-1.fc41.x86_64.rpm

This can be done either from the last deployment with issues, by accessing a text console (with Ctrl+Alt+F3 or Ctrl+Alt+F2), or from the working deployment^[2], if it’s easier to perform the command in a graphical session.

When the mesa fix lands in the repos, remove the override:

sudo rpm-ostree override reset \
mesa-dri-drivers \
mesa-filesystem \
mesa-libGL \
mesa-va-drivers \
mesa-libglapi \
mesa-libgbm \
mesa-libEGL \
mesa-vulkan-drivers

Download the first package from the list locally in case you receive time-outs. ↩︎
After having performed a rollback. ↩︎

fannullone · January 12, 2025, 9:15am

I opt for the ‘rpm-ostree override replace’ but getting the following error:

Summary

root@fc40sb:~# ./mesa-fix.bash
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-dri-drivers-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-filesystem-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libgbm-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libglapi-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libEGL-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libGL-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-vulkan-drivers-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-va-drivers-24.2.8-1.fc41.x86_64.rpm...done
Checking out tree 100dce8... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora copr:copr.fedorainfracloud.org:phracek:PyCharm google-chrome rpmfusion-nonfree-nvidia-driver rpmfusion-nonfree-steam updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2024-03-11T19:22:31Z solvables: 3
rpm-md repo 'updates' (cached); generated: 2025-01-12T01:33:29Z solvables: 16854
rpm-md repo 'fedora' (cached); generated: 2024-10-24T13:55:59Z solvables: 76624
rpm-md repo 'copr:copr.fedorainfracloud.org:phracek:PyCharm' (cached); generated: 2024-12-16T12:28:59Z solvables: 10
rpm-md repo 'google-chrome' (cached); generated: 2025-01-11T21:19:11Z solvables: 4
rpm-md repo 'rpmfusion-nonfree-nvidia-driver' (cached); generated: 2024-12-17T11:02:42Z solvables: 17
rpm-md repo 'rpmfusion-nonfree-steam' (cached); generated: 2024-12-02T07:45:38Z solvables: 2
rpm-md repo 'updates-archive' (cached); generated: 2025-01-12T02:01:31Z solvables: 22813
Resolving dependencies... done
error: Sorting packages: Loading pkgcache branch rpmostree/pkg/mesa-dri-drivers/24.2.8-1.fc41.x86__64: Missing metadata key rpmostree.sepolicy
root@fc40sb:~#

Please note that selinux is disbled and I have applied it from the broken deploy via an ssh session

tqcharm · January 12, 2025, 11:22am

This looks related to SELinux, and not necessarily to the workaround. This Bugzilla issue looks similar to what you’re experiencing, even though it was filed for an old version of Fedora.

Why not trying to run the override on the working deployment, i.e.:

boot into the working deployment and log in with a regular, wheel user
(optional: rpm-ostree rollback)
enable SELinux to enforcing (or permissive mode in case there is a reason you can’t use enforcing)
run sudo rpm-ostree override replace <mesa-packages> and reboot
run rpm-ostree upgrade in case the override was successful, in order to be on the latest version of the packages except for mesa.

fannullone · January 12, 2025, 3:02pm

Thanks a lot!
This worked ^^, running now gnome session with latest packages except mesa which is pinned to latest working version 24.2.8/1.fc41.
Short summary:

boot to broken deploy: 41.20250112.0 getting a dark screen
ssh to the machine
set selinux in permissive mode
reboot
reboot to broken deploy: 41.20250112.0 getting a dark screen
ssh to the machine
rpm-ostree override replace
set selinux back into disabled mode
reboot; now gnome session works

Shall we open a bugzilla to fix this mesa bug?

session log

root@fc40sb:~# cat mesa-fix.bash
rpm-ostree override replace \
  https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-dri-drivers-24.2.8-1.fc41.x86_64.rpm \
  https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-filesystem-24.2.8-1.fc41.x86_64.rpm \
  https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libgbm-24.2.8-1.fc41.x86_64.rpm \
  https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libglapi-24.2.8-1.fc41.x86_64.rpm \
  https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libEGL-24.2.8-1.fc41.x86_64.rpm \
  https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libGL-24.2.8-1.fc41.x86_64.rpm \
  https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-vulkan-drivers-24.2.8-1.fc41.x86_64.rpm \
  https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-va-drivers-24.2.8-1.fc41.x86_64.rpm

root@fc40sb:~# ./mesa-fix.bash
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-dri-drivers-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-filesystem-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libgbm-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libglapi-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libEGL-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-libGL-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-vulkan-drivers-24.2.8-1.fc41.x86_64.rpm...done
Downloading https://kojipkgs.fedoraproject.org//packages/mesa/24.2.8/1.fc41/x86_64/mesa-va-drivers-24.2.8-1.fc41.x86_64.rpm...done
Checking out tree 100dce8... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora copr:copr.fedorainfracloud.org:phracek:PyCharm google-chrome rpmfusion-nonfree-nvidia-driver rpmfusion-nonfree-steam updates-archive
Updating metadata for 'updates'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2024-03-11T19:22:31Z solvables: 3
rpm-md repo 'updates'; generated: 2025-01-12T01:33:29Z solvables: 16854
rpm-md repo 'fedora' (cached); generated: 2024-10-24T13:55:59Z solvables: 76624
rpm-md repo 'copr:copr.fedorainfracloud.org:phracek:PyCharm' (cached); generated: 2024-12-16T12:28:59Z solvables: 10
rpm-md repo 'google-chrome' (cached); generated: 2025-01-11T21:19:11Z solvables: 4
rpm-md repo 'rpmfusion-nonfree-nvidia-driver' (cached); generated: 2024-12-17T11:02:42Z solvables: 17
rpm-md repo 'rpmfusion-nonfree-steam' (cached); generated: 2024-12-02T07:45:38Z solvables: 2
rpm-md repo 'updates-archive'; generated: 2025-01-12T02:01:31Z solvables: 22813
Resolving dependencies... done
Relabeling... done
Applying 8 overrides and 28 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 256,7 MB (pkgcache branches: 0)
Downgraded:
  mesa-dri-drivers 24.3.2-2.fc41 -> 24.2.8-1.fc41
  mesa-filesystem 24.3.2-2.fc41 -> 24.2.8-1.fc41
  mesa-libEGL 24.3.2-2.fc41 -> 24.2.8-1.fc41
  mesa-libGL 24.3.2-2.fc41 -> 24.2.8-1.fc41
  mesa-libgbm 24.3.2-2.fc41 -> 24.2.8-1.fc41
  mesa-libglapi 24.3.2-2.fc41 -> 24.2.8-1.fc41
  mesa-va-drivers 24.3.2-2.fc41 -> 24.2.8-1.fc41
  mesa-vulkan-drivers 24.3.2-2.fc41 -> 24.2.8-1.fc41
Use "rpm-ostree override reset" to undo overrides
Run "systemctl reboot" to start a reboot
root@fc40sb:~#

tqcharm · January 12, 2025, 5:15pm

Glad to hear it worked!

You could do that, but it might be rejected as an upstream issue. See here the list of open issues against Mesa on Freedesktops’s GitLab. I see two of them referring to this issue on Fedora specifically^[1], you could add a comment on one of them.

Fedora run in virtual machines though. ↩︎

Topic		Replies	Views
Fedora Silverblue can't upgrade to Fedora 41 due to missing layered packages Ask Fedora rpm-ostree , silverblue , f40 , f41 , atomic-desktops	6	443	November 1, 2024
Upgrade Silverblue Fedora 37 to 38 "Unable to Upgrade" "Could not depsolve Transaction" "Conflicting Requests" Ask Fedora f37 , f38 , gnome-software , gnome , nvidia , silverblue	1	813	May 22, 2023
Stuck on 39.20231205.0 Ask Fedora gnome , intel , silverblue	6	12292	December 9, 2023
Problem upgrading from Fedora Silverblue 38 to 39 Ask Fedora silverblue-team , f38 , nvidia , silverblue	3	668	November 15, 2023
Silverblue can't boot after update Ask Fedora silverblue-team , f38 , silverblue	1	530	July 19, 2023

Silverblue fix for f41 mesa problem

Related topics