Repeated full-system freezes/OOM crashes on Silverblue + GNOME Wayland + NVIDIA, root cause found (known upstream Mutter bug)

Ask Fedora
1

Posting this because it took real digging to root-cause, and I think other NVIDIA + GNOME Wayland users on Silverblue are probably hitting it without knowing why.

Symptom: system freezes completely or crashes outright, requiring a hard power-cycle to recover. Happened 5 times in one day for me before I dug into it properly.

Setup: Fedora Silverblue 44, GNOME Shell 50.4 / Mutter 50.4, kernel 7.1.8-200.fc44.x86_64, NVIDIA driver 610.57.04 (akmod-nvidia), Dell XPS 15 9510, Wayland session.

Root cause: this is a known, still-open upstream Mutter/GNOME Shell bug (gnome-shell#5729, open since 2022) where memfd-backed shared memory buffers used for window pixel data aren’t released promptly under Wayland on NVIDIA. It’s NVIDIA-specific (the original report confirms it doesn’t happen on sway/wayfire/KWin, only gnome-shell), and grows with normal desktop use: window open/close, workspace switching, opening the overview.

In my case it’s escalated past “gradual growth” to full system-wide Shmem exhaustion. A real kernel OOM report from one crash:

shmem:27550308kB shmem_thp:20226048kB

That’s ~26GB of a 31GB system tied up in shared memory, ~19GB of it in transparent huge pages. /dev/shm, /tmp, and /run were all confirmed empty at the time, so it’s genuinely anonymous memfd growth, not something writable/clearable via a normal tmpfs mount.

How I confirmed it, in case it helps someone else diagnose the same thing:

  • journalctl -k --since for “invoked oom-killer” / “Killed process” to find real kernel OOM events (many freezes never reach this point and just hang instead, so absence of this log doesn’t mean it’s not an OOM-class issue).
  • Checked /proc/meminfo’s Shmem/ShmemHugePages fields directly against /dev/shm, /tmp, /run disk usage to confirm the growth is anonymous, not a named tmpfs mount filling up.
  • Checked per-process RssShmem in /proc//status: the growth was diffuse across many processes rather than concentrated in one obvious app, which points at the compositor holding buffers rather than a single leaking application.

No real fix on my end yet beyond periodic reboots. This is an upstream compositor/driver interaction issue, not something fixable from userspace config as far as I can tell. Posting mainly so:

  1. Anyone else hitting mysterious freezes/crashes on this exact combo (Silverblue + GNOME Wayland + NVIDIA) has something to search for and match against.
  2. If Fedora’s NVIDIA packaging/COPR maintainers are aware of any Fedora-specific mitigation or want additional diagnostic data, I’m happy to provide it (I have a live memory-sampling log running now to catch the growth trend in real time).

Has anyone else on this combo hit the same thing? first OOMED happened 8.8.2026 then daily

2

Growth trend (6 samples, 20:14 → 20:26, ~12 minutes): Shmem climbed from ~1.19GB to ~1.58GB about 390MB in 12 minutes, and it’s been on every single sample, not sporadic.

The key finding: the top processes by RssShmem only add up to ~30-40MB total (Firefox ~12-21MB, gnome-shell steady at ~9.9MB, Firefox’s content process ~8-12MB) but system-wide Shmem is now 1.58GB. Over 1.5GB is unaccounted for by any live process’s own RssShmem. That’s the strongest evidence yet for the theory in the draft reports: buffers get shared with the Mutter compositor via memfd, the originating client’s own accounting drops or shrinks, but the compositor (or the kernel, via a lingering reference) keeps the pages alive anyway so it shows up in the global Shmem total without belonging to any process you can point at and kill.

At the current average rate (~32MB/min), it would take roughly 13-14 hours to reach the ~27.5GB level seen at the actual crash, though today’s crashes happened faster than that at times, so the rate likely isn’t constant and probably speeds up with more window/workspace activity (matches the original upstream report’s trigger list). It’ll keep logging in the background either way, so we’ll have a longer trend line whenever you check next.

3

With the information you provided it would seem wise to report a bug (possibly as critical) on bugzilla.redhat.com
I don’t know if this is specific to your hardware or not, but it would seem that potentially something that was updated on 8/8/2026 may be the cause. You might also report whatever was updated at that time as a suspect. (mutter? or something else?)

4

Link to the upstream bug

5

removing nvidia drivers fixed no more cache filling and memory etc going up and all is stable at this moment

6

Hmm, that makes sense to me :slight_smile:

I have 64GB, so it takes some time running around on my Jellyfin server updating metadata, images, covers a.s.o.

The other days, it has been mainly getting the data ready for the server and getting Docker and other things in place, so 64GB probably lasts a bit longer doing that.

What command did you use to find the mem-usages?

7

I gave claude to task debug and find and create script to monitor to see how things was going faster since i get OOM in 2 hours of usage and results are here what found on that time before i OOMed again :joy:

8

Hmm, i might be starting to understand it. When using:

watch -n 5 free -m

I can see the use of memory staying around 5.2-5.4 GB playing a 720p or 1080p movie. Playing a 4K makes it jump to 5.8 GB, but it drops after closing the player.

But what just keeps going up and up and …

Is the “buff/cache” use - it never gets released and available memory just gets slightly smaller by every video i play. Also browser Youtube eats a good chunk by the seconds.

And it just grows and doesn’t get released again.

Is there a way to clear that “buff/cache” thing?

9

Hmm, i don’t understand why:

vmstat 1 5

Show the “buff” usage 2 GB less :thinking:

EDIT: Looked at the wrong number :wink:

10

Yeqh similar what i have cache/sharedmem keeps growing more and more the more i use browser and watch netflix or youtube. I only restarted when first time i get information high usage of menory by gnome and then back again untiöl i switched to cosmic atomic no issues on taht side yet so…

25

I see that the OP on this topic ( @thephatlee )has not posted since post 10 and there is no indication if his issue was solved. I also see that all discussion here in post 11 and following is related to the issues for @boblebad .

I am moving all discussion, post 11 and after to the other noted topic that was started by @boblebad

Also note that the OP here is using silverblue and @boblebad is on workstation to this is 2 completely different topics.

26

didint want to comment here as it was so active. My silverblue is not solved, but i solved all the issues moving to cosmic-atomic no more oom killers no more out of memories and so on so as OP post this seems to be related on gnome only

edit

on cosmic 6 hours full streaming videos, music and doing the work memory still all good 6GB RAM usage and 12GB cache usage