My system started experiencing instability with gnome after upgrading from fedora 41 to fedora 42. Things worked fine under fedora 41. There are problems with both gdm and gnome-session.
gnome-session:
When logged in, doing absolutely nothing, memory use steadily rises eventhough no particular process in the system monitor shows any process consuming much memory at all. “smem -kw” shows kernel memory steadily rising - both cached and non-cached. If left to go long enough (a few hours), the system (32Gb) will eventually start swapping severely, and random processes start getting killed. If I log out before it gets terrible, the memory is freed pretty much instantly - at least according to “smem -kw”. I installed kde plasma to test if it also has this behavior. It does not. KDE plasma appears to work without issue. When I first upgraded to fedora 42, there was no session choice in the gdm menu for “gnome on xorg”. At some point I guess it was brought back, because it is available now. If I run gnome on Xorg, I also do not experience the memory leak. gnome-session appears to work without error on Xorg.
gdm
GDM seems buggy. Moving the mouse over a login name might cause gdm to freeze on that login name, but accept no further button clicks, mouse motion, keyboard input etc… A quick Ctrl-Alt F2 / Ctrl-Alt F1 to go to a virtual console then back to gdm always unsticks it. When the gdm screen is active and no gnome-sessions are active “smem -kw” shows a similar memory growth pattern as seen when logged in. The Ctrl-Alt F2 / Ctrl-Alt F1 instantly frees up the memory. I tried another login manager, sddm. It does not have this problem. If I uncomment “#WaylandEnable=false” from /etc/gdm/custom.conf, gdm works without error.
From these two observations, I’ve concluded that my system has issues with gnome under wayland. Since it worked perfectly well on fedora 41, I’m assuming there is an issue in either the newer version of gnome or wayland that is causing me problems.
My system has an AMD 6600 XT and intel integrated graphics. I tried blacklisting the amdgpu to test with integrated graphics using the i915 driver. The same bug exists. So the problem doesn’t appear to be display driver related.
Has anyone encountered something similar? Any tips on how to investigate further? Does this seem like a bug that should be reported?
Install btop (other top programs exist, I just happen to have btop installed).
Take a screenshot of a fresh system, showing the memory usage by processes. Maybe there’s something baked into gnome for this already - I don’t use gnome so I have no idea.)
Use the system as you would normally, do whatever you do when you notice this memory usage increasing.
Take a second screenshot when you have come to the conclusion that all memory is in use.
Compare the processes and the memory they consume. What is the largest consumer of your ram?
For Btop, I used the left arrow key to scroll the process display to sort by decreasing memory, as in:
Here is a screenshot of btop (processes sorted by mem) when running gnome under wayland. The numbers are very suspicious. Summing the process memory comes out to around ~1.9G. But total Used memory is 13.9G. I think that means that the kernel itself is using the difference ~12G. This leaves an available memory of 17.1G. The Cached memory is listed at 16.6G. This is a bit suspicious only because I’ve done nothing in this shell other than just login and let it sit idle. What was accessing so many files causing the cache to build up? The free number is listed at 12.7G which makes no sense to me at all. My understanding is that number should roughly be the available memory minus the cached memory.
I don’t observer this behavior when running gnome under Xorg, nor when running KDE under wayland. Under those configurations, the cached memory remains low if the system is idle and the memory numbers add up in sensible ways. My suspicion is that something in the integration of gnome with wayland is causing the kernel itself to leak memory on my system. But exactly what, I’m not sure. The system worked as expected under f41.
My first suspicion was an issue with the amdgpu driver. I blacklisted the amdgpu driver, relying on the integrated graphics. The behavior still persisted. I’ve abandoned that as a possible cause.
(in joke, please disregard)