I seem to have some kind of memory issue on my Silverblue laptop. Unfortunately, I cannot find the problematic process using the system monitor or top.
As you can see here the memory usage is extreme. But when looking at the processes, there does not seem to be any problem. The system monitor is set to show all processes and I don’t see a difference when comparing with top.
Is there another way to find out what process is eating all my RAM? 
You can have top sort by percent of memory used. In the default configuration, it’s the column to the right of percent CPU. Since it sorts on the CPU column by default, you can just type a “>” and it will move right one column for the sort.
I see just the same in top as in system monitor. The memory consumption does not add up. Several gigabytes are used but only a few hundred are accounted for.
Try ps aux --sort -rss
I just ran this on my Silverblue system and the top hits from the list are the usual suspects for memory usage for me (Firefox, Thunderbird, GNOME shell, Signal…)
Maybe try checking how much RAM the kernel is using? If it really seems invisible it might be a kernel module / driver issue.
I have encountered the issue again after some long running selenium tests.
ps aux --sort -rss shows the same as top:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
lennart 9231 5.5 4.5 2090396 175828 tty2 Sl+ 14:53 21:38 firefox-bin
lennart 9944 0.0 3.5 1656720 140628 tty2 Sl+ 14:58 0:16 /app/lib/firefox/firefox -contentproc -childID 6 -isForBrowser -prefsLen 6791 -prefMapSize 171660 -schedulerPrefs 0001,2 -parentBuildID 20181031020034 -greomni /app/lib/firefox/omni.ja -appomni /app/lib/firefox/browser/omni.ja -appdir /app/lib/firefox/browser 3 tab
lennart 9359 11.3 2.1 1837868 84804 tty2 Sl+ 14:53 44:10 /app/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -prefsLen 83 -prefMapSize 171660 -schedulerPrefs 0001,2 -parentBuildID 20181031020034 -greomni /app/lib/firefox/omni.ja -appomni /app/lib/firefox/browser/omni.ja -appdir /app/lib/firefox/browser 3 tab
lennart 1568 0.4 2.0 3393472 81160 tty2 Dl+ nov01 11:46 /usr/bin/gnome-shell
lennart 9306 0.3 1.3 1642784 53316 tty2 Sl+ 14:53 1:24 /app/lib/firefox/firefox -contentproc -childID 1 -isForBrowser -prefsLen 1 -prefMapSize 171660 -schedulerPrefs 0001,2 -parentBuildID 20181031020034 -greomni /app/lib/firefox/omni.ja -appomni /app/lib/firefox/browser/omni.ja -appdir /app/lib/firefox/browser 3 tab
lennart 9455 0.5 0.9 2165560 36636 tty2 Sl+ 14:53 2:16 /app/lib/firefox/firefox -contentproc -childID 3 -isForBrowser -prefsLen 5444 -prefMapSize 171660 -schedulerPrefs 0001,2 -parentBuildID 20181031020034 -greomni /app/lib/firefox/omni.ja -appomni /app/lib/firefox/browser/omni.ja -appdir /app/lib/firefox/browser 3 tab
lennart 13462 0.5 0.5 1495020 22716 tty2 Sl+ 21:20 0:00 /app/lib/firefox/firefox -contentproc -childID 7 -isForBrowser -prefsLen 9199 -prefMapSize 171660 -schedulerPrefs 0001,2 -parentBuildID 20181031020034 -greomni /app/lib/firefox/omni.ja -appomni /app/lib/firefox/browser/omni.ja -appdir /app/lib/firefox/browser 3 tab
lennart 13508 2.1 0.5 560740 22600 ? Ssl 21:21 0:02 /usr/libexec/gnome-terminal-server
root 5454 0.1 0.5 250448 20972 ? Ss 12:58 0:58 /usr/libexec/sssd/sssd_kcm --uid 0 --gid 0 --logger=files
lennart 6244 2.1 0.3 578568 15076 tty2 Sl+ 13:12 10:28 gnome-system-monitor
root 596 0.0 0.2 59728 8092 ? Ss nov01 0:01 /usr/lib/systemd/systemd-journald
lennart 13627 0.0 0.1 226028 5136 pts/0 R+ 21:22 0:00 ps aux --sort -rss
lennart 13548 0.0 0.0 223624 3784 pts/0 Ss 21:21 0:00 bash
lennart 1940 0.0 0.0 1702180 3408 tty2 Sl+ nov01 0:11 /usr/bin/gnome-software --gapplication-service
lennart 1716 0.0 0.0 598984 2936 tty2 Sl+ nov01 0:04 /usr/libexec/gsd-sharing
dbus 775 0.0 0.0 25820 2876 ? Ssl nov01 0:10 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 816 0.0 0.0 270824 2776 ? S nov01 0:00 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
lennart 1626 0.0 0.0 527972 2740 tty2 Sl nov01 0:15 ibus-daemon --xim --panel disable
lennart 1742 0.0 0.0 1003512 2724 tty2 Sl+ nov01 0:02 /usr/libexec/gsd-color
lennart 3397 0.0 0.0 521388 2656 ? Sl nov01 0:00 /usr/libexec/gvfsd-trash --spawner :1.4 /org/gtk/gvfs/exec_spaw/0
lennart 1635 0.0 0.0 546888 2636 tty2 Sl nov01 0:03 /usr/libexec/ibus-extension-gtk3
lennart 1934 0.0 0.0 723472 2616 tty2 DNl+ nov01 0:00 /usr/libexec/tracker-miner-fs
root 851 0.0 0.0 1153140 2548 ? Ssl nov01 0:17 /usr/sbin/NetworkManager --no-daemon
lennart 1602 0.0 0.0 491016 2460 tty2 Sl+ nov01 2:06 /usr/bin/Xwayland :0 -rootless -terminate -accessx -core -listen 4 -listen 5 -displayfd 6
...
Output from top:
Tasks: 218 total, 1 running, 217 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3,6 us, 2,4 sy, 0,1 ni, 93,0 id, 0,1 wa, 0,7 hi, 0,2 si, 0,0 st
MiB Mem : 3815,4 total, 118,6 free, 3462,6 used, 234,2 buff/cache
MiB Swap: 3944,0 total, 2367,3 free, 1576,7 used. 72,4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9359 lennart 20 0 1894188 186156 42880 S 3,6 4,8 44:32.92 Web Content
9231 lennart 20 0 2067796 164264 52424 S 3,3 4,2 22:33.44 firefox-bin
1568 lennart 20 0 3392560 76632 35436 S 4,6 2,0 12:45.70 gnome-shell
9306 lennart 20 0 1644832 41324 20012 S 0,0 1,1 1:28.00 Web Content
5454 root 20 0 250448 21004 344 S 0,3 0,5 1:00.08 sssd_kcm
6244 lennart 20 0 578816 19916 12168 S 6,6 0,5 11:23.22 gnome-system-mo
13508 lennart 20 0 567052 18812 11584 S 0,3 0,5 0:07.37 gnome-terminal-
9455 lennart 20 0 2179896 18624 4752 S 0,3 0,5 2:19.67 WebExtensions
1939 lennart 20 0 794276 7100 4032 S 0,0 0,2 0:00.37 tracker-store
13462 lennart 20 0 1495020 4928 3196 S 0,0 0,1 0:00.80 Web Content
13911 lennart 20 0 226356 4920 4008 R 0,0 0,1 0:00.36 top
1934 lennart 39 19 723472 4848 4848 S 0,0 0,1 0:00.31 tracker-miner-f
596 root 20 0 59728 4288 3732 S 0,0 0,1 0:01.17 systemd-journal
851 root 20 0 1153140 4200 2348 S 0,0 0,1 0:17.93 NetworkManager
1626 lennart 20 0 527972 2832 1844 S 0,0 0,1 0:17.36 ibus-daemon
1712 lennart 20 0 906232 2544 1792 S 0,0 0,1 0:00.65 gsd-power
1635 lennart 20 0 546888 2528 1860 S 0,0 0,1 0:03.72 ibus-extension-
775 dbus 20 0 25820 2492 612 S 0,0 0,1 0:10.37 dbus-daemon
1716 lennart 20 0 598984 2352 1740 S 0,0 0,1 0:04.43 gsd-sharing
1742 lennart 20 0 1003512 2300 1788 S 0,0 0,1 0:02.97 gsd-color
1442 lennart 20 0 20036 2296 1644 S 0,0 0,1 0:00.37 systemd
1668 lennart 20 0 155948 2284 1764 S 0,0 0,1 0:00.14 dconf-service
1675 lennart 20 0 458916 2264 1608 S 0,0 0,1 0:00.41 gvfs-udisks2-vo
1526 lennart 20 0 443808 2104 1632 S 0,0 0,1 0:00.15 gvfsd
1693 lennart 20 0 439904 2100 1692 S 0,0 0,1 0:00.04 gvfs-goa-volume
1464 lennart 20 0 35836 2060 696 S 0,0 0,1 0:01.60 dbus-daemon
1833 lennart 20 0 370588 2032 1640 S 0,0 0,1 0:04.57 ibus-engine-sim
1743 lennart 20 0 640204 1936 1936 S 0,0 0,0 0:00.22 gsd-keyboard
1686 lennart 20 0 442460 1896 1576 S 0,0 0,0 0:00.04 gvfs-gphoto2-vo
1704 lennart 20 0 439832 1840 1472 S 0,0 0,0 0:00.04 gvfs-mtp-volume
I also tried to investigate how much the kernel is using, but I’m not sure I understood it completely. Anyway, this is what I found:
$ grep Slab /proc/meminfo
Slab: 213072 kB
Obviously, this is no where close to the missing memory.
It may be worth mentioning that I run the tests in containers using podman, and that all the containers were stopped and deleted when I took the readings above.
How about the other command in the post, awk '{print $1 " " $2 }' /proc/modules?
Sorry for the delay, here is the output from that command. I’m not sure how to interpret it unfortunately.
$ awk '{print $1 " " $2 }' /proc/modules
fuse 122880
ccm 20480
rfcomm 86016
devlink 61440
nf_conntrack_netbios_ns 16384
nf_conntrack_broadcast 16384
xt_CT 16384
ip6t_rpfilter 16384
ip6t_REJECT 16384
nf_reject_ipv6 16384
xt_conntrack 16384
ebtable_nat 16384
ip6table_nat 16384
nf_conntrack_ipv6 16384
nf_defrag_ipv6 20480
nf_nat_ipv6 16384
ip6table_mangle 16384
ip6table_raw 16384
ip6table_security 16384
iptable_nat 16384
nf_conntrack_ipv4 16384
nf_defrag_ipv4 16384
nf_nat_ipv4 16384
nf_nat 36864
iptable_mangle 16384
iptable_raw 16384
iptable_security 16384
nf_conntrack 147456
libcrc32c 16384
ip_set 45056
nfnetlink 16384
ebtable_filter 16384
ebtables 36864
ip6table_filter 16384
ip6_tables 32768
bnep 24576
sunrpc 430080
arc4 16384
rtl8723be 106496
btcoexist 172032
rtl8723_common 28672
rtl_pci 36864
intel_rapl 24576
rtlwifi 98304
x86_pkg_temp_thermal 16384
intel_powerclamp 16384
coretemp 16384
kvm_intel 237568
iTCO_wdt 16384
mac80211 913408
kvm 737280
iTCO_vendor_support 16384
snd_hda_codec_realtek 110592
snd_hda_codec_hdmi 57344
snd_hda_codec_generic 86016
snd_hda_intel 45056
irqbypass 16384
snd_hda_codec 151552
intel_cstate 16384
btusb 53248
intel_uncore 135168
btrtl 16384
btbcm 16384
cfg80211 778240
btintel 24576
snd_hda_core 94208
intel_rapl_perf 16384
uvcvideo 110592
snd_hwdep 16384
snd_seq 81920
videobuf2_vmalloc 16384
videobuf2_memops 16384
bluetooth 598016
videobuf2_v4l2 28672
snd_seq_device 16384
videobuf2_common 53248
snd_pcm 114688
videodev 217088
hp_wmi 16384
joydev 24576
wmi_bmof 16384
sparse_keymap 16384
media 45056
lpc_ich 28672
r8169 90112
ecdh_generic 24576
intel_pch_thermal 16384
snd_timer 36864
rtsx_pci_ms 20480
mei_me 45056
memstick 16384
rfkill 28672
snd 94208
mii 16384
mei 114688
soundcore 16384
hp_accel 28672
lis3lv02d 24576
input_polldev 16384
tpm_infineon 20480
pcc_cpufreq 16384
hp_wireless 16384
dm_multipath 32768
i915 2076672
rtsx_pci_sdmmc 28672
mmc_core 172032
i2c_algo_bit 16384
crct10dif_pclmul 16384
drm_kms_helper 196608
crc32_pclmul 16384
crc32c_intel 24576
drm 475136
ghash_clmulni_intel 16384
serio_raw 16384
rtsx_pci 73728
wmi 28672
video 45056
Hmm, nothing there either… Only other idea is that you have a tmpfs that’s getting too full. Do you see any in mount? Are you actively using any Flatpaks or containers?
I’m using the flatpak for Firefox Nightly and occasionally slack and telegram. I have been using some containers running with podman, but not all the times this has happened.
Last night, I left the laptop on with Firefox running. In the morning, I was greeted with a blinking cursor. Memory usage was at 100 % again. I was able to log in in text mode and get the output from mount:
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1926756k,nr_inodes=481689,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/fedora_legolas-root on /sysroot type ext4 (rw,relatime,seclabel)
/dev/mapper/fedora_legolas-root on / type ext4 (rw,relatime,seclabel)
/dev/mapper/fedora_legolas-root on /usr type ext4 (ro,relatime,seclabel)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18028)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel)
/dev/mapper/fedora_legolas-root on /var type ext4 (rw,relatime,seclabel)
/dev/sda1 on /boot type ext4 (rw,relatime,seclabel)
/dev/mapper/fedora_legolas-home on /var/home type ext4 (rw,relatime,seclabel)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /run/user/42 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=390696k,mode=700,uid=42,gid=42)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=390696k,mode=700,uid=1000,gid=1000)
Hmm, so then maybe it could be related to Firefox? Unfortunately, the mount command doesn’t show private mounts and therefore isn’t of much help… If this occurs again, try running:
flatpak run --command=df org.mozilla.FirefoxNightly
which should show the disk usage from inside the Flatpak.
Nothing weird there as far as I can tell:
$ flatpak run --command=df org.mozilla.FirefoxNightly
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 1952864 100 1952764 1% /.flatpak-info
/dev/mapper/fedora_legolas-root 51343840 27214388 21491628 56% /usr
tmpfs 1952864 0 1952864 0% /usr/share/runtime/share/icons
/dev/mapper/fedora_legolas-home 65928664 26391452 36145112 43% /var/data
tmpfs 390572 8248 382324 3% /run/host/monitor
tmpfs 1952864 0 1952864 0% /dev
devtmpfs 1924544 0 1924544 0% /dev/tty
tmpfs 1952864 0 1952864 0% /tmp/.X11-unix
tmpfs 1952864 92 1952772 1% /tmp/.X11-unix/X99
tmpfs 1952864 1668 1951196 1% /run/systemd/journal/socket
Some processes just seem to hide their memory usage after a few hours. They still show up in the system monitor, but their memory usage is N/A.
After a restart I get this instead.
Note for example all the bwrap and firefox processes. They don’t seem to consume that much memory, but since they sometimes report N/A I still suspect them.
Man this is weird…
I’d say maybe try leaving Firefox open one night, leaving it closed the next night, etc. to try and verify if it’s that. My best guess is still a tmpfs somewhere, though I’m admittedly not sure where… I can see if maybe I can conjure up some shell command to check different process mounts?
(FWIW I think N/A here just means memory usage is negligible?)
