Systemd-oomd not working on my desktop

Ask Fedora
, , ,
1

Due to an annoying memory leak with Firefox that happens seemingly randomly every 24 h or so, my system has been hard freezing a lot and requiring me to do a hard reset every time.

I did some research and found out about systemd-oomd and got confused as to why it wasn’t killing Firefox, so I did some experiments.

Running the command tail /dev/zero on my main desktop PC with systemd-oomd running, the system hard freezes just like when the Firefox memory leak occurs. However, if I instead use earlyoom and disable systemd-oomd, it kills the process as you would expect.

If I try the same command, on my laptop, using the default systemd-oomd, it works as expected and kills the process. Both my desktop PC and my laptop have nvidia gpus with the proprietary drivers running and very similar packages installed.

I am very confused as to why systemd-oomd works on my laptop, but not on my desktop. The only journal logs from systemd-oomd are it starting and stopping whenever I start / stop my pc, there are no logs other than that.

3

Here it is,

               total        used        free      shared  buff/cache   available
Mem:            31Gi        16Gi       318Mi       221Mi        15Gi        15Gi
Swap:          8.0Gi       4.2Gi       3.8Gi
Filename                                Type            Size            Used            Priority
/dev/zram0                              partition       8388604         4455552         100
NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle         8G  4.2G  1.6G  1.6G      12 [SWAP]
● systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer
     Loaded: loaded (/usr/lib/systemd/system/systemd-oomd.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Sat 2024-08-10 10:35:33 CEST; 4min 11s ago
TriggeredBy: ● systemd-oomd.socket
       Docs: man:systemd-oomd.service(8)
             man:org.freedesktop.oom1(5)
   Main PID: 66120 (systemd-oomd)
     Status: "Processing requests..."
      Tasks: 1 (limit: 38312)
     Memory: 1.3M (min: 64.0M low: 64.0M peak: 3.6M)
        CPU: 82ms
     CGroup: /system.slice/systemd-oomd.service
             └─66120 /usr/lib/systemd/systemd-oomd

aug 10 10:35:33 pc systemd[1]: Starting systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer...
aug 10 10:35:33 pc systemd[1]: Started systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer.
1 Like
4

No that I think it would be super useful but are some logs when triggering a memory leak with tail /dev/zero and manually activating the kernel oom with Alt + SysRq + f:

aug 10 15:53:35 pc kernel: kworker/8:2 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=-1, oom_score_adj=0
aug 10 15:53:35 pc kernel: Workqueue: events moom_callback
aug 10 15:53:35 pc kernel:  oom_kill_process.cold+0xa/0xaa
aug 10 15:53:35 pc kernel:  moom_callback+0x7a/0xb0
aug 10 15:53:35 pc kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
aug 10 15:53:35 pc kernel: [ 137814]   998 137814     4125      235      192       43         0    69632        0          -900 systemd-oomd
aug 10 15:53:35 pc kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/app.slice/run-r668b349b9a9d4a959011d53056ab4fe7.service,task=tail,pid=140920,uid=1000
aug 10 15:53:35 pc kernel: Out of memory: Killed process 140920 (tail) total-vm:16345100kB, anon-rss:16118016kB, file-rss:120kB, shmem-rss:0kB, UID:1000 pgtables:31616kB oom_score_adj:200
aug 10 15:53:35 pc systemd[4172]: run-r668b349b9a9d4a959011d53056ab4fe7.service: Failed with result 'oom-kill'.

I also found these issues which I think are related,

I guess I’ll just use earlyoom instead?

5

After two more hard freezes even with earlyoom I did some more research and experimenting and it turns out that my ram is most likely failing or just bad, since turning off the XMP profile completely solved everything.

Tldr: If you’re experiencing this, try turning off the XMP profile in UEFI settings

1 Like
6

Ignore that last comment lol, it’s broken again. I have done more digging and found this bug report 2248071 – systemd-oomd doesn't kick in on high memory pressure, leading to system lockup which is the same that I have.

7

Are you on an Intel system?

8

yes, i7 8700k