OOM out of memory errors, Bad RIP value; hardware tests passed

Ask Fedora Ask in English
1

I’ve run all firmware updates, dnf updates and still OOM’s on various processes a couple times an hour. Could this be a bug? The 5 hour hardware test within the Dell BIOS passed see screen shot, no errors. No errors in the Dell debug logs within the Lifecyle Controller. Kernel is 5.2.17-200.fc30.x86_6 but it’s been happening for a few kernel versions.

Oct  2 23:47:04 mandelbrot kernel: MLM invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Oct  2 23:47:05 mandelbrot kernel: CPU: 11 PID: 1335 Comm: MLM Not tainted 5.2.17-200.fc30.x86_64 #1
Oct  2 23:47:05 mandelbrot kernel: Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.3.10 08/15/2019
Oct  2 23:47:05 mandelbrot kernel: Call Trace:
Oct  2 23:47:05 mandelbrot kernel: dump_stack+0x5c/0x80
Oct  2 23:47:05 mandelbrot kernel: dump_header+0x51/0x2ff
Oct  2 23:47:05 mandelbrot kernel: ? sched_clock+0x5/0x10
Oct  2 23:47:05 mandelbrot kernel: ? oom_unkillable_task+0x9a/0xd0
Oct  2 23:47:05 mandelbrot kernel: oom_kill_process.cold+0xb/0x10
Oct  2 23:47:05 mandelbrot kernel: out_of_memory+0x1d4/0x480
Oct  2 23:47:05 mandelbrot kernel: __alloc_pages_slowpath+0xcc4/0xde0
Oct  2 23:47:05 mandelbrot kernel: __alloc_pages_nodemask+0x2ee/0x340
Oct  2 23:47:05 mandelbrot kernel: pagecache_get_page+0xb4/0x370
Oct  2 23:47:05 mandelbrot kernel: filemap_fault+0x683/0x9f0
Oct  2 23:47:05 mandelbrot kernel: ? __mod_lruvec_state+0x3f/0xe0
Oct  2 23:47:05 mandelbrot kernel: ? page_add_file_rmap+0xf4/0x230
Oct  2 23:47:05 mandelbrot kernel: ? alloc_set_pte+0x498/0x6b0
Oct  2 23:47:05 mandelbrot kernel: ? file_fdatawait_range+0x20/0x20
Oct  2 23:47:05 mandelbrot kernel: ? _cond_resched+0x15/0x30
Oct  2 23:47:05 mandelbrot kernel: __xfs_filemap_fault+0x6d/0x200 [xfs]
Oct  2 23:47:05 mandelbrot kernel: __do_fault+0x36/0x180
Oct  2 23:47:05 mandelbrot kernel: __handle_mm_fault+0x1091/0x1ac0
Oct  2 23:47:05 mandelbrot kernel: handle_mm_fault+0xc4/0x1e0
Oct  2 23:47:05 mandelbrot kernel: do_user_addr_fault+0x1f6/0x450
Oct  2 23:47:05 mandelbrot kernel: do_page_fault+0x33/0x120
Oct  2 23:47:05 mandelbrot kernel: ? page_fault+0x8/0x30
Oct  2 23:47:05 mandelbrot kernel: page_fault+0x1e/0x30
Oct  2 23:47:05 mandelbrot kernel: RIP: 0033:0x7fd8c0cfaffb
Oct  2 23:47:05 mandelbrot kernel: Code: Bad RIP value.
Oct  2 23:47:05 mandelbrot kernel: RSP: 002b:00007fd8c0bebe90 EFLAGS: 00010293
Oct  2 23:47:05 mandelbrot kernel: RAX: 0000000000000000 RBX: 000000000227ece0 RCX: 00007fd8c0cfaffb
Oct  2 23:47:05 mandelbrot kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Oct  2 23:47:05 mandelbrot kernel: RBP: 000000000227d260 R08: 00007fd8c0bebed0 R09: 0000000000000000
Oct  2 23:47:05 mandelbrot kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 00007fffb6aaf17e
Oct  2 23:47:05 mandelbrot kernel: R13: 00007fffb6aaf17f R14: 00007fffb6aaf200 R15: 00007fd8c0bebfc0
Oct  2 23:47:05 mandelbrot kernel: Mem-Info:
Oct  2 23:47:05 mandelbrot kernel: active_anon:15233105 inactive_anon:898356 isolated_anon:0#012 active_file:25 inactive_file:30 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:11388 slab_unreclaimable:30920#012 mapped:11 shmem:6 pagetables:44129 bounce:0#012 free:56588 free_pcp:0 free_cma:0
Oct  2 23:47:05 mandelbrot kernel: Node 0 active_anon:30085864kB inactive_anon:1774920kB active_file:0kB inactive_file:64kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:4kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Oct  2 23:47:05 mandelbrot kernel: Node 1 active_anon:30846556kB inactive_anon:1818504kB active_file:108kB inactive_file:56kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:68kB dirty:0kB writeback:0kB shmem:20kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Oct  2 23:47:05 mandelbrot kernel: Node 0 DMA free:15872kB min:20kB low:32kB high:44kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15972kB managed:15888kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  2 23:47:05 mandelbrot kernel: lowmem_reserve[]: 0 1396 31608 31608 31608
Oct  2 23:47:05 mandelbrot kernel: Node 0 DMA32 free:122816kB min:1968kB low:3396kB high:4824kB active_anon:1324160kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:1521664kB managed:1456128kB mlocked:0kB kernel_stack:0kB pagetables:2596kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  2 23:47:05 mandelbrot kernel: lowmem_reserve[]: 0 0 30212 30212 30212
Oct  2 23:47:05 mandelbrot kernel: Node 0 Normal free:42448kB min:42640kB low:73576kB high:104512kB active_anon:28761916kB inactive_anon:1774920kB active_file:0kB inactive_file:64kB unevictable:0kB writepending:0kB present:31457280kB managed:30937480kB mlocked:0kB kernel_stack:3336kB pagetables:84528kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  2 23:47:05 mandelbrot kernel: lowmem_reserve[]: 0 0 0 0 0
Oct  2 23:47:05 mandelbrot kernel: Node 1 Normal free:45216kB min:45476kB low:78468kB high:111460kB active_anon:30846556kB inactive_anon:1818504kB active_file:108kB inactive_file:56kB unevictable:0kB writepending:0kB present:33554432kB managed:33003212kB mlocked:0kB kernel_stack:2232kB pagetables:89392kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  2 23:47:05 mandelbrot kernel: lowmem_reserve[]: 0 0 0 0 0
Oct  2 23:47:05 mandelbrot kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15872kB
Oct  2 23:47:05 mandelbrot kernel: Node 0 DMA32: 0*4kB 0*8kB 4*16kB (U) 2*32kB (U) 1*64kB (U) 4*128kB (UM) 1*256kB (M) 4*512kB (UM) 3*1024kB (U) 1*2048kB (U) 28*4096kB (M) = 122816kB
Oct  2 23:47:05 mandelbrot kernel: Node 0 Normal: 204*4kB (UME) 231*8kB (UME) 102*16kB (UME) 95*32kB (UME) 21*64kB (UE) 6*128kB (ME) 4*256kB (UME) 3*512kB (UME) 10*1024kB (UME) 8*2048kB (UM) 1*4096kB (M) = 42728kB
Oct  2 23:47:05 mandelbrot kernel: Node 1 Normal: 358*4kB (UME) 188*8kB (UME) 102*16kB (UME) 43*32kB (UME) 17*64kB (UME) 10*128kB (UME) 3*256kB (UE) 4*512kB (UME) 4*1024kB (UME) 1*2048kB (E) 7*4096kB (M) = 45944kB
Oct  2 23:47:05 mandelbrot kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct  2 23:47:05 mandelbrot kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct  2 23:47:05 mandelbrot kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct  2 23:47:05 mandelbrot kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct  2 23:47:05 mandelbrot kernel: 2875 total pagecache pages
Oct  2 23:47:05 mandelbrot kernel: 2807 pages in swap cache
Oct  2 23:47:05 mandelbrot kernel: Swap cache stats: add 5867290, delete 5863717, find 830/4546
Oct  2 23:47:05 mandelbrot kernel: Free swap  = 0kB
Oct  2 23:47:05 mandelbrot kernel: Total swap = 23441404kB
Oct  2 23:47:05 mandelbrot kernel: 16637337 pages RAM
Oct  2 23:47:05 mandelbrot kernel: 0 pages HighMem/MovableOnly
Oct  2 23:47:05 mandelbrot kernel: 284160 pages reserved
Oct  2 23:47:05 mandelbrot kernel: 0 pages cma reserved
Oct  2 23:47:05 mandelbrot kernel: 0 pages hwpoisoned
Oct  2 23:47:05 mandelbrot kernel: Tasks state (memory values in pages):
Oct  2 23:47:05 mandelbrot kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Oct  2 23:47:05 mandelbrot kernel: [    857]     0   857    15284       30   143360      268             0 systemd-journal
Oct  2 23:47:05 mandelbrot kernel: [    882]     0   882    38364       12    61440       85             0 lvmetad
Oct  2 23:47:05 mandelbrot kernel: [    890]     0   890     7781        1    86016      599         -1000 systemd-udevd
Oct  2 23:47:05 mandelbrot kernel: [   1224]    32  1224     2674        0    61440      180             0 rpcbind
Oct  2 23:47:05 mandelbrot kernel: [   1236]     0  1236     1560        4    49152      124             0 rpc.idmapd
Oct  2 23:47:05 mandelbrot kernel: [   1242]    70  1242     2989        0    61440      196             0 avahi-daemon
Oct  2 23:47:05 mandelbrot kernel: [   1243]     0  1243   134580       61   139264      695             0 NetworkManager
Oct  2 23:47:05 mandelbrot kernel: [   1244]     0  1244    78035        0   110592      264             0 rngd
Oct  2 23:47:05 mandelbrot kernel: [   1246]     0  1246    19842       42    53248       46             0 irqbalance
Oct  2 23:47:05 mandelbrot kernel: [   1247]     0  1247     2693        2    61440      354             0 smartd
Oct  2 23:47:05 mandelbrot kernel: [   1248]     0  1248      624        0    45056       29             0 mcelog
Oct  2 23:47:05 mandelbrot kernel: [   1249]     0  1249    78329        0   110592      411             0 ModemManager
Oct  2 23:47:05 mandelbrot kernel: [   1251]     0  1251   179899        0   749568      245             0 rsyslogd
Oct  2 23:47:05 mandelbrot kernel: [   1262]     0  1262   114629        0   118784      483             0 abrtd
Oct  2 23:47:05 mandelbrot kernel: [   1263]    70  1263     2956        0    57344      173             0 avahi-daemon
Oct  2 23:47:05 mandelbrot kernel: [   1264]    81  1264    67178        0    61440      188          -900 dbus-broker-lau
Oct  2 23:47:05 mandelbrot kernel: [   1266]   994  1266    21125        3    69632      201             0 chronyd
Oct  2 23:47:05 mandelbrot kernel: [   1267]  1001  1267     1440       12    53248      135             0 lmgrd
Oct  2 23:47:05 mandelbrot kernel: [   1270]  1001  1270    39782        1    77824      219             0 MLM
Oct  2 23:47:05 mandelbrot kernel: [   1281]     0  1281    65915        2    86016      176             0 gssproxy
Oct  2 23:47:05 mandelbrot kernel: [   1339]    81  1339     1495       32    53248      244          -900 dbus-broker
Oct  2 23:47:05 mandelbrot kernel: [   1344]     0  1344   200667       17   757760      466             0 abrt-dump-journ
Oct  2 23:47:05 mandelbrot kernel: [   1345]     0  1345   202700       13   475136      460             0 abrt-dump-journ
Oct  2 23:47:05 mandelbrot kernel: [   1346]     0  1346   198586       16   385024      439             0 abrt-dump-journ
Oct  2 23:47:05 mandelbrot kernel: [   1359]     0  1359    58907      674    86016     3135             0 x2gocleansessio
Oct  2 23:47:05 mandelbrot kernel: [   1374]   997  1374   483383       45   237568     1689             0 polkitd
Oct  2 23:47:05 mandelbrot kernel: [   1666]     0  1666     9838        0   118784     6314             0 rpc.ypxfrd
Oct  2 23:47:05 mandelbrot kernel: [   1667]     0  1667     2993        0    61440      210         -1000 sshd
Oct  2 23:47:05 mandelbrot kernel: [   1677]     0  1677     9824        0   118784     6310             0 ypserv
Oct  2 23:47:05 mandelbrot kernel: [   1689]     0  1689     9882        0   114688     6315             0 rpc.yppasswdd
Oct  2 23:47:05 mandelbrot kernel: [   1694]    29  1694    15539        4   163840    12484             0 rpc.statd
Oct  2 23:47:05 mandelbrot kernel: [   1697]     0  1697     9633        6   118784     6309             0 rpc.mountd
Oct  2 23:47:05 mandelbrot kernel: [   1724]     0  1724    89680        2   192512    12489             0 ypbind
Oct  2 23:47:05 mandelbrot kernel: [   1751]     0  1751     3472       25    65536      471             0 sendmail
Oct  2 23:47:05 mandelbrot kernel: [   1759]    51  1759     3327        6    65536      387             0 sendmail
Oct  2 23:47:05 mandelbrot kernel: [   1774]     0  1774    11714       49   135168     6390             0 systemd-logind
Oct  2 23:47:05 mandelbrot kernel: [   1776]     0  1776      910        1    45056       46             0 atd
Oct  2 23:47:05 mandelbrot kernel: [   1780]     0  1780   196083        0   253952    12806             0 automount
Oct  2 23:47:05 mandelbrot kernel: [   1805]     0  1805    54298       27    53248      186             0 crond
Oct  2 23:47:05 mandelbrot kernel: [   1826]     0  1826    53864        0    45056       31             0 agetty
Oct  2 23:47:05 mandelbrot kernel: [   2517]     0  2517    16706        4   172032    12606             0 sshd
Oct  2 23:47:05 mandelbrot kernel: [   2542]     0  2542 27305777 16124016 175431680  5746588             0 (systemd)
Oct  2 23:47:05 mandelbrot kernel: [   2597]  6105  2597    16706       18   172032    12594             0 sshd
Oct  2 23:47:05 mandelbrot kernel: [   2604]  6105  2604    62558        1   126976     6645             0 bash
Oct  2 23:47:05 mandelbrot kernel: [   2703]  6105  2703    69007      959   184320    11566             0 su
Oct  2 23:47:05 mandelbrot kernel: [   2729]     0  2729     5043      208    81920      226             0 systemd
Oct  2 23:47:05 mandelbrot kernel: [   2734]     0  2734    43007      394   102400      470             0 (sd-pam)
Oct  2 23:47:05 mandelbrot kernel: [   2744]     0  2744    54406      218    57344      167             0 bash
Oct  2 23:47:05 mandelbrot kernel: [   2847]     0  2847    56386     1367    69632        0             0 x2golistsession
Oct  2 23:47:05 mandelbrot kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/init.scope,task=(systemd),pid=2542,uid=0
Oct  2 23:47:05 mandelbrot kernel: Out of memory: Killed process 2542 ((systemd)) total-vm:109223108kB, anon-rss:64496060kB, file-rss:4kB, shmem-rss:0kB
Oct  2 23:47:08 mandelbrot kernel: oom_reaper: reaped process 2542 ((systemd)), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Oct  2 23:47:08 mandelbrot systemd[1]: user@6105.service: Failed with result 'timeout'.

1d37ef1f448574c60aa52557738fd51e6414ff12.png
1d37ef1f448574c60aa52557738fd51e6414ff12.png3077×2189 266 KB

2

First association with OOM is that system is running out of memory. When you experience this issue, what’s the status of the system memory usage? If usage looks good and if this is multi-processor (multi socket) system it is also worth checking NUMA settings to see what’s the allocation policy.

3

All memory is in use:

free -hw
              total        used        free      shared     buffers       cache   available
Mem:           62Gi        62Gi       251Mi       0.0Ki       1.0Mi       130Mi       206Mi
Swap:          22Gi       5.3Gi        17Gi

Are these helpful?

numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10
node 0 size: 31625 MB
node 0 free: 31260 MB
node 1 cpus: 1 3 5 7 9 11
node 1 size: 32253 MB
node 1 free: 31383 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10


numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11
cpubind: 0 1
nodebind: 0 1
membind: 0 1

dmesg -T|grep -i numa
[Wed Oct  9 10:22:12 2019]  ? task_numa_fault+0x765/0x1230

Any idea what emotectl is? Whenever that’s in the process of getting killed the system is under a very high load:

top
top - 10:22:54 up 4 days, 20:54,  1 user,  load average: 5.97, 6.13, 5.23
Tasks: 308 total,   2 running, 306 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.8 us,  7.1 sy,  0.0 ni, 87.2 id,  1.5 wa,  0.1 hi,  0.1 si,  0.0 st
MiB Mem :  63879.6 total,  47462.3 free,  16138.6 used,    278.6 buff/cache
MiB Swap:  22892.0 total,  22133.9 free,    758.1 used.  47252.6 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
26916 root      20   0   52.2g  15.1g   4640 R  99.3  24.2   0:13.31 (emotectl)

I see errors like this:

Oct  9 10:31:22 ourserver kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/init.scope,task=(emotectl),pid=27265,u
id=0
Oct  9 10:31:22 ourserver kernel: Out of memory: Killed process 27265 ((emotectl)) total-vm:109224456kB, anon-rss:64322432kB, file-rss:4kB, shmem-rss:0kB
Oct  9 10:31:24 ourserver kernel: oom_reaper: reaped process 27265 ((emotectl)), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

And all of the Dell utilities, e.g., the TSR, return no errors. Could this me a memory leak?

4

From the first output about memory usage it is visible that all the memory on the system is utilized. As soon as some process tries to allocate a bit more memory (250MB isn’t much wiggle room to be honest) oom-killer gets invoked and reaps the process.

On Linux, process that triggers OOM gets killed by default if oom score is not set up otherwise.

Even though you have the swap space, there are pages that can’t be swapped to disk, and I assume your workload is just that, pages that can’t be swapped to disk.

5

As for emotectl it seems that’s Fedora Cockpit process? To be honest, haven’t used that service much so I’m not sure. But you could try poking around a bit to see what binary is being run and so on. For starters try listing process PID folder:

ls -alh /proc/26916

Where 26916 is the PID of that emotectl process.

6

That sounds right as cockpit is one of the services that is always killed. I KNOW I disabled it but it seems to want to start after a reboot. This time I even masked it:

systemctl status cockpit
● cockpit.service
   Loaded: masked (Reason: Unit cockpit.service is masked.)
   Active: failed (Result: timeout) since Wed 2019-10-09 10:40:58 EDT; 38min ago

Oct 09 10:38:04 ourserver systemd[1]: Stopped Cockpit Web Service.
Oct 09 10:38:04 ourserver systemd[1]: Starting Cockpit Web Service...
Oct 09 10:39:34 ourserver systemd[1]: cockpit.service: Start-pre operation timed out. Terminating.
Oct 09 10:40:58 ourserver systemd[1]: cockpit.service: Control process exited, code=killed, status=9/KILL
Oct 09 10:40:58 ourserver systemd[1]: cockpit.service: Failed with result 'timeout'.
Oct 09 10:40:58 ourserver systemd[1]: Failed to start cockpit.service.

But other services also get killed, e.g., d-anchor/unbound-anchor.service, systemd-rfkill.socket, and systemd. Could this be a memory leak some where? How can I debug further?

7

Yes, it would appear so. If you disabled the service (systemct disable cockpit) then it shouldn’t startup at boot. You can try uninstalling it if you don’t use it.

And if that process is the one that gets killed most of the time then it most certainly has some memory leak. As previously stated, process that triggers the OOM gets killed. So if system is low on memory and some random service starts allocating memory it gets killed, although it might not be the culprit. Looking by the usage I would say our first guess is correct and that Cockpit has some memory leak issues on this system.

To go further from there you could try checking how process exactly allocates the memory which can be achieved with utilities like pmap (pmap -x PID).

With all seen it is almost certainly issue with Cockpit then, in such situation I would suggest opening up a bug report. Once bug report is reviewed you might be asked to provide some additional information.

8

I had to use dnf remove cockpit* as there were several dependencies, hoping I didn’t uninstall something we need.

Dependencies resolved.
==============================================================================================================================================================================
 Package                                               Architecture                  Version                                            Repository                       Size
==============================================================================================================================================================================
Removing:
 cockpit-bridge                                        x86_64                        203-1.fc30                                         @updates                        1.2 M
 cockpit-networkmanager                                noarch                        203-1.fc30                                         @updates                        325 k
 cockpit-storaged                                      noarch                        203-1.fc30                                         @updates                        336 k
 cockpit-system                                        noarch                        203-1.fc30                                         @updates                        1.3 M
 cockpit-ws                                            x86_64                        203-1.fc30                                         @updates                        2.2 M
Removing unused dependencies:
 clevis                                                x86_64                        11-5.fc30                                          @fedora                         131 k
 clevis-luks                                           x86_64                        11-5.fc30                                          @fedora                          14 k
 iscsi-initiator-utils                                 x86_64                        6.2.0.876-8.gitf3c8e90.fc30                        @fedora                         2.0 M
 iscsi-initiator-utils-iscsiuio                        x86_64                        6.2.0.876-8.gitf3c8e90.fc30                        @fedora                         229 k
 isns-utils-libs                                       x86_64                        0.97-8.fc30                                        @fedora                         576 k
 jose                                                  x86_64                        10-4.fc30                                          @fedora                         153 k
 libjose                                               x86_64                        10-4.fc30                                          @fedora                         218 k
 libluksmeta                                           x86_64                        9-2.fc29                                           @fedora                          45 k
 luksmeta                                              x86_64                        9-2.fc29                                           @fedora                          32 k
 tpm2-tools                                            x86_64                        3.2.0-3.fc30                                       @updates                        3.5 M
 udisks2-iscsi                                         x86_64                        2.8.4-1.fc30                                       @updates                        282 k
 udisks2-lvm2                                          x86_64                        2.8.4-1.fc30                                       @updates                        559 k

Just before uninstalling systemd had an OOM:

(systemd) invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
[...]

Oct  9 12:23:36 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/init.scope,task=(systemd),pid=8923,uid=0
Oct  9 12:23:36 kernel: Out of memory: Killed process 8923 ((systemd)) total-vm:109226500kB, anon-rss:64266876kB, file-rss:0kB, shmem-rss:0kB
Oct  9 12:23:38 kernel: oom_reaper: reaped process 8923 ((systemd)), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Oct  9 12:23:38  systemd[1]: user@6105.service: Failed with result 'timeout'.

I’ll keep the pmap suggestion in mind as the OOM finished before I could use it. I had opened a bug (with no response yet) but perhaps I can reassign it to Cockpit and provide some of the OOM errors there…

9

Confirmed this is a bug with Cockpit remotectl OOM out of memory errors · Issue #12949 · cockpit-project/cockpit · GitHub