I’ve run all firmware updates, dnf updates and still OOM’s on various processes a couple times an hour. Could this be a bug? The 5 hour hardware test within the Dell BIOS passed see screen shot, no errors. No errors in the Dell debug logs within the Lifecyle Controller. Kernel is 5.2.17-200.fc30.x86_6 but it’s been happening for a few kernel versions.
Oct 2 23:47:04 mandelbrot kernel: MLM invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Oct 2 23:47:05 mandelbrot kernel: CPU: 11 PID: 1335 Comm: MLM Not tainted 5.2.17-200.fc30.x86_64 #1
Oct 2 23:47:05 mandelbrot kernel: Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.3.10 08/15/2019
Oct 2 23:47:05 mandelbrot kernel: Call Trace:
Oct 2 23:47:05 mandelbrot kernel: dump_stack+0x5c/0x80
Oct 2 23:47:05 mandelbrot kernel: dump_header+0x51/0x2ff
Oct 2 23:47:05 mandelbrot kernel: ? sched_clock+0x5/0x10
Oct 2 23:47:05 mandelbrot kernel: ? oom_unkillable_task+0x9a/0xd0
Oct 2 23:47:05 mandelbrot kernel: oom_kill_process.cold+0xb/0x10
Oct 2 23:47:05 mandelbrot kernel: out_of_memory+0x1d4/0x480
Oct 2 23:47:05 mandelbrot kernel: __alloc_pages_slowpath+0xcc4/0xde0
Oct 2 23:47:05 mandelbrot kernel: __alloc_pages_nodemask+0x2ee/0x340
Oct 2 23:47:05 mandelbrot kernel: pagecache_get_page+0xb4/0x370
Oct 2 23:47:05 mandelbrot kernel: filemap_fault+0x683/0x9f0
Oct 2 23:47:05 mandelbrot kernel: ? __mod_lruvec_state+0x3f/0xe0
Oct 2 23:47:05 mandelbrot kernel: ? page_add_file_rmap+0xf4/0x230
Oct 2 23:47:05 mandelbrot kernel: ? alloc_set_pte+0x498/0x6b0
Oct 2 23:47:05 mandelbrot kernel: ? file_fdatawait_range+0x20/0x20
Oct 2 23:47:05 mandelbrot kernel: ? _cond_resched+0x15/0x30
Oct 2 23:47:05 mandelbrot kernel: __xfs_filemap_fault+0x6d/0x200 [xfs]
Oct 2 23:47:05 mandelbrot kernel: __do_fault+0x36/0x180
Oct 2 23:47:05 mandelbrot kernel: __handle_mm_fault+0x1091/0x1ac0
Oct 2 23:47:05 mandelbrot kernel: handle_mm_fault+0xc4/0x1e0
Oct 2 23:47:05 mandelbrot kernel: do_user_addr_fault+0x1f6/0x450
Oct 2 23:47:05 mandelbrot kernel: do_page_fault+0x33/0x120
Oct 2 23:47:05 mandelbrot kernel: ? page_fault+0x8/0x30
Oct 2 23:47:05 mandelbrot kernel: page_fault+0x1e/0x30
Oct 2 23:47:05 mandelbrot kernel: RIP: 0033:0x7fd8c0cfaffb
Oct 2 23:47:05 mandelbrot kernel: Code: Bad RIP value.
Oct 2 23:47:05 mandelbrot kernel: RSP: 002b:00007fd8c0bebe90 EFLAGS: 00010293
Oct 2 23:47:05 mandelbrot kernel: RAX: 0000000000000000 RBX: 000000000227ece0 RCX: 00007fd8c0cfaffb
Oct 2 23:47:05 mandelbrot kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Oct 2 23:47:05 mandelbrot kernel: RBP: 000000000227d260 R08: 00007fd8c0bebed0 R09: 0000000000000000
Oct 2 23:47:05 mandelbrot kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 00007fffb6aaf17e
Oct 2 23:47:05 mandelbrot kernel: R13: 00007fffb6aaf17f R14: 00007fffb6aaf200 R15: 00007fd8c0bebfc0
Oct 2 23:47:05 mandelbrot kernel: Mem-Info:
Oct 2 23:47:05 mandelbrot kernel: active_anon:15233105 inactive_anon:898356 isolated_anon:0#012 active_file:25 inactive_file:30 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:11388 slab_unreclaimable:30920#012 mapped:11 shmem:6 pagetables:44129 bounce:0#012 free:56588 free_pcp:0 free_cma:0
Oct 2 23:47:05 mandelbrot kernel: Node 0 active_anon:30085864kB inactive_anon:1774920kB active_file:0kB inactive_file:64kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:4kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Oct 2 23:47:05 mandelbrot kernel: Node 1 active_anon:30846556kB inactive_anon:1818504kB active_file:108kB inactive_file:56kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:68kB dirty:0kB writeback:0kB shmem:20kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Oct 2 23:47:05 mandelbrot kernel: Node 0 DMA free:15872kB min:20kB low:32kB high:44kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15972kB managed:15888kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 2 23:47:05 mandelbrot kernel: lowmem_reserve[]: 0 1396 31608 31608 31608
Oct 2 23:47:05 mandelbrot kernel: Node 0 DMA32 free:122816kB min:1968kB low:3396kB high:4824kB active_anon:1324160kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:1521664kB managed:1456128kB mlocked:0kB kernel_stack:0kB pagetables:2596kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 2 23:47:05 mandelbrot kernel: lowmem_reserve[]: 0 0 30212 30212 30212
Oct 2 23:47:05 mandelbrot kernel: Node 0 Normal free:42448kB min:42640kB low:73576kB high:104512kB active_anon:28761916kB inactive_anon:1774920kB active_file:0kB inactive_file:64kB unevictable:0kB writepending:0kB present:31457280kB managed:30937480kB mlocked:0kB kernel_stack:3336kB pagetables:84528kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 2 23:47:05 mandelbrot kernel: lowmem_reserve[]: 0 0 0 0 0
Oct 2 23:47:05 mandelbrot kernel: Node 1 Normal free:45216kB min:45476kB low:78468kB high:111460kB active_anon:30846556kB inactive_anon:1818504kB active_file:108kB inactive_file:56kB unevictable:0kB writepending:0kB present:33554432kB managed:33003212kB mlocked:0kB kernel_stack:2232kB pagetables:89392kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 2 23:47:05 mandelbrot kernel: lowmem_reserve[]: 0 0 0 0 0
Oct 2 23:47:05 mandelbrot kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15872kB
Oct 2 23:47:05 mandelbrot kernel: Node 0 DMA32: 0*4kB 0*8kB 4*16kB (U) 2*32kB (U) 1*64kB (U) 4*128kB (UM) 1*256kB (M) 4*512kB (UM) 3*1024kB (U) 1*2048kB (U) 28*4096kB (M) = 122816kB
Oct 2 23:47:05 mandelbrot kernel: Node 0 Normal: 204*4kB (UME) 231*8kB (UME) 102*16kB (UME) 95*32kB (UME) 21*64kB (UE) 6*128kB (ME) 4*256kB (UME) 3*512kB (UME) 10*1024kB (UME) 8*2048kB (UM) 1*4096kB (M) = 42728kB
Oct 2 23:47:05 mandelbrot kernel: Node 1 Normal: 358*4kB (UME) 188*8kB (UME) 102*16kB (UME) 43*32kB (UME) 17*64kB (UME) 10*128kB (UME) 3*256kB (UE) 4*512kB (UME) 4*1024kB (UME) 1*2048kB (E) 7*4096kB (M) = 45944kB
Oct 2 23:47:05 mandelbrot kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 2 23:47:05 mandelbrot kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 2 23:47:05 mandelbrot kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 2 23:47:05 mandelbrot kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 2 23:47:05 mandelbrot kernel: 2875 total pagecache pages
Oct 2 23:47:05 mandelbrot kernel: 2807 pages in swap cache
Oct 2 23:47:05 mandelbrot kernel: Swap cache stats: add 5867290, delete 5863717, find 830/4546
Oct 2 23:47:05 mandelbrot kernel: Free swap = 0kB
Oct 2 23:47:05 mandelbrot kernel: Total swap = 23441404kB
Oct 2 23:47:05 mandelbrot kernel: 16637337 pages RAM
Oct 2 23:47:05 mandelbrot kernel: 0 pages HighMem/MovableOnly
Oct 2 23:47:05 mandelbrot kernel: 284160 pages reserved
Oct 2 23:47:05 mandelbrot kernel: 0 pages cma reserved
Oct 2 23:47:05 mandelbrot kernel: 0 pages hwpoisoned
Oct 2 23:47:05 mandelbrot kernel: Tasks state (memory values in pages):
Oct 2 23:47:05 mandelbrot kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 2 23:47:05 mandelbrot kernel: [ 857] 0 857 15284 30 143360 268 0 systemd-journal
Oct 2 23:47:05 mandelbrot kernel: [ 882] 0 882 38364 12 61440 85 0 lvmetad
Oct 2 23:47:05 mandelbrot kernel: [ 890] 0 890 7781 1 86016 599 -1000 systemd-udevd
Oct 2 23:47:05 mandelbrot kernel: [ 1224] 32 1224 2674 0 61440 180 0 rpcbind
Oct 2 23:47:05 mandelbrot kernel: [ 1236] 0 1236 1560 4 49152 124 0 rpc.idmapd
Oct 2 23:47:05 mandelbrot kernel: [ 1242] 70 1242 2989 0 61440 196 0 avahi-daemon
Oct 2 23:47:05 mandelbrot kernel: [ 1243] 0 1243 134580 61 139264 695 0 NetworkManager
Oct 2 23:47:05 mandelbrot kernel: [ 1244] 0 1244 78035 0 110592 264 0 rngd
Oct 2 23:47:05 mandelbrot kernel: [ 1246] 0 1246 19842 42 53248 46 0 irqbalance
Oct 2 23:47:05 mandelbrot kernel: [ 1247] 0 1247 2693 2 61440 354 0 smartd
Oct 2 23:47:05 mandelbrot kernel: [ 1248] 0 1248 624 0 45056 29 0 mcelog
Oct 2 23:47:05 mandelbrot kernel: [ 1249] 0 1249 78329 0 110592 411 0 ModemManager
Oct 2 23:47:05 mandelbrot kernel: [ 1251] 0 1251 179899 0 749568 245 0 rsyslogd
Oct 2 23:47:05 mandelbrot kernel: [ 1262] 0 1262 114629 0 118784 483 0 abrtd
Oct 2 23:47:05 mandelbrot kernel: [ 1263] 70 1263 2956 0 57344 173 0 avahi-daemon
Oct 2 23:47:05 mandelbrot kernel: [ 1264] 81 1264 67178 0 61440 188 -900 dbus-broker-lau
Oct 2 23:47:05 mandelbrot kernel: [ 1266] 994 1266 21125 3 69632 201 0 chronyd
Oct 2 23:47:05 mandelbrot kernel: [ 1267] 1001 1267 1440 12 53248 135 0 lmgrd
Oct 2 23:47:05 mandelbrot kernel: [ 1270] 1001 1270 39782 1 77824 219 0 MLM
Oct 2 23:47:05 mandelbrot kernel: [ 1281] 0 1281 65915 2 86016 176 0 gssproxy
Oct 2 23:47:05 mandelbrot kernel: [ 1339] 81 1339 1495 32 53248 244 -900 dbus-broker
Oct 2 23:47:05 mandelbrot kernel: [ 1344] 0 1344 200667 17 757760 466 0 abrt-dump-journ
Oct 2 23:47:05 mandelbrot kernel: [ 1345] 0 1345 202700 13 475136 460 0 abrt-dump-journ
Oct 2 23:47:05 mandelbrot kernel: [ 1346] 0 1346 198586 16 385024 439 0 abrt-dump-journ
Oct 2 23:47:05 mandelbrot kernel: [ 1359] 0 1359 58907 674 86016 3135 0 x2gocleansessio
Oct 2 23:47:05 mandelbrot kernel: [ 1374] 997 1374 483383 45 237568 1689 0 polkitd
Oct 2 23:47:05 mandelbrot kernel: [ 1666] 0 1666 9838 0 118784 6314 0 rpc.ypxfrd
Oct 2 23:47:05 mandelbrot kernel: [ 1667] 0 1667 2993 0 61440 210 -1000 sshd
Oct 2 23:47:05 mandelbrot kernel: [ 1677] 0 1677 9824 0 118784 6310 0 ypserv
Oct 2 23:47:05 mandelbrot kernel: [ 1689] 0 1689 9882 0 114688 6315 0 rpc.yppasswdd
Oct 2 23:47:05 mandelbrot kernel: [ 1694] 29 1694 15539 4 163840 12484 0 rpc.statd
Oct 2 23:47:05 mandelbrot kernel: [ 1697] 0 1697 9633 6 118784 6309 0 rpc.mountd
Oct 2 23:47:05 mandelbrot kernel: [ 1724] 0 1724 89680 2 192512 12489 0 ypbind
Oct 2 23:47:05 mandelbrot kernel: [ 1751] 0 1751 3472 25 65536 471 0 sendmail
Oct 2 23:47:05 mandelbrot kernel: [ 1759] 51 1759 3327 6 65536 387 0 sendmail
Oct 2 23:47:05 mandelbrot kernel: [ 1774] 0 1774 11714 49 135168 6390 0 systemd-logind
Oct 2 23:47:05 mandelbrot kernel: [ 1776] 0 1776 910 1 45056 46 0 atd
Oct 2 23:47:05 mandelbrot kernel: [ 1780] 0 1780 196083 0 253952 12806 0 automount
Oct 2 23:47:05 mandelbrot kernel: [ 1805] 0 1805 54298 27 53248 186 0 crond
Oct 2 23:47:05 mandelbrot kernel: [ 1826] 0 1826 53864 0 45056 31 0 agetty
Oct 2 23:47:05 mandelbrot kernel: [ 2517] 0 2517 16706 4 172032 12606 0 sshd
Oct 2 23:47:05 mandelbrot kernel: [ 2542] 0 2542 27305777 16124016 175431680 5746588 0 (systemd)
Oct 2 23:47:05 mandelbrot kernel: [ 2597] 6105 2597 16706 18 172032 12594 0 sshd
Oct 2 23:47:05 mandelbrot kernel: [ 2604] 6105 2604 62558 1 126976 6645 0 bash
Oct 2 23:47:05 mandelbrot kernel: [ 2703] 6105 2703 69007 959 184320 11566 0 su
Oct 2 23:47:05 mandelbrot kernel: [ 2729] 0 2729 5043 208 81920 226 0 systemd
Oct 2 23:47:05 mandelbrot kernel: [ 2734] 0 2734 43007 394 102400 470 0 (sd-pam)
Oct 2 23:47:05 mandelbrot kernel: [ 2744] 0 2744 54406 218 57344 167 0 bash
Oct 2 23:47:05 mandelbrot kernel: [ 2847] 0 2847 56386 1367 69632 0 0 x2golistsession
Oct 2 23:47:05 mandelbrot kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/init.scope,task=(systemd),pid=2542,uid=0
Oct 2 23:47:05 mandelbrot kernel: Out of memory: Killed process 2542 ((systemd)) total-vm:109223108kB, anon-rss:64496060kB, file-rss:4kB, shmem-rss:0kB
Oct 2 23:47:08 mandelbrot kernel: oom_reaper: reaped process 2542 ((systemd)), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Oct 2 23:47:08 mandelbrot systemd[1]: user@6105.service: Failed with result 'timeout'.