Kernel memory leak? (Fedora 40 + NVidia drivers + ollama)

Ask Fedora
, ,
1

Can anybody help me to find a memory leak in the kernel, please?

10GiB on 16GiB system is used by kernel dynamic memory after repeated call to Ollama (local AI runner) accelerated with NVidia/Cuda 12.

I turned off zswap and turned on regular swap on disk, but it doesn’t help.


root@fedora:~# uname -a
Linux fedora 6.11.4-201.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Oct 20 15:04:22 UTC 2024 x86_64 GNU/Linux

root@fedora:~# ps_mem
 Private  +   Shared  =  RAM used	Program

 (...cut off less than 1MiB)
  1.2 MiB +  46.5 KiB =   1.2 MiB	gvfsd-metadata
  1.0 MiB + 299.5 KiB =   1.3 MiB	mate-brightness-applet
  1.2 MiB + 168.5 KiB =   1.3 MiB	firewall-applet
  1.2 MiB + 156.5 KiB =   1.3 MiB	systemd-resolved
968.0 KiB + 567.5 KiB =   1.5 MiB	rsyslogd
  1.3 MiB + 246.5 KiB =   1.5 MiB	blueman-applet
  1.4 MiB + 152.5 KiB =   1.5 MiB	wsdd
  1.5 MiB + 109.5 KiB =   1.6 MiB	pcscd
  1.4 MiB + 282.5 KiB =   1.6 MiB	mate-power-manager
  1.6 MiB +  81.5 KiB =   1.7 MiB	xdg-desktop-portal
  1.5 MiB + 186.5 KiB =   1.7 MiB	dbus-broker (3)
  1.7 MiB + 121.5 KiB =   1.8 MiB	upowerd
  2.0 MiB + 147.5 KiB =   2.1 MiB	wpa_supplicant
  1.5 MiB + 696.5 KiB =   2.2 MiB	mate-typing-monitor
  2.6 MiB + 204.0 KiB =   2.8 MiB	bash (18)
  2.7 MiB + 309.5 KiB =   3.0 MiB	nm-applet
  3.1 MiB +  51.5 KiB =   3.1 MiB	NetworkManager
  2.8 MiB + 607.0 KiB =   3.4 MiB	systemd (2)
  3.2 MiB + 344.5 KiB =   3.6 MiB	pipewire
  3.7 MiB + 211.5 KiB =   3.9 MiB	wireplumber
  3.7 MiB + 404.5 KiB =   4.1 MiB	mate-settings-daemon
  3.4 MiB + 967.5 KiB =   4.4 MiB	systemd-journald
  2.8 MiB +   2.1 MiB =   4.9 MiB	systemd-userwork (3)
  4.8 MiB + 160.5 KiB =   5.0 MiB	polkitd
  5.3 MiB + 673.5 KiB =   6.0 MiB	marco
  7.5 MiB + 283.5 KiB =   7.8 MiB	pipewire-pulse
  8.6 MiB +  16.5 KiB =   8.6 MiB	ollama-0.3.14
 11.5 MiB + 891.5 KiB =  12.4 MiB	mate-terminal
 15.2 MiB +   1.1 MiB =  16.3 MiB	pluma
 17.7 MiB + 810.5 KiB =  18.5 MiB	mate-panel
 60.2 MiB + 737.5 KiB =  60.9 MiB	Xorg
  1.2 GiB +  97.2 MiB =   1.3 GiB	firefox (21)
---------------------------------
                          1.5 GiB
=================================

root@fedora:~# free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi        14Gi       519Mi       426Mi       952Mi       709Mi
Swap:          8,0Gi       4,3Gi       3,7Gi

root@fedora:~# cat /proc/meminfo
MemTotal:       16023912 kB
MemFree:          533400 kB
MemAvailable:     728248 kB
Buffers:             300 kB
Cached:           803368 kB
SwapCached:        21592 kB
Active:         10718048 kB
Inactive:        3155552 kB
Active(anon):   10445276 kB
Inactive(anon):  3052012 kB
Active(file):     272772 kB
Inactive(file):   103540 kB
Unevictable:      397060 kB
Mlocked:            5768 kB
SwapTotal:       8388604 kB
SwapFree:        3893072 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:               496 kB
Writeback:             0 kB
AnonPages:       1438796 kB
Mapped:           182552 kB
Shmem:            422240 kB
KReclaimable:     157592 kB
Slab:             438712 kB
SReclaimable:     157592 kB
SUnreclaim:       281120 kB
KernelStack:       22416 kB
PageTables:        54476 kB
SecPageTables:      2092 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16400560 kB
Committed_AS:   12297228 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      105192 kB
VmallocChunk:          0 kB
Percpu:            11328 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:    14336 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     9875176 kB
DirectMap2M:     6559744 kB
DirectMap1G:           0 kB

# slabtop
 Active / Total Objects (% used)    : 1525462 / 1823793 (83,6%)
 Active / Total Slabs (% used)      : 35921 / 35921 (100,0%)
 Active / Total Caches (% used)     : 432 / 607 (71,2%)
 Active / Total Size (% used)       : 304550,24K / 401514,03K (75,9%)
 Minimum / Average / Maximum Object : 0,01K / 0,22K / 32,54K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
165970  97911  58%    0,57K   5938	 28     95008K radix_tree_node
 99620  98940  99%    0,02K    586	170	 2344K avtab_node
 85376  84614  99%    0,03K    667	128	 2668K nvidia_pte_cache
 82212  61399  74%    0,04K    806	102	 3224K vma_lock
 78936  60733  76%    0,17K   1716	 46     13728K vm_area_struct
 74580  74421  99%    0,13K   2486	 30	 9944K kernfs_node_cache
 64668  28963  44%    0,12K   1902	 34	 7608K lsm_inode_cache
 52096  51892  99%    0,06K    814	 64	 3256K ebitmap_node
 51786  17584  33%    0,19K   1233	 42	 9864K dentry
 36480  25512  69%    0,06K    570	 64	 2280K anon_vma_chain
 35486  15866  44%    1,01K   1198	 31     38336K btrfs_inode
 25700  13216  51%    0,31K   1028	 25	 8224K btrfs_delayed_node
 23120  22270  96%    0,02K    136	170	  544K hashtab_node
 22464  16367  72%    0,10K    576	 39	 2304K anon_vma
 20910  17128  81%    0,02K    123	170	  492K lsm_file_cache
 19456  18944  97%    0,02K     76	256	  304K kmalloc-rnd-14-16
 18752  14953  79%    0,25K    586	 32	 4688K filp
 17752   9312  52%    0,07K    317	 56	 1268K vmap_area
 16320  10352  63%    0,25K    510	 32	 4080K kmalloc-rnd-01-256
 16072  16072 100%    0,07K    287	 56	 1148K Acpi-Operand
 15360  13312  86%    0,01K     30	512	  120K kmalloc-rnd-14-8
 14592  14384  98%    0,02K     57	256	  228K kmalloc-rnd-12-16
 14080  13569  96%    0,02K     55	256	  220K kmalloc-rnd-09-16
 12870   5894  45%    0,10K    330	 39	 1320K btrfs_extent_map
 12544  12288  97%    0,03K     98	128	  392K kmalloc-rnd-14-32
 12544  11264  89%    0,02K     49	256	  196K kmalloc-rnd-03-16
 12416   7035  56%    0,25K    388	 32	 3104K maple_node
 12032   9647  80%    0,03K     94	128	  376K kmalloc-cg-32
 11776  11292  95%    0,01K     23	512        92K kmalloc-rnd-09-8
2 
root@fedora:~# vmstat -m
Cache                       Num  Total   Size  Pages
Acpi-Namespace             9945   9945     48     85
Acpi-Operand              16072  16072     72     56
Acpi-Parse                 1241   1241     56     73
Acpi-ParseExt               816    816     80     51
Acpi-State                 1377   1377     80     51
active_node                2468   2528    128     32
aio_kiocb                    84     84    192     42
anon_vma                  17290  23010    104     39
anon_vma_chain            26873  36928     64     64
audit_buffer               2720   2720     24    170
audit_tree_mark               0      0     80     51
avc_node                   3248   3248     72     56
avc_xperms_data               0      0     32    128
avc_xperms_decision_node      0      0     48     85
avc_xperms_node               0      0     56     73
avtab_extended_perms          0      0     40    102
avtab_node                98940  99620     24    170
backing_aio                   0      0    128     32
bdev_cache                  100    100   1600     20
bfq_io_cq                     0      0   1360     24
bfq_queue                     0      0    576     28
bio-106                     936   1086   1088     30
bio-136                     252    252    192     42
bio-200                    1936   2064    256     32
bio-256                      50     50    320     25
bio-264                      75     75    320     25
bio-312                      75     75    320     25
bio-376                    1816   1816    384     42
bio-392                      36     36    448     36
bio-448                    1536   1568    512     32
bio-984                      32     32   1024     32
bio_crypt_ctx               204    204     40    102
bio_integrity_payload        42     42    192     42
bio_post_read_ctx           170    170     48     85
Cache                       Num  Total   Size  Pages
biovec-128                   16     16   2048     16
biovec-16                  1280   1408    256     32
biovec-64                   928   1060   1024     32
biovec-max                  380    400   4096      8
blkdev_ioc                  736    736     88     46
btree_node                    0      0    128     32
btrfs_delayed_extent_op       0      0     32    128
btrfs_delayed_node        13246  25700    320     25
btrfs_delayed_ref_head     1820   1820    152     26
btrfs_delayed_ref_node     2628   2628    112     36
btrfs_extent_buffer        3122   3128    240     34
btrfs_extent_map           6210  12870    104     39
btrfs_extent_state         3080   3136     72     56
btrfs_free_space          10226  10452    104     39
btrfs_free_space_bitmap     457    464   4096      8
btrfs_inode               15826  35486   1032     31
btrfs_inode_defrag            0      0     56     73
btrfs_ordered_extent       1599   1599    416     39
btrfs_path                  576    576    112     36
btrfs_prelim_ref            765    765     80     51
btrfs_trans_handle          384    384    168     24
buffer_head                2088   2535    104     39
configfs_dir_cache            0      0     88     46
cred                       2526   2646    192     42
damon_region                  0      0     56     73
dax_cache                    78     78    832     39
dentry                    23076  51828    192     42
dio                           0      0    640     25
dmaengine-unmap-128          30     30   1088     30
dmaengine-unmap-16           42     42    192     42
dmaengine-unmap-2            64     64     64     64
dmaengine-unmap-256          15     15   2112     15
dma-kmalloc-128               0      0    128     32
dma-kmalloc-16                0      0     16    256
Cache                       Num  Total   Size  Pages
dma-kmalloc-192               0      0    192     42
dma-kmalloc-1k                0      0   1024     32
dma-kmalloc-256               0      0    256     32
dma-kmalloc-2k                0      0   2048     16
dma-kmalloc-32                0      0     32    128
dma-kmalloc-4k                0      0   4096      8
dma-kmalloc-512               0      0    512     32
dma-kmalloc-64                0      0     64     64
dma-kmalloc-8                 0      0      8    512
dma-kmalloc-8k                0      0   8192      4
dma-kmalloc-96                0      0     96     42
dm_exception                  0      0     32    128
dm_snap_pending_exceptio      0      0    128     32
dm_uevent                     0      0   2888     11
dnotify_mark                  0      0     80     51
dnotify_struct                0      0     32    128
dquot                         0      0    256     32
drm_buddy_block               0      0     72     56
drm_i915_gem_object        1463   2044   1152     28
drm_sched_fence               0      0    192     42
ebitmap_node              51892  52096     64     64
ep_head                    4096   4096     16    256
eventpoll_epi              2173   3232    128     32
eventpoll_pwq              2240   2240     64     64
execute_cb                    0      0     64     64
ext4_allocation_context     260    260    152     26
ext4_fc_dentry_update         0      0    104     39
ext4_free_data              657    657     56     73
ext4_groupinfo_4k            44     44    184     44
ext4_inode_cache            378    378   1184     27
ext4_io_end                 640    640     64     64
ext4_io_end_vec            1280   1280     32    128
ext4_prealloc_space         360    360    112     36
ext4_system_zone            102    102     40    102
Cache                       Num  Total   Size  Pages
extent_status              1530   1530     40    102
fanotify_fid_event            0      0     72     56
fanotify_mark               153    153     80     51
fanotify_path_event           0      0     64     64
fanotify_perm_event           0      0     96     42
fasync_cache                  0      0     48     85
fat_cache                   306    306     40    102
fat_inode_cache             328    328    792     41
fib6_node                   448    448    128     32
file_lease_cache              0      0    160     25
file_lock_cache             672    672    192     42
file_lock_ctx              2482   2482     56     73
files_cache                 782    782    704     46
filp                      15771  18752    256     32
fs_cache                   1024   1024     64     64
fscrypt_inode_info            0      0    128     32
fsnotify_mark_connector    2890   2890     24    170
fsverity_info                 0      0    264     31
ftrace_event_field         9417   9417     56     73
fuse_inode                  180    180    896     36
fuse_request                338    338    152     26
hashtab_node              22270  23120     24    170
hfs_inode_cache               0      0    832     39
hfsplus_attr_cache            0      0   3840      8
hfsplus_icache                0      0    960     34
hugetlbfs_inode_cache        75     75    648     25
i915_dependency             512    512    128     32
i915_lut_handle            4371   4992     32    128
i915_priolist                 0      0     48     85
i915_request               2077   2162    704     46
i915_vma                   1303   1625    640     25
i915_vma_resource          1868   2475    320     25
ima_iint_cache                0      0    104     39
inet_peer                   462    462    192     42
Cache                       Num  Total   Size  Pages
inode_cache                6770   8462    640     25
inotify_inode_mark         1734   1734     80     51
intel_context              1848   1974    768     42
io                            0      0     64     64
io_buffer                     0      0     32    128
io_kiocb                      0      0    256     32
iommu_iova                    0      0     64     64
iommu_iova_magazine           0      0   1024     32
ip4-frags                     0      0    200     40
ip6_dst_cache               512    512    256     32
ip6-frags                     0      0    184     44
ip_fib_alias                876    876     56     73
ip_fib_trie                1020   1020     48     85
iwl_cmd_pool:0000:2e:00.   1664   1664    128     32
jbd2_inode                  512    512     64     64
jbd2_journal_handle        1022   1022     56     73
jbd2_journal_head           408    408    120     34
jbd2_revoke_record_s          0      0     32    128
jbd2_revoke_table_s         256    256     16    256
jbd2_transaction_s          320    320    256     32
jfs_ip                        0      0   1312     24
jfs_mp                       32     32    128     32
kcopyd_job                    0      0   3240     10
kernfs_iattrs_cache        4129   4284     80     51
kernfs_node_cache         74640  74640    136     30
key_jar                     832    832    256     32
khugepaged_mm_slot            0      0     40    102
kioctx                       56     56    576     28
kmalloc-128                1727   1888    128     32
kmalloc-16                 4096   4096     16    256
kmalloc-192                 966    966    192     42
kmalloc-1k                  832    832   1024     32
kmalloc-256                 768    768    256     32
kmalloc-2k                  304    304   2048     16
Cache                       Num  Total   Size  Pages
kmalloc-32                 2816   2816     32    128
kmalloc-4k                  264    280   4096      8
kmalloc-512                 608    608    512     32
kmalloc-64                 1920   1920     64     64
kmalloc-8                  8192   8192      8    512
kmalloc-8k                   48     48   8192      4
kmalloc-96                  714    714     96     42
kmalloc_buckets              72     72    112     36
kmalloc-cg-128             2016   2016    128     32
kmalloc-cg-16              4096   4096     16    256
kmalloc-cg-192             2725   2940    192     42
kmalloc-cg-1k              1217   1408   1024     32
kmalloc-cg-256             1440   1440    256     32
kmalloc-cg-2k              1431   1520   2048     16
kmalloc-cg-32              9647  12032     32    128
kmalloc-cg-4k               423    424   4096      8
kmalloc-cg-512             1554   1664    512     32
kmalloc-cg-64              2304   2304     64     64
kmalloc-cg-8               8192   8192      8    512
kmalloc-cg-8k               108    108   8192      4
kmalloc-cg-96              6530   7392     96     42
kmalloc-rcl-128            1664   1664    128     32
kmalloc-rcl-16                0      0     16    256
kmalloc-rcl-192             756    756    192     42
kmalloc-rcl-1k               64     64   1024     32
kmalloc-rcl-256             160    160    256     32
kmalloc-rcl-2k                0      0   2048     16
kmalloc-rcl-32                0      0     32    128
kmalloc-rcl-4k                0      0   4096      8
kmalloc-rcl-512              64     64    512     32
kmalloc-rcl-64             3200   3200     64     64
kmalloc-rcl-8                 0      0      8    512
kmalloc-rcl-8k                0      0   8192      4
kmalloc-rcl-96             2353   2604     96     42
Cache                       Num  Total   Size  Pages
kmalloc-rnd-01-128          640    640    128     32
kmalloc-rnd-01-16          4096   4096     16    256
kmalloc-rnd-01-192         2054   2310    192     42
kmalloc-rnd-01-1k          2005   2368   1024     32
kmalloc-rnd-01-256        10354  16320    256     32
kmalloc-rnd-01-2k           400    400   2048     16
kmalloc-rnd-01-32          5383   7808     32    128
kmalloc-rnd-01-4k           280    280   4096      8
kmalloc-rnd-01-512         6393  10720    512     32
kmalloc-rnd-01-64          3762   3840     64     64
kmalloc-rnd-01-8           8192   8192      8    512
kmalloc-rnd-01-8k            68     68   8192      4
kmalloc-rnd-01-96           840    840     96     42
kmalloc-rnd-02-128          608    608    128     32
kmalloc-rnd-02-16          6143   6144     16    256
kmalloc-rnd-02-192         1386   1386    192     42
kmalloc-rnd-02-1k           512    512   1024     32
kmalloc-rnd-02-256          576    576    256     32
kmalloc-rnd-02-2k           272    272   2048     16
kmalloc-rnd-02-32          6795   6912     32    128
kmalloc-rnd-02-4k           176    176   4096      8
kmalloc-rnd-02-512          576    576    512     32
kmalloc-rnd-02-64          5824   5824     64     64
kmalloc-rnd-02-8           8192   8192      8    512
kmalloc-rnd-02-8k            20     20   8192      4
kmalloc-rnd-02-96          1512   1512     96     42
kmalloc-rnd-03-128          512    512    128     32
kmalloc-rnd-03-16         11264  12544     16    256
kmalloc-rnd-03-192          714    714    192     42
kmalloc-rnd-03-1k           736    736   1024     32
kmalloc-rnd-03-256         1600   1600    256     32
kmalloc-rnd-03-2k           854    864   2048     16
kmalloc-rnd-03-32          4867   5376     32    128
kmalloc-rnd-03-4k           250    256   4096      8
Cache                       Num  Total   Size  Pages
kmalloc-rnd-03-512          512    512    512     32
kmalloc-rnd-03-64          8397   8704     64     64
kmalloc-rnd-03-8           9728   9728      8    512
kmalloc-rnd-03-8k            96     96   8192      4
kmalloc-rnd-03-96          3906   3906     96     42
kmalloc-rnd-04-128          928    928    128     32
kmalloc-rnd-04-16          4096   4096     16    256
kmalloc-rnd-04-192          756    756    192     42
kmalloc-rnd-04-1k          5216   5216   1024     32
kmalloc-rnd-04-256          512    512    256     32
kmalloc-rnd-04-2k           256    256   2048     16
kmalloc-rnd-04-32          3456   3456     32    128
kmalloc-rnd-04-4k           400    400   4096      8
kmalloc-rnd-04-512          544    544    512     32
kmalloc-rnd-04-64          2048   2048     64     64
kmalloc-rnd-04-8           8192   8192      8    512
kmalloc-rnd-04-8k            80     80   8192      4
kmalloc-rnd-04-96           798    798     96     42
kmalloc-rnd-05-128         2432   2432    128     32
kmalloc-rnd-05-16          8474   9216     16    256
kmalloc-rnd-05-192         2604   2604    192     42
kmalloc-rnd-05-1k           608    608   1024     32
kmalloc-rnd-05-256          608    608    256     32
kmalloc-rnd-05-2k           400    400   2048     16
kmalloc-rnd-05-32          2560   2560     32    128
kmalloc-rnd-05-4k           128    128   4096      8
kmalloc-rnd-05-512          512    512    512     32
kmalloc-rnd-05-64          3550   3904     64     64
kmalloc-rnd-05-8           8192   8192      8    512
kmalloc-rnd-05-8k             8      8   8192      4
kmalloc-rnd-05-96           672    672     96     42
kmalloc-rnd-06-128         1728   1728    128     32
kmalloc-rnd-06-16          4096   4096     16    256
kmalloc-rnd-06-192         1911   1911    192     42
Cache                       Num  Total   Size  Pages
kmalloc-rnd-06-1k          1216   1216   1024     32
kmalloc-rnd-06-256         1716   1824    256     32
kmalloc-rnd-06-2k           664    702   2048     16
kmalloc-rnd-06-32          4613   5120     32    128
kmalloc-rnd-06-4k           302    336   4096      8
kmalloc-rnd-06-512         1728   1728    512     32
kmalloc-rnd-06-64          1664   1664     64     64
kmalloc-rnd-06-8           8192   8192      8    512
kmalloc-rnd-06-8k           188    200   8192      4
kmalloc-rnd-06-96          1302   1302     96     42
kmalloc-rnd-07-128         1664   1664    128     32
kmalloc-rnd-07-16          4096   4096     16    256
kmalloc-rnd-07-192          672    672    192     42
kmalloc-rnd-07-1k           608    608   1024     32
kmalloc-rnd-07-256          736    736    256     32
kmalloc-rnd-07-2k           352    352   2048     16
kmalloc-rnd-07-32          4097   4352     32    128
kmalloc-rnd-07-4k           280    308   4096      8
kmalloc-rnd-07-512          640    640    512     32
kmalloc-rnd-07-64          6556   7104     64     64
kmalloc-rnd-07-8           8192   8192      8    512
kmalloc-rnd-07-8k           215    220   8192      4
kmalloc-rnd-07-96          2699   2730     96     42
kmalloc-rnd-08-128         1152   1152    128     32
kmalloc-rnd-08-16          4096   4096     16    256
kmalloc-rnd-08-192         1260   1260    192     42
kmalloc-rnd-08-1k          1078   1132   1024     32
kmalloc-rnd-08-256          640    640    256     32
kmalloc-rnd-08-2k           720    720   2048     16
kmalloc-rnd-08-32          2048   2048     32    128
kmalloc-rnd-08-4k           232    264   4096      8
kmalloc-rnd-08-512          672    672    512     32
kmalloc-rnd-08-64          3044   3328     64     64
kmalloc-rnd-08-8           8192   8192      8    512
Cache                       Num  Total   Size  Pages
kmalloc-rnd-08-8k            92    108   8192      4
kmalloc-rnd-08-96          1764   1764     96     42
kmalloc-rnd-09-128          640    640    128     32
kmalloc-rnd-09-16         13567  14080     16    256
kmalloc-rnd-09-192         1008   1008    192     42
kmalloc-rnd-09-1k           512    512   1024     32
kmalloc-rnd-09-256          544    544    256     32
kmalloc-rnd-09-2k           272    272   2048     16
kmalloc-rnd-09-32          3072   3072     32    128
kmalloc-rnd-09-4k           258    264   4096      8
kmalloc-rnd-09-512          512    512    512     32
kmalloc-rnd-09-64          1344   1344     64     64
kmalloc-rnd-09-8          11292  11776      8    512
kmalloc-rnd-09-8k            64     64   8192      4
kmalloc-rnd-09-96          1554   1554     96     42
kmalloc-rnd-10-128          512    512    128     32
kmalloc-rnd-10-16          4352   4352     16    256
kmalloc-rnd-10-192          714    714    192     42
kmalloc-rnd-10-1k           512    512   1024     32
kmalloc-rnd-10-256          512    512    256     32
kmalloc-rnd-10-2k           256    256   2048     16
kmalloc-rnd-10-32          2688   2688     32    128
kmalloc-rnd-10-4k           128    128   4096      8
kmalloc-rnd-10-512          544    544    512     32
kmalloc-rnd-10-64          1920   1920     64     64
kmalloc-rnd-10-8           8192   8192      8    512
kmalloc-rnd-10-8k           176    180   8192      4
kmalloc-rnd-10-96           672    672     96     42
kmalloc-rnd-11-128         3272   3360    128     32
kmalloc-rnd-11-16          4096   4096     16    256
kmalloc-rnd-11-192         2309   2562    192     42
kmalloc-rnd-11-1k          1907   2016   1024     32
kmalloc-rnd-11-256          864    864    256     32
kmalloc-rnd-11-2k           456    480   2048     16
Cache                       Num  Total   Size  Pages
kmalloc-rnd-11-32          3200   3200     32    128
kmalloc-rnd-11-4k           152    152   4096      8
kmalloc-rnd-11-512         2045   2144    512     32
kmalloc-rnd-11-64          4002   4224     64     64
kmalloc-rnd-11-8           8192   8192      8    512
kmalloc-rnd-11-8k           124    139   8192      4
kmalloc-rnd-11-96          3464   3654     96     42
kmalloc-rnd-12-128          704    704    128     32
kmalloc-rnd-12-16         14384  14592     16    256
kmalloc-rnd-12-192          672    672    192     42
kmalloc-rnd-12-1k           544    544   1024     32
kmalloc-rnd-12-256          928    928    256     32
kmalloc-rnd-12-2k           544    544   2048     16
kmalloc-rnd-12-32          8589   8832     32    128
kmalloc-rnd-12-4k           128    128   4096      8
kmalloc-rnd-12-512         1620   1696    512     32
kmalloc-rnd-12-64          4673   5440     64     64
kmalloc-rnd-12-8          10240  10240      8    512
kmalloc-rnd-12-8k            44     44   8192      4
kmalloc-rnd-12-96          1050   1050     96     42
kmalloc-rnd-13-128         1824   1824    128     32
kmalloc-rnd-13-16          4352   4352     16    256
kmalloc-rnd-13-192         1638   1638    192     42
kmalloc-rnd-13-1k           512    512   1024     32
kmalloc-rnd-13-256          768    768    256     32
kmalloc-rnd-13-2k           848    848   2048     16
kmalloc-rnd-13-32          2688   2688     32    128
kmalloc-rnd-13-4k           255    284   4096      8
kmalloc-rnd-13-512         5078   5248    512     32
kmalloc-rnd-13-64          3811   4032     64     64
kmalloc-rnd-13-8           8192   8192      8    512
kmalloc-rnd-13-8k            40     40   8192      4
kmalloc-rnd-13-96          1092   1092     96     42
kmalloc-rnd-14-128          544    544    128     32
Cache                       Num  Total   Size  Pages
kmalloc-rnd-14-16         18944  19456     16    256
kmalloc-rnd-14-192          672    672    192     42
kmalloc-rnd-14-1k           672    672   1024     32
kmalloc-rnd-14-256         1568   1568    256     32
kmalloc-rnd-14-2k           480    480   2048     16
kmalloc-rnd-14-32         12288  12544     32    128
kmalloc-rnd-14-4k           289    312   4096      8
kmalloc-rnd-14-512          512    512    512     32
kmalloc-rnd-14-64          2048   2048     64     64
kmalloc-rnd-14-8          13312  15360      8    512
kmalloc-rnd-14-8k            92     92   8192      4
kmalloc-rnd-14-96           798    798     96     42
kmalloc-rnd-15-128          512    512    128     32
kmalloc-rnd-15-16          4096   4096     16    256
kmalloc-rnd-15-192         1386   1386    192     42
kmalloc-rnd-15-1k          1131   1248   1024     32
kmalloc-rnd-15-256          864    864    256     32
kmalloc-rnd-15-2k           304    304   2048     16
kmalloc-rnd-15-32          2432   2432     32    128
kmalloc-rnd-15-4k           128    128   4096      8
kmalloc-rnd-15-512          640    640    512     32
kmalloc-rnd-15-64          2112   2112     64     64
kmalloc-rnd-15-8           8192   8192      8    512
kmalloc-rnd-15-8k            40     40   8192      4
kmalloc-rnd-15-96          2226   2226     96     42
kmem_cache                  960    960    256     32
kmem_cache_node            1344   1344     64     64
ksm_mm_slot                   0      0     48     85
ksm_rmap_item                 0      0     64     64
ksm_stable_node               0      0     64     64
kvm_async_pf                  0      0    136     30
kvm_mmu_page_header           0      0    184     44
kvm_vcpu                      0      0   8960      3
lsm_file_cache            17526  20910     24    170
Cache                       Num  Total   Size  Pages
lsm_inode_cache           32435  64668    120     34
maple_node                 7664  12416    256     32
mb_cache_entry                0      0     56     73
memdup_user-128             256    256    128     32
memdup_user-16             4096   4096     16    256
memdup_user-192              42     42    192     42
memdup_user-1k              512    512   1024     32
memdup_user-256               0      0    256     32
memdup_user-2k              256    256   2048     16
memdup_user-32             2048   2048     32    128
memdup_user-4k               80     80   4096      8
memdup_user-512               0      0    512     32
memdup_user-64             1024   1024     64     64
memdup_user-8              8192   8192      8    512
memdup_user-8k                0      0   8192      4
memdup_user-96              588    588     96     42
mfc6_cache                    0      0    192     42
mfc_cache                     0      0    192     42
migrate_vma_state_t           0      0  33320      1
minix_inode_cache             0      0    704     46
mm_struct                   552    552   1408     23
mnt_cache                  2298   2352    384     42
MPTCP                         0      0   1984     16
MPTCPv6                       0      0   2176     15
mqueue_inode_cache          544    544    960     34
msg_msg-128                   0      0    128     32
msg_msg-16                    0      0     16    256
msg_msg-192                   0      0    192     42
msg_msg-1k                    0      0   1024     32
msg_msg-256                   0      0    256     32
msg_msg-2k                    0      0   2048     16
msg_msg-32                    0      0     32    128
msg_msg-4k                    0      0   4096      8
msg_msg-512                   0      0    512     32
Cache                       Num  Total   Size  Pages
msg_msg-64                    0      0     64     64
msg_msg-8                     0      0      8    512
msg_msg-8k                    0      0   8192      4
msg_msg-96                    0      0     96     42
names_cache                 192    192   4096      8
net_namespace               189    189   4672      7
nf_conntrack               1248   1376    256     32
nf_conntrack_expect           0      0    208     39
nf-frags                      0      0    184     44
nilfs2_btree_path_cache       0      0   1792     18
nilfs2_inode_cache            0      0    872     37
nilfs2_segbuf_cache           0      0    240     34
nilfs2_transaction_cache      0      0     24    170
nsproxy                     896    896     72     56
numa_policy                 480    480    272     30
nvidia_p2p_page_cache         0      0     32    128
nvidia_pte_cache          84614  85376     32    128
nvidia_stack_cache            0      0  12288      2
nvme-chap-buf-cache          16     16   4096      8
pde_opener                 1632   1632     40    102
pending_reservation           0      0     32    128
perf_event                  432    480   1328     24
pid                        2936   2944    128     32
pid_2                        84     84    192     42
pid_3                       672    672    192     42
pid_namespace               280    280    144     28
PING                         96     96   1024     32
PINGv6                       78     78   1216     26
pma_address_batch             0      0    512     32
pool_workqueue             1664   1664    512     32
posix_timers_cache          128    128    256     32
proc_dir_entry             5166   5166    192     42
proc_inode_cache           5298   5566    712     46
pte_list_desc                 0      0    128     32
Cache                       Num  Total   Size  Pages
QIPCRTR                      78     78    832     39
radix_tree_node           97568 165998    584     28
RAW                          64     64   1024     32
RAWv6                       924    962   1216     26
request_queue                99     99    984     33
request_sock_subflow_v4       0      0    384     42
request_sock_subflow_v6       0      0    384     42
request_sock_TCP            416    416    312     26
request_sock_TCPv6            0      0    312     26
rpc_buffers                  16     16   2048     16
rpc_inode_cache              46     46    704     46
rpc_tasks                    32     32    256     32
rtable                      672    672    192     42
scsi_sense_cache            128    128    128     32
seq_file                    544    544    120     34
sgpool-128                    8      8   4096      8
sgpool-16                    32     32    512     32
sgpool-32                    32     32   1024     32
sgpool-64                    16     16   2048     16
sgpool-8                     32     32    256     32
shared_policy_node            0      0     48     85
shmem_inode_cache          4498   6027    792     41
sighand_cache               877    945   2112     15
signal_cache               1125   1148   1152     28
sigqueue                    816    816     80     51
skbuff_ext_cache              0      0    192     42
skbuff_fclone_cache         832    832    512     32
skbuff_head_cache          1708   1792    256     32
skbuff_small_head          1325   1450    640     25
sock_inode_cache           2352   2847    832     39
task_delay_info               0      0    144     28
task_group                  400    400    640     25
taskstats                   592    592    432     37
task_struct                1451   1461  10496      3
Cache                       Num  Total   Size  Pages
TCP                         351    351   2368     13
tcp_bind2_bucket            544    544    128     32
tcp_bind_bucket             544    544    128     32
TCPv6                       204    204   2560     12
trace_event_file           3696   3696     96     42
tracefs_inode_cache         360    360    672     24
tw_sock_TCP                 496    496    264     31
tw_sock_TCPv6                93     93    264     31
UDP                         448    448   1152     28
UDP-Lite                      0      0   1152     28
UDPLITEv6                     0      0   1344     24
UDPv6                       384    384   1344     24
ufs_inode_cache               0      0    840     39
uhci_urb_priv                 0      0     56     73
uid_cache                   462    462    192     42
UNIX                       1028   1120   1024     32
UNIX-STREAM                1308   1408   1024     32
userfaultfd_ctx_cache         0      0    192     42
user_namespace              400    400    640     25
uts_namespace               407    407    432     37
uvm_block_thrashing_info      0      0    192     42
uvm_gpu_chunk_4             330    330    136     30
uvm_gpu_chunk_5              31     31    264     31
uvm_gpu_chunk_t             876    876     56     73
uvm_page_mask_t             512    512     64     64
uvm_perf_callback_list     1360   1360     24    170
uvm_pinned_page_t             0      0     56     73
uvm_pmm_sysmem_page_reve      0      0     16    256
uvm_processor_mask_t       1632   1632     40    102
uvm_range_group_range_t       0      0     96     42
uvm_range_group_t           730    730     56     73
uvm_service_block_contex      4      4  17784      1
uvm_tools_block_map_remo      0      0     88     46
uvm_tools_block_migratio      0      0    120     34
Cache                       Num  Total   Size  Pages
uvm_tools_event_tracker_      0      0   1136     28
uvm_tools_map_remote_dat      0      0     48     85
uvm_tools_migration_data      0      0     56     73
uvm_tools_replay_data_t       0      0     80     51
uvm_va_block_context_t       33     33  10464      3
uvm_va_block_cpu_node_st      0      0    136     30
uvm_va_block_gpu_state_t      0      0    416     39
uvm_va_block_t                0      0   2952     11
uvm_va_policy_node_t          0      0    112     36
uvm_va_range_channel_t      272    272    240     34
uvm_va_range_device_p2p_      0      0    152     26
uvm_va_range_external_t      84     94  14512      2
uvm_va_range_managed_t      259    259    216     37
uvm_va_range_semaphore_p    112    112   2256     14
uvm_va_range_sked_reflec    352    352    184     44
uvm_vma_wrapper_t           595    595     48     85
vma_lock                  64741  82212     40    102
vmap_area                  9324  17752     72     56
vm_area_struct            64672  78982    176     46
x86_emulator                  0      0   2672     12
x86_lbr                      21     21   1536     21
...
3

I recommend you put zswap back for better system stability.

Which stat are you tracking that shows the leak?
When you quit the allama app does the memory get freed?

4

Most of memory (over 10GiB) is used by kernel after call to ollama. System starts to swap heavily. Large number of kernel dynamic memory is allocated: smem -wp shows that large percent of kernel dynamic memory is allocated to non-cache.

Aya model called by ollama:

root@fedora:~# smem -wp
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        70.17%     33.74%     36.43% 
userspace memory             24.05%      3.33%     20.72% 
free memory                   5.78%      5.78%      0.00%

qwen model called:

root@fedora:~# smem -wp
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        71.42%     24.57%     46.85% 
userspace memory             24.48%      3.32%     21.16% 
free memory                   4.10%      4.10%      0.00%

codestral model called:

root@fedora:~# smem -wp
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        71.42%      5.33%     66.10% 
userspace memory             20.82%      1.68%     19.14% 
free memory                   7.76%      7.76%      0.00%

aya model called again:

root@fedora:~# smem -wp
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        83.93%     11.17%     72.76% 
userspace memory              8.88%      1.42%      7.46% 
free memory                   7.18%      7.18%      0.00%
5

You are on a 16GiB system that may be too small to run the AI code.

Have other people said that you can run this work load in that size memory?

When my son runs AI models be needed a powerful 4090 GPU and a lots of RAM for example.

6

There may be a chance the system is not using the GPU, but instead using the CPU. I would check that first. Verify Ollama with which LLM you’re using is being offloaded onto the GPU.

1 Like
7

GPU is used by ollama, I checked it with ollama ps. My notebook has 16GiB of video memory (GeForce RTX 3080 Ti Mobile).

root@fedora:~# ollama ps
NAME                           ID              SIZE     PROCESSOR         UNTIL              
qwen2.5:32b-instruct-q3_K_M    44bd723cfd42    17 GB    8%/92% CPU/GPU    4 minutes from now    

root@fedora:~# ollama ps
NAME                       ID              SIZE     PROCESSOR          UNTIL              
codestral:22b-v0.1-q6_K    e77ab7ec51f6    19 GB    16%/84% CPU/GPU    4 minutes from now    

root@fedora:~# ollama ps
NAME                   ID              SIZE     PROCESSOR    UNTIL              
aya-expanse:8b-q8_0    dea9deb4709b    10 GB    100% GPU     2 minutes from now

When GPU is not used, the situation is much worse.

8

My system is:

11:03:05 james@toolbx ~ → smem -wp
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        25.11%     23.12%      1.98% 
userspace memory              8.41%      2.31%      6.10% 
free memory                  66.49%     66.49%      0.00%
9 
11:05:58 james@Z840 ~ → podman exec -it ollama ollama ps
NAME    ID    SIZE    PROCESSOR    UNTIL
10

I have similar statistic after reboot until I start to use ollama: 4-5GiB is used by Firefox, 1GiB by idle ollama service (0.3.14), <1GiB by the rest of system.

11

Are you using Fedora Workstation? I use Ollama with llama3.1:8b-instruct-q8_0 and openweb-ui in podman containers in Fedora Linux Silverblue 41. That maybe why I don’t see any output from ollama ps. Both were running when I posted my output.

12

Yes, I use Fedora Workstation 40, with Mate desktop, NVidia driver from negativo17.org repository (nvidia-driver-565.57.01) built using akmod, with BTRFS, encrypted disk, and my own key for secure bios. Ollama is installed locally in /usr/local/.

13

I can’t install locally in Silverblue. The system is read only.

I reloaded the model and tried again, and get this:

11:24:06 james@Z840 ~ → podman exec -it ollama ollama ps
NAME               ID              SIZE      PROCESSOR    UNTIL              
llama3.1:latest    42182419e950    6.7 GB    100% GPU     2 minutes from now
14 
11:26:05 james@toolbx ~ → smem -wp
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        32.27%     30.28%      1.99% 
userspace memory              8.48%      2.28%      6.20% 
free memory
15

What I would try at this point is to get a really fast USB and install fedora Workstation on it, and then enable the RPMfusion repositories. Add the stock Nvidia drivers. Install Ollama and add llama3.1:8b LLM and see how it performs. Those third party Nvidia driver can cause problems. Especially akmod modifying the kernel to use those alternate drivers. Do that just as a test, and at least you have a rescue drive anyway.

16

I cannot disable secure boot in BIOS, so I must build and sign NVidia drivers localy via akmod. I have better experience with NVidia and CUDA driver packaged by negativo17 than with RPMFusion. Both package the same closed source NVidia driver. Both are thirdparty to Fedora.

I’m long time Redhat and Fedora user (since Fedora 1.0). I need help to debug kernel memory leak.

17

I checked my NVidia driver, and found that negativo17 switched from closed source NVidia driver to open source NVidia driver. I switched driver back to closed source in /etc/nvidia/kernel.conf, and problem is gone, so problem is in open source version of the NVidia driver.

18

Excellent! Glad you got it fixed. I was thinking along the lines of the video driver being the problem. So you still have the secure boot also?

19

Yes, secure boot is still working, because I rebuilt akmod after switching using akmod --rebuild command.

However, memory leak is still present in opensource version of NVidia driver.

20

Yup, I’m using Nvidia opensource drivers. They are a mess. A lot of work still needs to be done. Glad it worked out for you. We all learn something from this.