Failed to start DNF makecache

Hi!
I am using a Fedora33 AWS EC2 instance. Once logged in I try to use dnf update however the process almost immediately gets killed. When running a check on the makecache service it says that it failed to start DNF makecache. Not sure what to try here. I have tried to do a dnf clean all and then an update but its the same issue.
Any help/suggestions would be greatly appreciated.
Thank you!

The killed message when trying to update:

Process could be killed due to numerous reasons, like triggering out of memory or whatever. Message itself doesn’t provide the details.

My suggestion:

  • Try to trigger the error
  • look at sudo journalctl -n 1000 -e (last 1000 entries) to see if anything stands out
  • if so, paste the output so we can review.

Could be drive space, could be memory, could be that you are not running it as root or with sudo. Check with df, free, and make certain the update is being run as root.

Dnf allows some things to be done by a standard user but not updates since the user cannot write into system spaces…

Yes so it looks like you are correct, its OOM. Not sure why this is occurring, space wise I have open space. How do I check or free up some more memory? Please let me know.

Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    441]     0   441     8141      157    61440        0         -1000 auditd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    472]   996   472    23463      176    65536        0             0 chronyd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    474]     0   474    11618      402   102400        0             0 sssd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    475]     0   475     5178      268    77824        0             0 systemd-homed
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    476]    81   476    67317      158    61440        0          -900 dbus-broker-lau
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    477]     0   477    12042      524   106496        0             0 sssd_be
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    478]     0   478    15351      320   147456        0             0 sssd_nss
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    479]    81   479     1381      145    49152        0          -900 dbus-broker
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    480]     0   480     5602      685    86016        0             0 systemd-logind
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    533]     0   533    65680      632   131072        0             0 NetworkManager
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    620]     0   620     5770      239    65536        0         -1000 sshd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    622]     0   622     2399       28    45056        0             0 agetty
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    623]     0   623     3041       34    49152        0             0 agetty
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [    637]     0   637     5053      243    81920        0             0 systemd-userdbd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2021]     0  2021     7455      269    86016        0             0 systemd-userwor
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2022]     0  2022     7455      268    86016        0             0 systemd-userwor
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2023]     0  2023     7455      267    81920        0             0 systemd-userwor
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2024]     0  2024    10770      324    90112        0             0 sshd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2028]  1000  2028     6104      937    90112        0             0 systemd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2031]  1000  2031    32764     1352   110592        0             0 (sd-pam)
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2038]  1000  2038     9135      328    81920        0             0 sshd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2039]  1000  2039     4044      161    61440        0             0 bash
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2064]     0  2064     9215      240    77824        0             0 sudo
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2066]     0  2066     8731      216    81920        0             0 su
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2067]     0  2067     4064      162    57344        0             0 bash
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [   2088]     0  2088   122954    85006   860160        0             0 dnf
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,t>
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: Out of memory: Killed process 2088 (dnf) total-vm:491816kB, anon-rss:340020kB, file-rss:>
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: oom_reaper: reaped process 2088 (dnf), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t>
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal systemd[1]: dnf-makecache.service: Main process exited, code=killed, status=9/KILL
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal systemd[1]: dnf-makecache.service: Failed with result 'signal'.
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal systemd[1]: Failed to start dnf makecache.
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal systemd[1]: dnf-makecache.service: Consumed 11.777s CPU time.
Jun 22 16:49:29 ip-172-31-15-164.us-west-1.compute.internal systemd[2028]: Starting Mark boot as successful...
Jun 22 16:49:29 ip-172-31-15-164.us-west-1.compute.internal systemd[2028]: grub-boot-success.service: Succeeded.
Jun 22 16:49:29 ip-172-31-15-164.us-west-1.compute.internal systemd[2028]: Finished Mark boot as successful.
Jun 22 16:49:30 ip-172-31-15-164.us-west-1.compute.internal audit[2101]: USER_ACCT pid=2101 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_>
Jun 22 16:49:30 ip-172-31-15-164.us-west-1.compute.internal audit[2101]: USER_CMD pid=2101 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t>
Jun 22 16:49:30 ip-172-31-15-164.us-west-1.compute.internal sudo[2101]:     root : TTY=pts/0 ; PWD=/home/fedora ; USER=root ; COMMAND=/usr/bin/journalctl -n>
Jun 22 16:49:30 ip-172-31-15-164.us-west-1.compute.internal audit[2101]: CRED_REFR pid=2101 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_>
Jun 22 16:49:31 ip-172-31-15-164.us-west-1.compute.internal sudo[2101]: pam_unix(sudo:session): session opened for user root(uid=0) by fedora(uid=0)
Jun 22 16:49:31 ip-172-31-15-164.us-west-1.compute.internal audit[2101]: USER_START pid=2101 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined>

So for some reason it tries to allocate shy of 500MB of memory.

Could you post the output of the following command?

free -m

You can also look at what’s using most of the memory with top -c and then sorting by memory.

If it’s a small instance (like 512MB of memory or so) then I’d recommend adding swap file, at least 1GB so you have some wiggle room for regular usage.

free -m produces the following: how much space does this requires?

MiB Mem :    463.2 total,    283.8 free,     97.2 used,     82.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.    338.4 avail Mem

As for the top command the only thing consuming space is the following

 1 root      20   0  109752  10308   5112 S   0.0   2.2   1:15.24 /usr/lib/systemd/systemd --switched-+

I am curious if a larger instance would solve the issue (more space)
Please let me know.

as seen, it’s like smallest instance. These days 500MB of RAM is a bit too little for normal day to day operation. 512-463=49 MB which are reserved/used by loaded kernel, so you have less 463 available for the rest. From that, software running on the system like systemd, cron etc. they all need some minimal amounts of RAM as well.

When installing package upgrade, DNF can run some post scripts etc. which can kick instance over the edge.

My suggestion is turning on SWAP file of 1GB or so, this should be enough.

Not sure what the workload of the instance is, but if it needs to be performant , and goes over 512MB of usage constantly, then yes, larger instance would be better. For simple machine which serves as a jump-box or whatever, swap should do the job.

1 Like

Thank you! Looks like using a larger AWS instance did the trick (more space).