Hi!
I am using a Fedora33 AWS EC2 instance. Once logged in I try to use dnf update however the process almost immediately gets killed. When running a check on the makecache service it says that it failed to start DNF makecache. Not sure what to try here. I have tried to do a dnf clean all and then an update but its the same issue.
Any help/suggestions would be greatly appreciated.
Thank you!
Process could be killed due to numerous reasons, like triggering out of memory or whatever. Message itself doesn’t provide the details.
My suggestion:
- Try to trigger the error
- look at
sudo journalctl -n 1000 -e(last 1000 entries) to see if anything stands out - if so, paste the output so we can review.
Could be drive space, could be memory, could be that you are not running it as root or with sudo. Check with df, free, and make certain the update is being run as root.
Dnf allows some things to be done by a standard user but not updates since the user cannot write into system spaces…
Yes so it looks like you are correct, its OOM. Not sure why this is occurring, space wise I have open space. How do I check or free up some more memory? Please let me know.
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 441] 0 441 8141 157 61440 0 -1000 auditd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 472] 996 472 23463 176 65536 0 0 chronyd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 474] 0 474 11618 402 102400 0 0 sssd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 475] 0 475 5178 268 77824 0 0 systemd-homed
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 476] 81 476 67317 158 61440 0 -900 dbus-broker-lau
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 477] 0 477 12042 524 106496 0 0 sssd_be
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 478] 0 478 15351 320 147456 0 0 sssd_nss
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 479] 81 479 1381 145 49152 0 -900 dbus-broker
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 480] 0 480 5602 685 86016 0 0 systemd-logind
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 533] 0 533 65680 632 131072 0 0 NetworkManager
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 620] 0 620 5770 239 65536 0 -1000 sshd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 622] 0 622 2399 28 45056 0 0 agetty
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 623] 0 623 3041 34 49152 0 0 agetty
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 637] 0 637 5053 243 81920 0 0 systemd-userdbd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2021] 0 2021 7455 269 86016 0 0 systemd-userwor
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2022] 0 2022 7455 268 86016 0 0 systemd-userwor
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2023] 0 2023 7455 267 81920 0 0 systemd-userwor
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2024] 0 2024 10770 324 90112 0 0 sshd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2028] 1000 2028 6104 937 90112 0 0 systemd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2031] 1000 2031 32764 1352 110592 0 0 (sd-pam)
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2038] 1000 2038 9135 328 81920 0 0 sshd
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2039] 1000 2039 4044 161 61440 0 0 bash
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2064] 0 2064 9215 240 77824 0 0 sudo
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2066] 0 2066 8731 216 81920 0 0 su
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2067] 0 2067 4064 162 57344 0 0 bash
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: [ 2088] 0 2088 122954 85006 860160 0 0 dnf
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,t>
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: Out of memory: Killed process 2088 (dnf) total-vm:491816kB, anon-rss:340020kB, file-rss:>
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal kernel: oom_reaper: reaped process 2088 (dnf), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t>
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal systemd[1]: dnf-makecache.service: Main process exited, code=killed, status=9/KILL
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal systemd[1]: dnf-makecache.service: Failed with result 'signal'.
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal systemd[1]: Failed to start dnf makecache.
Jun 22 16:48:11 ip-172-31-15-164.us-west-1.compute.internal systemd[1]: dnf-makecache.service: Consumed 11.777s CPU time.
Jun 22 16:49:29 ip-172-31-15-164.us-west-1.compute.internal systemd[2028]: Starting Mark boot as successful...
Jun 22 16:49:29 ip-172-31-15-164.us-west-1.compute.internal systemd[2028]: grub-boot-success.service: Succeeded.
Jun 22 16:49:29 ip-172-31-15-164.us-west-1.compute.internal systemd[2028]: Finished Mark boot as successful.
Jun 22 16:49:30 ip-172-31-15-164.us-west-1.compute.internal audit[2101]: USER_ACCT pid=2101 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_>
Jun 22 16:49:30 ip-172-31-15-164.us-west-1.compute.internal audit[2101]: USER_CMD pid=2101 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t>
Jun 22 16:49:30 ip-172-31-15-164.us-west-1.compute.internal sudo[2101]: root : TTY=pts/0 ; PWD=/home/fedora ; USER=root ; COMMAND=/usr/bin/journalctl -n>
Jun 22 16:49:30 ip-172-31-15-164.us-west-1.compute.internal audit[2101]: CRED_REFR pid=2101 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_>
Jun 22 16:49:31 ip-172-31-15-164.us-west-1.compute.internal sudo[2101]: pam_unix(sudo:session): session opened for user root(uid=0) by fedora(uid=0)
Jun 22 16:49:31 ip-172-31-15-164.us-west-1.compute.internal audit[2101]: USER_START pid=2101 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined>
So for some reason it tries to allocate shy of 500MB of memory.
Could you post the output of the following command?
free -m
You can also look at what’s using most of the memory with top -c and then sorting by memory.
If it’s a small instance (like 512MB of memory or so) then I’d recommend adding swap file, at least 1GB so you have some wiggle room for regular usage.
free -m produces the following: how much space does this requires?
MiB Mem : 463.2 total, 283.8 free, 97.2 used, 82.2 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 338.4 avail Mem
As for the top command the only thing consuming space is the following
1 root 20 0 109752 10308 5112 S 0.0 2.2 1:15.24 /usr/lib/systemd/systemd --switched-+
I am curious if a larger instance would solve the issue (more space)
Please let me know.
as seen, it’s like smallest instance. These days 500MB of RAM is a bit too little for normal day to day operation. 512-463=49 MB which are reserved/used by loaded kernel, so you have less 463 available for the rest. From that, software running on the system like systemd, cron etc. they all need some minimal amounts of RAM as well.
When installing package upgrade, DNF can run some post scripts etc. which can kick instance over the edge.
My suggestion is turning on SWAP file of 1GB or so, this should be enough.
Not sure what the workload of the instance is, but if it needs to be performant , and goes over 512MB of usage constantly, then yes, larger instance would be better. For simple machine which serves as a jump-box or whatever, swap should do the job.
1 Like
Thank you! Looks like using a larger AWS instance did the trick (more space).