Fedora 43: systemd-udevd 100% CPU and consuming all RAM

Ask Fedora
1

Hello!

After upgrading my ASUS vivobook (Intel Core Ultra 7) to Fedora 43 KDE, my system crashes after 3-4 hours working.

It starts ok, but after a few hours, systemd-udevd starts consuming 100% of one core and starts eating RAM until systemd-oomd starts killing processes and eventually the system crashses.

Sometimes it takes more time to start failing, but every time it eventually fails.

It’s happened with kernel 6.17-5 and also with just-updated 6.17-6

Is somebody having the same problem?

Thanks a lot

Regards

1 Like
2

Hello,

I’ve been having a similar issue on Fedora 43 & 6.17.7. After a few hours the memory usage spikes to 100% seemingly randomly and it’ll kernel panic “System is deadlocked on memory”. If I check htop right before the crash I also notice systemd-udevd as the culprit.

At first I thought it was a RAM issue so I ran memtest86 overnight, however it reported no errors.

Next I checked udevadm monitor which would output the same message hundreds of times per second:
KERNEL[836.223349] change /devices/virtual/thermal/thermal_zone1 (thermal)
KERNEL[836.223704] change /devices/virtual/thermal/thermal_zone1 (thermal)
UDEV [836.224307] change /devices/virtual/thermal/thermal_zone1 (thermal)
KERNEL[836.225016] change /devices/virtual/thermal/thermal_zone1 (thermal)
KERNEL[836.225144] change /devices/virtual/thermal/thermal_zone1 (thermal)
UDEV [836.225385] change /devices/virtual/thermal/thermal_zone1 (thermal)

This is probably the problem, cat /sys/class/thermal/thermal_zone1/type reports “INT3400 Thermal” which could indicate a bug with Intel’s thermal driver.

A possible temporary fix would be sudo systemctl stop thermald but this could lead to overheating, especially on your laptop. Otherwise, we may just have to wait for a fix in a future kernel update.

3

Hi!

I am having the same problem but do not have the aforementioned symptom with thermal zones: no thermald is installed and udevadm monitor is empty.

Currently, systemd-udevd is growing at a rate of about 30MB/s in RAM after a resume from suspend on an Intel Kaby Lake-R machine.

5

We need more details in order to understand the issue(s) and determine if they are symtoms of one underlying bug. man 8 systemd-udevd has many ways to increase the information provided by udevd:

The behavior of the daemon can be configured using udev.conf(5), its command line options, environment variables, and on the kernel command line, or changed dynamically with udevadm control.

Start with sudo udevadm -d monitor in a terminal window.

It is usually best if you can ensure that linux packages and vendor firmware are fully updated so you aren’t chasing a solved problem.

6

Thank you both. I’ll restart using the laptop (I gave up because of this), update all packages and try logging using udevadm as suggested.

I’ll let you know how it goes.

Regards

7

A little lost here…

After upgrading everything (again) the problem still exists.

When the laptop is plugged in, I get constant (once per second) messages about the battery:

KERNEL[5164.988492] change   /devices/platform/USBC000:00/typec/port0/port0-partner (typec)
KERNEL[5165.111155] change   /devices/platform/USBC000:00/power_supply/ucsi-source-psy-USBC000:001 (power_supply)
KERNEL[5165.111227] change   /devices/platform/USBC000:00/power_supply/ucsi-source-psy-USBC000:001 (power_supply)

After two hours, the problem appeared and top looked like this after a while:

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                
   621 root      20   0 6386388   6.1g   8568 R  99.3  19.7   3:17.53 systemd-udevd

I tried unplugging it. Immediately, the messages stopped appearing but systemd-udevd kept eating memory. Then, after 5-10 minutes, udevadm reported this several times:

UDEV  [5391.305849] change   /devices/platform/USBC000:00/power_supply/ucsi-source-psy-USBC000:001 (power_supply)

then systemd-udevd freed all the memory and returned to normal state.

I will keep doing tests

Regards
Miguel

8

in my case, I have kernel-6.17.11-300.fc43.x86_64 and udevadm -d monitor shows the following neverending sequence:

UDEV [59085.427720] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde2 (block) UDEV [59085.556309] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde1 (block) UDEV [59085.588631] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde4 (block) UDEV [59085.613123] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde3 (block) UDEV [59087.072490] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde (block) UDEV [59087.780640] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde1 (block) UDEV [59087.825326] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde2 (block) UDEV [59087.850730] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde4 (block) UDEV [59087.864484] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde3 (block) UDEV [59089.312058] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde (block) UDEV [59090.028316] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde2 (block) UDEV [59090.070654] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde1 (block) UDEV [59090.088768] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde4 (block) UDEV [59090.091803] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde3 (block) UDEV [59091.555267] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde (block) UDEV [59092.062615] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde4 (block) UDEV [59092.204165] change /devices/pci0000:00/0000:00:13.2/usb3/3-3/3-3.4/3-3.4.4/3-3.4.4:1.0/host13/target13:0:0/13:0:0:0/block/sde/sde1 (block)`

1 Like
9

Note: the high cpu usage survives a systemctl soft-reboot, but after said soft-reboot “udevadm -d monitor” shows nothing.

Also, the high CPU usage began while I was sharing an USB key (aka /dev/sde) with an OVH server remotely, via their own java kvm applet. Since all the reported activity was on /dev/sde, I suppose this should be related.

10

Sadly I was not able to reproduce this anymore, but it was related to logspam from a Samsung USB drive showing up as /dev/sda. The output was the same as the one mentioned above (with /dev/sde).

11

I have this exact issue. The stop thermald does nothing, I can only restart it or kill it with PID to free the memory, but a new process spawns immediately.

Did anyone find any more info?