Manually going through the logs via journalctl -p err I found the following errors, present even before I upgraded to F31:
Nov 09 23:06:39 wasteland systemd-udevd[704]: /usr/lib/udev/rules.d/60-nvidia-uvm.rules:1 Invalid value "/usr/bin/bash -c '/usr/bin/mknod -m 666 /dev/nvidia-uvm c $(grep nvidia-uvm /proc/de>
Nov 09 23:06:39 wasteland systemd-udevd[704]: /usr/lib/udev/rules.d/60-nvidia-uvm.rules:2 Invalid value "/usr/bin/bash -c '/usr/bin/mknod -m 666 /dev/nvidia-uvm-tools c $(grep nvidia-uvm /p>
Nov 09 23:06:39 wasteland systemd-udevd[704]: /usr/lib/udev/rules.d/60-nvidia.rules:2 Invalid value "/usr/bin/bash -c 'for i in $(cat /proc/driver/nvidia/gpus/*/information | grep Minor | c>
Any idea what it is related to?
I am using the nvidia driver, akmod-nvidia, from @rpmfusion. But before that I was using the official driver from the Nvidia website, installed using https://www.if-not-true-then-false.com/2015/fedora-nvidia-guide/. As far as I know, I’ve uninstalled it correctly, but who knows what’s left.
I started getting similar udevd errors after upgrading to F31, but not only for nvidia, which I have installed from the negativo repo.
Nov 06 13:41:36 systemd-udevd[978]: /etc/udev/rules.d/60-ipath.rules:4 Invalid value "kcopy/%02n" for NAME (char 7: invalid substitution type), ignoring, but please fix it.
Nov 06 13:41:36 systemd-udevd[978]: /usr/lib/udev/rules.d/60-nvidia.rules:2 Invalid value "/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidiactl c $(grep nvidia-frontend /proc/devices | cut -d \ -f 1) 255'" for RUN (char 61: invalid substitution type), ignoring, but please fix it.
Nov 06 13:41:36 systemd-udevd[978]: /usr/lib/udev/rules.d/60-nvidia.rules:3 Invalid value "/usr/bin/bash -c 'for i in $(cat /proc/driver/nvidia/gpus/*/information | grep Minor | cut -d \ -f 4); do /usr/bin/mknod -Z -m 666 /dev/nvidia${i} c $(grep nvidia-frontend /proc/devices | cut -d \ -f 1) ${i}; done'" for RUN (char 28: invalid substitution type), ignoring, but please fix it.
Nov 06 13:41:36 systemd-udevd[978]: /usr/lib/udev/rules.d/60-nvidia.rules:4 Invalid value "/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidia-modeset c $(grep nvidia-frontend /proc/devices | cut -d \ -f 1) 254'" for RUN (char 66: invalid substitution type), ignoring, but please fix it.
Nov 06 13:41:36 systemd-udevd[978]: /usr/lib/udev/rules.d/60-nvidia.rules:5 Invalid value "/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidia-uvm c $(grep nvidia-uvm /proc/devices | cut -d \ -f 1) 0'" for RUN (char 62: invalid substitution type), ignoring, but please fix it.
Nov 06 13:41:36 systemd-udevd[978]: /usr/lib/udev/rules.d/60-nvidia.rules:6 Invalid value "/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidia-uvm-tools c $(grep nvidia-uvm /proc/devices | cut -d \ -f 1) 1'" for RUN (char 68: invalid substitution type), ignoring, but please fix it.
Nov 06 13:41:36 systemd-udevd[978]: /usr/lib/udev/rules.d/65-md-incremental.rules:28 Invalid value "/sbin/mdadm -I $env{DEVNAME} --export $devnode --offroot ${DEVLINKS}" for IMPORT (char 58: invalid substitution type), ignoring, but please fix it.
Nov 06 13:41:36 systemd-udevd[978]: /usr/lib/udev/rules.d/99-vmware-scsi-udev.rules:5 Invalid value "/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'" for RUN (char 27: invalid substitution type), ignoring, but please fix it.
Nov 06 13:41:36 systemd-udevd[978]: /usr/lib/udev/rules.d/99-vmware-scsi-udev.rules:6 Invalid value "/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'" for RUN (char 27: invalid substitution type), ignoring, but please fix it.
It hasn’t caused me problems that I know of, so I’ve ignored it for now, but I would rather not have the errors.
You could try to issue this command: rpm -qf /usr/lib/udev/rules.d/60-nvidia-uvm.rules
In order to see to which package this file belongs to.
If it doesn’t return any result, probably such file was installed by a package no more installed on your system. If it returns the package owner, you could file a bug against it (of course on rpmfusion bug tracker, it it is the case).
[xxx@wasteland ~]$ rpm -qf /usr/lib/udev/rules.d/60-nvidia-uvm.rules | xargs -L 1 dnf info
Last metadata expiration check: 3:08:57 ago on Sun 10 Nov 2019 06:41:46 PM CET.
Installed Packages
Name : xorg-x11-drv-nvidia-cuda-libs
Epoch : 3
Version : 440.31
Release : 1.fc31
Architecture : x86_64
Size : 105 M
Source : xorg-x11-drv-nvidia-440.31-1.fc31.src.rpm
Repository : @System
From repo : rpmfusion-nonfree-updates
Summary : CUDA libraries for xorg-x11-drv-nvidia
URL : http://www.nvidia.com/
License : Redistributable, no modification permitted
Description : This package provides the CUDA driver libraries.
It seems systemd-udevd has tightened up its parsing of variable substitutions, since in every case the character flagged as the “invalid substitution type” is a $ character.
But not always — for instance, the $env{DEVNAME} and $devnode in 65-md-incremental.rules, in @fasulia’s output, don’t get flagged. That makes sense, as both of those are documented as valid syntax in the udev(7) man page. The syntax is fully spelled out there, so it’s relatively simple to figure out what’s wrong in each rules file and how to fix it.
It seems clear that this is a fairly recent change (the logging, at least, if not the strictness itself), and also that lots of upstream packages that provide udev rules haven’t been updated to avoid using unsupported/improper syntax. Probably because they weren’t even aware they were using any.
The “proper” long-term solution would be to reach out to each upstream and let them know their udev rules use invalid syntax (if they don’t already). Then, either get them to fix the rules so they conform to udev(7), or submit a patch and get them to accept it.
Short-term, until that can get dealt with by the upstreams, probably the most sensible course is just emulating systemd-udevd itself, and (as it says): “ignoring”.
Specifically, they’re $ characters that are meant for the underlying shell process, not for systemd-udevd itself. That probably works because when systemd-udevd saw unrecognized substitutions, it just silently ignored them. (Now, it noisily ignores them.)
But, as the udev(7) man page indicates, a literal $ can be inserted with $$, and a literal % with %%. So the solution to 90% of these is going just be replacing any flagged $ characters with $$, so they’ll pass through without triggering any complaints.
In the case of 65-md-incremental.rules, there’s no ${DEVLINKS} documented in the rules syntax, so I don’t know what that’s about. If it’s an envvar set somewhere else, then both $env{DEVLINKS} and $${DEVLINKS} should be valid replacements. (The first has systemd-udevd do the substitution, the other leaves it to the shell.)