And I found the problem was caused by wrong user:group of the file /var/cache/ddclient/ddclient.cache (which should be ddclient:ddclient instead of clevis):
-rw-------. 1 clevis 960 system_u:object_r:ddclient_var_t:s0 614 Dec 30 14:08 /var/cache/ddclient/ddclient.cache
I also found user:group of the file /etc/ddclient.conf is wrong:
-rw-------. 1 clevis 960 system_u:object_r:ddclient_etc_t:s0 10410 Oct 5 08:53 /etc/ddclient.conf
It seems that the user:group id of ddclient:ddclient has changed.
While the problem can be solved by running chown on the files. I still don’t know why suddenly user:group id changed. Anyone has any clues?
gid=960 is in the range of dynamically assigned ids and maybe the scripts in the rpm will shed some light on why they changed. Is your system an “atomic” version or purely RPMs?
I am using silverblue fc41. By checking the journalctl log, I found the error occurred after I updated the system, so it looks like the scripts in the rpm did this.
By inspecting the contents of /etc/passwd, /etc/group and /usr/lib/passwd, /usr/lib/group, I found that clevis user/group is defned in /etc/passwd and /etc/group while ddclient user/group is defned in the /usr/lib/passwd and /usr/lib/group.
$ grep ddclient /usr/lib/passwd
dclient:x:963:962:Dynamic DNS Client:/var/cache/ddclient:/sbin/nologin
$ grep ddclient /usr/lib/group
ddclient:x:962:
I think system users/groups in silverblue should all go to /usr/lib/passwd and /usr/lib/group. Maybe it is an upstream bug that handled user/group ids incorretly?
Searching around a bit shows problems that have been looked at but you have a case that is currently not working. More recent attempts in bootc exist. Not sure what is silverblue using today. The transition to a new way can’t be easy.
After reading the github issue, I have some clues why this happened but still not very sure.
I occasionally run sudo rpm-ostree apply-live --allow-replacement after rpm-ostree upgrade to avoid rebooting my system every time.
I think maybe rpm-ostree does not handle user/groupd id allocation correctly when executing apply-live, which caused clevis scripts in the rpm modified /etc/passwd and /etc/group instead of /usr/lib/passwd and /usr/lib/group. Therefore clevis appeared in /etc/passwd and /etc/group instead of /usr/lib/passwd and /usr/lib/group.
So I manually removed clevis from /etc/passwd and /etc/group and uninstalled some layered packages to enable rpm-ostree to update my system. After rebooting, clevis user/group now is defined in /usr/lib/passwd and /usr/lib/group. However, I am not sure this is the real cause because I can not reproduce this bug by running apply-live.
I think system users/groups in silverblue should all go to /usr/lib/passwd and /usr/lib/group. Maybe it is an upstream bug that handled user/group ids incorretly?