Dynamic users with sysusers and bootc/image mode

Project Discussion
1

Hi,

I am trying to setup a mail server based on bootc and stalwart (as people who came to my talk at devconf.cz might know), and one of the issue I stumbled upon is the interaction between bootc and sysusers snippets in some Fedora packages.

For my mail server, I picked caddy as a proxy for the admin interface and the caddy package use sysusers.d to create a caddy user. Then upon getting a certificate, it create files in /var/lib/caddy and reuse them later.

One day, caddy didn’t start, and a quick debug revealed that /var/lib/caddy wasn’t owned by the caddy (982) uid but 981. I do not know what happened, but my theory is that at a build time T, caddy got assigned a uid (981) in the container, and at a build time T+1, a different one was assigned in a new build. Since sysusers are assigned in %post in a first come first served fashion, I assume that the packages order installation changed, or I just installed another package earlier in my Dockerfile, etc.

Due to the way the 3 way merge with /etc work, the initial /etc/password with caddy 981 was replaced by one with caddy assigned to uid 982. But the ownership of files created before in /var/lib/caddy wasn’t changed, so caddy couldn’t read them under its new UID and failed.

There is 2 possible fixes. Either we make sure that the UID never change (eg, go back to static allocation like 15 years ago), or we make sure that the files in /var/lib/caddy get changed as well. I picked the 2nd one, and I have submitted the fix to the package (and applied in my bootc container).

But this problem is likely not limited to caddy. For example, bind9-next use sysusers, and have files in /var/ that might need to be fixed if uid is changed. Just on my own server, I found that prosody would face the same issue, coturn, and unbound. I am sure most packages do, because once a user is created, it doesn’t change its uid on a normal Fedora system so no one has a reason to face the issue (except specific case where you reinstall your system and keep existing /var).

And if we want to avoid surprises on Image Mode, this class of bugs should be IMHO fixed.
And so I would suggest to:

  • amend the policy to say that using sysusers.d should be paired with a tmpfiles.d fragment to correct /var/ files if it make sense (I am sure there is corner cases, hence this post).
  • add a rpmlint check for that.

propose to use tpmfiles.d that because this look like the least painful from a upgrade perspective. If I had to do it from 0, I would use systemd DynamicUsers in most services because this is easier and is made to allocate UID in a smarter way to reduce chown(). But if we want to upgrade existing packages to use DynamicUsers instead of sysuers.d, I think this could potentially break backups scripts (where /var/lib/foo is hardcoded) and requires some data migration in /var, which can be tricky to do in a rpm %postinst if someone used different partitions.

2

See Add support for chowning across upgrades · Issue #1263 · bootc-dev/bootc · GitHub and yes this is a pain but TL;DR one can work around with tmpfiles.d units that do chowning.

As an aside:

I am not a big fan of the use of the word “normal” in that way as it implies other systems are abnormal. I would say “dnf based” for example. There are advantages and disadvantages to both systems, I wouldn’t say one is more normal than the other. (There are spaces of the world where the only way to deploy operating systems is image based for example, package based would be not-normal there.

Also related and as noted there, this problem occurs with NFS scenarios, and also cases where the applications are containerized and not part of the host (which we perhaps overly encourage).