Podman container can't chown and create new directories inside directory, even though directory is owned by its user

I’m running a podman container via podman-compose, with the environment variables specifying that it should run as the same user as the one that owns its configuration directory, yet I get errors like this:

chown: changing ownership of '/config': Permission denied
[nginx] | **** Permissions could not be set. This is probably because your volume mounts are remote or read-only. ****
[nginx] | **** The app may not work properly and we will not provide support for it. ****
mkdir: cannot create directory ‘/config/keys’: Permission denied
mkdir: cannot create directory ‘/config/php’: Permission denied
mkdir: cannot create directory ‘/config/www’: Permission denied
mkdir: cannot create directory ‘/config/log’: Permission denied
mkdir: cannot create directory ‘/config/log’: Permission denied
mkdir: cannot create directory ‘/config/nginx’: Permission denied
s6-rc: warning: unable to start service init-folders: command exited 1

here’s my compose file:

services:
  nginx:
    image: lscr.io/linuxserver/nginx:latest
    container_name: nginx
    environment:
      - PUID=1001
      - PGID=1002
      - TZ=Etc/UTC
    volumes:
      - /var/home/container/nginx/config:/config
    ports:
      - 80:80
      - 443:443
    restart: unless-stopped

Here’s the directory:

drwsrwsr-x. 3 container containers   46 Apr 12 21:47 .
drwsrwsr-x. 9 container containers 4096 Apr 12 21:36 ..
drwsrwsr-x. 2 container containers    6 Apr 12 21:48 config
-rw-rw-r--. 1 container containers  292 Apr 12 21:47 docker-compose.yml

where the uid of the container user is 1001, and the gid of containers is 1002.

When opening a shell inside the container, all the files inside the container show as owned by root:root, but the ones mounted from outside show as owned by nobody:nobody.

How do I fix this? Is this a SELinux thing? Not even running these as root seems to work.

Added podman

Unless you actually have a file named config in the root directory and you’re trying to change its ownership with root privileges, that’s never going to work. And, I think you’ll find that chown always requires elevated privs to work.

Don’t you need to say that its bind mounted read-write?
Isn’t the default to bind mount read-only?

it’s /config in the container’s root directory, but that’s just a volume mounted to a regular folder (pictured above) on the actual server. Also yes, it does require elevated privileges, and within the container, the program does indeed have them, and its conception of “root” should be identical to the “container” user, therefore making it able to chown config, but it isn’t, because somehow the users aren’t mapping right and config is actually owned by “nobody”

I’ll give that a try and see if it helps

What does :
ls -Z /var/home/container/nginx/config
bring up return?

If SELinux is enabled and causing permission issues, you might need to adjust the SELinux context of your directories or files using the chcon command :

chcon -R -t container_file_t /var/home/container/nginx/config

1 Like

I think you need to add :z or :Z to your volume mount.

For more information search in the podman docs for labeling volume mounts .

2 Likes

That’s what I initially thought too, but that created an “lsetxattr: operation not permitted” (or something to that effect) error

That error is given when the user that is used to run podman doesn’t have access to the mounted directory. In your opening post I see that config is owned by containers, is containers also used to run podman? Did containers have access to the config directory when you tried to use :Z?

As a debug step you could try to disable SELinux separation for the container using --security-opt label=disable.

That error is given when the user that is used to run podman doesn’t have access to the mounted directory. In your opening post I see that config is owned by containers, is containers also used to run podman? Did containers have access to the config directory when you tried to use :Z?

Ah, that would be it. Looking at the problem with fresh eyes this morning, I realized that the PGID and PUID variables in the compose file were for setting the user and group inside the container, not what user and group the container itself was running as, so in effect I was running podman as myself, in a directory owned by another user (container), so that’s why it couldn’t change the permissions.

So now, really important question:

how do I run podman as another user when that user doesn’t have a password and doesn’t log in (so systemd isn’t started for it), it’s just a user I created specifically for containers?

To let systemd start for a user enable lingering: loginctl enable-linger $USER
Then systemd will automatically start the enabled services for that user, without the need for that user to login.

https://www.freedesktop.org/software/systemd/man/latest/loginctl.html

1 Like

Lovely, thank you!

It worked!!!

1 Like

Just a quick suggestion because you are using coreos (whoops, see it is only tagged and never mentioned ;p). Have you considered podman quadlets? I think they are pretty easy to work with and then you don’t need to layer podman-compose. From my experience podman-compose can have some strange edge cases.

If you are used to docker I think using podman-compose is fine, but if you are still learning and are only interested in podman, quadlets might be a better option. It has pretty good documentation: podman-systemd.unit — Podman documentation

2 Likes

Thank you, I’ll certainly check out your suggestion, but I am actually much more experienced with docker-compose, trying to port old docker-compose configs over, and I’m also on an image (ucore-minimal) that comes with podman-compose already layered, so it makes sense for me to use podman-compose at the moment I think.

1 Like

Purged my images and reran podman-compose do to an unrelated heisenbug and I am now getting the chown: changing ownership: operation not permitted errors even with the volumes mounted as :Z and even though I’m not getting an lsetxattr or whatever error, so the volumes must be mounting correctly as far as SELinux is concerned. And yet…

Is chown executed as the root user in the container? If not that might be the problem.

When using rootless podman the root user in the container is mapped to the host user. Host user here being the user that runs podman, so root in the container can access the files when owned by the host user.

But when using another user in the container it gets assigned a ‘random’ UID from the range set in /etc/subuid. This is from inside the container not visible, the UID there will be normal, but on the host it will be different. So if the file is owned by the host user and the service is executed as a non-root user in the container it might not have access.

It is possible to map the UID in the container to a specific UID on the host, if you want to try that it is explained here: podman-run — Podman documentation

Probably a better option is to chown the files from inside the container using the root user. The mapping should always be the same, so after doing this once it should no longer give problems. From the hosts point of view the files will be owned by a random UID without user attached.