Podman-compose how to connect to socket?

Ask Fedora
, ,
1

I can’t find documentation about how to do this. I am migrating my homeserver (docker-compose based) to Silberblue. I have read a lot and have adjusted my compose file for rootless podman-compose.

I have used caddy-docker-proxy for years, it automatically generates the caddyfile based on compose labels and takes care of https/http reverse proxy. But to do that it needs access to podman.sock. It supports podman, I have seen old issues in the Github repository and people confirming it works.

  caddy-proxy:
    container_name: net-caddy
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    restart: always
    networks: 
      - net-caddy
    environment:
      - CADDY_INGRESS_NETWORKS=net-caddy
      - CADDY_DOCKER_NO_SCOPE=true
    volumes:
      - /run/user/1000/podman/podman.sock:/var/run/docker.sock
      - $DOCKERDIR/network/caddy-proxy/caddy_data:/data:Z
      - $DOCKERDIR/network/caddy-proxy/config:/config:Z
    ports:
      - 443:443
      - 80:80
    labels:
      caddy.email: $EMAIL

Unfortunately it has trouble connecting to the socket. I did check with podman info which clearly shows

{"level":"info","ts":1738271453.6760228,"logger":"docker-proxy","msg":"Running caddy proxy controller"}
{"level":"error","ts":1738271453.676158,"logger":"docker-proxy","msg":"Docker ping failed","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head \"http://%2Fvar%2Frun%2Fdocker.sock/_ping\": dial unix /var/run/docker.sock: connect: permission denied"}
Error: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied

  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock

Same result after systemctl enable --now podman.socket. The socket is running:

systemctl status podman.socket
● podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/system/podman.socket; enabled; preset: disabled)
     Active: active (listening) since Thu 2025-01-30 22:09:43 CET; 23min ago
 Invocation: 2a2ff8c48fea4973bd79421fd4136c09
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/podman/podman.sock (Stream)
     CGroup: /system.slice/podman.socket

Jan 30 22:09:43 obelix.o systemd[1]: Listening on podman.socket - Podman API Socket.

Also, the user does have access to the socket, if I use the right path:

❯ ls -al /run/podman/podman.sock
ls: cannot access '/run/podman/podman.sock': Permission denied

❯ ls -al /run/user/1000/podman/podman.sock
total 0
drwxr-xr-x. 2 asterix asterix 40 Jan 30 21:58 .
drwxr-xr-x. 3 asterix asterix 60 Jan 30 21:58 .

So I expect it to work… Caddy is my only container that needs the socket access hence I did not bump into this before. It is my most essential container as it handles https access (and even local domain http access within my home network) for all my services.

I must miss something obvious here to get it working, but I can’t find any documentation or topic about this. What am I missing?

2

The error shows “permission denied”, so you should verify the permissions.

In addition to the standard Unix mode bits, Fedora Linux has SELinux. I think sudo journalctl -t audit should show if SELinux is blocking access to anything.

3

Just for fun and learning I ran the command and it output 40k lines. It there anything one can grep to get a more concise output?

4

I often add something like --since=today or --since=-5min to get just the more recent logs. Something like -g podman might be another option in your case (you are looking for denials when accessing podman.sock – the name as seen on the host, not from within the container).

Edit: I notice there is some confusion in your earlier post about /run/podman/podman.sock versus /run/user/1000/podman/podman.sock. I think the former would be for a “system” service (systemctl status podman.sock) whereas the latter would be for a “user” service (systemctl --user status podman.sock). I’ve never used podman.sock, but I would guess which is the right one to use would depend on whether you are running your container rootful or rootless.

1 Like
5

Yes, as mentioned i use Podman rootless, so at least I got that correct :slight_smile:
I’ll try with :Z that should help in case its SELinux… hope that works.

6

OK I got 1 step further by adding :Z:

Now I get: podman logs -f --tail 20 caddy-proxy

{"level":"info","ts":1738407477.0551832,"logger":"docker-proxy","msg":"Running caddy proxy server"}
{"level":"info","ts":1738407477.055994,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1738407477.056199,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1738407477.0562088,"logger":"docker-proxy","msg":"Running caddy proxy controller"}
{"level":"error","ts":1738407477.0563962,"logger":"docker-proxy","msg":"Docker ping failed","error":"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"}
Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Strangely… when I do sudo ls -al /run/podman/podman.sock

srw-rw----. 1 root root 0 Jan 30 22:09 /run/podman/podman.sock

But when I do it for the user, it’s empty:

ls -al /run/user/1000/podman/podman.sock
total 0
drwxr-xr-x. 2 asterix asterix 40 Jan 30 21:58 .
drwxr-xr-x. 3 asterix asterix 60 Jan 30 21:58 ..

So when I check that service specifically:

systemctl --user start podman.socket
Job failed. See "journalctl -xe" for details.

So now let’s check status: systemctl --user status podman.socket

podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: disabled)
     Active: failed (Result: resources)
 Invocation: b207bb9023df45ebaaa079780d4aa586
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/1000/podman/podman.sock (Stream)

Feb 01 11:57:31 obelix.o systemd[1449]: podman.socket: Failed with result 'resources'.
Feb 01 11:57:31 obelix.o systemd[1449]: Failed to listen on podman.socket - Podman API Socket.
Feb 01 11:57:37 obelix.o systemd[1449]: podman.socket: Failed to create listening socket (/run/user/1000/podman/podman.sock): Address already in use
Feb 01 11:57:37 obelix.o systemd[1449]: podman.socket: Failed to listen on sockets: Address already in use
Feb 01 11:57:37 obelix.o systemd[1449]: podman.socket: Failed with result 'resources'.
Feb 01 11:57:37 obelix.o systemd[1449]: Failed to listen on podman.socket - Podman API Socket.
Feb 01 12:01:22 obelix.o systemd[1449]: podman.socket: Failed to create listening socket (/run/user/1000/podman/podman.sock): Address already in use
Feb 01 12:01:22 obelix.o systemd[1449]: podman.socket: Failed to listen on sockets: Address already in use
Feb 01 12:01:22 obelix.o systemd[1449]: podman.socket: Failed with result 'resources'.
Feb 01 12:01:22 obelix.o systemd[1449]: Failed to listen on podman.socket - Podman API Socket.

Address already in use ?
even after I do
systemctl stop podman.socket and systemctl disable podman.socket and then:
systemctl --user enable podman.socket and systemctl --user start podman.socket, same result.

Makes no sense to me… why is Podman so hard ?

7

I started over again, podman system migrate and podman system reset also stopped the --user podman.socket service, disabled it, then enabled and started it.

systemctl --user status podman.socket

● podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: disabled)
     Active: active (listening) since Sat 2025-02-01 14:09:04 CET; 13min ago
 Invocation: 49ed1168dec44588ac20f24d5cb8b0e5
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/1000/podman/podman.sock (Stream)
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/podman.socket

Feb 01 14:09:04 obelix.o systemd[1449]: Listening on podman.socket - Podman API Socket.

All good.

Then podman-compose up -d (after creating the network).
podman logs -f --tail 20 caddy-proxy

{"level":"info","ts":1738416080.1704485,"logger":"docker-proxy","msg":"Running caddy proxy server"}
{"level":"info","ts":1738416080.1711955,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1738416080.171304,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1738416080.1713104,"logger":"docker-proxy","msg":"Running caddy proxy controller"}
{"level":"error","ts":1738416080.1714833,"logger":"docker-proxy","msg":"Docker ping failed","error":"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"}
Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

with compose.yml

#
version: "2.4"
services:
##_______network SERVICES________
##
##_____________________ Caddy [network/web-proxy]
  caddy-proxy:
    container_name: caddy-proxy
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    restart: always
    networks: 
      - caddy-net
    environment:
      - CADDY_INGRESS_NETWORKS=caddy-net
      - CADDY_DOCKER_NO_SCOPE=true
    volumes:
      - /run/user/1000/podman/podman.sock:/var/run/docker.sock:Z
      - $DOCKERDIR/network/caddy-proxy/caddy_data:/data:Z
      - $DOCKERDIR/network/caddy-proxy/config:/config:Z
    ports:
      - 443:443
      - 80:80
    labels:
      caddy.email: $EMAIL

What am I missing here?

also tried upping the version of my compose.yml to 3.7 makes no difference.

8

That output doesn’t look right to me. It appears to be showing that podman.sock is a directory, but it should be a socket. Sockets should show s in the first column of the ls -al output:

$ ls -al /run | grep '^s'
srw-rw-rw-.  1 root           root              0 Feb  1 10:43 gssproxy.default.sock
srw-rw-rw-.  1 root           root              0 Feb  1 10:43 gssproxy.sock
9

Thanks. That is strange. Not sure how to fix that. Stop via systemctl and delete that folder?

10

I did:

podman down
podman system migrate
podman system disable
systemctl stop podman.socket
systemctl disable podman.socket

Then check that path again:

ls -al /run/user/1000/podman/podman.sock
ls: cannot access '/run/user/1000/podman/podman.sock': No such file or directory

Now I start podman again:

systemctl --user enable podman.socket
Created symlink '/var/home/asterix/.config/systemd/user/sockets.target.wants/podman.socket' → '/usr/lib/systemd/user/podman.socket'.

~/services 
❯ systemctl --user start podman.socket

~/services 
❯ systemctl --user start podman

Then check the path again:

ls -al /run/user/1000/podman/podman.sock
srw-rw----. 1 asterix asterix 0 Feb  1 18:47 /run/user/1000/podman/podman.sock

Looks good…

podman-compose up -d

But now I am back at the original issue… podman logs -f --tail 20 caddy-proxy

{"level":"info","ts":1738432293.5259368,"logger":"docker-proxy","msg":"Running caddy proxy server"}
{"level":"info","ts":1738432293.526501,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1738432293.5265949,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1738432293.5266,"logger":"docker-proxy","msg":"Running caddy proxy controller"}
{"level":"error","ts":1738432293.5267446,"logger":"docker-proxy","msg":"Docker ping failed","error":"permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head \"http://%2Fvar%2Frun%2Fdocker.sock/_ping\": dial unix /var/run/docker.sock: connect: permission denied"}
Error: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied

I am jumping between “cannot connect to socket” and “permission denied” each time I do the EXACT same actions…
I could do podman down, podman migrate podman reset, delete the config folder for my container, stop the podman.socket service and disable.
Then enable that service and start it again… run podman up and I would be back at “cannot connect” .
Then do it all again and get “permission denied”…

edit: I tried but still stuck with permission denied, also when I remove :Z

11

I think the permission denied problem you are seeing is due to SELinux. Are you seeing any messages in the logs about SELinux blocking access to your /run/user/1000/podman/podman.sock file?

When that path shows as a directory, I’m guessing it is because podman-compose created it and assumed it should be a directory. You probably need to make sure your user instance of podman.sock is running before you start your container.

12

Can podman open privileged ports for rootless users?
have you tried to use ports >1024

10443:443
  1080:80
1 Like
13

Alternatively, you could lower the start of the unprivileged port range by creating a .conf file under /etc/sysctl.d containing the following line:

net.ipv4.ip_unprivileged_port_start=80

Use sysctl --system to tell your system to rescan the config files and sysctl net.ipv4.ip_unprivileged_port_start to display the active settting.

14

I will check this and report back, I suppose I have to check journalctl right?

I have already done that before I started testing :slight_smile: Eventually after I get everything to work I want to use firewalld instead of lowering the privileged ports.

15

Yes, jounalctl. Or, the same auditd messages should also be duplicated in /var/log/audit/audit.log.

I’m not sure what you have in mind. I guess since firewalld runs as a privileged service, it could be configured to forward one of the lower-numbered ports to a high-numbered port that your container is running on. It seems a little inefficient to do that much port forwarding though.

16

I do see this repeating in /var/log/audit/audit.log

type=AVC msg=audit(1738575086.060:2161510): avc:  denied  { write } for  pid=3190345 comm="caddy" name="podman.sock" dev="tmpfs" ino=986797 scontext=system_u:system_r:container_t:s0:c424,c434 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=sock_file permissive=0
type=ANOM_PROMISCUOUS msg=audit(1738575086.195:2161511): dev=veth0 prom=0 old_prom=256 auid=1000 uid=1000 gid=1000 ses=1AUID="asterix" UID="asterix" GID="asterix"
type=NETFILTER_CFG msg=audit(1738575086.220:2161512): table=netavark:924112 family=1 entries=8 op=nft_unregister_rule pid=3190368 subj=unconfined_u:unconfined_r:iptables_t:s0 comm="nft"
type=ANOM_PROMISCUOUS msg=audit(1738575086.274:2161513): dev=veth0 prom=256 old_prom=0 auid=1000 uid=1000 gid=1000 ses=1AUID="asterix" UID="asterix" GID="asterix"
type=NETFILTER_CFG msg=audit(1738575086.288:2161514): table=netavark:924113 family=1 entries=10 op=nft_register_chain pid=3190374 subj=unconfined_u:unconfined_r:iptables_t:s0 comm="nft"
type=NETFILTER_CFG msg=audit(1738575086.309:2161515): table=netavark:924114 family=1 entries=8 op=nft_register_rule pid=3190376 subj=unconfined_u:unconfined_r:iptables_t:s0 comm="nft"

I suppose this indicates SELinux is indeed blocking access, even if I use:

...
    volumes:
      - /run/user/1000/podman/podman.sock:/var/run/docker.sock:Z
      - $DOCKERDIR/network/caddy-proxy/caddy_data:/data:Z
      - $DOCKERDIR/network/caddy-proxy/config:/config:Z

Is that not strange? I thought this is what :Z was for…

17

Hi, since you are running podman in rootless mode, you may want to start the socket as the unprivileged user like this:

systemctl --user enable --now podman.socket

This should at least solve the service permission failure.
If you started the socket by root, e.g using sudo, then you need to stop and disable the service running by root again, in order to not to get a port conflict.

Edit
I see, this already has been suggested.
If SELinux is in “enforcing” mode (check it by commando getenforce), you may want to test it by temporally disabling selinux. If it is in “permissive” mode, you can ignore it.

Looks like Gregory also already mentioned that the unprivileged User is not allowed to control privileged ports. So you really need to ensure that

net.ipv4.ip_unprivileged_port_start=80

is set.

18

Rather than disabling SELinux entirely, you could set permissive mode for containers. It is still a “big hammer” and not ideal, but at least it would be a little more targeted.

$ sudo semanage permissive -a container_t
1 Like
19

That is exactly how I do it: Podman-compose how to connect to socket? - #10 by zilexa

and the unprivileged port starts at 53 already:

cat /etc/sysctl.d/podman-privileged-ports.conf
# Lowering privileged ports to allow to run common docker containers via Podman
net.ipv4.ip_unprivileged_port_start=53

I am just trying to understand how this is supposed to work on clean Universal Blue… I thought the whole idea was that you can use podman-compose perfectly… but this seems like quite a blocker that no regular user can solve, without disabling SELinux?

This works:

$ sudo semanage permissive -a container_t

Still weird I have to disable SELinux to allow a container read-permissions to podman socket, while we have :Z for that?

20

There might be something special about that podman.sock file. I don’t think it is normal for that file to be mapped into the container like that. It looks like you are trying to give the container control over itself, which probably defeats a lot of the container security anyway.