Fedora 39 fail to start podman.service as user

Ask Fedora
, , ,
1

Problem

I’m using Fedora server 39. As my user I’m trying to start podman.service.
$ systemctl --user start podman.service
In journal I have this error:
podman[59361]: time=“2024-03-19T19:42:59+01:00” level=info msg=“/usr/bin/podman filtering at log level info”
podman[59361]: Error: creating runtime static files directory “/containers/storage/libpod”: mkdir /containers: permission denied
systemd[41121]: podman.service: Main process exited, code=exited, status=125/n/a
But if I use the command:
$ /usr/bin/podman --log-level=info system service
I have no error.
If I run any the container with podman run it starts without problems.
If I try to start the container with
$ systemctl --user start mysql.service
I run into the same error.

I have created .config/containers/storage.conf file:
[storage]
driver = “overlay”
runroot = “${XDG_RUNTIME_DIR}/containers”
graphroot = “${XDG_DATA_HOME}/containers/storage”
rootless_storage_path = “${XDG_DATA_HOME}/containers/storage”
[storage.options]
additionalimagestores =
pull_options = {enable_partial_images = “true”, use_hard_links = “false”, ostree_repos=“”}
[storage.options.overlay]
mount_program = “/usr/bin/fuse-overlayfs”
mountopt = “nodev,metacopy=on”
[storage.options.thinpool]

and I moved the default /var/lib/containers/ to different location.
$ podman info
host:
arch: amd64
buildahVersion: 1.33.5
cgroupControllers:

  • cpu
  • memory
  • pids
    cgroupManager: systemd
    cgroupVersion: v2
    conmon:
    package: conmon-2.1.10-1.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: ’
    cpuUtilization:
    idlePercent: 99.93
    systemPercent: 0.05
    userPercent: 0.02
    cpus: 16
    databaseBackend: sqlite
    distribution:
    distribution: fedora
    variant: server
    version: “39”
    eventLogger: journald
    freeLocks: 2045
    hostname: fedora
    idMappings:
    gidmap:
    • container_id: 0
      host_id: 1000
      size: 1
    • container_id: 1
      host_id: 524288
      size: 65536
      uidmap:
    • container_id: 0
      host_id: 1000
      size: 1
    • container_id: 1
      host_id: 524288
      size: 65536
      kernel: 6.7.9-200.fc39.x86_64
      linkmode: dynamic
      logDriver: journald
      memFree: 30987816960
      memTotal: 33015783424
      networkBackend: netavark
      networkBackendInfo:
      backend: netavark
      dns:
      package: aardvark-dns-1.10.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
      package: netavark-1.10.3-1.fc39.x86_64
      path: /usr/libexec/podman/netavark
      version: netavark 1.10.3
      ociRuntime:
      name: crun
      package: crun-1.14.4-1.fc39.x86_64
      path: /usr/bin/crun
      version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
      os: linux
      pasta:
      executable: /usr/bin/pasta
      package: passt-0^20240220.g1e6f92b-1.fc39.x86_64
      version: |
      pasta 0^20240220.g1e6f92b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
      https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      remoteSocket:
      exists: false
      path: /run/user/1000/podman/podman.sock
      security:
      apparmorEnabled: false
      capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
      rootless: true
      seccompEnabled: true
      seccompProfilePath: /usr/share/containers/seccomp.json
      selinuxEnabled: true
      serviceIsRemote: false
      slirp4netns:
      executable: /usr/bin/slirp4netns
      package: slirp4netns-1.2.2-1.fc39.x86_64
      version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
      swapFree: 8589930496
      swapTotal: 8589930496
      uptime: 9h 51m 32.00s (Approximately 0.38 days)
      variant: “”
      plugins:
      authorization: null
      log:
  • k8s-file
  • none
  • passthrough
  • journald
    network:
  • bridge
  • macvlan
  • ipvlan
    volume:
  • local
    registries:
    search:
  • registry.fedoraproject.org
  • registry.access.redhat.com
  • docker.io
  • quay.io
    store:
    configFile: /home/cristian/.config/containers/storage.conf
    containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
    graphDriverName: overlay
    graphOptions:
    overlay.mount_program:
    Executable: /usr/bin/fuse-overlayfs
    Package: fuse-overlayfs-1.13-1.fc39.x86_64
    Version: |-
    fusermount3 version: 3.16.1
    fuse-overlayfs: version 1.13-dev
    FUSE library version 3.16.1
    using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,metacopy=on
    graphRoot: /raid1/containers/storage
    graphRootAllocated: 999579512832
    graphRootUsed: 69182865408
    graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: “false”
    Supports d_type: “true”
    Supports shifting: “true”
    Supports volatile: “true”
    Using metacopy: “false”
    imageCopyTmpDir: /var/tmp
    imageStore:
    number: 2
    runRoot: /run/user/1000/containers
    transientStore: false
    volumePath: /raid1/containers/storage/volumes
    version:
    APIVersion: 4.9.3
    Built: 1708357294
    BuiltTime: Mon Feb 19 16:41:34 2024
    GitCommit: “”
    GoVersion: go1.21.7
    Os: linux
    OsArch: linux/amd64
    Version: 4.9.3

Cause

Not yet known.

Related Issues

Workarounds

Not yet known.

2

It is a system service and not designed to run as a user service as
it needs privs to do its job as far as I know.

3

From Proposed Common Issues to Ask Fedora

4

Maybe, but if I try to start it as root it works. Also from cockpit → Podman you can start it as root but if I try to start it as user it fails with the same error message.
Even if I don’t try to start it by myself when I start the container with systemctl I get the same error message.
I don’t know where it tries to create /containers directory.

5

It states that it is in ${XDG_RUNTIME_DIR}/containers so that should be /var/run/user/1000/containers which is linked to /run/user/1000/containers.

6

Thank you, you pointed me in the right direction.
I solve it like this:

  1. remove containers directory in /run/user/1000
  2. I commented runroot line in .config/containers/storage.conf file from $HOME
  3. $ podman system migrate
  4. $ podman system reset
    and everything worked fine.

I followed the steps from here podman/docs/tutorials/rootless_tutorial.md at main · containers/podman · GitHub and I added runroot field to storage.conf.
But what is strange that even runroot pointed to same directory I was getting the error permission denied.

3 Likes