Hi.
I’m still getting this error. I’m using Fedora Silverblue 32. Each time I do an rpm-ostree upgrade
and then I reboot, I get the same error:
$ toolbox enter
toolbox: failed to start container fedora-toolbox-32
With verbose:
$ toolbox -v enter
toolbox: running as real user ID 1000
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user juanje
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: running on a cgroups v2 host
toolbox: current Podman version is 2.0.2
toolbox: migration not needed: Podman version 2.0.2 is unchanged
toolbox: Fedora generational core is f32
toolbox: base image is fedora-toolbox:32
toolbox: container is fedora-toolbox-32
toolbox: checking if container fedora-toolbox-32 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container fedora-toolbox-32
toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-32
Error: unable to start container "7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873": setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error
toolbox: failed to start container fedora-toolbox-32
At this upgrade the package Podman changed to the 2.0 version, but at the previous upgrades the error was the same. At least the last two lines.
The container exists and was working just fine before the reboot:
$ toolbox list
IMAGE ID IMAGE NAME CREATED
dc930c1469f5 localhost/fedora-toolbox:32 3 weeks ago
CONTAINER ID CONTAINER NAME CREATED STATUS IMAGE NAME
7e628d2413b6 fedora-toolbox-32 1593148819 Created localhost/fedora-toolbox:32
I can create a new one and it’ll work, but my previous container won’t work, I’ve to remove it.
Here is the full error when I try to start the container with podman and set the log level to debug:
$ podman --log-level debug start fedora-toolbox-32
INFO[0000] podman filtering at log level debug
DEBU[0000] Called start.PersistentPreRunE(podman --log-level debug start fedora-toolbox-32)
DEBU[0000] Ignoring libpod.conf EventsLogger setting "/var/home/juanje/.config/containers/containers.conf". Use "journald" if you want to change this setting and remove libpod.conf files.
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{{[] [] containers-default-0.14.4 [] private enabled [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL CAP_MKNOD CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_CHROOT] [] [] [] [] [] true [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] false false false private k8s-file -1 slirp4netns false 2048 private /usr/share/containers/seccomp.json 65536k private host 65536} {true systemd [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ctrl-p,ctrl-q true /run/user/1000/libpod/tmp/events/events.log file [/usr/share/containers/oci/hooks.d] docker:// /pause k8s.gcr.io/pause:3.2 /usr/libexec/podman/catatonit shm false 2048 /usr/bin/crun map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] missing false [] [crun runc] [crun] [kata kata-runtime kata-qemu kata-fc] {false false false false false false} /etc/containers/policy.json false 3 /var/home/juanje/.local/share/containers/storage/libpod 10 /run/user/1000/libpod/tmp /var/home/juanje/.local/share/containers/storage/volumes} {[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman /etc/cni/net.d/}}
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/home/juanje/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/home/juanje/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /var/home/juanje/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /var/home/juanje/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] using runtime "/usr/bin/runc"
DEBU[0000] using runtime "/usr/bin/crun"
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] using runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 25
DEBU[0000] overlay: mount_data=lowerdir=/var/home/juanje/.local/share/containers/storage/overlay/l/LXKWYMJF43CUKNXMBQZRYXV74X:/var/home/juanje/.local/share/containers/storage/overlay/l/MNRBLEPXKBGAXA4E4CQZ32EMUI,upperdir=/var/home/juanje/.local/share/containers/storage/overlay/bbdbc7daaf9a85a31389fde56a2ce41d7d126a7f6a4d0493925294efdcf2494a/diff,workdir=/var/home/juanje/.local/share/containers/storage/overlay/bbdbc7daaf9a85a31389fde56a2ce41d7d126a7f6a4d0493925294efdcf2494a/work,context="system_u:object_r:container_file_t:s0:c353,c423"
DEBU[0000] mounted container "7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873" at "/var/home/juanje/.local/share/containers/storage/overlay/bbdbc7daaf9a85a31389fde56a2ce41d7d126a7f6a4d0493925294efdcf2494a/merged"
DEBU[0000] Created root filesystem for container 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 at /var/home/juanje/.local/share/containers/storage/overlay/bbdbc7daaf9a85a31389fde56a2ce41d7d126a7f6a4d0493925294efdcf2494a/merged
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU[0000] Setting CGroups for container 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 to user.slice:libpod:7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873
DEBU[0000] set root propagation to "rslave"
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Created OCI spec for container 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 at /var/home/juanje/.local/share/containers/storage/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -c 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 -u 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 -r /usr/bin/crun -b /var/home/juanje/.local/share/containers/storage/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata -p /run/user/1000/containers/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata/pidfile -n fedora-toolbox-32 --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket -s -l k8s-file:/var/home/juanje/.local/share/containers/storage/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/juanje/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg /usr/bin/crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
DEBU[0000] Received: -1
DEBU[0000] Cleaning up container 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] unmounted container "7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873"
Error: unable to start container "7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873": setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error
For me, the most suspicious errors are:
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
and
Error: unable to start container "7e62...873": setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error
I tried to find something about those errors, but no luck…
Here is my current OS layer (but it failed at all the previous versiones/upgrades):
$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/32/x86_64/silverblue
Version: 32.20200710.0 (2020-07-10T00:44:21Z)
BaseCommit: 6aea1d9096f14f30a5edf97ce9b4c25b8f978696834d522747556689c5f50e86
GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
LayeredPackages: acpi exfat-utils fedora-workstation-repositories ffmpeg fuse-exfat gnome-boxes gstreamer1-plugin-openh264 gstreamer1-plugins-ugly-free guake htop libvirt mozilla-openh264 nmap-ncat qemu-kvm
setroubleshoot-server systemd-container vagrant vagrant-cachier vagrant-libvirt vim virt-install
LocalPackages: rpmfusion-nonfree-release-32-1.noarch rpmfusion-free-release-32-1.noarch
I hope this info helps to find the issue, I’ll keep looking. If you need more info, just ask.
Thanks.