Toolbox not working, again

Error: unable to start container "fedora-toolbox-31": container '50f4c0b124eebaf7b01c68ce018bd1dde11b213c9b28193477269b33e7403844' already exists: OCI runtime error

A new day begins and toolbox breaks after today’s update.

I am not sure if I can recommend Silverblue anymore, this release has been absolute pain.

EDIT: I am on Silverblue 31

Version: 31.20191204.0 (2019-12-04T00:37:38Z)
BaseCommit: 29760a7e84c23d06eaa98d0db35ba1f908dae2a563e55f8d9213a76fa1220045
2 Likes

At this point I think there is a serious issue on how these packages are getting pushed with almost zero consideration on toolbox.

3 Likes

Hello @deathwish,

Was this from creating a container or entering one already created? I have both coretoolbox and toolbox currently in use on my F31SB and there are no issues for me.

Yep, broken for me too. I get a different error. This is opening a previously created container. This is just getting ridiculous.

toolbox: failed to create /run/.toolboxenv in container fedora-toolbox-31

FWIW: I’m currently using 31.20191204.0 too (same basecommit too), but toolbox works here for me, both for running existing containers and creating (and running) new ones.

But, yeah, agreed: We’ve all been hitting too many issues with toolbox on Silverblue lately and it does need better testing before being pushed out to all of us. (Most of these were during the beta, but that random crun-based issue after an official release stung.)

What version of Silverblue did you update from? Perhaps there’s a configuration issue related to old versions of ~/.config/containers/libpod.conf (or your containers in ~/.local/share/containers/) and new.

@deathwish, @mmorrell2016: Can you create new containers and run them at least?

toolbox create -c test
toolbox enter -c test
1 Like

I see a very similar reason with just plain podman, without toolbox, on current Fedora CoreOS (see https://github.com/cockpit-project/bots/pull/305). I just reported that as 1780161 – podman restart fails: container '<id>' already exists: OCI runtime error

I get the error both when running an old toolbox and creating a new one. After a rollback to

Version: 31.20191130.0 (2019-11-30T00:35:48Z)
BaseCommit: 3acfe02779b60bf385950fac7b0bcf1de58b10bedd263e39dd2d70d6448903b5

toolbox is working again. In discord one user at least reported that it was working fine and I didn’t see any package that could break anything during the update (talking from ignorance).

I feel the same way about Silverblue all the time, but usually the problems work themselves out. What keeps me around is that I believe in the technology behind it and I knew when I downloaded Silverblue that I was getting into something that was not a finished product but a work in progress.

Just to be clear, I been on silverblue for some time now and it has only become an issue with f31 as this is happening almost like once a month.

Solved in 31.20191205.1

1 Like

I am still getting same error with 31.20191207.0. Here is what I get when I try to toolbox -v enter:

[mike@mustang ~]$ toolbox enter
toolbox: failed to create /run/.toolboxenv in container fedora-toolbox-31
[mike@mustang ~]$ toolbox -v enter
toolbox: running as real user ID 1000
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user mike
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: running on a cgroups v2 host
toolbox: current Podman version is 1.6.2
toolbox: migration not needed: Podman version 1.6.2 is unchanged
toolbox: Fedora generational core is f31
toolbox: base image is fedora-toolbox:31
toolbox: container is fedora-toolbox-31
toolbox: checking if container fedora-toolbox-31 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container fedora-toolbox-31
toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-31
toolbox: inspecting entry point of container fedora-toolbox-31
toolbox: entry point of container fedora-toolbox-31 is toolbox
toolbox: waiting for container fedora-toolbox-31 to finish initializing
Error: unable to find user root: no matching entries in passwd file
toolbox: failed to create /run/.toolboxenv in container fedora-toolbox-31

Here’s some more info after doing some troubleshooting. If I remove my container with toolbox rm container-id and then do a toolbox enter and say yes to create a new container, that works. When I exit the toolbox and do a systemctl restart, after the reboot I get the same error…

[mike@mustang ~]$ toolbox enter
toolbox: failed to create /run/.toolboxenv in container fedora-toolbox-31
[mike@mustang ~]$ toolbox list
IMAGE ID      IMAGE NAME                                        CREATED
c7b500cb5741  localhost/fedora-toolbox-mike:30                  7 months ago
b8e18bacde99  registry.fedoraproject.org/f30/fedora-toolbox:30  8 months ago
3a3fb0a29265  registry.fedoraproject.org/f31/fedora-toolbox:31  2 months ago

CONTAINER ID  CONTAINER NAME     CREATED        STATUS             IMAGE NAME
54efa9849dbe  fedora-toolbox-31  3 minutes ago  Up 27 seconds ago  registry.fedoraproject.org/f31/fedora-toolbox:31

Seems like it might be a permission issue maybe?

Seems to be fixed since the 31.20191208.0 update. Not sure what changed but glad it is fixed again.

Same issue here. Did an rpm-ostree upgrade and toolbox broke, giving the same error cited above:

toolbox: failed to create /run/.toolboxenv in container fedora-toolbox-31

Rolling back the deployment fixed it. For reference:

$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/31/x86_64/silverblue
                   Version: 31.20191219.0 (2019-12-19T00:41:28Z)
                    Commit: 675ab14ffbae92afce5424c756b281fec5614208400291215e39599748cdaaf9
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

  ostree://fedora:fedora/31/x86_64/silverblue
                   Version: 31.20191221.0 (2019-12-21T00:41:41Z)
                    Commit: ec1cbfc67d9534ef6dadf6d243a3fb2996c173452a5f6bde02c6fff11a056b4a
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

Looking at the toolbox script, it looks like this error happens when it tries create the .toolboxenv hidden file in the containers /run directory, but the file already exists so it errors out. The file may exist already because another shell already has entered the toolbox container, or it was not cleaned up when the toolbox was exited. Maybe this happens when toolbox is entered and a system reboot happens. To fix this, I think you need to shell into the running container and remove the /run/.toolboxenv file. Then exit the container shell and try toolbox enter again.

I’ve been hit by this same bug (ie. if I remove all containers, create new then works, upon reboot stops working). How do I “shell into the container to manually rm /run/.toolboxenv”? I can’t seem to execute any “podman exec” commands because it complains about “unable to find user root”.

Yeah, I would think

podman run -ti image-id /bin/bash

would connect an interactive terminal to the container. Not sure why you get the “unable to find user root” error. I did get the same error as you recently and I removed all my containers and images and also deleted my local container storage directory ~/.local/share/containers/storage and started fresh. So far, I have not had the problem reoccur. Currently running version 31.20191221.0. I have my base on ext4 partition and my home directory is btrfs mounted filesystem. No sure if having my container storage on btrfs filesystem makes a difference, but it shouldn’t.

Got that error after doing a rpm-ostree upgrade. Did a rollback and toolbox would work again!!

Filed an issue: toolbox: failed to create /run/.toolboxenv in container fedora-toolbox-31 · Issue #359 · containers/toolbox · GitHub

rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/31/x86_64/silverblue
Version: 31.20200107.0 (2020-01-07T01:34:27Z)
BaseCommit: 3ea3a112a5a5e89559435fecb9377549edbbe93eb44a3888deadaf2c4f995d37
GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
LayeredPackages: fedora-workstation-repositories filelight google-chrome mupdf virt-manager

ostree://fedora:fedora/31/x86_64/silverblue
Version: 31.20200108.0 (2020-01-08T11:26:07Z)
BaseCommit: 13625cd496d7cccc8729af79df460d8e1ca75a446ce0da0537c86ec4cc67302c
GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
LayeredPackages: fedora-workstation-repositories filelight google-chrome mupdf virt-manager

Hi.
I’m still getting this error. I’m using Fedora Silverblue 32. Each time I do an rpm-ostree upgrade and then I reboot, I get the same error:

$ toolbox enter
toolbox: failed to start container fedora-toolbox-32

With verbose:

$ toolbox -v enter
toolbox: running as real user ID 1000
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user juanje
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: running on a cgroups v2 host
toolbox: current Podman version is 2.0.2
toolbox: migration not needed: Podman version 2.0.2 is unchanged
toolbox: Fedora generational core is f32
toolbox: base image is fedora-toolbox:32
toolbox: container is fedora-toolbox-32
toolbox: checking if container fedora-toolbox-32 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container fedora-toolbox-32
toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-32
Error: unable to start container "7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873": setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error
toolbox: failed to start container fedora-toolbox-32

At this upgrade the package Podman changed to the 2.0 version, but at the previous upgrades the error was the same. At least the last two lines.
The container exists and was working just fine before the reboot:

$ toolbox list
IMAGE ID                                              IMAGE NAME  CREATED
dc930c1469f5 localhost/fedora-toolbox:32 3 weeks ago              

CONTAINER ID  CONTAINER NAME     CREATED     STATUS   IMAGE NAME
7e628d2413b6  fedora-toolbox-32  1593148819  Created  localhost/fedora-toolbox:32

I can create a new one and it’ll work, but my previous container won’t work, I’ve to remove it.

Here is the full error when I try to start the container with podman and set the log level to debug:

$ podman --log-level debug start fedora-toolbox-32
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called start.PersistentPreRunE(podman --log-level debug start fedora-toolbox-32) 
DEBU[0000] Ignoring libpod.conf EventsLogger setting "/var/home/juanje/.config/containers/containers.conf". Use "journald" if you want to change this setting and remove libpod.conf files. 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{{[] [] containers-default-0.14.4 [] private enabled [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL CAP_MKNOD CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_CHROOT] [] []  [] [] [] true [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] false false false  private k8s-file -1 slirp4netns false 2048 private /usr/share/containers/seccomp.json 65536k private host 65536} {true systemd [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ctrl-p,ctrl-q true /run/user/1000/libpod/tmp/events/events.log file [/usr/share/containers/oci/hooks.d] docker:// /pause k8s.gcr.io/pause:3.2 /usr/libexec/podman/catatonit shm   false 2048 /usr/bin/crun map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] missing false   [] [crun runc] [crun] [kata kata-runtime kata-qemu kata-fc] {false false false false false false} /etc/containers/policy.json false 3 /var/home/juanje/.local/share/containers/storage/libpod 10 /run/user/1000/libpod/tmp /var/home/juanje/.local/share/containers/storage/volumes} {[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman /etc/cni/net.d/}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/juanje/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/juanje/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/juanje/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/juanje/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] Setting parallel job count to 25             
DEBU[0000] overlay: mount_data=lowerdir=/var/home/juanje/.local/share/containers/storage/overlay/l/LXKWYMJF43CUKNXMBQZRYXV74X:/var/home/juanje/.local/share/containers/storage/overlay/l/MNRBLEPXKBGAXA4E4CQZ32EMUI,upperdir=/var/home/juanje/.local/share/containers/storage/overlay/bbdbc7daaf9a85a31389fde56a2ce41d7d126a7f6a4d0493925294efdcf2494a/diff,workdir=/var/home/juanje/.local/share/containers/storage/overlay/bbdbc7daaf9a85a31389fde56a2ce41d7d126a7f6a4d0493925294efdcf2494a/work,context="system_u:object_r:container_file_t:s0:c353,c423" 
DEBU[0000] mounted container "7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873" at "/var/home/juanje/.local/share/containers/storage/overlay/bbdbc7daaf9a85a31389fde56a2ce41d7d126a7f6a4d0493925294efdcf2494a/merged" 
DEBU[0000] Created root filesystem for container 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 at /var/home/juanje/.local/share/containers/storage/overlay/bbdbc7daaf9a85a31389fde56a2ce41d7d126a7f6a4d0493925294efdcf2494a/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 to user.slice:libpod:7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 at /var/home/juanje/.local/share/containers/storage/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 -u 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 -r /usr/bin/crun -b /var/home/juanje/.local/share/containers/storage/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata -p /run/user/1000/containers/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata/pidfile -n fedora-toolbox-32 --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket -s -l k8s-file:/var/home/juanje/.local/share/containers/storage/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/juanje/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg /usr/bin/crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container 7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873" 
Error: unable to start container "7e628d2413b6875d5ff30b64dc6121d4007280cce62bf79951bc0845ccacd873": setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error

For me, the most suspicious errors are:

[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

and

Error: unable to start container "7e62...873": setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error

I tried to find something about those errors, but no luck…

Here is my current OS layer (but it failed at all the previous versiones/upgrades):

$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/32/x86_64/silverblue
                   Version: 32.20200710.0 (2020-07-10T00:44:21Z)
                BaseCommit: 6aea1d9096f14f30a5edf97ce9b4c25b8f978696834d522747556689c5f50e86
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
           LayeredPackages: acpi exfat-utils fedora-workstation-repositories ffmpeg fuse-exfat gnome-boxes gstreamer1-plugin-openh264 gstreamer1-plugins-ugly-free guake htop libvirt mozilla-openh264 nmap-ncat qemu-kvm
                            setroubleshoot-server systemd-container vagrant vagrant-cachier vagrant-libvirt vim virt-install
             LocalPackages: rpmfusion-nonfree-release-32-1.noarch rpmfusion-free-release-32-1.noarch

I hope this info helps to find the issue, I’ll keep looking. If you need more info, just ask.
Thanks.

I try to export the container and create a new one with Podman, but nothing.
Also, I checked with podman inspect the old toolbox container (the one from before the reboot) and the new one (the one I created after the reboot) and there are some interesting differences:

Old:

Env: "container=oci"

New:

Env: "container=podman"

Old:

"Ulimits": [
    {
        "Name": "RLIMIT_NOFILE",
        "Soft": 524288, 
        "Hard": 524288
    },
    {
        "Name": "RLIMIT_NPROC",
        "Soft": 62530,
        "Hard": 62530
    }
],

New:

"Ulimits": [],

Old:

"IpcMode": "host",

New:

"IpcMode": "private",

I didn’t change any of these options by hand, I just created and used a container with Toolbox.

But maybe those are just changes from the last packages. Anyway, it would be nice if the toolbox’s containers would work after each upgrade and reboot.

Any ideas?
Thanks.