#1

Do you know why I get the error message Error: executable file ls not found in $PATH: No such file or directory: OCI runtime command not found error? The error message goes away when I bind-mount an empty directory over /var/lib/containers by adding -v ./emptydir:/var/lib/containers:rw

[core@fedora ~]$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/next
                   Version: 33.20201117.1.0 (2020-11-18T16:30:21Z)
                    Commit: 4769149819c16b7d60c930c2f3fe8a8161de82549f9e24b8cc3ae651718a29e0
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31

  ostree://fedora:fedora/x86_64/coreos/next
                   Version: 33.20201104.1.0 (2020-11-05T14:20:13Z)
                    Commit: f7bff56c9710b5fedb22d50651bcd875856bb83510f7115e10d21167bd9a3b75
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
[core@fedora ~]$ podman run -v /dev/fuse:/dev/fuse:rw --rm --ulimit host --privileged --rm quay.io/podman/stable podman run --user 0 docker.io/library/alpine:3.12.1 ls
Trying to pull docker.io/library/alpine:3.12.1...
Getting image source signatures
Copying blob sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964
Copying config sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0
Writing manifest to image destination
Storing signatures
Error: executable file `ls` not found in $PATH: No such file or directory: OCI runtime command not found error
[core@fedora ~]$ mkdir emptydir
[core@fedora ~]$ podman run -v ./emptydir:/var/lib/containers:rw -v /dev/fuse:/dev/fuse:rw --rm --ulimit host --privileged --rm quay.io/podman/stable podman run --user 0 docker.io/library/alpine:3.12.1 ls
Trying to pull docker.io/library/alpine:3.12.1...
Getting image source signatures
Copying blob sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964
Copying config sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0
Writing manifest to image destination
Storing signatures
bin
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var
[core@fedora ~]$

Some background information: I am trying to run Podman in Podman which is needed for a software project that I’m writing (https://github.com/eriksjolund/slurm-container-cluster) . I found a Github comment that mentioned how to do it. (I shared that information in a Stackoverflow answer)

Use buildah in toolbox
#2

there is some code in Podman that detects whether the a container is already mounted (checking whether the destination path is overlay) that gets confused when the Podman storage is already on overlay. This happens when running Podman in Podman, as /var/lib/containers is on overlay as the entire container rootfs.
The solution is to avoid the overlay on overlay situation. One of the two Podman must be using a different driver (e.g. vfs), or a better solution as you’ve already found out is to mount a volume at /var/lib/containers, or try using fuse-overlayfs.

Podman could detect this situation and give a better error message, but it won’t solve the root cause that overlay on overlay is not supported by the kernel.

1 Like
#3

Thanks for clarifying the mystery! I tested it. Indeed adding --storage-driver vfs to the right-most podman makes it work. Here is an example of that (-v /dev/fuse:/dev/fuse:rw was also removed)

[core@fedora ~]$ podman run --rm --ulimit host --privileged --rm quay.io/podman/stable podman --storage-driver vfs run --user 0 docker.io/library/alpine:3.12.1 ls | head -5
Trying to pull docker.io/library/alpine:3.12.1...
Getting image source signatures
Copying blob sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964
Copying config sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0
Writing manifest to image destination
Storing signatures
bin
dev
etc
home
lib
[core@fedora ~]$

I thought I was already using fuse-overlayfs. I don’t quite follow.

Anyway, in the project I’m writing, I now use --volume to bind-mount a directory over /var/lib/containers. It works! I don’t see the error message anymore.