Error pulling from a Fedora's container registry in F31 beta

TL;DR if you hit a TLS error when using podman or toolbox to pull from registry.fedoraproject.org set the GODEBUG=tls13=0 environment variable.

There is a known issue where pulling from registry.fedoraproject.org is not working perfectly in Fedora 31+. This is because of a few reasons:

1 - golang in f31+ enabled TLS 1.3
2 - There was a bug in openssl that caused an error when a TLS 1.3 client connected to our services hosted by Fedora infra. Maybe it is they way they are configured that surfaced the bug, I don’t know.
3 - The bug in openssl was fixed we think but Fedora infra still needs time to test/deploy the change.

3 Likes

Thank you!

Is a known issue that user containers are broken under podman when turning off cgroupsv2 (needed to run Docker)?

Thanks @dustymabe. But after the container creation, It can’t start the container:

toolbox -v enter
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user thiago
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.5.1-dev is unchanged
toolbox: Fedora generational core is f31
toolbox: base image is fedora-toolbox:31
toolbox: container is fedora-toolbox-31
toolbox: checking if container fedora-toolbox-31 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container fedora-toolbox-31
toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-31
Error: unable to start container "fedora-toolbox-31": time="2019-09-14T19:06:27-03:00" level=warning msg="signal: killed"
time="2019-09-14T19:06:27-03:00" level=warning msg="no such directory for freezer.state"
time="2019-09-14T19:06:27-03:00" level=warning msg="no such directory for freezer.state"
time="2019-09-14T19:06:27-03:00" level=error msg="container_linux.go:346: starting container process caused \"process_linux.go:297: applying cgroup configuration for process caused \\\"mountpoint for cgroup not found\\\"\"\n"
container_linux.go:346: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"mountpoint for cgroup not found\"": OCI runtime error
toolbox: failed to start container fedora-toolbox-31
1 Like

I would say it’s probably not a known issue. Maybe report a bug with your steps and the maintainers of podman can find out? If you’re trying to run both Docker and podman on the same system I’m not sure if that’s a supported configuration.

I’m not running F31SB yet but a cloud VM I have spun up seems to be having trouble with toolbox too:

[vagrant@vanilla-f31-beta ~]$ GODEBUG=tls13=0 toolbox -v enter
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user vagrant
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: Fedora generational core is f31
toolbox: base image is fedora-toolbox:31
toolbox: container is fedora-toolbox-31
toolbox: checking if container fedora-toolbox-31 exists
WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument 
toolbox: container fedora-toolbox-31 not found
toolbox: found 0 containers
No toolbox containers found. Create now? [y/N] y
toolbox: Fedora generational core is f31
toolbox: base image is fedora-toolbox:31
toolbox: container is fedora-toolbox-31
toolbox: checking value /run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket
toolbox: parsing value /run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket
toolbox: checking if 'podman create' supports --dns=none and --no-hosts
toolbox: 'podman create' supports --dns=none and --no-hosts
toolbox: looking for image localhost/fedora-toolbox:31
toolbox: looking for image registry.fedoraproject.org/f31/fedora-toolbox:31
Image required to create toolbox container.
Download registry.fedoraproject.org/f31/fedora-toolbox:31 (500MB)? [y/N]: y
toolbox: pulling image registry.fedoraproject.org/f31/fedora-toolbox:31
Trying to pull registry.fedoraproject.org/f31/fedora-toolbox:31...
Getting image source signatures
Copying blob 333e25f0b57f done
Copying blob 4861f19bcaa5 done
Copying config 46131ebf80 done
Writing manifest to image destination
Storing signatures
toolbox: base image fedora-toolbox:31 resolved to registry.fedoraproject.org/f31/fedora-toolbox:31
toolbox: checking if container fedora-toolbox-31 already exists
toolbox: /home/vagrant canonicalized to /home/vagrant
toolbox: checking if /home is a symbolic link to /var/home
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: creating container fedora-toolbox-31
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container fedora-toolbox-31
toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-31
Error: unable to start container "fedora-toolbox-31": setrlimit (RLIM_MEMLOCK): Operation not permitted: OCI runtime error
toolbox: failed to start container fedora-toolbox-31

Podman by iself seems fine:

[vagrant@vanilla-f31-beta ~]$ rpm -q podman
podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64
[vagrant@vanilla-f31-beta ~]$ GODEBUG=tls13=0 podman run -it --rm registry.fedoraproject.org/fedora:30
WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument 
Trying to pull registry.fedoraproject.org/fedora:30...
Getting image source signatures
Copying blob ed60cb1abc2e done
Copying config 02781e9f50 done
Writing manifest to image destination
Storing signatures
[root@2514e0314d67 /]# 
[root@2514e0314d67 /]# echo "hi from inside container" 
hi from inside container
[root@2514e0314d67 /]# 
[root@2514e0314d67 /]# exit
exit

Maybe @rishi know’s what is going on?

I think your issue is related to this: https://github.com/containers/libpod/issues/3976

In any case, after fixing the issue with cgroups you will still probably experience the following: https://github.com/containers/libpod/issues/4024

Yes, the initial error log shown by @brogos seems to be related to using runc as the runtime with Cgroups v2, instead of crun. It could have been because of using an old Podman build that’s setup to use runc (eg., https://github.com/debarshiray/toolbox/issues/246) or having runc in `~/.config/containers/libpod.conf (as @returntrip pointed out).

1 Like

@dustymabe so the error is:

Were you using toolbox(1) from Git or from one of the released RPMs? Was this a container that you already had, potentially created with Cgroups v1? Or was it freshly created?

Does --ulimit host work for you? eg., podman run -it --ulimit host --rm ....

It was the toolbox-0.0.13-1.fc31.noarch rpm. This was a fresh f31 vagrant box with just installed toolbox.

The --ulimit does seem to work:

[vagrant@vanilla-f31-beta ~]$ GODEBUG=tls13=0 podman run -it --ulimit host --rm  registry.fedoraproject.org/fedora:30
Trying to pull registry.fedoraproject.org/fedora:30...
Getting image source signatures
Copying blob ed60cb1abc2e done
Copying config 02781e9f50 done
Writing manifest to image destination
Storing signatures

Ok. I wonder if this is a difference between podman run, and a sequence of podman create and podman start.

Can the error be related to https://fedoraproject.org/wiki/Changes/CGroupsV2#Scope ?