Docker plugins on F41 not working?

Ask Fedora
,
1

Hi, after upgrading to F41 I tried to build a docker image with the same command I always use and got a weird error.

$ docker build -t localhost/test-base:latest -f Dockerfile.baseimage .
unknown shorthand flag: 't' in -t

I thought, F41 might have come with a new docker CLI and the option changed. So I checked man docker-build but the -t option was still there.

Next I tried to run the build without arguments and I found the culprit:

$ docker build
docker: 'buildx' is not a docker command.
See 'docker --help'

Do I have the buildx plugin?

$ rpm -qa | grep docker
docker-cli-27.3.1-2.fc41.x86_64
docker-buildx-0.18.0-1.fc41.x86_64

Yes. So where is the plugin installed then?

$ rpm -ql docker-buildx | grep docker-buildx$
/usr/libexec/docker/cli-plugins/docker-buildx
/usr/share/doc/docker-buildx
/usr/share/licenses/docker-buildx

Quickly checking the docs at Install the Compose plugin | Docker Docs and Docker Plugin API | Docker Docs they reference the following plugin directories:

  • ~/.docker/cli-plugins
  • /usr/local/lib/docker/cli-plugins
  • /usr/lib/docker/cli-plugins
  • /etc/docker/cli-plugins

But there is no /usr/libexec/docker/cli-plugins mentioned anywhere.

So I tried the following:

$ mkdir -p ~/.docker/cli-plugins
$ cd ~/.docker/cli-plugins
$ ln -s /usr/libexec/docker/cli-plugins/docker-buildx .

Then tried the build command again:

$ docker build       
ERROR: "docker buildx build" requires exactly 1 argument.
See 'docker buildx build --help'.

Usage:  docker buildx build [OPTIONS] PATH | URL | -

Start a build

It worked! And now also my original build works with as before.

Is this a change in the plugin path accidental or did I miss to make some configuration change between F40 and F41?

I’ve also noticed that docker plugin commands don’t work, but I don’t know if it was working before on F40, never used that before.

$ docker plugin ls
Error response from daemon: Path /v1.41/plugins is not supported
3

And docker info shows this:

docker info
Client:
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.18.0
    Path:     /var/home/mihalyr/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose

I think, I installed compose manually on that path, I don’t have any RPM for that.

4

Oh, and I have also this

$ env | grep -i docker
DOCKER_SOCK=/run/user/3000/podman/podman.sock
DOCKER_HOST=unix:///run/user/3000/podman/podman.sock

I mostly use Podman, but I had to install docker as some images I have no control over did not work with Podman.

$ rpm -qa | grep podman
podman-5.2.5-1.fc41.x86_64
podman-compose-1.2.0-2.fc41.noarch

And there is also

$rpm -qa | grep moby  
moby-filesystem-27.3.1-2.fc41.noarch
moby-engine-27.3.1-2.fc41.x86_64
moby-engine-nano-27.3.1-2.fc41.noarch
$ rpm -ql moby-filesystem-27.3.1-2.fc41.noarch
/usr/libexec/docker
/usr/libexec/docker/cli-plugins
5

Interestingly the buildx repository has this in the README

On Unix environments:

  • /usr/local/lib/docker/cli-plugins OR /usr/local/libexec/docker/cli-plugins
  • /usr/lib/docker/cli-plugins OR /usr/libexec/docker/cli-plugins

Another mention of libexec path here CLI Plugins Design · Issue #1534 · docker/cli · GitHub

And it seems the sources also contain the libexec path: cli/cli-plugins/manager/manager_unix.go at 6c76914532de7a39a202b3ef22519da319558560 · docker/cli · GitHub

But for some reason my docker won’t find the thing under libexec.

6

A workaround without symlink is to configure the extra dir in ~/.docker/config.json

{
  "auths": {},
  "cliPluginsExtraDirs": [
    "/usr/libexec/docker/cli-plugins"
  ]
}
7

Hm, very strange, /usr/local/libexec/docker/cli-plugins path also works, only /usr/libexec/docker/cli-plugins seems to be not working, the one where the plugin is installed from the RPM…

$ docker info
Client:
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.18.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
8

I don’t really know what happened, I was poking around and suddenly this started to work with the /usr/libexec/docker/cli-plugins path as well…

$ docker info
Client:
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.18.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
9

Found it! Permissions!

Here is what happens:

$ sudo chmod 750 /usr/local/libexec
$ docker build             
docker: 'buildx' is not a docker command.
See 'docker --help'
$ sudo chmod 755 /usr/local/libexec
$ docker build
ERROR: "docker buildx build" requires exactly 1 argument.
See 'docker buildx build --help'.

Usage:  docker buildx build [OPTIONS] PATH | URL | -

Start a build

It seems that docker will iterate through the configured plugin libraries in a certain order and it checks /usr/local/lib and /usr/local/libexec before /usr/lib and /usr/libexec which is correct. But there seems to be lack of error handling when a directory does not have the proper permissions. If the user running docker has no permission to access the /usr/local/libexec folder, Docker fails to search the remaining locations and will not find the plugins.

My issue was that /usr/local/libexec had permissions 750, while it didn’t contain any docker/cli-plugins folder, Docker failed on this and did not check further for plugins under /usr/libexec even though that directory had the correct permissions. When I was poking around and testing whether /usr/local/libexec/docker/cli-plugins will work, I fixed the permissions and then everything started to work.

I think, this is a bug in docker-cli, unrelated to Fedora. The permissions on /usr/local/libexec was probably due to me creating that folder with default umask 027 and forgetting about changing it later.