Docker help on Fedora (SELinux issue?)

I have run docker run hello-world successfully, so I went for the next step and tried:

$docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
f476d66f5408: Pull complete 
8882c27f669e: Extracting [==================================================>]     851B/851B
d9af21273955: Download complete 
f5029279ec12: Download complete 
docker: failed to register layer: unknown file type for /media/data/docker-data/overlay/ed414086db1bddb0515b1da4c980bf0d4a25b81fdb8d72ba225f4619cbb8ca3a/root/dev/full.
See 'docker run --help'.

And at the same time as this, SELinux alert browser shows:

SELinux is preventing bash from read access on the file /usr/lib64/libc-2.28.so.

What is the problem?

I haven’t run docker in a while, and this is the first since upgrading to Fedora 30. I did have docker running this demo about a year ago without issue. At that time I had installed it from the docker-ce repo. One change I made was to move the docker root directory from the default /var/lib/docker to /media/data/docker-data`.

Now on Fedora 30 I also removed docker-ce (which doesn’t have a stable build for Fedora 30 yet) and installed moby-engine.

I’m a total noob with docker so it’s entirely possible this is simply user error.

docker info
$ docker info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 2
Server Version: dev
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 5.0.11-300.fc30.x86_64
Operating System: Fedora 30 (Thirty)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.4GiB
Name: ssb2
ID: MY2I:IOZS:PVPO:DRGT:VQF7:PNEF:VMQR:X6N2:2Y3U:HD7S:ODAI:LNG4
Docker Root Dir: /media/data/docker-data
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true
1 Like

I got rid of the “failed to register layer” docker error. I’m not sure if this was the solution, but I switched to the “overlay2” storage driver since that is recommended. Docker storage drivers | Docker Docs

Containers work as expected if selinux is in permissive mode.

Still not solved though as containers fail to launch with selinux enforcing:

$ docker run -it ubuntu bash
bash: error while loading shared libraries: libtinfo.so.5: cannot change memory protections

SELinux is preventing bash from read access on the file /lib/x86_64-linux-gnu/libtinfo.so.5.9.

If SELinux is the issue then it’s likely this change:

One change I made was to move the docker root directory from the default /var/lib/docker to /media/data/docker-data`.

Presumably there are SELinux rules - policies - in the Docker package, and now that the path is not standard, they don’t trigger the right way.

SELinux is preventing bash from read access on the file /usr/lib64/libc-2.28.so.

The alert browser has a way to show suggested fixes, which are custom SELinux rules you could save to a file and activate. These are based on actual blocked system calls, and should account for your custom paths.

2 Likes

I think you’re out of space or you will:

Could you describe What step did you follow to move the default docker data directory to another one?

Why don’t you install docker from the main fedora repo?

sudo dnf install docker

You have to enable it:

sudo systemctl enable docker
sudo systemctl start docker

See the place where docker have this files:

docker info | grep “Docker Root Dir”

Change the default location:

You have to edit /etc/sysconfig/docker , and add the -g option in the OPTIONS variable. If there’s more than one option, make sure you enclose them in “”. In my case, that file contained:

OPTIONS=--selinux-enabled # more parameters here

so it would become for example

OPTIONS="--selinux-enabled -g /media/data/docker-data"

After a restart ( systemctl restart docker ) , Docker should use the new directory

Confirm the changes:

docker info | grep “Docker Root Dir”

NOTE: I run hello and I don’t get any warning…

COMMENT:

There are other alternatives also availables in the main fedora repo, for examples:

Buildah : Create containers
Podman : Run containers

Regards.,

To test this I changed the docker root directory back to /var/lib/docker and the containers launched without errors or SELinux complaints.

One confusion is that I moved the default docker root directory to /media/data/docker-data a year ago and it used to work fine then. I’m not sure if moby-engine has different SELinux rules than the docker-ce installation. I would install the latter to test, but they don’t have a stable build for Fedora 30 at the moment. I might enable testing or nightly builds if I get curious enough…

Using my non-default docker root directory again, I tried the suggestions from the SELinux alert browser but they didn’t work. See the screenshots below. There is no /lib/x86_64-linux-gnu directory on my host machine, but it does exist in the ubuntu container.

selinux_err1 selinux_err2

Surely changing the docker root directory is a pretty standard procedure. How is it supposed to be done on Fedora / SELinux systems? What I did was simply following the docker documentation by creating the file /etc/docker/daemon.json with the contents:

{
    "data-root": "/media/data/docker-data",
    "storage-driver": "overlay2"
}

I also tried adding the -g /media/data/docker-data option to etc/sysconfig/docker and restarting the docker service, but that didn’t work. I think that option was deprecated?

The docker package in the repo is quite outdated. The moby-engine package also in the fedora repo seems to be more recent, lagging only a little behind docker-ce from docker’s own repo.

I was using docker-ce originally, but since they take weeks/months after each new Fedora release to issue a stable docker-ce build, I decided to try moby-engine.

docker-ce is currently at version docker-ce-18.09.6-3.fc29.x86_64.rpm (notice fc29, no stable build for fc30 yet)

moby-engine is a few point releases behind:

$ dnf info moby-engine
Installed Packages
Name         : moby-engine
Version      : 18.06.3
Release      : 2.ce.gitd7080c1.fc30
Architecture : x86_64
Size         : 226 M
Source       : moby-engine-18.06.3-2.ce.gitd7080c1.fc30.src.rpm
Repository   : @System
From repo    : fedora
Summary      : The open-source application container engine
URL          : https://www.docker.com
License      : ASL 2.0
Description  : Docker is an open source project to build, ship and run any application as a
             : lightweight container.
             : 
             : Docker containers are both hardware-agnostic and platform-agnostic. This means
             : they can run anywhere, from your laptop to the largest EC2 compute instance and
             : everything in between - and they don't require you to use a particular
             : language, framework or packaging system. That makes them great building blocks
             : for deploying and scaling web apps, databases, and backend services without
             : depending on a particular stack or provider.

docker looks like it hasn’t had commits in 3 years based on the github url:

$ dnf info docker
Available Packages
Name         : docker
Epoch        : 2
Version      : 1.13.1
Release      : 67.git1185cfd.fc30
Architecture : x86_64
Size         : 26 M
Source       : docker-1.13.1-67.git1185cfd.fc30.src.rpm
Repository   : updates
Summary      : Automates deployment of containerized applications
URL          : https://github.com/projectatomic/docker
License      : ASL 2.0
Description  : Docker is an open-source engine that automates the deployment of any
             : application as a lightweight, portable, self-sufficient container that will
             : run virtually anywhere.
             : 
             : Docker containers can encapsulate any payload, and will run consistently on
             : and between virtually any server. The same container that a developer builds
             : and tests on a laptop will run at scale, in production*, on VMs, bare-metal
             : servers, OpenStack clusters, public instances, or combinations of the above.

Out of curiosity I tried it anyways, and docker won’t launch due to the data-root option not being recognized:

dockerd-current[27085]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives don’t match any configuration option: data-root

systemd[1]: Failed to start Docker Application Container Engine.

Could you please try running docker run -it ubuntu bash and let me know if that works for you? The hello-world demo does not run into this issue.

-g and --graph flags on dockerd

Deprecated In Release: v17.05.0

The -g or --graph flag for the dockerd or docker daemon command was used to indicate the directory in which to store persistent data and resource configuration and has been replaced with the more descriptive --data-root flag.

{
  "data-root": "/new/path/to/docker-data"
}

That is your error…

Regards.,

Yes, I mentioned in my earlier post that I am using daemon.json. I tried your suggestion of -g just to be thorough. It’s been years since that option worked.


I tried docker-ce (for fc29) again (Index of linux/fedora/29/x86_64/stable/Packages/ ) and there are no SELinux problems with the non-default docker root directory. So maybe this is an issue with moby-engine.

I also have problem to run your command, It was a SELinux content on the new directory:

What I did?

  1. Remove the new destination directory…

sudo rm -rf /run/media/hhlp/samsung/docker

  1. Copy Again preserving SELinux context from the Original DIrectory:

sudo cp --preserve=context -r /var/lib/docker /run/media/hhlp/samsung/docker

Check again with -Z option , display SELinux content:

ls -lZ /run/media/hhlp/samsung/docker

Regards.,

Thanks for confirming the issue! That was with moby-engine yes?

I did not copy to the new directory, but let docker create it. If a specified root directory does not exist, docker creates it when started.

Is there a way to fix the SELinux context without deleting the new directory? I just finished downloading a bunch of images again.

$ sudo ls -lZ /var/lib/docker
total 48
drwx------. 2 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 builder
drwx------. 4 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 buildkit
drwx------. 2 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 containers
drwx------. 3 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 image
drwxr-x---. 3 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 network
drwx------. 3 root root system_u:object_r:container_share_t:s0   4096 May  7 15:18 overlay2
drwx------. 4 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 plugins
drwx------. 2 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 runtimes
drwx------. 2 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 swarm
drwx------. 2 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 tmp
drwx------. 2 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 trust
drwx------. 2 root root system_u:object_r:container_var_lib_t:s0 4096 May  7 15:18 volumes


$ sudo ls -lZ /media/data/docker-data/
total 52
drwx------.  2 root root system_u:object_r:mnt_t:s0 4096 May  7 01:09 builder
drwx------.  4 root root system_u:object_r:mnt_t:s0 4096 May  7 01:09 buildkit
drwx------.  3 root root system_u:object_r:mnt_t:s0 4096 May  7 01:09 containerd
drwx------.  6 root root system_u:object_r:mnt_t:s0 4096 May  7 14:32 containers
drwx------.  3 root root system_u:object_r:mnt_t:s0 4096 May  7 01:09 image
drwxr-x---.  3 root root system_u:object_r:mnt_t:s0 4096 May  7 01:09 network
drwx------. 21 root root system_u:object_r:mnt_t:s0 4096 May  7 14:32 overlay2
drwx------.  4 root root system_u:object_r:mnt_t:s0 4096 May  7 01:09 plugins
drwx------.  2 root root system_u:object_r:mnt_t:s0 4096 May  7 14:22 runtimes
drwx------.  2 root root system_u:object_r:mnt_t:s0 4096 May  7 01:09 swarm
drwx------.  2 root root system_u:object_r:mnt_t:s0 4096 May  7 14:22 tmp
drwx------.  2 root root system_u:object_r:mnt_t:s0 4096 May  7 01:09 trust
drwx------.  2 root root system_u:object_r:mnt_t:s0 4096 May  7 01:09 volumes

The SELinux errors mentioned mnt_t. I really don’t know much about SELinux though, it’s always been kind of confusing to me…

If you’ve moved the docker root directory, you need to ensure it carries the correct SELinux label on it. The following command will inform SELinux to label everything under /media/data/docker-data as it would under /var/lib/docker:

sudo semanage fcontext -a -e /var/lib/docker /media/data/docker-data

And then do this to relabel the whole thing:

sudo restorecon -R /media/data/docker-data

Edit: added sudo before the command, added restorecon invocation

2 Likes

too see the content for docker:

sudo semanage fcontext -l | grep docker

Regards.,

1 Like

Thank you! @nixclusive0’s post solves the problem. Now moby-engine has no problems working with my non-default docker root directory.

Gradually I’m learning bits and pieces of selinux. This has been more helpful than just reading.

One little detail: docker info now shows Server Version: dev while with docker-ce it contained the version number, eg. 18.09.6. Is this intentional or an oversight? Where might it be suggested for moby-engine to include the version number?

I’ve run into another selinux problem, this time when creating and accessing docker volumes for containers:

$ docker run --mount source=ubuntu-home,target=/home/ ubuntu
/bin/bash: error while loading shared libraries: libtinfo.so.5: cannot change memory protections

From SETroubleshoot details:

Source Context                system_u:system_r:container_t:s0:c770,c809
Target Context                system_u:object_r:container_var_lib_t:s0
Target Objects                /lib/x86_64-linux-gnu/libtinfo.so.5.9 [ file ]

What is the correct way to deal with this conflict between source and target contexts? container_t vs container_var_lib_t

Looks like the solution is to use the z mount option. i.e. docker run -v ubuntu-home:/home:z ubuntu

This is an article by Dan Walsh on exactly this topic of problems with docker volumes when using SELinux: Using Volumes with Docker can Cause Problems with SELinux — Project Atomic

There is a small mention of this in docker’s documentation: Bind mounts | Docker Docs

If you use selinux you can add the z or Z options to modify the selinux label of the host file or directory being mounted into the container. This affects the file or directory on the host machine itself and can have consequences outside of the scope of Docker.

  • The z option indicates that the bind mount content is shared among multiple containers.
  • The Z option indicates that the bind mount content is private and unshared.

Of concern:

Use extreme caution with these options. Bind-mounting a system directory such as /home or /usr with the Z option renders your host machine inoperable and you may need to relabel the host machine files by hand.

Also:

The --mount flag does not support z or Z options for modifying selinux labels.

Interestingly, this is mentioned in the documentation only for bind mounts. The page for volume mounts doesn’t have anything about z or selinux.

The risks and details make it seem like quite a headache to use selinux with docker, even though it is arguably a prime use case for it.

Since in the team we started to use cache and delegated storage options, I need to upgrade my docker.
I was using the docker package, but since it’s too old it does not support that options, so I had to replace docker with moby-engine.
The procedure was very trivial with a removal of docker and it’s dependencies and then install moby-engine. Also I’ve performed an docker system prune to remove the old images.
Also I’ve created an /etc/docker/deamon.js with:

 {
  "data-root": "/home/docker"
}

because I keep all my docker data under that dir.

Anyone with the same problem? How can I debug this?

@fidalgo

Yes, of course we cover this issue in this section, you have all the information here:

https://discussion.fedoraproject.org/t/docker-help-on-fedora-selinux-issue/68807 and I move it noe :blush:

There is a more elegant to accomplish this task using Docker systemd documentation.

  1. Create directory and file for custom configuration:
sudo mkdir -p /etc/systemd/system/docker.service.d
sudo $EDITOR /etc/systemd/system/docker.service.d/docker-storage.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --data-root="/mnt"
  1. Restart Docker:
sudo systemctl daemon-reload
sudo systemctl restart docker
  1. Check it:
docker info|grep "Docker Root Dir"

Regards…

I’ve posted the other post after reading this one.
The I had already docker working nice with a custom dir, I’ve even created a post http://fidalgo.pt/2017/12/08/docker-and-selinux.html on how to do it.

Right now, i have:
sudo semanage fcontext -l | grep docker

/etc/docker(/.*)?                                  all files          system_u:object_r:container_config_t:s0 
/etc/docker-latest(/.*)?                           all files          system_u:object_r:container_config_t:s0 
/etc/docker/certs\.d(/.*)?                         all files          system_u:object_r:cert_t:s0 
/root/\.docker                                     all files          system_u:object_r:container_home_t:s0 
/usr/bin/docker-current                            regular file       system_u:object_r:container_runtime_exec_t:s0 
/usr/bin/docker-latest                             regular file       system_u:object_r:container_runtime_exec_t:s0 
/usr/bin/docker-novolume-plugin                    regular file       system_u:object_r:container_auth_exec_t:s0 
/usr/bin/docker.*                                  regular file       system_u:object_r:container_runtime_exec_t:s0 
/usr/lib/docker/[^/]*plugin                        regular file       system_u:object_r:container_runtime_exec_t:s0 
/usr/lib/docker/docker-novolume-plugin             regular file       system_u:object_r:container_auth_exec_t:s0 
/usr/lib/systemd/system/docker.*                   regular file       system_u:object_r:container_unit_file_t:s0 
/usr/libexec/docker/.*                             regular file       system_u:object_r:container_runtime_exec_t:s0 
/usr/libexec/docker/docker.*                       regular file       system_u:object_r:container_runtime_exec_t:s0 
/var/lib/docker(/.*)?                              all files          system_u:object_r:container_var_lib_t:s0 
/var/lib/docker-latest(/.*)?                       all files          system_u:object_r:container_var_lib_t:s0 
/var/lib/docker-latest/.*/config\.env              all files          system_u:object_r:container_share_t:s0 
/var/lib/docker-latest/containers/.*/.*\.log       all files          system_u:object_r:container_log_t:s0 
/var/lib/docker-latest/containers/.*/hostname      all files          system_u:object_r:container_share_t:s0 
/var/lib/docker-latest/containers/.*/hosts         all files          system_u:object_r:container_share_t:s0 
/var/lib/docker-latest/init(/.*)?                  all files          system_u:object_r:container_share_t:s0 
/var/lib/docker-latest/overlay(/.*)?               all files          system_u:object_r:container_share_t:s0 
/var/lib/docker-latest/overlay2(/.*)?              all files          system_u:object_r:container_share_t:s0 
/var/lib/docker/.*/config\.env                     all files          system_u:object_r:container_share_t:s0 
/var/lib/docker/containers/.*/.*\.log              all files          system_u:object_r:container_log_t:s0 
/var/lib/docker/containers/.*/hostname             all files          system_u:object_r:container_share_t:s0 
/var/lib/docker/containers/.*/hosts                all files          system_u:object_r:container_share_t:s0 
/var/lib/docker/init(/.*)?                         all files          system_u:object_r:container_share_t:s0 
/var/lib/docker/overlay(/.*)?                      all files          system_u:object_r:container_share_t:s0 
/var/lib/docker/overlay2(/.*)?                     all files          system_u:object_r:container_share_t:s0 
/var/run/docker(/.*)?                              all files          system_u:object_r:container_var_run_t:s0 
/var/run/docker-client(/.*)?                       all files          system_u:object_r:container_var_run_t:s0 
/var/run/docker/plugins(/.*)?                      all files          system_u:object_r:container_plugin_var_run_t:s0 
/var/run/docker\.pid                               regular file       system_u:object_r:container_var_run_t:s0 
/var/run/docker\.sock                              socket             system_u:object_r:container_var_run_t:s0 
/home/docker = /var/lib/docker

so I think the SELinux has no issues

Also from my docker info, it seems good as well:

Containers: 26
 Running: 6
 Paused: 0
 Stopped: 20
Images: 124
Server Version: dev
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 5.0.16-300.fc30.x86_64
Operating System: Fedora 30 (Thirty)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.519GiB
Name: alpha
ID: KO22:RYWZ:UPRH:NN4O:UCHB:HQCH:XOMS:KGZZ:OYH5:K3OM:4CLL:YCDH
Docker Root Dir: /home/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

and journalctl -t setroubleshoot gives me

May 17 09:25:43 alpha setroubleshoot[1690]: failed to get filesystem list from rpm
May 17 09:26:04 alpha setroubleshoot[2131]: failed to get filesystem list from rpm
May 17 09:58:35 alpha setroubleshoot[17471]: failed to get filesystem list from rpm

My system is sluggish after the switch (from docker to moby) and I feel that building images and running things inside docker is a lot more slow now.

Anyone has some points were I can try to find out?