STAT errors with Podman and Containers on Fedora 35

I’m trying to run Podman that distributed now with Fedora 35 workstation and the problems are multiplied: Podman stores some data in the User’s home, User’s home is located in /home btrfs subvolume and Container-storage package attempts to create OCI containers file structure using either btrfs driver creating subvolumes or overlay driver creating folders under the $HOME. It fails on the STAT call. When I execute all this sequence manually as rootless user everything looks good except very incoherent labelling of home folders and container’s files. When STAT called programmatically by container-storage it returns 125??? I Is it possible that SELinux labels can affect STAT I prefer to createcontainer-storage root as subvolume and try to relabel everything manually step-by-step and see how SELinux labelling affects STAT.

I have looked in audit log. SELinux block creation/deletion of a new user due to blocked access to mail folder (???). Failed stat doesn’t leave any records in the log files but the btrfs container-storage driver source code has no code for the STAT error processing - only exits with received error code. It looks like nothing is logged. Maybe, somebody faced an issue with stat caused by SELinux or saw documentation describing how home area should be labeled and what is the trigger for relabeling in the Fedora with homed service enabled and running?

Hello @pavelsosin ,
Welcome to :fedora: discussion. This area is usually for contributor related discussion in Fedora project. For user and using Fedora Linux, support questions are best asked at ask.fp.o.
Podman is to be run as rootless normally, since that was the purpose it was created for, rootless containers.

It is exactly what I’m doing! In the Fedora workstation environment root user is hardly useful also as development user. It used to work in Fedora 34, not workstation. In the Fedora 35 workstation with btrfs as the default and systemd-homed introduction it is broken. User’s created via Settings → Users Add UI have no home area. Root can create containers and have no other problems. But it stores containers out of the home area. Stat utility invoked from Terminal->bash works OK. What is special in the btrfs->stat?

Can you provide steps for reproducing the problem? And if you can run journalctl -f before you reproduce it, and post the log messages that appear when the problem happens, that might give a clue including the full selinux AVC. Next post the output from podman system info will help give an overview of the configuration. Thanks

The most surprising for me is that journal contains no hints pointing to SeLinux issue. I suppose that the root cause is that btrfs subvolume create command/call creates unlabeled directory node and every subvolume creation has to be followed by the setcontext call of selinux userspace library to make the subvolume visible to the “unconfined” user that has created the subvolume. The command btrfs subvolume create itself has no options to supply SELinux context for the subvolume root and inner files. I afraid that the corresponding API call has no such parameters too. Maybe, it should be governed by the targeting policy? I have no idea.

Hmm, yeah interesting:

$ ls -lZ
total 4
drwxr-sr-x+ 1 root systemd-journal system_u:object_r:var_log_t:s0     16214 Jan 16 12:47 6f69bf39af5a4dbdbb627b3f3971e626
drwxr-xr-x. 1 root root            system_u:object_r:unlabeled_t:s0       0 Jan 16 13:01 test
drwxr-sr-x+ 1 root systemd-journal unconfined_u:object_r:var_log_t:s0     0 Jan 16 13:01 test2

The $machineid dir was already present; then btrfs sub create test and mkdir test2.

$ pwd
/home/chris/.local/share/containers/storage/btrfs/subvolumes
$ ls -lZ
total 0
dr-xr-xr-x. 1 chris chris system_u:object_r:unlabeled_t:s0                42 Dec 21 16:03 038d5e4fb91fc82dfbfe16f161e26e42b8bb85121e1699963c149b1afa67a37a
dr-xr-xr-x. 1 chris chris system_u:object_r:container_file_t:s0:c659,c962 42 Dec 21 15:53 2634d4a8cf3f5927e25c912effcebb6684983693202b19ca6284c743d91a96ac
dr-xr-xr-x. 1 chris chris system_u:object_r:container_file_t:s0:c145,c358 42 Dec 21 15:54 61137c9e33d59f71c1bd676e9deb631a9dff008646ea3afb1521c9996f839a76
dr-xr-xr-x. 1 chris chris system_u:object_r:unlabeled_t:s0                10 Dec 21 15:53 e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359
$ 

Looks like podman does have a way to change the context. What I vaguely understand is upon running a container from a particular image, the image subvolume is first read-write snapshot, and the container runs from the read-write snapshot.

Of course, this is not a default configuration. Fedora is using the overlay graph driver by default, so no Btrfs subvolumes or snapshots, instead any copy-up operation by overlayfs (or fuse-overlayfs) results in reflink copies on Btrfs and XFS.

I already tried to apply Oracle’s Podman BTRFS configuration that uses the overlay driver but it caused the podman info failure. I think that Oracle’s Linux OS build is different from Fedora 35. The btrfs and selinux libraries and SElinux configuration may not match even in the case when Oracle’s Container-storage implementation is perfect.

Good news:I tried to create container from scratch using buildah “from scratch” and the result looks much better: “Working-container” is created and can be mounted. Then everything looks expected: podmanunshare podman mount, etc.

ls -alZ /home/pavelsosin/.local/share/containers/storage/btrfs/subvolumes/e3982c6d30c0d0582d16ab4cc4305aaaa26fb3da8be3f3c158ac7f3861a77d1a

total 0

dr-xr-xr-x. 1 pavelsosin pavelsosin system_u:object_r:container_file_t:s0:c180,c597

and
stat /home/pavelsosin/.local/share/containers/storage/btrfs/subvolumes/e3982c6d30c0d0582d16ab4cc4305aaaa26fb3da8be3f3c158ac7f3861a77d1a

But Podman info and Buildah info show different btrfs versions: 15.4 - working and 5.15 - failing. Does it point to regression? The library version is 102 and configuration is shared too.

Check if it’s btrfs-progs-5.15.1-1.fc35though offhand I’m not aware of a related bug fix in 5.15.1.
https://btrfs.wiki.kernel.org/index.php/Changelog#btrfs-progs_v5.15_.28Nov_2021.29

What specifically fails with btrfs-progs-5.15?

It used to be OK in the previous Podman release 3.4.1 and broken in 3.4.4, about the time of btrfs 5.15 release date. The dependency between Podman and btrfs is transient, via Container-storage → drivers - > btrfs. Indeed, the last change of the btrfs driver was 9 months ago, i.e. changes of 5.15 can be missed.
I tried to build a standalone Container-storage CLI to test btrfs driver in-vitro but my make failed in Fedora 35.

More good news: the same ubuntu docker image pulled by buildah as rootless user using buildah from … and then committed by buildah into the local storage can be used by podman to create container as rootless user without any issue. The created container still refuses to start with OCI permission error. But it is more uid/gid mapping of Fedora human user to the container’s technical user root.

And now … we need help from zpytela/container-selinux - human unconfined user muse be able to access and stat container subvolume and access its inner files.