Troubleshooting SELinux on FCOS

I’m new to coreOS and Fedora/RH in general. Before that I used Ubuntu Server.

My homeserver is running FCOS39 to provide some services like pihole, nextcloud-aio, caddy-reverseproxy, evcc and homeassistant. I needed some time dealing with podman/quadlet (instead of docker) and Butane/Ignition (love this; took me to a recovery scenario of maximum five minutes).

Finally I have this up and running, but with SELinux in permissive mode.

All container data is located in my home directory or in named volumes. I use :z / :Z options for most of the volumes.

Now I would like to enforce SELinux policies again, but I got “permission denied” for some files during boot. When I start quadlet units by myself from ssh session everything works fine. I’m quite sure that this is a labeling issue.

Can someone tell me how to troubleshoot SELinux on FCOS?

I found some guides for Fedora or RHEL, but it seems that some packages are not part of FCOS.

Any help would be appreciated.

You should look for AVC error messages in the journal log or audit log (/var/log/audit/audit.log).

This commands should also give you some results:

ausearch -m AVC,USER_AVC,SELINUX_ERR,USER_SELINUX_ERR -ts recent

It’s basically the beginning of Troubleshooting Problems Related to SELinux :: Fedora Docs that you’ve found already.

Thanks for your advice.
So far so good. I did that already and assumed that this is a labeling problem.

But I don’t know what to do with these AVC messages.
How to analyze and fix them?

If it is a labeling problem you could do a blanket relabel with either
sudo touch /.autorelabel which would do the relabel the next time you reboot.
or
sudo restorecon -R / to redo the labeling while the system is running. If done this way you would need to ensure that command has completed before shutdown/sleep/suspend/reboot.

Either method will take some time to complete.

Since FCOS is one of the immutable spins this may not work properly so the fix might have to wait until the next system upgrade or restrict the context restore to the portions that may be altered – some parts of /etc and /var.

This will not work on Silverblue/Kinoite/CoreOS/IoT as / is immutable.

This is dangerous. See Troubleshooting :: Fedora Docs.

What you should do instead is selectively restore the labels for a subset of directories and files with:

$ restorecon -RFv /path/in/var /path/in/etc

If you can share them here then we might be able to help.

Sorry for my delayed reply.
And thanks for your help in advice.
My AVC messages basically look like this for different files.
I think podmans volume options :Z and :z are labeling this files/volumes which is a problem when systemd/quadled starts the containers on boot?

type=AVC msg=audit(1700938442.398:124): avc:  denied  { read } for  pid=1270 comm="evcc" name="evcc.yaml" dev="sdb4" ino=118489291 scontext=system_u:system_r:container_t:s0:c176,c221 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938445.773:130): avc:  denied  { write } for  pid=1449 comm="python3" name="config" dev="sdb4" ino=120586451 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=dir permissive=1

type=AVC msg=audit(1700938445.773:131): avc:  denied  { add_name } for  pid=1449 comm="python3" name="home-assistant.log.fault" scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=dir permissive=1

type=AVC msg=audit(1700938445.773:132): avc:  denied  { create } for  pid=1449 comm="python3" name="home-assistant.log.fault" scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938445.773:133): avc:  denied  { append } for  pid=1449 comm="python3" path="/config/home-assistant.log.fault" dev="sdb4" ino=120600467 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938445.773:134): avc:  denied  { ioctl } for  pid=1449 comm="python3" path="/config/home-assistant.log.fault" dev="sdb4" ino=120600467 ioctlcmd=0x5413 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938445.807:135): avc:  denied  { write } for  pid=1449 comm="python3" name="home-assistant.log" dev="sdb4" ino=120600469 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938445.807:136): avc:  denied  { remove_name } for  pid=1449 comm="python3" name="home-assistant.log.1" dev="sdb4" ino=120600468 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=dir permissive=1

type=AVC msg=audit(1700938445.807:137): avc:  denied  { unlink } for  pid=1449 comm="python3" name="home-assistant.log.1" dev="sdb4" ino=120600468 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938445.807:138): avc:  denied  { rename } for  pid=1449 comm="python3" name="home-assistant.log" dev="sdb4" ino=120600469 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938445.814:139): avc:  denied  { read } for  pid=1449 comm="python3" name=".HA_VERSION" dev="sdb4" ino=120586452 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938450.191:185): avc:  denied  { read } for  pid=1449 comm="python3" name="core.analytics" dev="sdb4" ino=121635057 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938450.193:186): avc:  denied  { ioctl } for  pid=1449 comm="python3" path="/config/.storage/core.analytics" dev="sdb4" ino=121635057 ioctlcmd=0x5413 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938450.375:187): avc:  denied  { write } for  pid=1449 comm="python3" name="home-assistant_v2.db" dev="sdb4" ino=120600470 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938450.379:188): avc:  denied  { lock } for  pid=1449 comm="python3" path="/config/home-assistant_v2.db" dev="sdb4" ino=120600470 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938450.380:189): avc:  denied  { write } for  pid=1449 comm="python3" name="config" dev="sdb4" ino=120586451 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=dir permissive=1

type=AVC msg=audit(1700938450.380:190): avc:  denied  { add_name } for  pid=1449 comm="python3" name="home-assistant_v2.db-wal" scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=dir permissive=1

type=AVC msg=audit(1700938450.380:191): avc:  denied  { create } for  pid=1449 comm="python3" name="home-assistant_v2.db-wal" scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938450.380:192): avc:  denied  { setattr } for  pid=1449 comm="python3" name="home-assistant_v2.db-wal" dev="sdb4" ino=120600468 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938450.380:193): avc:  denied  { write } for  pid=1449 comm="python3" name="config" dev="sdb4" ino=120586451 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=dir permissive=1

type=AVC msg=audit(1700938450.380:194): avc:  denied  { add_name } for  pid=1449 comm="python3" name="home-assistant_v2.db-shm" scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=dir permissive=1

type=AVC msg=audit(1700938450.389:195): avc:  denied  { remove_name } for  pid=1449 comm="python3" name="home-assistant_v2.db-shm" dev="sdb4" ino=120600472 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=dir permissive=1

type=AVC msg=audit(1700938450.390:196): avc:  denied  { unlink } for  pid=1449 comm="python3" name="home-assistant_v2.db-shm" dev="sdb4" ino=120600472 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938455.450:198): avc:  denied  { nlmsg_read } for  pid=2235 comm="ss" scontext=system_u:system_r:container_t:s0:c985,c989 tcontext=system_u:system_r:container_t:s0:c985,c989 tclass=netlink_tcpdiag_socket permissive=1

type=AVC msg=audit(1700938462.170:217): avc:  denied  { read } for  pid=1449 comm="python3" name="mobile_app" dev="sdb4" ino=121635848 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938462.171:218): avc:  denied  { ioctl } for  pid=1449 comm="python3" path="/config/.storage/mobile_app" dev="sdb4" ino=121635848 ioctlcmd=0x5413 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938470.380:219): avc:  denied  { create } for  pid=1449 comm="python3" name="tmpupvx6pt1" scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938470.381:220): avc:  denied  { write } for  pid=1449 comm="python3" path="/config/.storage/tmpupvx6pt1" dev="sdb4" ino=121635063 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1
----
time->Sat Nov 25 19:54:30 2023
type=AVC msg=audit(1700938470.387:221): avc:  denied  { setattr } for  pid=1449 comm="python3" name="tmpupvx6pt1" dev="sdb4" ino=121635063 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938470.390:222): avc:  denied  { rename } for  pid=1449 comm="python3" name="tmpupvx6pt1" dev="sdb4" ino=121635063 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938470.391:223): avc:  denied  { unlink } for  pid=1449 comm="python3" name="core.restore_state" dev="sdb4" ino=121649956 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

type=AVC msg=audit(1700938470.421:224): avc:  denied  { lock } for  pid=1449 comm="python3" path="/config/home-assistant_v2.db-shm" dev="sdb4" ino=120600472 scontext=system_u:system_r:container_t:s0:c186,c289 tcontext=system_u:object_r:container_file_t:s0:c27,c542 tclass=file permissive=1

From the logs:

scontext=system_u:system_r:container_t:s0:c176,c221
tcontext=system_u:object_r:container_file_t:s0:c27,c542

You are likely sharing a volume between two different containers. Thus you need to use z instead of Z to tell that to SELinux: --volumes-from=CONTAINER[:OPTIONS] — Podman documentation

See also: docker - Podman volume mounts: When to use the :z or :Z suffix? - Unix & Linux Stack Exchange

Thanks for your reply.
Yes, as I mentioned initially, I do share files between container. There is a SFTP container I use for editing config files of other container with WinSCP.

And I already use :z and :Z. Without that I had permission denied on any access from a container. With that options it is working when my user starts the container units via systemctl --user. During reboot I got the above AVC messages.

In any case that matters: I should mention that the container files are initially created due to an SFTP transfer. And all files are in core’s home directory.

Those are system containers (rootful podman containers) from the SELinux contexts so you can manage them via the systemd user instance.

You’re not giving me enough context to be able to help. Sorry.

Okay, I’ll try to give more context.

There are two rootful containers. Their volume files are within /etc. And labeled with :Z. No problem with them and SELinux. Just mentioning them because one is my SFTP server.

By using this SFTP server I transfer other files to a folder in my home directory and create a couple of rootless containers. As I would like to edit those files using the SFTP server, they are shared between two containers (sftp and e. g. homeassistant) and I’m using volumes with the :z option.

And as I said it is working when my user starts the rootless container units via systemctl --user . During reboot I got the above AVC messages.

Feel free to ask any question which makes you able to help.

You can not share files between rootful and rootless containers, they don’t have the same SELinux labels. Sharing /etc in containers is also generally not recommended as it will create issues on the host if you relabel files.

I do not share /etc. Just files in subfolders. So that should be fine. But I will take a look.

But different labels for rootful and rootless containers make sense. Thanks.

I run the sftp server rootful because it run and creates files with uid 1000 inside the container. So files through sftp are accessable for my fcos user and rootless containers running with root.
Any suggestion how to deal with it?

Either run all the containers that need to share data as rootless or rootful, not a mix.