How to use Nvidia container-toolkit without disabling selinux?

Ask Fedora
, , , , ,
1

Hello, I am looking to use the ollama container run LLMs using podman. I have intalled nvidia container toolkit and it seems to only have access to the GPU if I add --security-opt=label=disable to the podman run command. Without it, I get the following error: Failed to initialize NVML: Insufficient Permissions.

How can I go about using my GPU with containers without disabling selinux? Any policy I have to install, or selinux configuration I need to change?

Not familiar with selinux, so I appreciate any guidance provided. Thank you! :pray:

3

Hello I see the following entries:

AVC avc:  denied  { getattr } for  pid=4836 comm="nvidia-smi" path="/dev/nvidiactl" dev="devtmpfs" ino=909 scontext=system_u:system_r:container_t:s0:c227,c662 tcontext=system_u:object_r:xserver_misc_device_t:s0 tclass=chr_file permissive=0

AVC avc:  denied  { getattr } for  pid=4836 comm="nvidia-smi" path="/dev/nvidiactl" dev="devtmpfs" ino=909 scontext=system_u:system_r:container_t:s0:c227,c662 tcontext=system_u:object_r:xserver_misc_device_t:s0 tclass=chr_file permissive=0

AVC avc:  denied  { read } for  pid=4836 comm="nvidia-smi" name="nvidiactl" dev="devtmpfs" ino=909 scontext=system_u:system_r:container_t:s0:c227,c662 tcontext=system_u:object_r:xserver_misc_device_t:s0 tclass=chr_file permissive=0
1 Like
5

For those that found this question via a search engine there is a solution that works that worked for me.

This assumes that you use the rpmfusion nvidia packages.

These instructions provide a working solution: https://copr.fedorainfracloud.org/coprs/g/ai-ml/nvidia-container-toolkit/

6

The problem is that by default, container_t is not allowed to access devices (files) labelled with xserver_misc_device_t (i.e, /dev/nvidia*).

$ matchpathcon /dev/nvi\*
/dev/nvidia0    system_u:object_r:xserver_misc_device_t:s0
/dev/nvidia-caps        system_u:object_r:device_t:s0
/dev/nvidiactl  system_u:object_r:xserver_misc_device_t:s0
/dev/nvidia-modeset     system_u:object_r:xserver_misc_device_t:s0
/dev/nvidia-uvm system_u:object_r:xserver_misc_device_t:s0
/dev/nvidia-uvm-tools   system_u:object_r:xserver_misc_device_t:s0

The solution is revealed if we search the SELinux policy for any rules that might allow this interaction:

$ sesearch -A -s container_t -t xserver_misc_device_t
allow container_domain device_node:blk_file { append getattr ioctl lock map open read write }; [ container_use_devices ]:True
allow container_domain device_node:chr_file { append getattr ioctl lock map open read write }; [ container_use_devices ]:True
allow container_t xserver_misc_device_t:chr_file getattr; [ container_use_xserver_devices ]:True
allow container_t xserver_misc_device_t:chr_file map; [ container_use_xserver_devices ]:True
allow container_t xserver_misc_device_t:chr_file { append getattr ioctl lock open read write }; [ container_use_xserver_devices ]:True

This means that there are rules to allow the access, but that they are only effective if the container_use_xserver_devices boolean is enabled.

This can be done permanently with semanage boolean -m --on container_use_devices.

FYI this boolean is documented in container_selinux(8):

If you want to allow containers to use any xserver device volume mounted into container, mostly used for GPU acceleration, you must turn on the `container_use_xserver_devices` boolean. Disabled by default.

2 Likes