[Fedora Silverblue] Ollama with ROCM - failed to check permission on /dev/kfd: open /dev/kfd: invalid argument

Ask Fedora
, , , , ,
10

Here’s how I’ve run ollama with podman:

podman run --pull newer --detach --security-opt label=type:container_runtime_t --replace --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm

(The largest difference: This avoids label=disable mentioned in the above solutions, and still works in a container with the GPU on an AMD system in a Podman user account.)

And to run the ollama command manually, do this:

podman exec -it ollama ollama

(Add “run” or whatever after if you want to run a model, like podman exec -it ollama ollama run mistral — this will download the model the first time you run it.)

Then, if you want a web UI for it, there’s this too:

podman run --replace --pull newer -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main

After you have ollama and the web UI running, it can be accessed at http://localhost:8080/

If you want to be really fancy, you can create quadlets (podman containers as a systemd service):

Create a file at .config/containers/systemd/ollama.container with this as the contents:

[Container]
ContainerName=ollama
Image=ollama/ollama:rocm
Volume=ollama:/root/.ollama
Pull=newer
AddDevice=/dev/kfd
AddDevice=/dev/dri
PublishPort=11434:11434
SecurityLabelType=container_runtime_t

And this as .config/containers/systemd/ollama-webui.container

[Container]
ContainerName=open-webui
Image=ghcr.io/open-webui/open-webui:main
Pull=newer
Volume=open-webui:/app/backend/data
Environment=OLLAMA_BASE_URL=http://127.0.0.1:11434
Network=host

Then run these commands to have it start as services when you log in:

systemctl --user daemon-reload
systemctl --user enable --now ollama ollama-webui

Notes:

  1. If you don’t want the web UI, just use the ollama container.
  2. If you don’t want this to run on start, use systemctl --user start ollama ollama-webui instead to start on demand. (The daemon-reload command is only needed after adding or changing the systemd-related files.)
  3. Like the above podman commands, it will take a while on first start, as it needs to download. It also will take a little while to start when the containers change, as it’ll download the new parts. You’ll see the changes when running the podman command, but the quadlet version via systemctl will download changes in the background.
  4. This has persistent storage, thanks to the volume.
  5. There’s a bunch of neat stuff you can do with an LLM (translations in a more conversational way, suggestions of all kinds of things in various topics, writing a rhyme based on some text, etc.), but don’t trust it. The level of trust you should place in its outputs is about the same as a random person you don’t know at a bar who starts talking with you: They might know what they’re talking about or might’ve had something to drink and are completely making things up, intentional or not. Or somewhere in-between.
  6. Seriously again; this is worth repeating: Don’t trust it. To get a feel for it, start by asking it several things on topics you know very well and then apply that same skepticism to anything else it ever says. (This goes for all LLMs, not just ones that can run in ollama.)