Running Large Language Models (LLMs) locally has become increasingly popular for development, privacy, and offline testing. Ollama makes this incredibly straightforward, allowing you to run models like Llama 3 or Mistral directly on your machine. By leveraging Podman on Fedora Linux, you can isolate Ollama inside a container. This approach keeps your host system clean while making it effortless to spin up, manage, and tear down your AI development environment. What is Ollama? Ollama is an open-source framework designed for running, creating, and sharing large language models. It packages model weights, configuration, and data into a unified management system. Running…
Read More
Can I also utilize NPU instead of CPU/GPU?
I mean if you want to use some kind of LLM I think it’s far easier to just use llamafiles. - Pre-built llamafiles | llamafile | Mozilla.ai Docs
All you do is download a model and then run it like an executable. Gives you access on the CLI and/or via a webpage (runs on port 8080 or something by default IIRC.)
Sample systemd file you can proxy to with nginx, etc.
/etc/systemd/system/llamafile.service
[Unit]
Description=Llamafile Service
[Service]
Type=simple
User=llama
Group=llama
ExecStart=/bin/sh /opt/llama/llamafile/llava-v1.6-mistral-7b-Q4_K_M.llamafile --port 9998 --host 0.0.0.0 --server
Restart=on-failure
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
1 Like
You need dedicate support in the runtime for NPU based inference. Unfortunately people gravitate towards ollama because of its low starting threshold, but it does not include the latest developments in various runtimes, not to mention they have a poor history of crediting other FOSS projects it builds on.
Anyway, to answer your question, your best bet is to try the latest llama.cpp release for your hardware, or if you have AMD hardware you could try lemonade. For both, you would need to do some digging to ensure you have enabled it with the right CLI flag or config option. There’s also FastFlowLM, but I have not tried it yet.
Hi. Thanks for podman configuration for Ollama. I’m using it on a bare metal right now, since it should occupy the box almost completely.
I found that Ollama is often not a problem, but clients are. Keeping models from interacting too much with my files is constantly pushing me back from AI. So I did build my own images with Aider.
Dockerfile
from quay.io/centos/centos@sha256:f964a3791678e692fcb4beacc5ea2c47fac78e4d4f62ca693833fb3a81fa20ea
USER 0
RUN dnf --assumeyes install python3 python3-pip python3-devel git
RUN useradd -u 1001 aideruser
RUN usermod -L aideruser
RUN mkdir -p /opt/aider_ai
RUN chown -R 1001:0 /opt/aider_ai
USER 1001
WORKDIR /opt/aider_ai
RUN pip install --upgrade "pip==26.1.2"
ADD requirements.txt /tmp/
RUN pip install --requirement /tmp/requirements.txt
RUN git config --global user.name "AIDER AI"
RUN git config --global user.email "aideruser@localhost"
ENV PATH="${PATH}:/home/aideruser/.local/bin"
ENV AIDER_GIT=true
ENV AIDER_DARK_MODE=true
ENV OLLAMA_API_BASE=http://ollama.host.local:11434/
ENV AIDER_ANALYTICS_DISABLE=true
ENV AIDER_CHECK_UPDATE=false
ENV AIDER_SHOW_RELEASE_NOTES=false
ENV AIDER_YES_ALWAYS=true
ENV OLLAMA_MODEL=qwen3.5:9b
CMD aider --model=ollama_chat/${OLLAMA_MODEL}
Change ollama.host.local to Your dedicated (or You can dream of it) Ollama host.
requirements.txt
aider-chat==0.86.2
aiohappyeyeballs==2.6.1
aiohttp==3.13.3
aiosignal==1.4.0
annotated-doc==0.0.4
annotated-types==0.7.0
anyio==4.12.1
asgiref==3.11.1
attrs==25.4.0
backoff==2.2.1
beautifulsoup4==4.14.3
certifi==2026.1.4
cffi==2.0.0
charset-normalizer==3.4.4
click==8.3.1
ConfigArgParse==1.7.1
diff-match-patch==20241021
diskcache==5.6.3
distro==1.9.0
fastapi==0.128.8
fastuuid==0.14.0
filelock==3.20.3
flake8==7.3.0
frozenlist==1.8.0
fsspec==2026.2.0
gitdb==4.0.12
GitPython==3.1.46
grep-ast==0.9.0
h11==0.16.0
hf-xet==1.2.0
httpcore==1.0.9
httpx==0.28.1
huggingface_hub==1.4.1
idna==3.11
importlib_metadata==7.2.1
importlib_resources==6.5.2
Jinja2==3.1.6
jiter==0.13.0
json5==0.13.0
jsonschema==4.26.0
jsonschema-specifications==2025.9.1
litellm==1.81.10
markdown-it-py==4.0.0
MarkupSafe==3.0.3
mccabe==0.7.0
mdurl==0.1.2
mixpanel==5.0.0
mslex==1.3.0
multidict==6.7.1
networkx==3.4.2
numpy==1.26.4
openai==2.20.0
orjson==3.11.7
oslex==0.1.3
packaging==26.0
pathspec==1.0.4
pexpect==4.9.0
pillow==12.1.1
posthog==7.8.6
prompt_toolkit==3.0.52
propcache==0.4.1
psutil==7.2.2
ptyprocess==0.7.0
pycodestyle==2.14.0
pycparser==3.0
pydantic==2.12.5
pydantic_core==2.41.5
pydub==0.25.1
pyflakes==3.4.0
Pygments==2.19.2
pypandoc==1.16.2
pyperclip==1.11.0
python-dateutil==2.9.0.post0
python-dotenv==1.2.1
PyYAML==6.0.3
referencing==0.37.0
regex==2026.1.15
requests==2.32.5
rich==14.3.2
rpds-py==0.30.0
scipy==1.15.3
shellingham==1.5.4
shtab==1.8.0
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
socksio==1.0.0
sounddevice==0.5.5
soundfile==0.13.1
soupsieve==2.8.3
starlette==0.52.1
tiktoken==0.12.0
tokenizers==0.22.2
tqdm==4.67.3
tree-sitter==0.25.2
tree-sitter-c-sharp==0.23.1
tree-sitter-embedded-template==0.25.0
tree-sitter-language-pack==0.13.0
tree-sitter-yaml==0.7.2
typer==0.23.0
typer-slim==0.23.0
typing-inspection==0.4.2
typing_extensions==4.15.0
urllib3==2.6.3
watchfiles==1.1.1
wcwidth==0.6.0
yarl==1.22.0
zipp==3.23.0
Build and run:
podman build -t aider .
mkdir desk
podman unshare chown 1001:0 desk
podman unshare chmod 770 desk
podman run --rm -it --env OLLAMA_MODEL='qwen3.5:9b' --name=aider_alfa --volume ./desk/:/opt/aider_ai:rw,Z aider
It has a few glitches, mainly I don’t even use newer pip I pull in Dockerfile, but it can be done. Have fun!