I mean if you want to use some kind of LLM I think it’s far easier to just use llamafiles. - Pre-built llamafiles | llamafile | Mozilla.ai Docs
All you do is download a model and then run it like an executable. Gives you access on the CLI and/or via a webpage (runs on port 8080 or something by default IIRC.)
Sample systemd file you can proxy to with nginx, etc.
/etc/systemd/system/llamafile.service
[Unit]
Description=Llamafile Service
[Service]
Type=simple
User=llama
Group=llama
ExecStart=/bin/sh /opt/llama/llamafile/llava-v1.6-mistral-7b-Q4_K_M.llamafile --port 9998 --host 0.0.0.0 --server
Restart=on-failure
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target