Hi.
Log Detective, the project I’m working on, is providing LLM based analysis of failed scratch RPM builds, as well as builds in Copr, or other systems. We’ve been maintaining our own inference servers, but it is becoming increasingly clear, that while it is an interesting experience, it doesn’t scale very well.
I have been wondering if there are other, similar projects, serving community, what kind of solutions they have implemented and if they would be interested in a common approach.
It’s not very official projects, but I’ve been building two tools that use LLMs for the Fedora community recently:
Right now they’re using Google Gemini but I’ve been looking for ways to use a more open LLM (ideally Olmo from Ai2). If there was a way to run the LLM queries from within the Fedora Infrastructure, I think they would be better accepted by the community. So I’m interested in getting that working as well!
Bedrock offers relatively wide selection of models. I don’t see Olmo in the list, but I do suspect that it will get included, if it appears popular enough.
What sort of model size is needed for Log Detective to work well? What do your existing inference servers look like? I’m curious about whether @abompard has tried smaller models with his apps, as well.
My team at Red Hat is interested in helping to serve apps like these in more community-accessible/controllable infra, probably using nodes at AWS, though, so there are some model size constraints.
We are using several vllm instances with nginx for load balancer. Each with gemma-4-E4B-it model loaded. Due to memory constraints we can’t have the full context, but it doesn’t really matter for us, since the full run rarely goes over 20000 tokens. Previously we were using gpt-oss-20b. That actually had better results, but the harness we are using wasn’t doing well with the harmony output format.