@andreboscatto and I would like to start a conversation about the use of Artificial Intelligence tools within the Fedora community—specifically for code quality analysis, code review assistance, and related evaluation tasks.
Why this matters:
AI-based tools have rapidly evolved and are increasingly used to support code quality, identify bugs, suggest improvements, and even assist with documentation. However, their usage also brings up important questions around trust, transparency, licensing, and alignment with Fedora’s values and open source principles.
Let’s open a community-wide discussion with the goal of reaching a shared understanding, and ideally a consensus on:
Which types of AI tools, if any, are acceptable in Fedora workflows, such as patch review, commit message suggestions, and static analysis?
What criteria should we use to evaluate and approve AI tools, including open-source licensing, reproducibility, data handling, and model transparency? How do we ensure any AI tools align with Fedora’s commitment to openness and community collaboration?
What does open source truly mean in the context of AI, Large Language Models (LLMs), and models? We need to establish standards to judge whether a model is truly open.
What AI tools are available, and are there open-source options we should consider? Are there existing guidelines or precedents in Fedora for AI-assisted tooling?
Looking forward to hearing your thoughts and building something together!
After publishing this post I came across the talk you gave last year at flock to fedora that showed the initial results of the fedora AI survey. I think the full results of this survey might answer some of the questions we asked. Have you published them somewhere?
We’d also like to get this opportunity to foster the discussion with the community. Do you think that makes sense?
I think AI tools can be potentially useful for review/analysis purposes. I’m not sure I’d want any directly suggesting code at this point, based on various evaluations of their capabilities and the legal questions around the ownership of the resulting code.
If they’re only, essentially, commenting on pull requests - not directly involved in the writing or building process - I don’t think we have to be super strict on F/OSS principles. Plenty of F/OSS projects, for instance, use codecov. But obviously open source models (however you define that, which is a tricky point at the moment…) would be best.
I think there’s a practical question regarding what’s actually useful. @martinpitt and his team tried out Sourcery and GitHub Copilot and the results were not great. I think something fairly focused like Log Detective for crash analysis might be useful, but I’m not super enthusiastic to get the kind of attempted PR review feedback Martin found.
I am afraid of using AI-based tools for code quality beyond some reasons.
We have already seen Curl complains about pointless AI “security reports”.
We already have some quite nasty false positives found by non-AI tools, and sometimes they are beyond human’s analysis and require significant efforts to evaluate. I don’t see a good solution for this anyway.
It is not a novel observation, but many LLMs are not really open. There are, however, efforts to evaluate that and define what “open” means. The other important consideration is that a LLM may contain inherent and invisible bias based on the training data used.
Inherently, a lot of development happens in the upstream as it should, and so choices made here and discussion here are only a subset of that. I think everyone knows that but it’s important to state. (What choices we make here can influence FOSS in general of course!)
Also related to that I think criteria should at least reference those defined by other organizations
I do personally use some proprietary tools, including proprietary AI models in some cases, but I also try to keep up to date on what’s happening in FOSS and actively seek to use it too.