Fedora AI Developer Desktop Initiative (updated)

Project Discussion
62

First,
It is my opinion that any technical work that would be expected to need to go through the ChangeProposal process that happens within the scope of any initiative. Technical work is_intended_ to go through a Change Proposal process. As I have said elsewhere, I think that needs to be made explicit in the initiative documentation. There is a lot of confusion over that.

Remixes are a wide space. Part of that space is a grey space for highly aligned things that don’t quite fit. Part of the work in that aligned grey space, is trying to move in a trajectory that minimizes the scope of policy exceptions that keeps it as a remix.

Lastly, no matter what process exists, there will be grey areas and there will be governance that is stood up to consider policy exceptions. Whether that be trademark policy as controlled by Council or technical policy exceptional processes used by FESCo or the packaging policy exceptions process used by the packaging committee. Each governance body that works at project scale as oversight has a tool set that lets them work in the grey and grant exceptions to policy they have oversight over. I don’t know if you are as new to the community as your FAS account suggests, with a creation date of May 18th 2026, but assuming your are… you need to understand that governance at multiple levels has a set of tools to accommodate exceptions and policy revisions. You have to be okay with exceptions and revisions to policy as a governance process or your involvement will be overly frustrating.

Every exception request is a discussion about tradeoffs. Exception requests, may even end up leading to documented policy changes as recognition that a policy created 20 years ago no longer fits as well as it did before. The changes to packaging policy with regard to golang bundling is a great example of policy exception that gets encoded as policy. golang challenges bundling packaging policy originally created for a lived experience packaging C shared libraries, and the tradeoffs inherent in that. That policy choice in the early days of the project, makes golang application packaging in Fedora cumbersome, and that policy has been revised to better suit the needs of the people doing the work to bring golang application to Fedora. Having been associated with the the development of a golang application in a previous paid gig, I’m personally thrilled at that policy exception for golang. The bundling policy was a significant reason why i didn’t even attempt to contribute golang applications to the project during that period when I was neck deep in golang ecosystem.

1 Like
63

Since vLLM has come up a couple of times and I maintain pytorch, my 2 cents.

PyTorch is a package and an ecosystem of related packages. Fedora’s pytorch package is relatively recent and has only the ROCm accelerated backend. Though has cpu modes for x86 and aarch64, these are in practice so slow that pytorch is not usable. If someone wants to add an accelerated backend, please bring this up in the ai/ml sig. The Fedora pytorch ecosystem is very modest, enough to run some examples. The limiter is adding packages to the ecosystem.

Chasing the latest python version is always a challenge, there are parts in the pytorch package that check the version and fail ungracefully. If Fedora had an n-1 version of python, I would be interested in moving to it. I would not be interested in having both a n-1 and n version.

vLLM is one of those ecosystem packages. The version of python is not the issue, I have looked at its dependence chain a while ago, it is wide and deep. If folks want vLLM, I would recommend bringing it up with the ai/ml sig and see there is interest.

‘AI’ as a term is vague. Within the context of pytorch, I think AI means the AI training usecase. Is AI training really something for Fedora ? Training to me assumes a datacenter, not a laptop. I would like AI in Fedora to be for the common user who wants AI working out of the box in their favorite apps. Could we be clear which usecases this initiative focuses on ?

1 Like
64

If we want part of getting to the most ethical available future.. we may have to provide a framework that stands up both inference and training as use cases that Fedora plays a role in. We may find that operational training workload is better suited for slower moving community efforts. We may find that in practise some of this stuff ends up being primarily consumed as EPEL branches for example. Or we may find that training workloads want to move faster than that. But at the very least, I fully expect home-labbers will be interested in training as a use case as a way to gain experience and understanding as part of the journey for the ecosystem as a whole to control and shape where this technology is going using hardware they control.

65

I’ve offered vLLM/PyTorch/Python as one example of a class of problems, but this is not a Python problem exclusively.

I’ll offer another example, and one that doesn’t have a simple solution: Qt. OBS Studio was broken in Fedora for a while because it was linked against a version of Qt whose behavior had changed before the OBS Studio developers had a chance to resolve that issue. Since ELF shared objects are linked against a soname with only a major version, there really isn’t any way to provide OBS Studio with a compatible version of Qt in the distribution. (Someone will probably object to this example because Qt Group doesn’t publish patch releases for Qt after a new minor release series begins, but I do see them publish patches for security issues in the same location they publish patch releases. For example, Index of /archive/qt/6.9 contains five CVE patches published after the release of 6.9.3)

I don’t think this problem is resolved, for many reasons. “Talk to us about their needs” means this is the dependency owner’s responsibility, and I don’t think that’s fair to the dependency owner. It also means that if the dependency owner declines to provide what an application needs, then the application maintainer is simply out of luck. Many packages today contain patches or commands that modify dependency specifications in order to align stated dependencies with the distribution’s set, and sometimes that works but in other cases it doesn’t and it creates bug reports upstream due to an untested composition of components.

And again, these are examples of a class of problems. The examples are not the problem per se. Adam suggested that some software shouldn’t be packaged by distributions and should be distributed in other ways. While this class of problems remains unresolved, we remain in a situation where perhaps people would like to build software on Fedora but they end up somewhere else because it’s simply too difficult to do here. And that software is going to include a lot of what we’d describe as “non-trivial software, especially in emerging technology fields”, including AI related software generally.

Because “First” is a foundation of Fedora, surely we should focus on the needs of emerging technology developers.

66

I am playing devils advocate to make the point we should articulate what usecases are important focus the discussion otherwise we will be boiling the ocean.

Let me start.

I am interested in Text-To-Speech for accessibility, so I have packaged up the python-piper-tts package. But to be usable out of the box, the models need to also be packaged. In the spirit of if we ship it we build it, it would be reasonable to have a requirement that models we ship in Fedora be built with Fedora. A rough break down of what we need is AI/ML Special Interest Group - Fedora Project Wiki a mix a pytorch ecosystem and tts packages.

This distills to a usecase/goal of

Fedora requires AI training of models shipped in Fedora to be trained in Fedora with appropriately free and open source material.

We do not need this initiative to do this, it just an example of concrete usecase of some outcome someone wants.

If we can not articulate what AI things we want to do, then downscope this initiative and remove AI from it. I would rather have a clear view on a smaller set.

67

Well, “we” - the Fedora folks who are interested in Python stuff - definitely don’t think this. We (primarily @churchyard ) spend several weeks/months around every major Python release doing rebuilds of everything Python-y packaged in Fedora, seeing what fails, and submitting fixes upstream. Quite a lot of porting things to new Python releases is done by us, because we front-run Python versions like this.

The relevance to your topic: it seems to me an alternative to “to package this stack for Fedora, we must accept its upstream’s insistence that it needs to run on an outdated Python” would be “to package this stack for Fedora, we will commit to porting it to new Python versions and submitting those fixes upstream”. After all, this is a thing we already do for quite a lot of other Python bits.

2 Likes
68

The risk if we keep piling up examples, though, is the “spinning out of control” that Miro referred to. OK, so we add Python 3.12! We add an old Qt! Where do we stop? At what point does our project stop being an operating system and start being a bunch of bits you have to try and build your own functional operating system out of, and if you plug the wrong bits together we say it’s your fault?

Who has to maintain the bits? If we add an old Qt for OBS Studio is that the OBS Studio packager’s responsibility? But they’re probably not a Qt packaging expert. Is it the Qt packager’s responsibility? But they might not want to maintain five versions of Qt for different downstreams of it which all decided they want to pin their own perfect version because updating is a drag…

69

Almost sounds like Fedora is being requested to reintroduce modularity (or RHEL Application Streams), or something like that.

70

And as the OBS Studio maintainer in Fedora, I will point out that I am aware of Qt and know it quite well. The issues they complained about weren’t issues in Fedora’s Qt build because we actively fixed and backported stuff both in OBS (sometimes) and Qt (more often).

As a general rule, we try to adapt software for newer stuff and contribute that work back upstream wherever it may be relevant. I don’t want Fedora shipping old stuff just because we can, I’d rather maintain that friction so that our packages are incentivized to fix things as Fedora itself evolves.

AI tooling is not special in that regard, it’s just the latest line of upstream developers who lack the ability or desire to practice good hygiene with their software stack.

71

I know that. “We” in this context is a figure of speech. It begins the answer to “I would like to know more about why this thing is difficult” from a place of sympathy.

I love that we do that when it is possible, but if that becomes a requirement for packaging, then we’ve raised the bar for packagers quite a lot.

A hypothetical system that allows package maintainers to build an application using the dependencies that have been tested only really requires a package maintainer to have experience with the build system itself.

A system that requires them to port an application and potentially any number of dependencies when some critical dependency is updated may require a package maintainer to be an expert developer, not only in the project they are packaging, but in the critical updated dependency as well.

That’s a huge difference, and while we certainly have that kind of expertise for many packages and stacks, I think it would be good if that were not the baseline expectation for packagers.

That question doesn’t make sense to me, because we already plug bits together and blame the upstream project when it doesn’t work.

If an application needs a specific version of a dependency, then they should have both the responsibility and the power to compose a build that includes the application and the supported versions of dependencies. Placing that responsibility on the applications that require it is the thing that keeps everything from “spinning out of control”.

But while everything above is trivial, the following point is something I think is very important:

From the perspective of a developer, a dependency release stream that is compatible and still maintained upstream is not “an old” release. Until its maintenance is discontinued, it is one of numerous current releases.

Fedora 43 is not an old release of Fedora. Fedora 43 and Fedora 44 are both current releases.

The idea that there is only one current release and that anything else is an “old” release is a world view in which everything is a rolling release. The fundamental purpose of the stable release model is to allow teams to work on asynchronous schedules. We have at least 50 years of software development literature supporting the common conclusion that trying to synchronize large groups of developers slows them down, and that asynchronous work is the key to moving faster.

Earlier, you suggested that some things “are inherently a bad target for distribution packaging, and should not be packaged by distributions.” And in part I agree. The Fedora RPM package repo is a collection of synchronized releases that we expect users to be able to compose into a working system. In some cases that’s just not going to work well. If OBS Studio can’t roll from Qt release to release reliably, then maybe it shouldn’t be an RPM in the repos. But you also suggested “they should be distributed in other ways,” and I agree with that, too! For example, Fedora could ship OBS Studio as a Flatpak, along with a compatible version of Qt. Isolating the application AND dependencies could allow them to build and ship on asynchronous schedule.

Fedora’s release model requires everything to synchronize with Fedora’s release schedule. Many of the things that make packaging for Fedora difficult are the result of this self-inflicted injury. Fedora’s cadence and maintenance window are good, in that they allow the project to ship most software while the release streams are still actively maintained upstream. But they’re not inherently the right cadence, or maintenance window, or release date for every upstream project.

I think that purely historical data supports that conclusion. But when we look at the rapidly increasing rate of CVE discovery, and when we consider adversarial adoption of AI to deploy exploits, the only reasonable conclusion (IMO) is that release velocity is likely to increase in order to address the security needs of the world today and tomorrow. That increase is probably going to continue to make it even more difficult to synchronize releases in one big set working on one release schedule.

Supporting asynchronous releases isn’t just about supporting “old” software that isn’t keeping up with Fedora, it’s also about supporting rapidly developing software that’s releasing and adopting dependency updates much faster than Fedora does. I don’t think we can reasonably expect to maintain “First” as a foundation in a world where our release schedule is significantly slower than the leading edge of technology.

72

well, we’re getting way off in the weeds here, but I have a potentially idiosyncratic answer to that, which is - much as my job involves lots of stressing about the Fedora “release cycle”, I personally almost never run a stable Fedora. I run Rawhide or Branched Silverblue. It works fine 99% of the time (because we actually test stuff now! It’s not 2014 any more and Rawhide isn’t fundamentally broken all the time), and if it doesn’t, I just roll back to the last snapshot which did.

I know whenever this argument comes up the ‘rolling release’ side always loses, but meh, in my headcanon at least it’s kind of a solved problem and Fedora is a sekrit rolling release…

anyway, to try and give a more general perspective: nothing you say is wrong, but the thing is, this is an area where there is no 100% right or 100% wrong answer. Changing Fedora’s model won’t make things unambiguously better, it’s just changing the tradeoffs. Our current model has benefits and drawbacks. Any other model we can adopt will also have benefits and drawbacks. Does this mean we should never possibly change model? Nah. But I think any proposal to change it has to acknowledge this reality and, rather than just concentrating on the benefits of the new model it’s proposing and saying “and thus we should adopt this new model to gain these benefits!”, say “ok, this new model comes with these benefits and these drawbacks, and we believe that combination is superior to the benefits and drawbacks of the current model”…

this was one of the many issues with Modularity for me, when we proposed and adopted that, we focused too much on “look at the things modularity will let us do!” and didn’t really think about the “…but hang on, what problems might it cause?” side of things…

2 Likes
73

Putting aside persistent social-related issues regarding the initiative title and LLM-assisted software development application.

From the perspective of current server-side software development trends, the most desirable situation is having absolute control of dependencies the software is directly interfacing with (using language-aligned dependency managers like ivy/maven/conan/npm/yarn/cargo/whatever, usually) while minimizing attention requirements of everything else. This is essentially the story of Spring Framework and Enterprise Application Servers in the Java ecosystem. Currently, Spring Boot makes it easy to bundle Servlet-level dependencies (web server) as part of an application archive. Stability is one way to minimize attention requirements of everything else, another one is upstreams and packaging involved not changing observable behavior.

Looking at CUDA variants of vLLM and llama.cpp containers, they both use NVIDIA-supplied base built on top of Ubuntu 22.04/24.04, PyPI to get locked versions of dependencies. Both projects have preferences about required GCC versions; vLLM also wants specific Rust toolchain version. According to NVIDIA CUDA repository, Fedora 43 received support at 2026-03-03. It takes time and effort to rebase one leading edge onto another.

Right now it is possible to grab “nvidia-container-toolkit-base” from NVIDIA Container Toolkit repository, tweak SELinux booleans and run llama.cpp-cuda13 container using podman on Fedora 44 with drivers from RPMFusion, although SELinux is unhappy with pasta attempting to access dri_device_t.

1 Like
74

Speaking of synchronization slowing down work, i’ve been less focused on moving these bits from proof of concept to usable systems while I’ve been discussing the proposal.

With discussion tapering off, I’m going to return to working on these in earnest. Not all of these need to be done in this order:

1: Every bike shed must be painted, and so the Remix work will need a name. I don’t want to spend too much time on it… Once there is a name, we’ll need to create forge orgs. Probably best to get them on GitHub&GitLab&Codeberg. And once there are orgs, we can move dist-git, terraform, ostree config, etc repos.

2: Once a name is chosen, I’ll also tear down the code signing VPC and build a new one with signing certs that contain the project’s identity.

3: Continue work on triggering kernel module builds based on fedora message bus events.

4: (Possibly) set up longterm kernel builds

5: (Possibly) set up a new base layer build.

6: Set up a desktop image build with interesting add-ons.

7: Set up COPRs for package (e.g. podman-desktop)

8: Set up “pages” for contributed guides, including:

  • how to contribute to the project

  • how the project works

  • how to use / eval software using the project’s systems

  • (avoid “how to configure your system” if that work could instead be a contribution to the image configs)

75

Is there any chance you could enable aarch64 builds for this as well? I’d love to use this but as an Asahi user, I’m currently limited to using my slightly out of date copr.

And again, feel free to reach out if you need help with any of the Podman Desktop packaging stuff. Always happy to learn and help :slightly_smiling_face:

76

Sure! Actually, I made some progress toward getting podman-desktop to build offline with targeted system packages (e.g. vite, rollup, esbuild) last week and was planning to ask you and @sergiomb if you were interested in helping wrap that up.

Source for the builds in my COPR should all be at gordonmessmer - Codeberg.org

Pending work includes (among other things):

  • build on aarch64 (new, per your request)
  • update electron to a newer version (@ngompa suggested a specific process for syncing code with chromium, but I can’t find it today)
  • finish stripping ELF binaries from the node_modules directory, which may require more rpm packages

Let’s continue discussion in “issues” in the codeberg repos.

77

As a starting place for future work, I’ve set up:

https://github.com/gordonmessmer/cerulean (which builds ghcr.io/gordonmessmer/cerulean) and

https://github.com/gordonmessmer/cerulean-cuda (which builds ghcr.io/gordonmessmer/cerulean-cuda)

78

You can see the process in the cef package in Fedora.

1 Like
79

While this conversation has stalled, I’ve created spaces on codeberg where people who are interested in collaborating on any of these things can work:

A couple of repos are still skeletons… blank slates while we touch up some processes and make sure everything works together.

6 Likes
80

Hi folks,

The Council have published an announcement that regards this proposal. Please see this post for details Fedora Council Statement on the Future of Community Initiatives and the AI Developer Desktop Proposal.

2 Likes