Fedora AI Developer Desktop Objective

Project Discussion
1

The Fedora AI Developer Desktop Initiative aims to build a thriving community around AI technologies by focusing on three key areas: equipping developers with the necessary platforms, libraries, and frameworks; ensuring users experience painless deployment and usage of AI applications; and establishing a space to showcase the work being done on Fedora, connecting developers with a wider audience.

Community building and effective outreach are skilled endeavors, often distinct from a developer’s core expertise. This initiative will bridge this gap, helping developers find an audience for their work and ensuring that their users can easily run and engage with their applications.

The baseline outputs of this objective are expected to work on all platforms, from ARM, AMD, Intel, and NVIDIA accelerators. We also hope to provide hardware optimized experiences that go beyond the baseline outputs.

In short, the objective will focus on developer tools, community building, and platforms and packaging.

Non-goals:

The system image will not be pre-configured with applications that inspect or monitor how users interact with the system or otherwise place user privacy at risk.

Tools and applications included in the AI Desktop will not be pre-configured to connect to remote AI services.

AI tools will not be added to Fedora’s existing system images, Editions, etc, by the AI Desktop initiative.

Current Issues

Complex initial setup turns away prospective users

Fedora provides applications and an operating system on which to run them. A great deal of the work we do to package applications is ensuring that they require minimal post-install configuration in order to be usable. This isn’t always the case with AI tooling, and local setup complexity varies significantly across hardware vendors. The ideal operating system is built and tested by the Fedora Project, and deployed on users’ hosts; we would like to minimize the amount of build and configuration that happens at the deployed system, and therefore the amount of expertise required to install, configure, and use AI tools and workflows.

Technical

The purpose of the stable release process is to allow users to test a new release and upgrade on asynchronous schedules, by continuing to support previous releases.

Fedora provides a stable release for most software, but provides a rolling-release kernel. For mature hardware that requires minimal testing, that’s fine, but the AI space is one example of the challenges of working with rolling release software. Both out of tree kernel software and user-space components can be impacted by the changes typical of a kernel minor release.

Reliable systems generally follow a pattern of build, test, deploy, but many users today are using software that requires a deploy, build, test pattern. Users frequently disrupt the build because it doesn’t provide enough feedback, and if an update doesn’t work properly there may not be an obvious rollback process.

The post-install build system is also challenging for Atomic systems, and at odds with the fundamental motivation for those systems. For the security-minded, the status quo supports Secure Boot only through the use of a local signing key, which offers only the illusion of security, since malware can use the officially supported module install infrastructure to install itself.

Logistical

Many applications lack guides for Fedora, in part because the fact that the OS requires configuration that varies from hardware vendor to vendor means that concerns cannot be effectively separated. Application vendors cannot take for granted a working OS. They may see the need for documentation that varies across distributions, and further across hardware vendors, and choose to focus on the systems that require the least variation in configuration before the user reaches the point of being able to install and use their application.

Legal / Policy

Fedora is dedicated to Free and Open Source Software. We build and distribute software that users are free to modify to suit their needs and to share with others. While Fedora can’t deliver non-Free software directly, we can build relationships with developers who want to provide users with complete systems that include their software. Those partnerships can expand the community to include users and developers who rely on low-level APIs that are not Free Software.

Deliverables

A number of changes would produce an operating system image that would improve Fedora as a platform for AI software. This work may include:

1. Build an LTS kernel to deliver the benefit of a stable release process consistently, across the release.

2. Build and sign the NVIDIA out-of-tree kmod, OpenRM (until the Nova driver is ready), to eliminate the need for signing keys on the device where the keys are used, to provide a standard best practice build-test-deploy sequence, and to enable an Atomic system to support NVIDIA hardware.

3. Publish an Atomic system image focused on support for accelerated AI workloads, as a Fedora Spin.

3.1 Publish a variant of that Atomic system image with CUDA runtime support, as a Fedora Remix.

4. Publish an Atomic system that supports the CUDA toolkit. (If Fedora cannot distribute this image due to license or policy issues that we can’t resolve, I’d like to ask NVIDIA if they would publish the image we build.) This layered image will be a Fedora Remix.

5. Include user-friendly tools common to various AI back-ends, such as Goose CLI and Podman Desktop.

6. Provide a contribution guide and invite community developers to directly improve the image rather than writing post-installation configuration guides.

7. Provide a space for AI software developers to showcase their work, provide short quick-start guides, and help users find applications that solve their problems and projects that they can contribute to.

8. Publish optimized container images for machine learning applications, aimed at AI developers.

Timeframe

F45 release: Platform work, 1-5
F46 release: Community work, 6-7
F47 release: Developer tools, 8

Outreach

Fulfillment of this initiative will require outreach both inside Fedora and beyond. Specifically contemplated contacts include:

  • FESCo: Revisit policies that prohibit the option of a stable Fedora kernel, as well as identifying developers that would be interested and capable of supporting the stable kernel.

  • Universal Blue: How do we work together? Can we make their maintenance easier by upstreaming portions of their work?

  • NVIDIA: Is there an interest in publishing Fedora’s image with CUDA Toolkit that only they can redistribute?

  • AMD/ARM/Intel: Provide first-class acceleration where there is upstream support

Interested People

Máirín Duffy

Objective Lead

Gordon Messmer gmessmer@redhat.com

History

Notes

A preview of the desktop Remix is here: https://quay.io/repository/gordonmessmer/atomic-desktop/silverblue

The ostree config for that remix is here: https://pagure.io/fork/gordonmessmer/workstation-ostree-config/tree/f43-longterm-plus-cuda

The stable kernel with nvidia module used in that remix is here: https://copr.fedorainfracloud.org/coprs/gordonmessmer/kernel-longterm-6.12-plus/

4 Likes
Community Update - Week 14 2026
2

@gordonmessmer Clarifying, are you seeking open feedback or are you actually proposing this as a formal, new, Initiative for 2026? If the latter, this needs to be opened as a Fedora Council ticket so we can work it into an upcoming meeting agenda. We do not triage Fedora Discussion topics for meetings, although it is easier for longer-form discussion. So the ticket may ultimately point here for the details.

2 Likes
3

According to the initiatives docs site “the Lead should first discuss on the #council tag on Fedora Discussion. Next, if well-received, file a ticket.”

1 Like
4

This feels more like a fully-baked proposal than an open question or discussion. :wink:

1 Like
5

It’s proposal-shaped largely because I am habitually formal.

4 Likes
6

Overall I like the idea and the proposed plan!

This makes sense as, to move faster at a higher level, the lower levels need to have less disruptive changes. Would this LTS be specifically for AI Developer Desktop or would it be an option for traditional and atomic desktops too? What I worry about is the maintenance factor on backports or testing. The wider the net the more help, but also the greater the surface area and want/need for changes for use cases.

1 Like
7

Fair enough! There is a lot of sensitivity around AI anyways, so being intentional and deliberate seems wise.

Are you in a time crunch for this? I think what will help the Fedora Council here is driving toward a deadline where this ends up as a ticket in our ticket queue. We can do informal review and feedback here in the interim before the ticket gets opened, but it will likely take some cat-herding to do. So, an ideal deadline for official submission would be helpful feedback! (I don’t want to see this topic get lost or buried in the noise either.)

Edit: I am super overloaded this week on meetings, Flock, and Outreachy, but I will try and carve out some cycles to mentally process what is here and give some thoughtful feedback.

8

Finally, glad to see this being proposed!
Personally, this aligns with me, and I would like to be involved with this objective sooner rather than later.

9

Did NVIDIA relicense their driver and CUDA to an open source license?

Also, it’s not clear how a “stable” kernel would improve the AI experience. The community seems fairly effective in maintaining the existing driver stack against the kernels Fedora ships today.

1 Like
10

I have a deep-seated bias toward supporting collaboration. One of the ways we make collaboration easier is through the stable release process, which allows teams to work asynchronously. I expect a stable kernel to benefit people in many audiences:

  • In the general case, I often hear people complain about hardware support regressions after kernel feature upgrades. I myself was unable to use Fedora kernels 6.15+ on Fedora 42 because my trackpad wasn’t usable on newer kernels. A stable kernel ensures security patches to users affected by regressions.
  • In the case of systems that use out-of-tree modules (including NVIDIA’s in the context of this proposal, but also things like VirtualBox or ZFS), a stable kernel provides support for the use of third-party kernel modules while they are ported to new kernel interfaces.

An additional kernel would present greater QE overhead. I don’t want to deny that. But I do think that the testing process around Fedora kernels today has serious flaws. The rolling kernel necessitates testing days mid-release, which does not align well with the concept of a stable release. But more importantly, even if users participate in testing days and report regressions, there just isn’t any realistic alternative to shipping the new release series as an update. By the time Fedora prepares a new kernel release series and organizes a test day, support for the previous release series has ended upstream. We can take bug reports regarding regressions, but there isn’t much we can do with them.

2 Likes
11

There has been an open source kernel driver for several years: https://developer.nvidia.com/blog/nvidia-transitions-fully-towards-open-source-gpu-kernel-modules/

We don’t build it because it’s out-of-tree, and there is a realistic risk that support for new kernel interfaces could lag behind the rolling kernel release, which might delay the release of new kernel packages.

That’s perfectly reasonable in the context of Fedora’s policies, which require only one kernel package: the rolling release kernel.

Building the NVIDIA driver, or any other open source but out-of-tree driver, is infeasible without a stable kernel.

12

The open-source kernel driver still depends on closed-source userspace components though, doesn’t it?

13

Yes, sorry… I got distracted and left the last reply incomplete.

CUDA runtime is not open source, but it is redistributable. A desktop image including the CUDA runtime is intended to be a Fedora Remix.

A preview of that Remix is here: Quay

1 Like
14

Would you please provide the sources used to build the container image?

15

The ostree configs are: Making sure you're not a bot!

The kernel source and binary builds are: https://copr.fedorainfracloud.org/coprs/gordonmessmer/kernel-longterm-6.12-plus/

1 Like
16

I’m somewhat skeptical of the validity of this objective as a distinct artifact separate from our existing deliverables. “AI developer” in itself isn’t useful unless it’s clarified with a target subtype. There are other further decompositions that are important: are we talking about input refinement (data collection, training synthesis, etc.) or outcome oriented (model synthesis, RAG, etc.)? These are completely different.

Ehh…

I don’t think this is a useful change. Firstly, LTS kernels are almost exclusively used for development of components that aren’t integrated or supported upstream in any meaningful way. Not to mention, the complexity of supporting alternative kernels in Fedora software and update infrastructure is significant. And finally, it screws over our ability to coherently identify hardware support.

I should note that this doesn’t require (1) either. The OpenRM module works just fine with the latest kernel code. There are much more significant problems with kernel module packages (KMPs):

  • The KMP infrastructure in Fedora is in very bad shape relative to other RPM distributions ever since we abandoned kmods 20 years ago.
  • We don’t have tooling to do automatic rebuilds to chain KMPs to new kernel updates.

As a point of comparison, the SUSE folks have all this worked out and consequently are able to ship things like that much more easily. They have first-class support and automation because they’ve always been automation-centric whereas Red Hat/Fedora has historically been manpower-centric.

Also, I don’t think it’s a good idea for us to provide a driver that is effectively useless out of the box. We’re not distributing the required userspace components ourselves, and nobody bothered to hook up Mesa to OpenRM on Linux. That’s not to say this stuff shouldn’t be fixed: I think it should for the various community developed out of tree kernel modules that do work with fully open source stacks.

Finally, there is another issue changing our policy to support and endorse out of tree kmods in Fedora as KMPs: the likelihood our user’s systems will be considered tainted and ineligible for support from upstream kernel developers goes up significantly. Kernel developers prefer Fedora and Fedora users over others because of the existing state of things.

Ehh, okay? Not sure why it should be atomic, but whatever.

Are we equating AI = CUDA now? Because I think generally Fedora and Fedora affiliated initiatives should not do that. Regardless of the value proposition of the current crop of AI things, I would hope our goal should be to encourage full stack OSS for this. Of course, this is proposed as a Remix, so it strictly doesn’t matter here, but I definitely would not want this to be Fedora-endorsed because it sends a dark signal that we don’t care enough to push for open source driven AI technology stacks.

This cannot be resolved against the philosophy of Fedora contributors. I don’t really care about the legal issues, because having worked through them for another distribution, I know they are resolvable. But philosophically I feel this is against Fedora’s own mission. I would really not want this to be done.

First get Podman Desktop in Fedora. :wink:

I have mixed feelings about this. A downside of this is that we’re not teaching people how to own the computing experience and have local ownership of their system. We’re also reinforcing opinionated setups over adaptable ones. An “appliance” workstation environment feels antithetical to me. But I know that there’s a lot of people really wanting this, so sure. But this also feels like Remix territory rather than Fedora territory.

With my FESCo hat on, I would signal pretty strongly against this. I don’t think there’s enough “meat” on those bones.

I also think your rationale for this is rather weak, since it isn’t even needed for OpenRM, and since you intend to go Atomic for your spin, you especially don’t need it since atomic images can choose to compose using the updates-archive repository and leverage cherry-picking of updates as they so please.

That image would need to drop the Fedora branding per our guidelines. I personally would be unhappy about a Fedora cobranded image with this given our philosophy as a project.

We are already doing this? There’s an AI/ML SIG, NeuroFedora, and other groups already focusing and iterating on that.

If I just scope this to the stuff I think we should do for this, it’s probably achievable in one Fedora release as a Change, since you’re basically making an opinionated spin that preloads a particular collection of software. The main gap is Podman Desktop isn’t in Fedora and that needs to be resolved. Adding Electron to Fedora is not much worse than the effort it took to add CEF, but someone has to be willing to keep up with it afterward. :upside_down_face:

One of the problems Fedora has right now is that its primary sponsor and major employer of kernel hackers does not generally allocate their contributors to participate in Fedora. They don’t review Fedora bugs, they don’t assist users, they don’t help much with the Fedora kernel in general. It is telling that ARK and the kernel package actually define Fedora as the special case, rather than RHEL being the special case as the subset downstream. :frowning:

As a counterpoint, Btrfs in Fedora is well-supported because the Btrfs SIG has upstream Btrfs developers as part of the team and actively does work on that. When reports come in, they are triaged and responded to either within Fedora or upstream. The fact that we ship the latest stable kernels means that those fixes are delivered quickly too.

Nearly all of the other subsystems in the Linux kernel in Fedora simply do not have anywhere close to that kind of support. Not in power and memory management (over a decade of nobody fixing hibernate with lockdown!), not in block and other filesystems, not in graphics, not in networking, and not in anything else.

Before we start talking about adding kernels, we need people to care about the stuff people report now.

4 Likes
17

Sure, I agree with you. But incremental problem solving is good. Think of it as a feature branch. It doesn’t seem like much when it’s first created, but it’s a place to work on the feature independent of the other branches. It is a space in which to work in semi-isolation to solve problems that you might not be able to solve directly in a production branch.

Work on this class of systems is happening now, outside of Fedora. I’d like to create a space for that work to happen where the intent is to upstream to Fedora things that are appropriate, rather than drifting further away.

That’s true today, but there is no guarantee that will be the case at any given point in time, which is the reason I’ve been given that the OpenRM module will not be built in Fedora’s kernel package.

I think the kernel maintainers are being perfectly reasonable to reject the out-of-tree module on the grounds that it could create conditions that would prevent upgrading to a new release series. But the need for a stable kernel is a natural consequence of that position.

I agree that Fedora should give significantly more regular maintenance work to the automation tools we use today, but from my point of view that’s outside the scope of this thread.

It’s incomplete, but I don’t think that’s the same as “useless.” It supports and enables the user-space bits. It allows users to install and run software, even if Fedora can’t pre-install that software.

Supporting the software that user want to run is not useless.

No, we are not. As I said in the proposal, some of the work we need to do is engineering work (packaging and platforms), and some of the work we need to do is community building.

NVIDIA gets mentioned a lot here because there are specific engineering tasks needed to close a gap in hardware support across major vendors.

For other platforms, where we’ve gotten more active support or better aligned support from vendors, we have the luxury of doing community building and promotional work.

I’m working on it.

I don’t know if I’ve phrased something badly, but that’s really specifically what I want to do. My goal is always to tell users how to participate, and give them entry points to do so.

Use of the updates-archive is not an alternative to a stable kernel. When there are hardware support regressions, or when there are interface changes that affect third-party kernel modules, the only option within Fedora is to stop updating the kernel.

A stable kernel package would continue to deliver security patches and serious bug fixes to users who can’t rebase to a new kernel series because of regressions or software support.

I think I agree with you on the facts, throughout your reply and on this point specifically. I just don’t come to all of the same conclusions.

RHEL is a product. Red Hat determines the intended use cases to fit the markets they want to participate in. They develop the product to support those use cases.

Fedora is not a product. The software we provide is unsupported, which is to say that we are not accountable to users. If the thing we build the publish works, they may use it, and if it doesn’t work, they can maybe participate and try to fix it or they can use something else. But there’s no mechanism for them to hold us accountable otherwise. Unlike Red Hat’s model, in which they are offering continued maintenance of components independent of upstream, Fedora does significantly less in-distro development, and is mostly doing distribution.

We might agree that Red Hat isn’t allocating developers to do significant development of the Fedora kernel, but I don’t think it’s reasonable to expect them to do that, nor do I think that a rolling kernel is structured to enable them to do that.

The purpose of a stable release system is that it allows groups to collaborate asynchronously. It allows the developers of a product to publish a release series to get a feature set in to the hands of interested users. The groups of users for whom that series works can upgrade immediately, and the groups for whom it doesn’t work can work on porting and bug fixing until it does. The rolling kernel timelines don’t overlap enough for that to be realistic. Regressions and porting cannot reasonably be addressed within the overlap. Users would suffer regressions, even if more developers were actively working on bug reports against the Fedora kernel. In other words, even if developers were addressing bug reports, we would need more overlap in maintenance windows AND we’d need a mechanism by which users could upgrade from one kernel release to the next on their own schedule, rather than shipping one rolling kernel package.

…which is exactly why a stable kernel that spans a release would benefit many users. I’m actually quite surprised to see anyone argue simultaneously that there are not enough developer resources assigned to the rolling kernel release and that the stable kernel isn’t useful or desirable. Those seem like contradictory points of view, to me. The latter is the solution to the former.

1 Like
18

Are LLMs meant to be included in this project?

If yes, since legal and policy questions are apparently part of this proposal, was there ever any discussion about how LLMs seem to have unresolved significant moral and ethical issues due to apparently not respecting FOSS licenses of the training data, yet apparently regularly they spit out the training data it out as-is?

Lawyer demoing Co-Pilot, he says “This is a copyright infringement” at some point: Consider not allowing LLM and AI contributions · Issue #38072 · mastodon/mastodon · GitHub

German court decision that seems to question fair use: Landmark ruling of the Munich Regional Court (GEMA v OpenAI) on copyright and AI training - Bird & Bird (I could be wrong, check it yourself and draw your own conclusion.)

Field study that seems to suggest up to %5 plagiarism rate in regular use: https://dl.acm.org/doi/10.1145/3543507.3583199

High-profile incident of Microsoft apparently accidentally plagiarizing despite not intending to do so: Microsoft uses plagiarized AI slop flowchart to explain how Github works, removes it after original creator calls it out: 'Careless, blatantly amateuristic, and lacking any ambition, to put it gently' | PC Gamer

Study suggesting high performance of models corresponds to high amount of plagiarism: An evaluation on large language model outputs: Discourse and memorization - ScienceDirect

This article puts the plagiarism at around 10% and claims even latest stopgaps to make the models not accidentally do that don’t work: AI's Memorization Crisis - The Atlantic

I’m not a lawyer, this isn’t legal advice. I’m merely pointing out experts seem to think LLM use is under a lot of unresolved scrutiny.

If this AI desktop is meant to include LLMs for code, or to enable those, I think this raises questions about Fedora’s position regarding all of this. Also, if there are any plagiarism mitigation tools I guess that would also be helpful, although the common position seems to be (especially if you look at the last article I linked from The Atlantic) that no mitigations currently really work.

If this was ever discussed before or this is too off-topic, then I apologize.

1 Like
19

Wearing my FESCo hat on, I am questioning the staffing level needed to support this and if anyone will be assigned to it / if there are enough volunteers.

The way we currently maintain and test kernels is not ideal, and while LTS will sidestep some of the issues it also introduces others. Is there a cross-distribution interest in supporting LTS kernels on the desktop (as opposed to on servers, on Android etc.)? Perhaps a joint effort to qualify a given kernel plus some of the low-level userspace components (Mesa etc.) could help, separate from this.

Wearing my “I work in a company that has an ongoing effort to offer Linux as a desktop choice” hat, yes, the hardware regression on new kernels is annoying, especially since we already buy ThinkPads! But this leads back to the previous question - maybe hardware qualification should be a joint effort as clearly even being on the leading edge by the time we test and find issues it’s way too late.

As for the messaging around AI… those who knows me know I am very conflicted on this, so I don’t really want to contribute to that discussion.

1 Like
20

Thanks @gordonmessmer for putting this together and starting the conversation.

I think there is a lot of overlap with what the Ublue community has been doing, in particular GitHub - ublue-os/akmods: A caching layer for pre-built Fedora akmod RPMs · GitHub. I wonder if there isn’t an opportunity to join forces here and consolidate on this work. Have you looked at what they are doing?

IMO this brings some benefits to both projects:

  1. Currently akmods is using @kwizart’s LTS build Making sure you're not a bot! , there might be a good opportunity to consolidate the maintenance effort for an LTS build.

  2. akmods builds ~15 kernel modules daily across 6 kernel and 2 architectures using GitHub actions. There is probably a lot of learning to get from the automation angle and reducing the manpower-centric effort Neal has described.
    There might be an opportunity for a shared build service or shared infra.

  3. This could be an opportunity for Fedora to provide a module signing service, so that both projects could share a trusted key chain, eliminating the need for users to enroll multiple keys.

Personally, I would love to see some collaboration happening around these topics.

1 Like