Fedora Initiative for a Potential AI Ecosystem

Project Discussion
1

Synopsis

It has come to my attention that some individuals within this community desire a distribution, with some extra focus placed on AI.

The various justifications may include the following:

  • Market forces
  • User convenience
  • Establishment of an AI developer community

These factors have pushed Gordon Messmer to publish this initiative, which can then go on to the Fedora Council to be approved, if the community accepts it: Fedora AI Developer Desktop Initiative (updated)

However, I feel that this initiative falls into a gray area under Fedora’s rules. Many community members may put into question the purpose of this proposal. As such, I feel that it is my duty to come up with a better proposal, that the community members are more likely to agree with. I will add that I don’t consider myself a community member, since I have only joined here just recently. However, I think you will still find value in what I have to say.

Fedora AI Ecosystem

I propose to the council what will likely be an excellent opportunity for Fedora to compete on the world stage. In response to certain market forces, and for the strengthening of Fedora’s community, I propose that Fedora places part of their focus on building an AI ecosystem, which would be similar to NVIDIA’s. Many large companies are doing the same: https://www.channelinsider.com/news-and-trends/intel-open-ecosystem/, so wouldn’t Fedora’s community members also wish to figure out what all the AI hype is about, and to conduct a significant amount of market research, while also connecting with developers around the world?

Communication

I believe that Fedora’s primary focus in regards to building this ecosystem, should be building communication channels. Fedora should open channels of communication to individuals who hold the following list of titles, and who work with AI, with the most prioritized title being at the top:

  • Developers
  • Office workers
  • Service workers
  • Trade workers (if applicable)
  • Government workers
  • Executives

Executives are de-prioritized, since they are not very well in tune with process of their respective companies, and are likely unaware of real concerns that their employees may have that are for, or against AI.

Developers are the most prioritized, since they can provide personal anecdotes, and can also contribute directly to the ecosystem’s code, given they have the time. Of course, various workers of all kinds will also provide important anecdotes.

Technology Stack

GPU API

This stack, for one, should have a GPU API associated with it. There are three choices currently (which are open source, and are acceptable under Fedora’s rules):

  • OpenCL
  • OpenGL
  • Vulkan

OpenCL is likely the worst choice, since it is extremely slow to update, as you can see here: OpenCL Cooperative Matrix Extensions Are Here. These were only added recently. As outlined here, Vulkan and OpenGL has had this feature for a few years now.

Cooperative matrix extensions essentially allow for one to use a GPU’s tensor cores, or whatever the equivalent may be on any given hardware.

I can only guess that this is due to there being far more graphics programmers who use the latter two APIs, than there are GPGPU programmers who use OpenCL. If Fedora is able to advocate the faster addition of new extensions to OpenCL, than it will be a much more viable option.

In the case that these three APIs are not sufficient for whatever reason, I have a third API “specification” to propose, which isn’t very formal at the moment. But I hope that it will make GPU hardware more approachable, and less intimidating overall.

Other options are welcome as well.

The necessity of a GPU API is due to the nature of the hardware itself. CPU programs don’t need an API to execute within a vacuum. Instead, they rely on a CPU Instruction Set Architecture (ISA), which defines what the instructions within the program will do when executed.

GPUs also have an ISA, however, they are also an external chip that the CPU must talk to. Certain operations that involve CPU-GPU communication, such as DMA transfers, must be done by a program within kernel space. Otherwise, you risk the security of the computer as a whole.

Software built on top of the GPU API

Fedora has several options for what can be built on top of the GPU API.

  1. Copy NVIDIA

This is the easiest option, although it isn’t likely to yield good results. Everyone is used to using NVIDIA’s stack, so it would be hard to break into the market this way.

  1. Build software based on feedback from above communication initiative

This is the harder option, but it will yield far more results. By establishing communication channels, Fedora will have a direct method of establishing precisely what software they should write.

If you have any other options, please outline them. I’m sure Fedora community members will appreciate it.

User experience

Fedora already has an excellent forum, where users can post about the issues they are experiencing. Support is widely available. If any user has a problem with Fedora’s AI ecosystem, they can simply post on the forum, and get their help there.

It would be excellent if the community keeps this effort. This helps far more people than you know.

Development Process

I’m sure that Fedora has a well established development process already. It isn’t my place to dictate how Fedora should handle this process.

Packaging

I’m sure the packagers can figure out for themselves how this stack will be packaged. If they can’t, then I’m sure the council can.

Conclusion

I believe that now is an excellent opportunity for Fedora, and its community, to compete on the world stage, and to make a name for itself. And I don’t believe Fedora is required to compromise their values to do so. I believe that Fedora is in a unique position, since they have the trust and allegiance of many within FOSS.

I would like to hear the community’s thoughts on this initiative, and what they think the future of Fedora should be. I give you my thanks,

– C. S. Sushi

2

Community Initiatives are for work proposals. There are some interesting thoughts here, but I am not reading this as, “I am here to do the work” but “I see some interesting things, someone should do this.”

Are you raising your hand to do this work, or do you have a team to help you execute this work?

3

I most certainly am capable of doing the work, and I wouldn’t mind participating. I am a SW engineer with some free time, after all, and I’ve worked with GPUs, using their compute & 3D features. I may be a bit slow for some people’s liking however, since I seldom use AI tools to build software.

1 Like
4

I’ll quell some of the confusion some community members may have.

  • This is a development initiative

This means that the primary goal of this initiative is to develop software. I will also be participating directly in the development process, whether as a development leader, or as an advisor & developer.

Lastly, would you like for me to start some kind of GPU engineering SIG? This would help instantiate the stack’s development process, and it will iron out the initial steps.

I don’t know how many currently qualified developers are in this community. As such, I will aid in the training of new developers, if necessary. And, it doesn’t seem like many community members believe the initiative I proposed should be thrown out, since it didn’t immediately get poked at by some of the members.

I would really like to hear the community’s thoughts. In particular, about the idea of building a custom AI/GPU-compute stack from near scratch above the APIs themselves. Do you think this is a daunting task? Do you think that this is feasible?

5

This means that the primary goal of this initiative is to develop software

By software, are you referring to packaging already existing software such as llama.cpp and as such making it available in Fedora so that users can simply do dnf install ai-tool-of-their-choice or do you want to do actual software development and build something that’s an alternative to those existing tools?

With Fedora being a Linux distribution, I’m not sure that doing software development would be the way to go for a Fedora initiative. Thinking back to when I started using Fedora around 6 years ago, I mostly weighed my pros and cons against availability of software and ease of use, not about who wrote that software. Fedora won because back then I constantly had to switch Java versions and it was super painful with other distros. So discussing and improving the tooling and infrastructure backing Fedora is a lot more important for such an initiative than pure software development work.

Now, that doesn’t mean it’s completely out of the question but if I wanted to improve the AI landscape for Fedora using new software, I personally wouldn’t limit it to Fedora only but consider Linux as a whole. This avoids the whole “Fedora is Redhat/IBM corporate, I won’t touch this because I don’t trust them” narrative some people seem to follow and would probably make community engagement a little easier. If it’s software to make enabling and distributing existing tools easier for Fedora, I think this is where the AI/ML SIG or the HC SIG would match quite well as the purpose is:

The Fedora AI/ML SIG is a grouping of like-minded individuals and groups who are working towards improving the state of AI/ML and the associated toolchains in Fedora

and

To encourage the packaging and accessibility of heterogeneous computing projects in Fedora and EPEL. This includes machine learning, OpenCL, and scientific computing.

In particular, about the idea of building a custom AI/GPU-compute stack from near scratch above the APIs themselves

It’s definitely a daunting task but “everything’s possible” as software developers usually say. I will say though, that the AI/GPU-compute stack is already quite fragmented in my opinion (vLLM, llama.cpp, MNN, Burn to name just a few I can think of right now) so I’m not sure if having another option would make things easier because it means having another thing to package in the end (and the packaging aspect can be quite hard depending on the underlying software used).

Fedora already has an excellent forum, where users can post about the issues they are experiencing. Support is widely available

This is where things get interesting. I’m not entirely sure if numbers exist for people using these forums compared to Fedora installs vs. people seeking help on other communication channels such as Matrix or the Fedora subreddit (coincidentally, I think @gordonmessmer could say more about this as he’s one of the subreddit moderators) but from what I’m observing in the Asahi subforum, it’s not used that much compared to the other two. I can only guess and say that for AI if tooling improved and you could just dnf install your favorite stack, it would probably also help bring more people to this forum if they need help but I also believe that seeking help should be an edge case and if there is some way to dnf install your favorite AI tools, it should mostly just work out of the box (people say using it should be “boring” nowadays but I don’t like that term). This is one of Fedora’s strengths as I’ve rarely seen any packages break or not work. What I believe would help more is setting up useful documentation over at the Fedora docs once what you envision is in place. The docs page is where people would usually look first (if Google does a good job sending them there), so having a plan for that will definitely be beneficial and something the SIGs I mentioned would definitely appreciate help with.

Would you say one of the SIGs already does what you wanted to propose here or did I misinterpret anything?

6

I wish to do software engineering, and possibly develop alternatives to this software. I think the scope should include GPU-based software, as well. That is why I proposed a GPU engineering SIG. If members believe that the AI/ML SIG is sufficient, than I will leverage that.

In regards to packaging, if we use OpenGL or Vulkan, that means that Fedora’s stack will depend on Mesa3D. This shouldn’t a huge worry, since a given packager would simply build the software with a given Mesa version. If there’s something I’m missing, let me know (since I’m not a packager).

There is something to be said about how the software is to be distributed regarding its own internal dependency chains. I’ll make sure to keep this chain as simple as possible. I don’t hope to burden the packagers too much.

I think that the biggest benefit of these forums, is that the engineers working on the stack would now be able to communicate directly with its users on the Fedora forum, since the stack would be Fedora-maintained. I believe this communication is paramount to the success of this stack, since the engineers will have direct knowledge of what the users want, along with their pain points. The engineers will be able to produce new software, if the users’ needs change.

This user-to-engineer communication is one of my primary goals with this initiative. One of my pain points with a lot of software personally, is that if I use any APIs that are exposed to other software engineers, I don’t have any avenues to figure out how the software functions, since communication with their developers is limited. Thus, I can only make limited changes, or, I would have to study the code for a long time. AI is still subpar at this, at the moment.

Likewise, if people know that there are engineers at Fedora who can implement software, than they are likely to consult Fedora to build new FOSS software, rather than consult a consulting agency, who may or may not be as well suited to implementing this software.

Utilizing discussion.fedoraproject.org would kill two birds with one stone.

Fedora is under Redhat/IBM’s umbrella. I agree that this may bring some bad publicity, and some people may be suspicious that the stack is Redhat/IBM derived. However, being under this umbrella gives Fedora leverage not only within FOSS, but within the corporate world as well. I dislike the corporate world, however, they are the ones who hold the most power. It would be beneficial for Fedora to be involved in some sense.

If Fedora doesn’t use these levers, than there is a high chance that the corporate world will dissolve Fedora’s FOSS mission over time.

Ultimately, I don’t have much to say here. I’ll let the community decide whether or not Fedora should take on software work, since this is fairly new and unusual.

We have to take a ride on the AI train, whether we like it or not. We will be discussing AI, whether we like it or not. So, we may as well rise up to the occasion, and take a piece of the AI pie for ourselves.

7

I also plan on extensively documenting the stack, all in regards to its internal workings, its APIs, and any user manuals/references. This will also be another aspect of the stack, which will make it shine.

8

Before anyone asks, yes, I’ll do the work myself if I have to, in order to get the ball rolling, although this may choke the initiative a little bit.

9

That’s a very noble goal. The reason I asked was actually because I wanted to clarify the intent here a bit more so someone with more knowledge about the inner workings of Fedora, SIGs, Initiatives, etc. than us can give you an answer on where this best belongs. My gut feeling (note that, similar to you, I’m just a “regular” Fedora user with minimal packaging experience) is that the SIG would be a good match to exchange ideas in that regard. Whether there’s a SIG that focuses on GPU-based software is something I’m not aware of currently, so I can’t really comment more on this.

In regards to packaging, if we use OpenGL or Vulkan, that means that Fedora’s stack will depend on Mesa3D. This shouldn’t a huge worry, since a given packager would simply build the software with a given Mesa version. If there’s something I’m missing, let me know (since I’m not a packager).

Me neither but the good thing as you also rightfully pointed out is that there’s people here who have that experience and who can give a definitive answer. From what I know, Mesa is already installed in the base system. Vulkan and OpenGL should be too. The only thing that are usually missing if you want to build from source are the -devel packages but they’re also easy to install.

The way a spec file (which is how Fedora infra builds packages) works is that you basically define your runtime dependencies and build dependencies (they’re called requires in the spec file). To give you a concrete example, the build process for llama.cpp is currently defined like this.

I think that the biggest benefit of these forums, is that the engineers working on the stack would now be able to communicate directly with its users on the Fedora forum, since the stack would be Fedora-maintained. I believe this communication is paramount to the success of this stack, since the engineers will have direct knowledge of what the users want, along with their pain points. The engineers will be able to produce new software, if the users’ needs change.

Would you envision this as something like a standalone category similar to the one for Project Discussions or a post like the meeting minutes where interested users could comment on? I’m a little worried about the number of posts because technology moves quite fast, especially when it comes to current trends like AI.

One of my pain points with a lot of software personally, is that if I use any APIs that are exposed to other software engineers, I don’t have any avenues to figure out how the software functions, since communication with their developers is limited

I fully share that sentiment. The good thing is that with FOSS, you can at least look at how the software functions and address issues upstream. I still remember having hour long debates with HERE (the geocoding/routing provider) about their updated APIs sending bogus responses and doing the same with Vodafone half a year later… fun times but a completely different story.

This is actually a common misconception. Fedora is as much of a community owned distro as any other distro out there. The main difference is that RedHat sponsors Fedora development and many RedHat folks also actively engage in the Fedora community because RHEL is based on Fedora. But of course this also means that if Redhat has an interest in certain pieces of software, they’ll try getting it into Fedora as well giving Fedora more leverage than other distros (maybe).

Before anyone asks, yes, I’ll do the work myself if I have to, in order to get the ball rolling, although this may choke the initiative a little bit.

I think as soon as we as a community have figured out what we want, there’ll be more people who will put in some work, so don’t worry, I don’t think you’ll have to do all of it by yourself and will get help from experts in fields you’re not as experienced in

10

Thanks!

I agree that the potential for the amount of posts can absolutely go out of hand. In fact. there may be a need to have extensive strategies dedicated to controlling these posts, ensuring that they don’t go off topic, or out of hand in any way that would go against Fedora itself, or divert attention away from other important matters related to Fedora itself.

I also envision that a standalone category will also be necessary.

If you wish, I may also discuss matters regarding these strategies further in the replies.

Nice. B-)

Thanks for the clarification.

Acknowledged, thanks! :grinning_face:

11

We should add ROCm to this list. https://fedoraproject.org/wiki/SIGs/HC#AMD’s_ROCm

AI ecosystem is a broad topic, what AI use case(s) are we trying to support ?

The one I am working on is supporting local llm through ollama here AI/ML Special Interest Group - Fedora Project Wiki . It has both ROCm and Vulkan backends.

12

I’ll state some of my personal goals with this initiative.

Noted. When I post a revision, ROCm will be added as a potential API.

I’m not sure. At the moment, I don’t have proper communication channels to establish what the various use cases are. This is one of the primary reasons I am proposing this initiative in the first place. If I and others were to develop software for this initiative, copying Ollama would likely be ineffective. In this case, I’d rather spend my time working on personal projects.

I ultimately seek to understand the various ways AI is actually used on the ground, and to reason about its viability, while also implementing various improvements/optimizations. I can’t help but notice the AI is used for X headlines; they make me extremely curious as to precisely how AI accomplishes these tasks.

I’m still not an AI expert (I don’t know how to implement a back propagation algorithm from scratch as of date), however, I desire to learn and improve myself in this aspect, so that I may become more knowledgeable in the future.

I also seek to develop software for GPUs in general, since I see them to be quite useful, even for some general purpose tasks.

Also, what on earth is this??!? -

13

I took a look at my day-to-day tools and looked for how they could be improved with some ai plugin or feature. Now I have a emacs native coding agent. If we get that into Fedora everyone will. (There is also something also for vim).

14

Almost anyone can integrate AI agents, and build platforms that use LLMs, given they already know how to use AI. That kind of tooling can probably be handled by the AI/ML SIG (and likely already is), and they fall outside of this topic’s scope.

The place where this topic intersects AI, is the implementation of algorithms like forward propagation, back propagation, and specific neural network structuring (CNNs vs RNNs vs DNNs, Etc.). Otherwise, what would be the point of this initiative, if the AI/ML SIG is already there, and we were just packaging/building agentic AI tooling, or copying Ollama? These tools are already out there.