I expect people to continue to ask what is in scope and what is out of scope, so I’d like to convey that I would like to enable as much hardware as possible. Removing the names is more accurate, but less meaningful. Do you think that’s the right trade-off?
“Packaging” has a section of several paragraphs. I think that one of the challenges we face with packaging generally (and with complex software like AI applications especially) is that Fedora treats every component as a rolling release by default. We can provide multiple release streams for a component (e.g. python312, python313, etc) but that’s not the default, and I think that’s something that needs to be addressed.
Whether that’s “base Fedora” is something I’ve explicitly left undefined. I think this is the sort of thing that software collections, and modularity, and containers all address in different ways, and we will need to address it in some way to deliver complex software.
I’m willing to hear counter-arguments, but my knee-jerk reaction is that having a non-goals section at all is weird and awkward, and putting it first reinforces the notion that the most important thing is reassuring people who aren’t participating in this work.
Putting my FPL hat on again in a quite snug fashion…
I’m going to strongly disagree with that.
Non-goals should never be the lead for anything.
Non-goals are supplemental material to provide context.
The people who need to be inspired by this and are going to want to participate need to see what the goals are, what the vision is, not the infinite space of what it is not. The people whom you are concerned about are likely already not in that group.
FWIW, I think the lesson we should learn is that we should not try and solve this. Every attempt to solve it has been, basically, bad. I don’t think it is a solvable problem in general.
Software stacks that want to pin very precise versions of their dependencies are inherently a bad target for distribution packaging, and should not be packaged by distributions. They should be distributed in other ways.
Containers are a good way to do it, and I guess if we want to get into the business of providing complex version-tied software stacks in containers, we can do that. It seems fairly orthogonal to the task of building operating systems, though, which I always thought was the thing we were supposed to be doing.
When a proposal is bound to have knee-jerk reactions due to the subject being divisive it’s important to me to lay out ‘this is not what youre going to assume’ very early in the post. This keeps the discussion clear of having to explain what it is not many times in the replies which only detracts from the proposal.
It seems this thread so far is doing fine, but the previous thread definitely had a lot of that going on.
I get where you’re coming from. As we’ve seen before, it’s hard to build complex software in sync with the rest of a Fedora release. Fedora’s cadence works well for software with a similar schedule, which tends to be software that’s relatively mature, where development has slowed down, or software that’s relatively less complex. Complexity and velocity make software harder to synchronize.
One of the things that I suspect we’ll have to contend with in the near future is that even if complexity doesn’t increase, development velocity probably will. Some of the increase will come from developers adopting assistants, but some of it is going to be driven by adversarial use of assistants. No matter what choices we make about how we develop software, our systems have to adapt to a world that is under pressure to respond to exploits being developed, released, and deployed much faster than they were in the past. My guess is that moving faster is going to mean less synchronization, because that’s always been the point of the stable release process. Teams can move faster when they don’t have to move in sync.
Complexity is the other axis on the same plane. When I look at complex software like vLLM, I see the same basic problems to solve. When you say that complex applications should be distributed in other ways, I hear what you’re saying, and to an extent I agree with you. I would be surprised if we emerge with vLLM RPMs in a Fedora repo of the type we have today. But if we give up building complex applications, I think we give up everything. A meta-build system that can build complex applications can build everything else, too. If we are content to build big complex applications with BuildStream (hypothetically), then why not build the OS with BuildStream as well, like GNOME OS and KDE Linux do? Fedora, today, serves a whole lot of use cases that GNOME OS and KDE Linux don’t, but I don’t think that will always be the case if we’re not looking for ways to improve the process.
I don’t think this is an insolvable problem. There are lessons we can learn from component registries (e.g. PyPI, crates, etc) and from other build systems like BuildStream. And I think we need to learn those lessons in order to take part in this field.
At this point, this is not substantially different enough to address my concerns in the original thread. Rather than repeat myself again, I’m going to once again assert that I do not think this initiative should be accepted and if you want my reasoning in detail, please look at the original thread. There’s not much that’s meaningfully worth it to do this way that isn’t already doable through other mechanisms, and I do not want to have Fedora endorsing pathways to abrogate our own principles in this manner.
Also, I take serious issue with this persistent (and I’m starting to suspect intentional) confusion of the “stable” with the longterm kernel series. (Once again, I feel like I have to point out that no kernel series guarantees any stability on any axis. That’s not something kernel developers care about. And the lack of engagement in the existing Fedora kernel efforts by the Red Hat kernel team does not make me enthusiastic in adding more pain to all of us with more kernel trees.)
I’m tired of this being pushed because it feels like everyone around me is forcing us to find a way to openwash it and make it look presentable. Between Red Hatters and the Fedora Council pushing this, it is now an extremely unpleasant topic and I do not really want to engage on this topic any further, even though I know I will have to.
I would really prefer if this initiative proposal was completely withdrawn and instead @gordonmessmer were to participate as a member of the AI/ML SIG instead. If he wants to do things with NVIDIA stuff, then he can continue doing the Fedora Remix, but Fedora endorsing the remix isn’t okay with me either, given the purpose of it.
This proposal suffers from a lack a specifics. Every accelerator will have its own challenges. On just the kernel side drivers will conflict. In the upstream these conflicts are worked out as part of the normal kernel review process. Someone will have to manage these conflicts. Who is that person ? Will there be a remix per accelerator ? That does not scale.
For some accelerators this proposal is burden and there should be an outreach to those vendors before associating them with this proposal. Speaking for AMD/ROCm in Fedora, ROCm and Vulkan is already well functioning.
This is extreme handwaving, multiple python versions would be a tremendous change, needing likely a couple of years to roll out. To get to AI or even Desktop, there needs to be a path and an idea of who is going to work on that path. If the path is not in Fedora, then why have this as a proposal.
Distill this proposal to what is concrete and rename it appropriately.
I’m jaded on this, if I take the generous approach that this is meant to explain to people how to build their own remix then it’s a good thing. If it’s meant as a ‘this is how you bend Fedora to run proprietary things’ I don’t particularly think it’s something that an initiative should be standing behind.
So, Universal Blue?
Atomic system images cannot currently be created, nor published, within Fedora infrastructure. Koji’s container builds cannot be signed; and image-builder’s bootc support (to convert those containers to concrete artifacts such as disk images and ISOs) has not been enabled in Koji/pungi as there was no previous interest due to the first problem.
Are you planning to address these issues as a Fedora Spin would need to be built inside Fedora infrastructure? Are you instead planning to use alternative build infrastructure?
Atomic system image also refers to multiple things at this point in time. Are we actually talking about bootc, or are we talking about ostree (the current way Fedora IoT and the Fedora Atomic Desktops are produced in Koji).
Same as above, but this would have to be built outside of Fedora infrastructure.
There are already a large variety of random Containerfiles for bootc floating around on GitHub that do the “same thing”, how will this image be different from those?
Electron isn’t packaged; are you planning to do this work inside Fedora (packaging Electron) or outside Fedora (plopping it into your Containerfile and thus leaving it out of the Fedora Spin).
Since Fedora doesn’t respin images (we should) (there’s a respins sig, but they only respin the live ISOs at the moment) I assume this is about the potential outside-of-Fedora artifacts you plan to produce and thus these contributions would likely not benefit Fedora itself?
Since you are talking about Atomic system and I am assuming that you mean bootc-based systems by that (see my earlier points) this would likely not benefit Fedora as a whole as the documentation would explain how to run dnf or other installation commands inside a containerfile.
Where, and what will make these container images optimized?
This is going to be very ambitious, but if you manage to crack it: awesome.
Reading through all of this the less I feel this is something Fedora should want to promote and pool resources behind; the barriers are large (not necessarily a problem for the technical ones) and the vibe feel very off. It feels much more like something that would fit in the Universal Blue ecosystem, where I think such efforts are already ongoing or even solved without there having been a need for an initiative from the Fedora side.
Well, let’s be reasonable: this is a proposed initiative, not a proposed Change. An initiative is sort of “someone wants to work on Big Idea X, and we agree to provide some space and structure for that”. It’s not really within the scope for every concrete detail to be pinned down; if the idea was already at that stage this would presumably be a Change proposal not an initiative proposal.
I think the non-goals section that’s been added to this version of the proposal was a solid improvement and appropriate within the scope of an initiative, but asking for all the deliverables and work items to be nailed down would be going too far.
I do agree there remains a risk of Fedora being perceived to be officially “blessing” things that should be considered external to Fedora, and if the Council accepts this I think it would be a good idea to make it clear that, so far as the project is concerned, the initiative is primarily about the parts that are in-scope for Fedora itself; I think the appropriate stance on the outside-of-Fedora work would be “we’re watching this with interest but we don’t own or Officially Bless it”.
FWIW, my overall opinion on this version of the initiative is that I’m not sure it’s the right Big Idea, but I’d be fine with Council approving it, with appropriate caveats and clarifications about the Fedora-external parts and a clear understanding that anything branded “Fedora AI Developer Desktop” without any “remix” identifier needs to follow all existing Fedora policies and principles.
I don’t know if I would want to use a Fedora AI Developer Desktop, but that’s fine. I don’t use KDE or i3 or the Design Suite or the Scientific Lab either. It’s not a reason to stand in the way of its existence.
I also don’t think it’s tenable for a broad church project like Fedora to reject the entire field of AI / LLMs on principle. There are going to be people in the project who want to play with the stuff; the existence of the AI/ML SIG already implies both that this is the case and that the Fedora project does not reject them outright. Given that, I can’t really see a from-principle argument for rejecting an AI-related initiative purely because it’s AI-related. With the inclusion of the non-goals, particularly the last two, I don’t really see a strong reason to stand in the way of folks who want to try this.
Jumping into this discussion mostly to add my 2 cents
4. Include user-friendly tools common to various AI back-ends, such as Goose CLI and Podman Desktop.
I’m all for this part of the initiative and I’d be interested in helping out specifically for Podman Desktop but I think it’s not really in the correct place. While I agree with Goose CLI, Podman Desktop seems odd to me. As someone who maintains a copr for Podman Desktop (which should mostly align with the packaging guidelines), I’d love to see this landing in mainline Fedora but as both Neal pointed out in previous thread and Simon mentioned in this thread, it’s missing a lot of basic infrastructure. To give a few links to former discussions that I’m aware of where similar ideas were discussed:
Most of these discussions have stalled somewhere around 2022 and I don’t know if there are any talks going on that I’m not aware of but they probably stalled for a reason, one of which I think is complexity. Shoving this into an AI Developer Desktop initiative feels both too constrained as Podman Desktop isn’t the only electron application and too broad for this specific initiative. I do know someone proposed an electron SIG in one of those discussions which might be better suited and broaden the scope of a potential electron integration instead of only focusing on a single app out of many.
Apart from the packaging issue, it’s currently unclear to me how Podman Desktop would help. From what I know, it’s mostly used for containers. There is an AI Labs extension (which would probably require extra packaging and open a whole other can of worms?) but the primary focus of it isn’t AI (I might be wrong on this one, though)
As for the general idea of this initiative, I’m torn between thinking this is a good idea in principle but a bad idea in practice because, as Neal and others rightfully pointed out, it’s currently too vendor-centric (specifically towards NVIDIA). To give a few examples of what I mean with this:
NVIDIA
NVIDIA: Is there an interest in a Fedora Remix for their developer community?
AMD/ARM/Intel: Provide first-class acceleration where there is upstream support
NVIDIA + other vendors
One thing this initiative points out is using vLLM or Goose, both of which are vendor/hardware agnostic. To quote vLLM
Support for NVIDIA GPUs, AMD GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more.
Note that they currently do not support Vulkan which is a pain point for people with even more exotic systems (Asahi Linux is a prime example here) and might cause people on those systems to get frustrated because all the work happens on CPU instead of GPU.
Goose would suffer from the same problem because it’s using llama.cpp for local inference but the Vulkan backend isn’t enabled in the Fedora build at the moment, making it a prime example for the AI/ML SIG (also, did I mention Podman Desktop wasn’t the only application needing electron? Seems like the Goose desktop client is another one).
Instead of focusing on showing people that Fedora makes it easy to integrate with certain vendors by making it easy to build out-of-tree modules for said vendors, I think it’d be much more beneficial for Fedora and the entire AI ecosystem to show people that they don’t need all of the vendor lock-in and can build high-quality, performant solutions leveraging open, well known standards (or fully open vendor specific stuff if they so want). Fedora as a distro should innovate not by opening up to vendors but by showing them what the correct way is by raising their index finger and pointing at what’s available (analogy to Torvalds’ often cited NVIDIA line that had a lasting impact at least from a marketing perspective).
Unfortunately, I don’t see any mention of the AI/ML SIG (or other SIGs for that matter) even though it seems most of this initiative would be a good match because that’s what the SIG was made for. Maybe it’d be possible to add relevant SIGs to the Outreach category to see where topics align/diverge and find a consensus that more people can agree on?
Side note: llama.cpp has quite good multi-modal support nowadays and has been acquired by Hugginface recently, so packaging it to work with GPU acceleration across the board would be a big win already and should be listed in Outreach as well (using AI/ML SIG if possible). Whether we like it or not, most if not all companies doing AI publish on HF, so Fedora being as close to upstream llama.cpp and as such HF as possible will definitely give the push for open source more leverage than Fedora promoting out of tree modules in a special Remix. If anyone’s curious why I think so, ngxson, one of the core maintainers, gave a nice talk on llama.cpp at FOSDEM this year where they explain why having a model-agnostic approach is what they envision for the project long-term. Sure, they’ll also have interest in implementing proprietary, vendor specific stuff but if the open core is good enough for 99% of the people, nobody will bother compiling llama.cpp with that vendor specific, proprietary backend enabled and it’ll hopefully degrade over time
I’m not comfortable specifically calling out one single remix as being unofficial.
I believe the operational standard is a remix becomes blessed once it gets trademark approval…and that will require a specific action…and a separate discussion.
I’m going to read up more fully on the history surrounding how asahi made the jump to “blessed.” That effort certainty didn’t have all the pieces in place when the request for trademark approval came in but it was on a trajectory to do so. (For example there were COPR built bits being used in the image when the trademark request was filed.. including a alternative kernel.. if I’m reading the initial request and discussion correctly)
If a remix from this initiative gets on a trajectory like asahi, we’ll likely have a discussion about blessing it. Asahi as a thing in some form, existed for like what 3 years before the trademark request came in? And the Fedora SIG working specifically towards a Fedora Remix as a deliverable around it appears to have been formed up at least a year prior…with a fas group and fedora mailinglist and pagure project space spun up to gather the people around the idea of needing to do a remix that was adjacent to the project.
Well, the idea of “blessing” a remix is a new one that was invented for Asahi. Really the point of remixes is the opposite: they’re supposed to be distanced from Fedora a bit, because they do stuff that is not fully in line with “Fedora”, or else they wouldn’t need to be remixes at all.
In this specific case I think we should be hesitant to “bless” a remix emerging from this initiative too hard, if the reason it’s a “remix” is that it includes proprietary components. AIUI, Asahi does not include proprietary components. My understanding is that Asahi is essentially only a remix because it has to front-run Fedora on kernel patches. To me there’s a big difference between “blessing” that, and “blessing” something that contains proprietary bits. Doing the latter seems to me like it would substantially compromise the Freedom foundation.
And Asahi was just given permission to use the name Fedora Asahi Remix instead of something like Asahi Fedora Remix. I don’t think it was quite “blessed” and it was not part of an official Fedora Initiative like this is.
I acknowledge that I have a tendency to get in the weeds on things, but my reason for asking these questions is to point out that things like packaging electron or entirely changing the way Python libraries and interpreter versions are packaged in Fedora (which is what Fedora AI Developer Desktop Initiative (updated) - #9 by gordonmessmer implies) are complex tasks in their own right and also that they don’t seem to belong in a high-level initiative about AI.
The fact that Asahi needs to use a proprietary toolchain to build pre-built elements of its installer, because we don’t have a working cross-compile toolchain for macOS.. is kinda… you know.. and interesting grey space. Are the prebuilt components proprietary if they must be built with a proprietary toolchain? I could argue they are and Asahi got a policy exception on hardware enablement grounds.
Asahi gets an exception from certain policies because ultimately hardware enablement is a complicated grey space that requires nuance. I understand that. I’m not here to punch Asahi in the eye for asking for that exception. I think it was a reasonable ask. A necessary ask given what Asahi wants to accomplish.
What I am concerned about however is the pattern of trying to cast all hardware enablement policy exceptions into the language of firmware.. because its the longest standing exception for proprietary elements in this project. Even in the ticket I reference, the request has to cast macOS as firmware-ish in order to fit itself into the long standing firmware exception language. I think that’s tragically dishonest to the hardware enablement nuance that Asahi faces with the Apple hardware. MacOS is much more than just the firmware, and its a hardware enablement hurdle, specific to that family of hardware, that cannot be overcome with upstreaming kernel driver support. It would require completely reverse engineering the Xcode based development tools… if im reading the discussion in that ticket correctly.
The reality is.. hardware enablement policy exceptions are always in scope for discussion and they will be nuanced and they will hav trade-offs. What we are really talking about is how far we are willing to go in order to make hardware enablement possible for different classes of hardware. What do we consider criticlal hardware support, that qualifies for an exception, and what can we live without.
So I’ll repeat what I wrote. IF a particular hardware enablement remix ends up on a trajectory similar to where Asahi was when it made its trademark request…then that will be an interesting thing to discuss IF that happens.
We may have similar unforseen challenges with the Fedora-V Force Risc-V enablement remix out there, where there are lingering “proprietary” bring up bits that require exceptional policy. I hopefully that won’t happen, but you just don’t know.. hardware enablement is difficult.