He also stated that the packaging members seem to have acted on their own, in spite of what was stated in the meeting:
I surmise that he likely had more words to say on this topic, however, he was being polite.
Asahi Linux’s endorsement by Fedora, with its closed-source toolchain binaries, seems to be more of an accident, rather than something properly approved.
I’ll drop back into the conversation briefly here:
If hardware enablement is really that huge of a problem, than Fedora can use its leverage to have hardware manufacturers open source their hardware documentation, so that open source kernel/userspace drivers can easily be written, and reasoned about.
Apple is always a special case, since practically all of their documentation regarding internal software and hardware is closed-source. I don’t believe they should set the standard of the direction Fedora should aspire to go down.
The best path towards hardware enablement, is to leverage Fedora’s position in the FOSS world, and to have more hardware companies open source their hardware documentation. If the option proposed in the above post is chosen, Fedora is guaranteed to lose its power over time.
This isn’t a good path to go down, no matter if it is now technically in a gray area due to Asahi’s contributions.
For Fedora Asahi Remix we document all deviations from stock Fedora on Making sure you're not a bot!
This is correct. Asahi was not “blessed” in any way, and AFAIK there is no concept of a “blessed” Remix in the first place. We just asked Council for a trademark exception to be able to call it Fedora Asahi Remix, because that’s what the policy requires if one wants to call a Remix anything other than “X, Fedora Remix”.
I’m not entirely sure what this has to do with the current discussion in this thread, but you may want to read Fedora approves shipping pre-built macOS binaries [LWN.net] which IMO summarizes the situation nicely. The prebuilt binares we ship are more akin to firmware than anything else – they’re only used for the installer, which only runs on macOS. We didn’t actually need to ship these in Fedora at all, we could have continue to just use the upstream Asahi Linux installer builds, but we felt strongly it was important to use as many Fedora-built components as possible, and shipping the installer in Fedora allowed us to use a Fedora-built stage1 bootloader for it.
Thanks! Extra context is always useful.
The Asahi project is in a gray area, and is technically an extension of Fedora itself. However, if Fedora keeps endorsing projects (or Remixes) like this, than the slippery slope problem will rear its ugly head.
I suggest that the OP rewrites this initiative!
And for what it’s worth, all the components we ship can be built from source, just not in Fedora because the macOS SDK is not available in a license that lets us ship a cross compiler that is useful. In the future, when Darling is expansive enough to make that possible and it is packaged in Fedora, we could very well start building out a Darwin cross-compilation stack like we did for MinGW.
But Darling only just dropped their usage of an incompatibly licensed kernel module a couple years ago, and nobody has really picked up evaluating it to build out a cross-toolchain in Fedora yet.
That said, the firmware argument applies here because Apple Silicon Macs use macOS as the boot firmware and the initial boot environment. iBoot starts up macOS which boots macOS (or Linux, in the case of Fedora Asahi Remix).
And this was deliberate. Under no circumstances did we ever want to put Asahi through the Council Objectives/Initiatives process. We don’t have a need for it and it’s not appropriate for something of this nature.
“Blessed” remixes (whatever that is) are not a thing, and they aren’t intended to be a thing. Even the trademark exception we got for FAR was based on the premise that we maintain the smallest possible delta with Fedora Linux and document that delta, which we do.
For future reference, FESCO’s meeting notes are publicly available, so you can read up on the whole discussion that was had here.
The rest would drive the discussion too much off-topic but if anyone’s interested in Asahi specific stuff, there’s a forum for it here as well (as well as a Matrix group)
I may have framed the Asahi team as being more sleazy than they actually are, in what I’ve said above. I apologize for this.
Thanks again for the context. it is always appreciated!
Overall, the addition of binary toolchain packages to the build process within Asahi is regarded as being in a gray area. Those were only added, since perhaps in a future date, those additions would be taken out. This seems to be an addition made in haste, so that they could get the release of Asahi Linux under Fedora done and over with.
What if new initiatives that utilize this gray-area precedent are used to justify the addition of gray-area binary blobs, which are never taken out?
I am also under the impression that Asahi Linux’s addition should’ve never been a council initiative in the first place.
To be clear: Asahi was not and has never been a Council initiative. I fully agree a Council initiative would not have made sense for this, and we never sought one in the first place. The only engagement with Council for Fedora Asahi Remix was the trademark exception request to allow us calling it Fedora Asahi Remix. You can see that whole discussion here: Fedora-Council/tickets ticket #432: Trademark approval request: Fedora Asahi Remix/Spin
What if new initiatives that utilize this gray-area precedent are used to justify the addition of gray-area binary blobs, which are never taken out?
If this was about binary blobs for software you would distribute within Fedora, I’d say the requested exception wouldn’t have been approved in the first place because you absolutely have to build from source. But assuming it would’ve been, the process would’ve been pretty similar to what happened with the Deepin packages, probably.
As for the link Davide posted, note that Asahi is still a fast moving target, so Mesa is no longer Asahi-specific for example. The goal has always been to upstream everything that’s possible. An installer that runs on macOS which is what the exception was made for doesn’t really have anything to do with the upstreaming process of Asahi or Linux in general. It was simply nice to have because it adds an extra layer of trust for users.
There was one remaining instance of the word “stable” used in a context that might refer to the labels used by the kernel.org developers, and I’ve changed it. The remaining uses of the term “stable” refer to the release model and are consistent with Fedora’s use of the term:
I’m not sure I understand that… I am building a kernel outside of Fedora in order to not place any additional burden on the existing Fedora kernel efforts. I expect to also build kernel modules, and hopefully to demonstrate the utility of the Fedora message bus for coordinating module builds. That shouldn’t impact existing workflows.
I do hope that we find ways to contribute, but the work that is underway now is substantially motivated by the desire to illustrate how third party packagers can build software for Fedora in a secure manner, to explore workflows that will improve reliability for users, and to do that without creating addition work for Fedora’s maintainers.
Systems today detect available hardware at runtime and dynamically load support. I don’t think we need multiple remixes for accelerators where there is no inherent conflict.
If there is a need for multiple remixes because of some conflict that has to be resolved at build time rather than at runtime, and if we can create the necessary framework for the users who need that support to maintain it and share it, I think we should do that. I think participation is the whole point of Free Software systems.
Well, no… Universal Blue is using the akmods framework to build kernel modules in CI, and unless I’m mistaken they’re signing those modules with a certificate that’s kept in a GitHub secret.
Both of those are things I think are flawed. I think Fedora provides the infrastructure necessary to release pre-built kernel modules without relying on a framework that builds and installs a new RPM package during an install transaction. CI secrets are great for things that can be (and are) rotated on a regular basis, but IMO not so great for signing keys.
As I understand it, there is already an initiative underway for image mode. I’d like to support that work in any way we can, and to make use of improvements that come out of the initiative.
I’m working on that, too:
Most of the individual tasks outlined here are intended to illustrate and document how to contribute, and to create on-ramps for participation.
A lot of Fedora’s tools are designed around the default assumption that software can be synchronized at one cadence and released as a collection, and an awful lot of the difficult things in packaging are difficult because that isn’t always true.
Fedora’s tools are very good for software that is simple and self contained, or mature software that (along with its dependency set) has slowed its development pace. When we look at the sorts of things we’d like to ship in the AI space, we see the opposite of that. It’s complex software in an emerging technology field where development pace is very rapid.
“First” is one of Fedora’s foundations, so it seems to me that addressing complex packaging tasks that make it difficult to build and distribute emerging technology, including the infrastructure that supports AI workloads and applications that fall under the “AI” umbrella does belong in the initiative. I don’t see how the initiative could complete its goals without doing that.
When it comes to outreach, maybe Lenovo’s linux team would be interested?
I saw this Flock talk from 2024 the other day where it was mentioned that some close source components were preventing Lenovo from offering workstations with NVIDIA GPUs, MIPI, and WWAN components with Fedora – so they offer Ubuntu for their workstations instead.
They had the idea of doing a remix to solve this issue but I don’t know what happened since.
I am still hung up on the fact that the core of this initiative seems to encompass work that is not happening in Fedora proper, but I appreciate your responses about the other details. I replied inline.
Is the idea now for the Nvidia kernel modules to be built and signed by Fedora Infrastructure? Based on my understanding, the proposal said it was going to be done with external infra.
It’s unclear to me what packaging Podman Desktop (containerization) has to do with AI (apologies if you’ve answered this already), but obviously I see that vllm does fall under that umbrella.
In any case, I am interested in what exact problems you are facing with vllm packaging and how we can improve the current Python tooling in Fedora. Happy to discuss that more on python-devel@ or in a breakout topic. You mentioned that Vllm “might need PyTorch for Python 3.12, while Fedora is building PyTorch for Python 3.14” which is problematic, since we only build Python libraries and applications against the latest system Python. This policy is partially to support Fedora’s First Foundation–inspired early Python integration work (see Making sure you're not a bot!) and prioritize working with upstreams to fix compatibility issues with the latest Python version.
Some of it, yes. In Fedora, we can produce a spin that caters to AI users and developers, and in that spin we can experiment with supporting infrastructure that the existing images don’t want for technical or political reasons.
Outside of Fedora, we can improve the process of building kernel modules for Fedora as a third party. We can provide an infrastructure definition for signing that we can offer to partners. We can document these processes and encourage people to participate.
I’m really trying to focus on Fedora as a project, not merely as a product.
No, when I said “I think Fedora provides the infrastructure” I mean that I think a third party can probably use the Fedora message bus to automatically trigger kernel modules builds when Fedora builds a new kernel, and doing so should allow them to offer kernel module packages that don’t lag behind Fedora kernel releases.
A lot of AI software ships as containers today, and Podman Desktop comes up very frequently in related conversations.
I think we tend to conceive of Python 3.14 as a backward compatible upgrade to Python 3.12, but for non-trivial software that isn’t necessarily the case. In some cases, it’s not merely a matter of building against the new release, the application has to actually be ported.
And that’s an area where Fedora’s policies don’t really fit reality. Fedora’s policies are all well reasoned, but they reflect a world view that doesn’t take into account that upstream projects do stable releases, too. There’s nothing wrong (IMO) with developers shipping an application that uses Python 3.12, as long as they’re still in 3.12’s release cycle and it’s still maintained upstream.
At the same time, I don’t want it to be the PyTorch maintainers’ job to support Python 3.12 for vLLM. If someone wants to package vLLM and they need Python 3.12 with PyTorch, I think it should be their responsibility to manage that.
I think that modularity, and software collections, and containers all come at that problem from different directions, and if we want to ship non-trivial applications in emerging technology fields, I think we’re going to have to try to solve the problem again.
(Writing this with my Python Maint hat on, not Council.) Frankly, I consider that problem solved on a technological level. See e.g. multiple Python stacks in RHEL 10 and EPEL 10. That can be done in Fedora as well. The only reason we are not doing it is that we don’t have the capacity[1] and there was no obvious demand that would justify making the capacity. If there are people who want Python 3.12 for PyTorch and its dependencies, all they need to do is talk to us about their needs.
Rather than “capacity to maintain a handful of extra python3.12-foo packages”, I mean the “capacity to make sure this does not spin out of control”. ↩︎
Just a thought from a moderator perspective: I suggest to split this issue into different new topics and leave this one sleep.
There are many intertwined discussions here, and people answer within one post to many at once. This makes every post very long, and this might exclude many people from discussion: unless someone has the time to read everything, they read nothing, not even the parts they would be interested in. You might already found out that the participating audience has decreased in size (a subtle indicator).
Also, this becomes confusing (e.g., not every post here can be allocated to any of the contained discussions at first glance) and impossible to moderate (e.g., it makes no sense to move posts to split-topics if they belong to many at once, as alternatively, we provoke again the same development). (Also, moderators have themselves limited time;)
Just some thoughts 
Let me know if you have or want to create new topics, and if there is a use to move individual posts from here to any of them, or to close this one.
In the past, I have found it confusing when posts are split out from a discussion that is already happening, especially if, as you said, there is not a clear boundary on where to split posts. Because then I go from having to follow one discussion to having to follow three discussions. And I don’t think anything here is so off topic that it needs to be split off.
I guess in the future, we could encourage only replying to one person/covering one distinct issue per comment to make splitting topics easier, but I’m not sure having more comments is necessarily the right answer.
As for the decrease in activity, I think many people who have reservations with the initiative as a whole have already said their piece, including me. I wanted to continue discussing the “Improve Fedora’s support for complex applications” step, since that work is intertwined with other parts of this proposal and also would impact the rest of the distribution (see the above points about alternative Python stacks).
I know it is technically possible to package alternative Python stacks in Fedora, and I agree that Python Maint has done a good job of handling this in RHEL without having to deal with SCLs or Modularity or any other ghosts. But I’d really like to avoid packaging a bunch of separate python3.12-* RPMs in Fedora unless there’s a really strong reason to do so.
For RHEL, the approach of having the python3.XX-* libraries as separate source packages makes sense, as it allows the older/system Python version to keep older/stable versions of libraries and have the newer alternative Python stacks made available later in the RHEL lifecycle use newer library versions that support the newer Python versions. But for Fedora, if it really is necessary to have Python applications and libraries for alt Python versions older than the system Python, I’d rather that we update the macros to make it easier to create additional python3.XX-* subpackages out of the existing python-* components.
