Would your package be easier to build if you could select specific releases of dependencies?

I’ve used Red Hat / Fedora systems for almost 30 years. As an SRE, one of the things that has been most valuable to me has been not just the distribution, but RPM as a build tool. The spec files offered reusable, composable build workflows designed by people with experience integrating components into a coherent system.

Earlier this year, I proposed an initiative that was mostly aimed at improving our build processes. Fedora’s systems are great for building Fedora, but it’s sometimes difficult to extend them to build applications *for* Fedora. In particular, large, complex applications tend to be compatible with specific releases of their dependencies, and as their dependency tree grows deeper, it also becomes less likely that it can coexist inside the distribution with similarly complex applications.

I recently completed a prototype of a couple of tools that I used to package vLLM, because it exhibits those characteristics. The tools themselves are very similar to tools we have in place now. The configuration for an application build is very similar to the configuration we use for a Flatpak build. The main difference is that adding branches to a component’s dist-git repo which mirror upstream release branches allows a developer to select the release series of each component, independent of the version that appears in a Fedora release.

That doesn’t necessarily work well in a mutable package-mode system, but I think it works very well for isolated builds, such as container images (including Flatpak).

I’d like to discuss adding “upstream-release branches” in Fedora dist-git. Although it’s a small change, I expect we’ll need to discuss various aspects in detail, in a series of threads. Among other topics, we will probably discuss whether those branches can be automated to avoid increasing maintainer workload (I think we might have the confidence to do that since they wouldn’t be used directly by distribution releases). We’ll need to review previous branch-management implementations to see if any previous work is reusable or needed. We’ll need to discuss whether there are concerns specific to Forgejo. We’ll probably discuss whether we can use quay.io and the existing Flatpak registry or if a separate registry is needed.

But before we discuss any of those topics, I’d like to hear first from people who think that this would help them. Would having branches that track upstream releases, so that they can build an application and dependencies decoupled from the distribution release, targeting a container output make it easier to build a package that you maintain? Would it make it possible to build something that you would like to build but currently cannot? Would these tools be useful to you outside of Fedora, to build software for your own?

My experience building vLLM is described here.

Please reply if you are interested!

This is good idea only as long as you think about branches for your main tooling (or your main dependency). But your project have other dependencies and they have other dependencies. And that would require other branches and the other dependencies will have other branches.

It would imply:

  • more work because of more branches. Upstream often introduces versioned dependencies for various reasons. They limit the upper version because they are afraid that future deps may not work with their code. They limit the lower version just to force you to use a version with a security fix. As a maintainer, I had to decipher this and relax these requirements. In more branches, that will mean more work.

  • more branches mean more possibilities and there is no guarantee that your dependecies will be resolvable at all. You may be able to resolve it for your project eventually, but if users want to install two different applications, they may hit an unresolvable tree of dependencies. The more branches you have the more likely this scenario is.

Fortunately, we have Copr, and you can create a proof of concept there. You can create a project there for every branch and see how it works.

To answer the title, yes.

This is something I’ve been mulling over for a while and don’t have a completely satisfactory idea for, but for most of the applications I’m interested in, greater flexibility and a more streamlined maintenance experience would make me more likely to package and maintain things I care about in Fedora, and also more likely to look to Fedora to provide applications I care about.

I don’t love the idea of representing them as branches in dist-git, in part because I would really prefer to see all the specfiles live in one repository, but I don’t love any of the designs I’ve thought of, either. Most of them revolve around making the look-aside cache smarter - basically closer to a curated mirror of language registries, making ingesting a source tarball a review point, and allowing packages to opt-in to those smarts to compose their dependencies at build time. There’s a lot of hand-waving there, I haven’t looked hard at how this would work in practice yet. The most obvious downside to this is that the answer to the question “what’s available in Fedora to build against” gets more complicated than “whatever dnf says is in the repository”.

I, of course, also want the ensure it’s simple for a maintainer to backport patches once for a library and have one button to deploy that patch to all the applications, etc. And all this is mostly interesting for languages that statically link like Rust, although I guess something like Python could do horrible crimes with venvs.

I’d like to talk about all of this more in a future thread, but real quick:

Yes, that is exactly what I demonstrated in the vLLM build.

When upstream-release branches are available, it means you don’t need to do the work you’re describing at all.

The system I’m describing will probably be used mostly for container builds, where there’s no conflicts because a different application will be in a different container.

Copr is for builds, and my project is largely about the earlier phase: source management. I think it would be helpful for package maintainers to collaborate on dependencies so that they don’t have to develop their own dist-git for every release of every dependency.

Awesome! What software would it help you package? :slight_smile:

We can certainly talk about other possible implementations. This one is merely a prototype.

I will say that the reason this prototype uses branches is that that’s how the software is managed in most projects upstream. By using the same practice that the projects themselves use, the tools naturally support source-git repos as well.

That creates really interesting workflows, were we can build a container directly from source, without the need for releases per se or dist-git at all. Commit a change, build the container. Minimal steps between commit and build. :star_struck:

Probably not the sort of thing the distribution will be interested in, but the developers who’ve seen this project love it.

Mostly Rust things, many tools are either missing or outdated in Fedora. Atuin, for example, is finally available but is a release from February. Zellij, starship, etc.

Can’t I already do that by just ignoring the distribution anyway, though? That’s what most projects do.

I’ve been mostly thinking about container things, but “Rust things” makes a lot of sense as a great use case. With no shared libraries to conflict, Rust packages actually could ship into the distribution repos.

Sure, you can. And yes, many developers do. But I’d like distributions to be relevant to developers, too. :slight_smile:

How much is a distribution still offering if not the package selection, vetting, testing, and integration? :slightly_smiling_face:

We have ideas and solutions for those of us who want to build everything from sources at various versions (nix, buildstream, etc).


As for the initial idea, I’d prefer more vendoring and automated tooling to handle vendoring well than shipping more versions of packages themselves.

Since I’m familiar with the Go ecosystem, but you also see some of this in Python where people regularly pin dependencies to specific commits instead of even upstream tags at which point you can blow up the dependency graphs by several orders of magnitude which is … probably going to lead to finding very fun problems.

1 Like

In general, I’m not looking at this as a way to ship more versions of packages. For the most part, dist-git branches that reflect upstream releases are a form of metadata that we can use to build applications.

I would like to be able to ship applications that we cannot ship today, and ship some of the applications that we ship today in a configuration that their developers have actually tested. It gets us applications that are more reliable, and a larger collection of applications.

It doesn’t help with me being the one (one of those) responsible for maintaining those different dependency versions. One of the main problems is that the Fedora maintenance group is understaffed, so adding more maintenance work isn’t the best idea.

The analogy with upstream or language ecosystems doesn’t work well here. All the Fedora distribution work is about integrating all the specific software versions together (and providing support during the release cycle). Upstream typically maintains just one version (the latest one in main, where work on that specific version ends at release time). As for language stacks, they don’t really do maintenance at all—it’s just a matter of uploading software to a repository at the time of the upstream release. That software then becomes immutable, which leads directly to dependency hell everyone faces.

The only known workaround here, in my opinion, is bundling/vendoring :frowning: (which makes it possible to bring dependency-hell-affected software to Fedora). However, this is a security disaster, given that we are responsible for fixing CVEs and should aim to have each library in the system only once.

On a more philosophical note, we need to keep visible human support in the loop somehow—to offer real added value as a distribution. Otherwise, we could just robotically rebuild RPMs in many different versions and lose the ability to provide actual maintainer support (for that dep-hell use-cases, people can just consume software directly from language stacks).

I hear you.

I want to make package maintenance less work.

Fedora has a ton of great automation tools. We’ve got packit, Anitya, the-new-hotness, a bunch of tools that automatically generate or update specs using information from external registries. And most of them are opt-in.

I understand that a lot of maintainers don’t trust the tools enough to hand over regular maintenance to automation, for Fedora releases.

But maybe we would for branches that we treat as purely metadata?

I’d like to pursue this further, because I think it would be less work, not more. It was certainly a whole lot easier to build vLLM using upstream-release branches than it would have been with fedora-release branches.

To be clear, I didn’t say that I don’t trust automation or that we shouldn’t accelerate our efforts there. I was trying to express the actual added value of the distribution itself, using a simple equation: really integrating and maintaining a single version of a library requires far less effort than doing the same thing for multiple versions. Bringing in more versions of unmaintained libraries adds very little value (better to bundle, or use cargo/pip/etc. directly).

For example, look at the pypi-in-copr experiment (where we attempt to build all PyPI packages as RPMs). That is an environment where no maintainers are involved—feel free to open a Copr ticket if you have ideas for this project (e.g., that you’d like to keep multiple versions of each RPM) or would like to see an equivalent for Rust.

1 Like

I don’t understand why you bring up Rust here?

And, ~surprise~, our tooling already fully supports shipping multiple “streams” of dependencies in parallel, and we use this extensively.

It would help to look into how things actually already worked before making statements like that.

You mean “packaging multiple versions” right?

In Rust, it is possible. But it’s also an extremely high friction process, because the when a crate is already packaged, an application developer who needs a specific release series for that crate has to ask another package maintainer to create an entirely new package. And then they have to wait for an indefinite amount of time.

The person who needs the work to be done is not empowered to do the work, and that is a sustainability problem.

There are straightforward ways to make this much easier in Rust, and possible in other languages where it currently is not possible.

Thanks for writing up the proposal! I saw your VLLM blog post on Fedi but did not have a chance to comment. It would help if you clarified the intended usecases for this new process other than “package complex applications” — I tried to infer usecases in this response.

Allowing an infinite amount of conflicting dependency versions into the primary Fedora repositories seems contrary to Fedora’s position as a Linux Distribution, whose goal is to integrate a set of components into a coherent repository/distribution/deliverable. It sounds nice if we allow each package to use whatever build dependency versions it wants, but this only works (from a technical/theoretical perspective, not necessarily a maintainability/sustainability perspective) for ecosystems that use static linking, since otherwise, the dependencies still need to be installable at runtime, and we don’t want to introduce conflicts between unrelated packages/applications in the primary repos because they depend on conflicting versions of the same dependency.

For the Rust case, which does build binaries statically linked against deps in rust-*-devel packages, we already have the ability to create compat packages and for individual packages to constraint their crate dependencies when needed. There is a some very well thought out packaging automation in the Rust SIG, and it already allows maintaining parallel versions of library dependencies. I’m not sure why this is a “high friction process”; any package maintainer can create a new compat package. Are you suggesting to change the Rust compat package model with this proposal?

Also, Fedora’s push toward using new toolchain and library versions is intentional, as it allows the ecosystem to move forward, and it drives our Upstream-First approach. Distro packagers are often the ones sending patches upstream to enable compatibility with new Python versions, library versions, etc. The Python community, in particular, has recognized the benefit our work here provides to the broader ecosystem by adjusting the CPython release schedule to better suit Fedora’s and allow for our early integration work. This is not something I want to discourage by adopting more version pinning in Fedora. (Even in the Rust SIG, compat packages are treated more as a last resort, and efforts are taken to port software when feasible. Keeping around old dependency versions forever in Fedora is not sustainable. Go’s switch to vendoring in Fedora was also a special-cased, last resort due to the unique constraints the Go ecosystem has that other ecosystems in Fedora do not.) I do want to have policies that encourage upstream collaboration and improve the packager workflow, while creating oppurtunity for automation.


If the goal here is just to provide a better way to build containers for single applications (due to the aforementioned propensity for package dependency conflicts between multiple applications using the proposed approach) based on RPMs, that’s a different discussion. Is that what you see as the usecase for this new approach? If so, what benefit does that provide over the existing way that modern applications build containers using language-specific tooling (pip, uv, cargo, etc.) based on lockfiles (uv.lock, Cargo.lock, etc.) that are auto-generated based on upstream dependency metadata and data from the corresponding language package registry (PyPI, crates.io, etc.)? The example in the VLLM blog post seems like a step back from the dependency generator automation that Fedora currently utilizes in Python and other ecosystems to avoid having to duplicate every single upstream dependency specification in downstream metadata. In this proposed model, dependencies would be specified in three places: upstream metadata; RPM metadata; and the manifest config format.

Even if we agree that this is a good model to cater to the single-application-in-container usecase, I am worried about the burden that having additional branches in distgit will create for existing package maintainers, especially considering that it represents a shift from Fedora’s role as a software distribution (see Simon’s earlier comment about what a distribution is supposed to offer). Can these side branches be hosted and maintained separately under a SIG in Fedora Forge, instead?

2 Likes

I think supplying upstream-release branches would be most helpful for applications that ship in a container image, including Flatpak. I tried to be clear about that both in the write-up on vLLM and in this thread.

I’m not suggesting that we do that.

That’s not true if the build output is a container image. The demonstration project proves that.

I did that once, and the owner of the primary package got quite upset about it.

In the rust ecosystem, I think it only really works if you need an older release series, and it’s potentially problematic to create a “compat” package for a release that’s newer than the main package.

And in non-rust ecosystems, I don’t think similar things are supported and probably not supportable.

Upstream-release branching makes one consistent practice available for all packages, regardless of what language is used.

I know. I want us to use new releases, too.

I’ll copy/paste a comment I sent to devel@ earlier this morning:

The idea that applications will lag and use old libraries is a common objection, which is weird because the only project has has been demonstrated with “upstream-release” branches needs them because they’re ahead of the distribution.

It’s Fedora, today, that is using old versions, not decoupled applications.

I suspect that many people will dismiss that as insufficient evidence, so I checked Rawhide recently:

https://drive.google.com/drive/folders/1axHOgq6LsODDmAA-D-C12StPs2jpI2Qq?usp…

About 1/6 of packages in Rawhide are, in my opinion, significantly behind upstream. Meaning they’re on the current release series but more than 3 patches behind, or they’re not on the current release series at all. Another 1/6 of packages don’t have release-monitoring set up nor a Url that references a VCS (which, I know, is not required) and are difficult to check without manual work. So, somewhere between 1/6 and 1/3 of packages are lagging, in Rawhide.

If you are concerned about shipping old packages, then upstream-release branching is the *solution*, not the problem.

The process we have in place today, where we try to build everything around one release of every shared component, actually make it harder to keep things up to date. They require more coordination than a system that offers upstream-release branches. Shared components require extra care to update because they might impact applications, and applications sometimes can’t be updated because the shared components aren’t ready yet.

Yes.

Those approaches mostly use pre-built binaries. And, among other problems, that doesn’t help us bring up software on architectures that the upstream developers aren’t building packages for.

The process that I’m describing supports our practice of building everything from source, maintaining our commitment to license compliance, to verifiability, to the ability to patch software when required, and to supporting hardware architectures of our choosing.

The Forge maintainers have told me that the new Forge is not to be used for package, and also that it is not to be used for anything that is not an official Fedora output.

So as a policy matter, no, that doesn’t appear to be possible today.

I see that this approach could work for developers to build single applications in a container, but I’m not sure how we would maintain or test these branches together if we could have any number of upstream versions for each application and library dependency. I also don’t understand the need for the separate manifest file (“manifest file” refers to the YAML in the blog post) that manually specifies each dependency and branch for an application. This seems duplicative to me. Why can’t this utilize existing RPM buildtime dependency metadata to figure out the correct upstream-release branch to use for each requirement?

I would like to avoid shipping old packages in the primary repositories, but I don’t see how this proposal accomplishes that.

I don’t think this is universally the case. For Python, extension modules are uploaded to PyPI as pre-built binaries in wheels, but pure Python code is not. Either way, the standardized tooling stills allows building sdists from source instead of using pre-built wheels. Ecosystems like Go and Rust compile all Go module and crate dependencies, respectively, from source.

From a downstream Fedora perspective, it sounds like this proposal would create separate side branches that require more maintenance work and can’t be included in the distribution’s primary RPM repositories, which would make supporting our practices more difficult.

That sounds wrong to me. If you formed a SIG around this work, it should be in scope of the Fedora Forge Usage policy. I don’t think it falls under dist-git’s scope which only contains RPM sources (rpms/*) and Flatpak manifests (flatpaks/*) (and associated gating tests (tests/*)) that are built in Koji and included in Fedora and EPEL repos.

Today, a source RPM’s BuildRequires are used to select binary RPMs that will be installed in a build root (e.g. one built by Mock).

If I understood him correctly, @ngompa suggested that a system that used src.rpm was possible and would not need a build manifest, and offered OpenPKG as an example of such a system. If I misunderstood him, I hope he’ll correct me.

In any case, the manifest isn’t really the key. It was easy to implement and allowed a lone developer to prototype and demonstrate a system in which upstream-release branches enabled a build that simply cannot be done in Fedora otherwise.

I think I understand one of your earlier comments better now than the first time I read it:

The dependency generation automation we use in Fedora is still used in the prototype. The dependency information in the RPMs is still generated, as it is today. There’s a copy in the rpm, yes, but it’s not new maintenance.

The manifest is a second copy of dependency information, but I don’t know if that would be required in a production implementation. I think a lot of improvements are possible, but a production implementation is something I hope we would take on as a project, not something I undertake as a lone developer. That’s why this was initially proposed as an initiative.

The vLLM build did not put any new RPMs in the primary repositories.

The build sequence creates binary RPMs, but they can be private as they were in Modularity or in Copr projects.

Python modules do not necessarily have sdists. At least one of the Python modules involved in the vLLM demo didn’t.

And even where sdists are present, we may need supplementary data to complete the BuildRequires.

Why couldn’t they be included in the primary dist-git repositories?

I’m going to be a little blunt here: please stop treating what other people are saying as things to just route around. We already had this conversation on Mastodon, and you just walked over the rebuttal as if it’s something to ignore. You tried to use Gentoo as a counterpoint, but I pointed out there that it too does curation based on shared maintenance capability and imposes limits similar to Fedora. You’re doing it again in this forum thread, and you need to change this.

But for everyone else here… This only benefits one person: the person packaging for themselves for their system. It foregoes every other axis for a shared system (auditability, maintainability, and supportability). Linux distributions fundamentally curate, simplify, and streamline for shared system maintenance. It’s also the crux of what makes the project scale.

What you’re proposing also creates technical debt. Fedora’s policies are deliberately designed to introduce friction for this, because otherwise nobody would be motivated to do work to drive upstreams to update their dependencies. In general, it’s a Bad Idea :trade_mark: to have a policy that promotes and maintains multiple versions in parallel as the default model, because it results in infinite combinations that have to be managed somehow. This breaks any reasonable community or commercial maintenance model since behavioral and security flaws can’t be fixed at any level of the stack reliably. Fedora Modularity also proved that doing this was just technically hard. If this model propagated to RHEL at scale, I don’t think Red Hat would consider RHEL a successful product anymore as the costs would massively eat away at the profit margin and destroy the value proposition of the product.

It’s also a fundamentally antisocial model. And what you did with vLLM was ultimately the wrong thing. You should have looked at it and said “how do I fix it to be more flexible and handle newer stuff?” rather than “how do I bend everything to ship whatever it says it needs?”. The former is the “how do I make it better?” mindset, and the latter is “how do I just have it?” mindset. Fedora’s philosophy and policies encourage the former, and attempt to discourage the latter.

To make it abundantly clear: your approach should be to work with other Red Hat contributors to vLLM to make vLLM better to support it in Fedora (which would propagate to RHEL and other things), because that’s the social way to make open source work.

OpenPKG does not use a build manifest because it uses the same mechanism that Fedora does: it works at head. It has a repository of SRPMs and there are tools that work through them to figure out what to build. It uses a custom repodata format that expresses source dependencies and source options for the openpkg tool to process to build what the user wants.

The company behind OpenPKG offers a commercial service for providing customized subsets for individual (non-redistributed) use-cases.

Fedora ultimately doesn’t offer this because we are fundamentally a binary distribution. If we wanted to offer “source distribution” features, then yes, we could add these features to our stack and extend Mock accordingly. But it doesn’t obviate the need for curation and streamlining what is available to make it maintainable.

4 Likes

Neal, I would like to reply to that, but I don’t know what it means.

I don’t think I’m doing that. I’m replying to the concerns that people bring up, as I understand them.

I didn’t ignore your point, at all.

I asked if components in Fedora should update to new release series when new release series are available, regardless of their impact on packages in Fedora, and you said “Yes, you break them. That’s the point. They either get fixed or they get failed out of the distribution.”

I think that’s unrealistic. Which is to say that I don’t think that’s how things should work and I don’t think that’s how things do work.

Not only did I not ignore your response, I reviewed the state of Rawhide to prove that’s not how things work today.

That wasn’t a reply to you, that was an observation made because it seemed funny.

It was something I wrote before there was a discussion on Mastodon.

Can you explain why you think that was a counterpoint? I’m genuinely confused here.

(Though, to your point about curation: while it’s possible that I misunderstand Gentoo’s build system, very nearly every library that I look at offers multiple versions for use in a source build, similar to the system I’m suggesting.)

Please refer to my earlier statement: I am trying to allow applications to update their dependencies.

There are at least three major problems with that idea.

First, I think that sets an unrealistic bar for package maintainers, because now you not only need to be familiar with the packaging policies and build systems generally, but you also need to be an expert developer not only in the languages that you ship, but possibly in other languages to handle the fallout of your changes. I’m glad that Fedora has developers among its contributors. But I don’t think that should be the baseline requirement.

Second, you are suggesting that I would have needed to make vLLM more tolerant of newer shared libraries, but most of the components in Fedora were too old, not too new. Patching for compatibility in that case would not create new value anywhere. Upstream will virtually never accept such patches.

And third, Fedora frequently ships adaptations of the kind you suggest and they don’t work well. We adjust things well enough to get something to build and maybe even pass post-build tests, but we end up shipping something that the upstream developers have never seen. And then they get bug reports for this thing they’ve never seen. And then they ask us to stop doing that, or to stop shipping their software entirely.

The most successful development in our industry in our lives has been the advancement of the concept that you test what you ship and ship what you test. Largely in the form of containers, which allow an application to be tested in conjunction with its dependencies. It is a practice that treats the application holistically rather than as a piece in a puzzle to be assembled elsewhere in the hope that it will work. The practice of modifying an application’s documented dependency set is the exact opposite of that. It leads to shipping things that have never been tested.