Should we rethink what software to package?

Project Discussion
1

Following the recent discussion on the devil list about “lightly-maintained packages” that did not lead to anything, I started to think about an issue that is, in my opinion, related, and shares the same root cause:

The Fedora project does not have enough maintainers for the currently existing (and rising) number of packages. Or rather, there is a small group of very active package maintainers who are responsible for hundreds of packages (I count myself to be in that group, being the primary maintainer of almost 400 packages).

To me it feels like a disconnect happened between the group of “main contributors” (> 100 packages) and others who are only responsible for a very small number of packages (< 10), with very few people in the “middle ground” (double digits). (Maybe I am wrong, since I do not yet have data to back this up.)

Since I don’t think that the current situation is sustainable in the long term, we should probably think about a deeper problem: Does Fedora actually need packages for certain kinds of software? I’m not talking about stuff that needs to be integrated with the system in any way (like system(d) services, “base” libraries, desktop applications/plugins, etc.), but rather about standalone applications that do not need integration with the system.

For example, some new statically linked tools like bat, lsd, or ripgrep are perfectly fine if you install them with cargo, and applications that are usually deployed with a different mechanism anyway (e.g. with bundler for Ruby gems) do not necessarily need to be packaged for Fedora at all, unless they’re needed by other packages.

However, those “leaf” applications tend to pull in a large number of additional dependencies, which the maintainer of the leaf application package is usually not interested in. They then usually get added to SIG groups, where they receive only limited maintenance (I can confirm this for the Java and Rust stacks).

TL;DR: I know this sounds like blasphemy in the context of a linux distribution, but I think we should consider whether it’s actually beneficial to provide packages for certain types of software, primarily if “leaf” applications are concerned (where other mechanisms like flatpak, cargo, or bundler are perfectly fine distribution mechanisms), or software that is usually used and deployed with a mechanism that is not RPM (node, pip, bundler, cargo, etc.).

6 Likes
2

As a user/sysadmin in my former life, I often found this to be the case with web stack applications — if I needed to deploy a complicated Django or Java app, having it packaged in Fedora (either the main collection or EPEL) provided little value.

Another category we might want to look at is applications where the developer is highly invested in providing a good Flatpak experience. Why duplicate?

I’m a little concerned about random utility leaf applications like ripgrep as you mention — I might not care exactly how they’re packaged, but as a user I would like a consistent install experience. These things are different from the big web app case, because the former is likely a single-purpose focused thing where having a separate procedure is basically natural.

I’m also concerned with making sure users have trustable packages — that’s a lot of the value we provide — how do I know that I’m not getting a version with a cryto miner built in?

3 Likes
3

Yes. We should do something about it. Those steps should be small and incremental. I thought that BuildRequires can be actually a good start. I created Introduce external BuildRequires by xsuchy · Pull Request #648 · rpm-software-management/mock · GitHub as a start. If/when people will likely we can extend it to DNF support it for run-time dependencies.

Disclaimer: I am making it “technically possible” while understanding that it may take time to be allowed in Fedora (or never at all). The target group is 3rd party packagers.

1 Like
4

As a user/sysadmin in my former life, I often found this to be the case with web stack applications — if I needed to deploy a complicated Django or Java app, having it packaged in Fedora (either the main collection or EPEL) provided little value.

Yes, that is the situation I was primarily thinking about.

Another category we might want to look at is applications where the developer is highly invested in providing a good Flatpak experience. Why duplicate?

I agree. It won’t make much sense for important first-party packages (e.g. the applications installed by default in Fedora Workstation), but a lot of third-party developers provide their applications on flathub themselves. There, packaging those projects for fedora does not provide much additional value.

I’m a little concerned about random utility leaf applications like ripgrep as you mention — I might not care exactly how they’re packaged, but as a user I would like a consistent install experience. These things are different from the big web app case, because the former is likely a single-purpose focused thing where having a separate procedure is basically natural.

Well, I brought up ripgrep because it seems to be popular with fedora developers. It pulls in more than 50 Rust libraries, and not a small part of them are only required by ripgrep and no other Rust packages in fedora. On the other hand, installing it with cargo is as simple as “cargo install ripgrep” (assuming you have your PATH set up correctly).

I’m also concerned with making sure users have trustable packages — that’s a lot of the value we provide — how do I know that I’m not getting a version with a cryto miner built in?

That is true. But I think we need to weigh both the advantages and the disadvantages here. I think we need to realize that packaging some software provides little additional value compared to the amount of maintenance work that is required to keep it running and up-to-date. Otherwise our users would be better served by using a flatpak version or installing it with the language-specific tool instead.

1 Like
5

I’ll be contrarian here and say that I have always found it to be tremendously valuable to have such applications packaged in Fedora. Having those there has made it easier for me to get up and running with something useful, regardless of the environment I’m trying to operate in.

I have operated as a sysadmin in environments where it’s functionally impossible to use Cargo, Maven, Pip/Poetry, etc. because the poor quality of provenance and reproducibility guarantees simply disqualify them from being used for development and deployment. You might think this is “old world”, but this is still a large portion of how systems are run today.

Even the container world continues to demonstrate major problems in this regard:

  • Lack of signatures on container images to verify authenticity and lack of tampering
  • Lack of proper reproducibility due to poor definition and handling with Containerfiles/Dockerfiles
  • Lack of provenance handling for all tasks (source verification, mirroring with verification, etc.)
  • Lack of useful source distribution for compliance

Flathub is basically a crapshoot. There are several major issues that make it extremely difficult to trust it as a quality source, much of those are shared with other “universal package manager” systems.

Of the reasons I’m willing to list, these are the ones that I think make it impossible to properly rely on:

  • Inability to apply our higher standard for legal compliance
  • No medium in which sources are collected and distributed for being able to build and rebuild Flatpaks
  • The laissez-faire model of responsibility for quality of service means there’s no way to provide a consistent, high-quality experience

I think it’s important to recognize that if we start doing this, we lose a great deal of the value Fedora itself provides to its users and developers. It’s in the name of the product: we are a Linux distribution and that means our value is directly proportional to the amount of useful software we provide. Fundamentally, you start taking that away, we start having less of a reason to exist.

Also, I want to point out another angle to this: we have generally been quite bad at the on-boarding process to get contributors up and going to contribute to the project as a whole, but this is especially true with supporting our language stack SIGs. The development of the Fedora Join SIG is a step in the right direction to fix this, but I firmly believe one of the reasons why we’re having these problems is because we do not have a good story for converting users into contributors to any and all aspects of the distribution.

4 Likes
6

For me, if this was to be implemented, I would be looking for another distro.

3 Likes
7

It seems that some of what I wrote was misunderstood. I like to use distribution-native RPM packages. I like making them (most of the time). Otherwise I would not be primary maintainer for almost 400 packages in Fedora.

I just think that we should reconsider packaging everything, and instead focus on the things that are actually used and actually useful.

3 Likes
8

By virtue of how things even get added into the distribution in the first place, we are intrinsically already doing that. I’m not going to just shovel in the whole of PyPI into Fedora, for example. People submit stuff that they want to use, we don’t just automatically pull stuff into Fedora.

1 Like
9

I will probably do the same if Fedora stops offering RPM packages for Flatpaked applications.

1 Like
10

Admittedly, my practical experience here is ten years old, but I haven’t seen much indication of things getting better. The specific problems I (repeatedly) encountered are:

  1. Web applications that expect modifications to config files, etc., in their application tree. This makes a messy mix with RPM content.
  2. Applications which don’t upgrade cleanly just by having their files replaced. Some which expect you to upgrade through their own web interface.
  3. Particularly with Java, specific jar versions being much more important than the Fedora packaging knows
  4. Application bugs fixed in vendored packages in a way that doesn’t go back upstream (maybe even won’t be accepted upstream)

… but, yeah, I also agree that having RPMs or something like them brings a lot of upsides, and that the various language ecosystems and container approaches have thrown at least some of the baby out with the bathwater.

So, this is actually why I’d like to keep at the “lightly maintained” packages idea. It hasn’t led to anything yet but there’s not like there’s an expiration date on the problem.

11

This is a big issue. I have been trying to figure out how to help on various fronts (I have many interests, choosing where to help would be logical first step) for some time, but can’t easily tell which instruction(s) to follow.
To begin with, old versions of instruction pages, of which there seems to be several different versions, should direct to the most recent ones.
Most people will likely give up and go somewhere else to help if data is not readily available or leads up to nothing.

2 Likes
12

While I don’t maintain much packages, I noticed a lot of software I work with regularly are the kinds that could not be packaged in accordance to current Fedora packaging guidelines … as they are primarily complex server software that requires version locks to specific library versions (Hadoop, server apps written in Python, etc).

They are usually installed by me using vendored RPMS if available or, i just create a vendored RPM myself , ie: install on /opt, carry their own dependencies, integrate well by having systemd files, sane default config that respect filesystem hierarchy standard, entry point scripts in /use/bin, etc … On the same time, C libraries and headers provided by fedora RPMS are extremely useful and helpful for building these softwares and for installing their dependencies … without those RPMS, it would be frustrating …

Imho, … C based software with development headers or kernel modules should be always packaged as RPMS … while things that carry their own package managers like Python, node, ruby, etc, fedora should ideally provide the RPMS for the core interpreter and perhaps also provide tooling to help make it easy to create vendored RPMS for own use or for submitting to copr …

Over the years, I’m noticing quite a bit of old unmaintained package still land on latest fedora , simply because their builds never failed … but the software already no longer working … these imo need to be cleaned up sooner or later … And a better governance on what should or what doesn’t really need to be packaged hopefully would help …

2 Likes
13

The standards required for rpms to be included in Fedora are the reason I prefer them over flatpak. I know I can reproduce the build. I agree there is a problem packaging newer languages with their own distribution method. Not requiring individual packages for each library is probably the best we can do for the language specific distribution problem. I agree there is a problem with web appllications in particular.

I have only encountered one broken package in Fedora and it was removed later I presume because of that. I have found older version of software which have been annoying. Usually we are pretty good at getting new software. I created a VM of rawhide to get the latest godot at one point. Still waiting for nim 1.2. (I tried updating the package myself but couldn’t get it to work.)

2 Likes