Deprecating the EPEL Packagers SIG

Project Discussion
1

Back in 2020, the EPEL Packagers SIG was created. This was a separate and smaller group from the overall EPEL SIG. The intent was for this SIG to function somewhat like “proven packagers for EPEL”, but also have some similarities with language or topic based SIGs such as the Python SIG. The idea was that members of the SIG could request branches for new EPEL versions and share in the ongoing maintenance of those EPEL packages. It was implemented as a FAS group that needed to be granted permission on a package-by-package basis, so it was opt-in for Fedora maintainers.

Fast forward to today, and I do not believe the EPEL Packagers SIG is in a healthy state.

  • The SIG has permissions on a huge number of packages that have nothing to do with each other, so many that the notifications are overwhelming. It has gotten so bad that some people have voluntarily left the SIG.
  • The SIG is mostly used for requesting branches, with very little ongoing maintenance via SIG permissions. This often results in Fedora maintainers being stuck with bugs they never signed up for, which is not fair to them.
  • We’ve run into situations where people don’t respond to bugzillas for EPEL packages that they themselves have branched and built via SIG permissions. This type of behavior was recently mentioned on the epel-devel mailing list.
  • In our package request guide, there is a template for SIG members to use that asks for the user and the SIG to be added as co-maintainers. This is exacerbating the other problems, especially the lack of long term ownership as it is resulting in situations that I would summarize as “oh I didn’t mean it, I was just copying the template”.
  • A tracker bug was created with the intent that SIG members could use it to find packages to help with, but to my knowledge it has never been used for that purpose.

We’re in a situation now where this SIG is hurting the overall reputation of EPEL.

:warning: I propose that we officially deprecate the EPEL Packagers SIG, and eventually remove it entirely. :warning:

If we move forward with this, the first step would be to remove mention of the SIG from the request template. When maintainers are requesting to co-maintain packages to build them for EPEL, they would do so only via their direct account, not group membership. Ideally this would foster a greater sense of ownership. We wouldn’t need to remove the SIG from every package it has access to overnight, but we could consider an automated approach, or even look for a way to not carry this group forward to the new dist-git forge when that migration eventually happens.

I think one of the flaws that got us to this point with this current SIG is that the scope is far too wide (literally any packages that could go into EPEL). For closely related sets of packages, it may make sense to replace the EPEL Packagers SIG with another more focused SIG with fewer members. I think a small SIG with access to a related set of packages that all the members care about is a better model than a large SIG with access to far too many unrelated things.

Proven packager is still available for maintainers that do truly want elevated permissions to many packages, and we should encourage EPEL maintainers to pursue that if they are interested. It doesn’t allow people to request new EPEL branches, but at this point I think that is for the best. If you want to request branches, you should be a direct maintainer of the package.

We’re in the middle of an exciting bring up of EPEL 10, and it has been moving quickly in part due to the ease of requesting branches via EPEL Packager SIG membership. Deprecating the SIG will slow this effort down to a degree, but it is probably for the best. To avoid being blocked in EPEL waiting for dependencies in the future, I recommend co-maintaining the dependencies of your packages whenever possible.

3 Likes
2

Thanks for the suggestion. It makes sense to me for all the reasons that you mention.

Looking forward, we should keep in mind that the SIG was meant to solve problems which may still exist, and that we may need other solutions. I"ll refrain from spelling this out here so that the thread remains focussed. (proven packager is a shotgun, though)

3

I maintain Ceph in Fedora and CentOS CBS (Storage SIG).

When CBS added the use of EPEL packages that was a solid win because, fortuitously, all (or the vast majority) of the dependencies were already built in EPEL and I could stop building and maintaining all the dependencies in CBS. And all the conflicts between mismatched rpms between EPEL and CBS went away.

But not all the dependencies are in EPEL10 yet. And many maintainers are reluctant to build EPEL or EPEL 10 versions of their rpms, and once again I find myself building and maintaining dozens of packages, most or all of them substantially unrelated to Ceph itself.

The difference between being an EPEL packagers SIG member and proven packager seems pretty minor to me. (I’m not even sure I’d be granted proven packager, or if I even want it.) But if you remove the EPEL packagers SIG then I’d no longer have an avenue to get many of the needed dependencies built in a timely manner.

So I guess I give up the idea of getting Ceph built for Stream 10. Or I wait (for months, years maybe?) until all the dependencies finally land in EPEL.

C’est la vie.

4

One of the items which I get a LOT of emails about daily is to the epel-packagers-sig@lists.fedoraproject.org mailing list which gets pretty much every bugzilla associated with the group. There seem to be a large number in the last 48 hours for adding more packages to the epel-packagers-sig. I think that ‘removing’ the group is going to be hard in multiple ways:

  1. It is going to leave a lot of packages in EPEL with no maintainer. The package maintainer may have done exactly what the email says:
If you do not wish to maintain python-pikepdf in epel10,
or do not think you will be able to do this in a timely manner,
the EPEL Packagers SIG would be happy to be a co-maintainer of the package;
please add the epel-packagers-sig group through
https://src.fedoraproject.org/rpms/python-pikepdf/addgroup
and grant it commit access, or collaborator access on epel* branches.

This is means that now the packager who assumed that a group was going to maintain things is getting bugs and no idea of who is fixing them. I think any sort of ‘killing’ of the group is going to need a multi-stage thing where:

  1. Every package assigned to this group gets listed and triaged for who is going to really maintain it.
  2. It either gets an EPEL maintainer added OR gets listed as going to be removed.
  3. Help is going to be needed for people to ‘spin up’ appropriate maybe CentOS SIGs for storage etc so that groups can help deal with the load.
  4. A deadline is given for the changes to occur.
  5. Appropriate announcements are done.
  6. Removal
1 Like
5

Indeed, the challenge of branching for a new EPEL version will remain, because we intentionally don’t create these new major version branches automatically. I know you’re familiar with this because we recently discussed it in bugzilla, but I’ll list those reasons again here for everyone else’s benefit.

  • Packages are added or removed in new RHEL major versions, changing what is eligible to be in EPEL. Branch requests are when we validate that.
  • We don’t want to assume that a maintainer who is interested in maintaining one EPEL branch will be interested in maintaining the next.
  • Libraries in the base operating system change quite a bit in the years between RHEL major versions. We can’t assume that branching from rawhide will work every time, as sometimes we need to branch from older branches to ensure compatibility.
  • EPEL branches have a much longer lifecycle than Fedora, so opt-in branching is a useful signal that maintainers believe the branch is appropriate for most (if not all) of that lifecycle.

We’ve improved on the EPEL branching friction in recent years by automating branch requests, but the requestor still must have permissions on the package. Having that permission is one of the main purposes of the EPEL Packagers SIG. I think the better alternatives we should move towards are these two items I mentioned above.

It will take time to transition away from the EPEL Packagers SIG to smaller topical SIGS and direct maintainership, but I think it would be worthwhile.

6

It is no doubt a good idea to migrate common dependencies from CentOS SIGs to EPEL so that they can be maintained in a more collaborative manner and utilized as dependencies for other EPEL packages.

We can’t force Fedora maintainers to participate in EPEL. If we want Fedora packages built for EPEL, we must be willing to do that work ourselves as co-maintainers if the Fedora maintainer is not interested. It is not appropriate to approach these requests with a goal of merely avoiding work.

There are two main differences.

  • Proven packager gives you commit access to almost all packages without a group being explicitly added as a co-maintainer. The SIG only gives you commit access to packages where the SIG group has been added as a co-maintainer.
  • Proven packager does not give you the ability to request new branches. The SIG does give you the ability to request new branches for packages where the SIG group has been added as a co-maintainer.

For these reasons, proven packager is not a complete replacement for the SIG, but it can be used for the “I need to push a commit to fix this EPEL package” type of problem. For example, I’ve used this permission to rebuild EPEL packages that no longer installed after RHEL changed a library soname in a new minor version.

Yes, you would. The avenue is becoming a co-maintainer of those dependencies, same as it worked before the SIG existed.

I have no problem with Ceph in the CentOS Storage SIG being delayed if it means its dependencies in EPEL are better maintained. If that doesn’t work for you, then you can go back to maintaining all of the dependencies however you see fit in the CentOS Storage SIG.

7

Indeed, and this assumption is part of the problem with our current situation. There is an adage that goes something like “if it’s everyone’s job, then it’s nobody’s job”. Some maintainers even set the SIG as the default assignee for EPEL bugs, which all but guarantees that new bug notifications are missed in the flood of emails from the SIG. It’s not a sustainable situation.

Assuming the Steering Committee votes to move forward with this overall idea, I like these steps as the method we go about it.

8

FWIW - I was the person who drove the current iteration of EPEL Packagers SIG, and in hindsight I agree with Carl that it’s at a breaking point.

Per recent discussions at the steering committee we’re pretty much aligned that we should basically stem the bleeding by removing mentions of EPEL Packagers SIG from the templates, so this should hopefully happen soon.

Beyond that, Smooge’s plan makes sense. We should document the decision somewhere so we can ask releng to do the ACL changes in one go - since I suspect for many of these packages asking the main admin to change things will be time consuming.

Personally, I’ve already joined the GNOME SIG just so that I can maintain EPEL packages wearing that hat rather than the EPEL Packagers hat, and likewise with Golang SIG

1 Like
9

One problem that came up with the Golang bringup, by the way, is that thanks to Google deciding to use antlr the bringup hinges on bringing up a lot of Java tooling… but the Java SIG keeps dying on us so there is no group ownership of those packages.

This should probably be fixed at some point (I suspect Golang SIG will also just recommends vendoring, though IDK if that sidesteps the need for antlr … we’ll need to see)

10

So, I am very much in favor of this retirement.

I think the idea of more targeted sigs might be a good one. Perhaps (if there’s enough folks willing) there could be a epel-ceph sig/group founded to help out with that collection?

11

I am in favor of disbanding the epel-packagers-sig, and where possible, creating smaller SIG’s.

But I also realize that it will be a big job.

Let’s assume this proposal is passed and look at the work that needs to be done.

  • Easy Work: Some people (like myself) would be perfectly happy maintaining some of the packages themselves instead of the sig. We could create a list of all the packages the SIG owns, and ask people if they want to maintain any of those packages on EPEL. We then transfer them from the SIG to the person

  • Harder Work: Packages that can be grouped, create a specialized sig and assign to them. Then have people join that sig to maintain them. Such as the epel Ceph or rust sig.

  • ??? Work: The leftovers. Packages that don’t fall into either of the above two things. I currently don’t have any ideas what to do with these.

12

Well, if someone other than the usual responsible parties built them in EPEL, it would seem that those packagers should be assigned as the EPEL maintainer if no one else take it first (you built it, you now own it).

13

I like that idea. It seems logical.