Looking back at EPEL 10, and forward to EPEL 11

Project Discussion
1

EPEL 10 launched about a year and half ago. By all accounts it has been a tremendous success. After an initial soft launch period, it officially launched with over 10k packages available. It has since grown to over 25k packages. Notably, EPEL 10 introduced minor versions, which is the biggest changes in EPEL’s history. Many maintainers have expressed their appreciation for how this has improved their workflow.

As can be expected with a change of this magnitude, there have been a few minor rough spots with the transition. Our original plan was to have CentOS systems request the major version repo, and have RHEL systems request the major.minor version corresponding to the OS. We eventually realized that this would cause upgrade problems going from one minor version to the next, and had to make a small pivot to the implementation.

We settled on a solution using a combination of MirrorManager redirects and mirror symlinks. The DNF variable $releasever_minor is defined on RHEL, but not defined on CentOS. Using DNF’s parameter expansion, we insert additional characters if that variable is defined. That works out as follows for the current minor versions:

metalink repo name redirected to used by
epel-z-10 epel-z-10.2 RHEL
epel-10 epel-z-10.3 CentOS
example baseurl path symlinked to used by
pub/epel/10z pub/epel/10.2 RHEL
pub/epel/10 pub/epel/10.3 CentOS

This works well enough and has been in production for over a year now. Users generally don’t need to understand these implementation details, and can just install epel-release as they always have.

However, some users consume EPEL by syncing the content to a private mirror rather than using epel-release. We’re starting to see reports of these users making the mistake of mirroring pub/epel/10 (instead of pub/epel/10z) for use on RHEL. Often times this will work initially, then later result in package dependency issues.

That sets the background for two ideas I want to put forward.

Can we do this better in EPEL 11?

I think the EPEL 10 minor version design has overall proven itself and should remain the foundation for EPEL 11. I also think we can iterate and improve on the redirect and symlink portion of the implementation.

I mentioned before that we keyed off the $releasever_minor variable in EPEL 10. Hypothetically we can instead key off the $stream variable for EPEL 11, which is set on CentOS but not on RHEL. That would invert which distro uses extra characters in its redirects and symlinks, as well as simplifying the repo names in MirrorManager.

metalink repo name redirected to used by
epel-11 epel-11.0 RHEL
epel-11s epel-11.1 CentOS
example baseurl path symlinked to used by
pub/epel/11 pub/epel/11.0 RHEL
pub/epel/11s pub/epel/11.1 CentOS

The change in the repo file would look something like this:

-#baseurl=https://download.example/pub/epel/$releasever${releasever_minor:+z}/Everything/$basearch/
+#baseurl=https://download.example/pub/epel/$releasever${stream:+s}/Everything/$basearch/

-metalink=https://mirrors.fedoraproject.org/metalink?repo=epel${releasever_minor:+-z}-$releasever&arch=$basearch
+metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever${stream:+s}&arch=$basearch

There would be no change for users utilizing epel-release, they would just install it and get the appropriate repos, just like they do today. Users creating private mirrors can use the previously pattern of pub/epel/$major to get the appropriate content for RHEL. We would be shifting the “I mirrored the wrong path and got the wrong content” problem to CentOS users, but I’m not terribly worried about that for two reasons:

  1. I think it is significantly less likely for CentOS users than RHEL users to be using a private EPEL mirror.
  2. Package dependency issues are less likely when using an older EPEL minor version than you should on CentOS, compared to using a newer EPEL minor version than you should on RHEL.

If we change this for EPEL 11, should we change EPEL 10 to match?

I’m a sucker for consistency, so naturally I’ve been asking myself if we can transition EPEL 10 to work the same way. Asking people who create private EPEL mirrors to change their behavior for EPEL 10, and then change it again for EPEL 11, is not ideal. Granted some are already doing that, but the vast majority of CentOS/RHEL users are not using 10 yet. These users will likely never become aware of these infrastructure implementation details if we make this change in EPEL 10 before they start using it.

I do think there is a realistic migration path we can use here. Normally we are moving the redirects and symlinks forward as we transition through EPEL minor versions. For example, next month we will be creating EPEL 10.4, and under the current process will be changing epel-10 and pub/epel/10 to point to 10.4. But what if we left those references at 10.3, and created new ones with the 10s pattern pointing to 10.4?

Here’s how that would look at the creation of EPEL 10.4 (while RHEL is still on 10.2):

metalink repo name redirected to used by
epel-z-10 epel-z-10.2 RHEL
epel-10 epel-z-10.3 CentOS with old epel-release
epel-10s epel-z-10.4 CentOS with new epel-release
example baseurl path symlinked to used by
pub/epel/10z pub/epel/10.2 RHEL
pub/epel/10 pub/epel/10.3 CentOS with old epel-release
pub/epel/10s pub/epel/10.4 CentOS with new epel-release

Once RHEL 10.3 is released in the fall, it would look like this:

metalink repo name redirected to used by
epel-z-10 epel-z-10.3 RHEL with old epel-release
epel-10 epel-z-10.3 RHEL with new epel-release,
CentOS with old epel-release
epel-10s epel-z-10.4 CentOS with new epel-release
example baseurl path symlinked to used by
pub/epel/10z pub/epel/10.3 RHEL with old epel-release
pub/epel/10 pub/epel/10.3 RHEL with new epel-release,
CentOS with old epel-release
pub/epel/10s pub/epel/10.4 CentOS with new epel-release

epel-release would be updated with these new patterns, but only published in the EPEL 10.3 repo. CentOS users would switch to the new pattern once they update epel-release. Once RHEL 10.3 is released, epel-z-10 and pub/epel/10z would move forward to 10.3 resources as expected. RHEL users would switch to the new pattern as they upgrade, using epel-10 and pub/epel/10 going forward. We can keep moving the epel-z-10 and pub/epel/10z references forward as well for compatibility.

Interestingly, this also aligns better with the changes that will happen when RHEL 10 reaches maintenance phase. After that there will be no more minor versions of RHEL 10, and epel-10 and pub/epel/10 will point to the 10.10 repos. This move would be aligning the no-suffix references with the current RHEL minor version early, made possible by CentOS switching to the suffixed references of epel-10s and pub/epel/10s.

Summary

Please share your feedback on these ideas. In particular, I’m interested in any drawbacks I’ve overlooked, how this would impact any third-party tooling, and other impacts on consumers of EPEL.

I also want to emphasize that this isn’t an all-or-nothing proposal. While I think both are good ideas, it’s also reasonable to conclude that the change only makes sense for EPEL 11 going forward, and EPEL 10 should be left as-is. When I bring this to the EPEL Steering Committee I’ll explicitly ask for separate votes.

5 Likes
F46 Change Proposal: Changes Discussion Only On Devel List
2

Maybe it should be 11stream instead of just 11s (nobody is typing this after all, right?), or something similar to make this more obvious what it’s for?

If 10 can be adapted in a compatible fashion, that’s probably for the best.

2 Likes
3

Thanks for the detailed write up! I am coming at this wearing my fedrq maintainer hat and also my person who likes consistency hat.

Re. fedrq, I have to keep tinkering around with how to best handle this change if it goes forward. I might end up creating a separate ticket in fedrq’s issue tracker about the implementation details if my attempt at summarizing the problem here gets too long.

I noticed an inconsistency with the proposed metalink repo names for minor versions between EPEL 10 and 11. epel-z-10.3 and epel-z-10.4 differ from epel-11.0 and epel-11.1. Would it be possible to unify these?

4

I really like this idea. I’ve always thought it makes more sense to add a suffix to the Stream variant rather than to the Enterprise Linux variant. One small change I would suggest, if possible, is to use 11-stream to stay consistent with https://mirror.stream.centos.org.

That said, there is one problem: you’re not the first person to propose this approach. The CentOS Kmods SIG originally used the $stream variable in version 10 to distinguish between CentOS Stream and Enterprise Linux, but later reverted that change. The reason is that most Enterprise Linux distributions define $stream even though they are RHEL derivatives rather than CentOS Stream. They do this so their users can install and use the centos-release-* packages provided by the CentOS SIGs, even though those packages are intended for CentOS Stream. This effectively claims a level of compatibility that cannot be guaranteed.

In my opinion, this is a mistake on the part of those distributions. However, since this behavior has already been adopted, it would first need to be changed - at least for version 10 - before your proposal could be implemented. Otherwise, we would effectively limit the compatibility of epel-release to RHEL and CentOS Stream.

1 Like
5

This proposal does not answer how the DNF “stream” variable emerges on the CentOS Stream systems. Will it be installed by one of the *-release packages differing between CentOS Stream and other distributions, or will it be magically invented by DNF5? I think I don’t have to say that I would prefer the first option.

6

The $stream variable is set by centos-stream-release package, see centos-stream-release.spec · c10s · Red Hat / centos-stream / rpms / centos-stream-release · GitLab

As mentioned in my previous post, some RHEL derivatives also define this variable, even though they should not, as they are not Stream derivatives, e.g., see almalinux-release/almalinux-release.spec at a10 - almalinux-release - AlmaLinux OS Foundation Git Server

7

FWIW - I’ve filed #10 - AlmaLinux is not a CentOS Stream derived distro - almalinux-release - AlmaLinux OS Foundation Git Server

8

Sounds pretty reasonable as long as the ‘stream’ variable use can get sorted out. :slight_smile:

9

The proposal is really about inserting extra characters in the epel-release URLs on CentOS but not on RHEL (the inverse of what we do now). Almost no one will ever read those characters as long as epel-release works, so I’m not terribly concerned about what is “obvious”. We can create some docs on mirroring EPEL (that again almost no one will ever read) and call it a day.

To me, what those specific characters are is bikeshedding. Now I don’t mind a good bikeshed conversation, as long as it is recognized as such. Personally I have a strong preference against using the full Stream name. Besides a preference towards shorter names in general, my original thinking with 11s was that it’s similar to the dist tag from CentOS SIGs (e.g. el9s, el10s). I realize that s is from the name Stream, but there’s enough negative connotation around the Stream name that I’d prefer not to spell it out. Another possible (perhaps even better) candidate would be 11c to associate with the CentOS name instead of the Stream name.

10

Possibly, but it would complicate the implementation a bit as I believe that will require MirrorManager database changes. And unlike the redirect changes, it would be more difficult to have a clear cutover. It would also hurt the backwards compatibility a bit for people using the existing epel-release and overriding $releasever to use older minor versions. As long as it’s not too much trouble for tool maintainers to special case the 10 minor repos, this part is probably not worth the effort of changing on the infrastructure side. I’m open to relaxing my reservations about this if someone that knows MirrorManager internals better than me can lay out a transition plan for the repo names.

11

I’m not sure that it is most, as RHEL (the reference implementation) does not define it, and neither does Oracle Linux.

Yeah I remember the background on this, and I share your compatibility concerns. I will point out that they did this at a time when most CentOS SIGs only built against CentOS, because RHEL chroots had only recently been added to CBS and it took a bit of time for SIGs to start using them. I’m hoping this is just a legacy thing that can be dropped from those derivatives fairly easily.

I agree, distros aiming for RHEL compatibility should not be defining it. Whether or not EPEL uses this variable should not be dictated by compatibility bugs in RHEL downstreams.

1 Like
12

Is it possible to add additional aliases (e.g., epel-10.3 > epel-z-10.3) like was done in https://forge.fedoraproject.org/epel/releng/issues/98? That would help fedrq so we don’t need to add even more special casing for (EP)EL.

13

More aliases can be added, but the problem is adding an alias requires lots of manual button clicking in MirrorManager’s admin web interface. Multiply that by the debug and source repos, and the testing variants of each, and it gets out of hand quickly. We already have to manage a hefty number of redirect changes four times a year (two mass branches, two RHEL minor releases), so I’m not real keen on adding even more redirects for every minor version. The request you linked was reasonable because it was major version only and a one time thing, not requiring any ongoing maintenance.

One thing that would appease my concerns on this is if there was a way to automate the redirect changes, especially if we can integrate it into our ansible playbooks. But as far as I’m aware, MirrorManager doesn’t expose an API.

14

We (AlmaLinux’s ALESCo) spoke about this in our meeting last Thursday and the tl;dr is that we are onboard with the changes to the $stream variable and will adjust however is needed. The reason we carry the variable is so that CentOS Stream SIG content can be consumed within AlmaLinux. We did not discuss this change for 10 as I thought it was only concerning 11.

For the mirroring and dropping of the z, we were all 100% onboard that it’s a good thing for 11. For 10 the sentiment was a little different but I told the rest of ALESCo that as an EPEL steering member, I intend to vote in favor of dropping the z if it can be done along with 10.3. Ultimately everyone reluctantly agreed that it’s kind of a crap situation with no perfect solution.

If this is for whatever reason delayed past 10.3 to 10.4 or later I may not be in favor of it anymore as adoption will be significantly higher of EL10.