EPEL 10 proposal

Status update:

On 2023-02-08 the EPEL Steering Committee voted and agreed to the overall direction of this proposal, summarized as “per-minor-version EPEL 10 repos”. The exact implementation details are still being worked out. There are a few spin-off threads to focus on specific details:


Now that EPEL 9 is in full swing, I’d like to start planning ahead for what comes next. CentOS Stream 10 is expected to be available in 2024. We should be able to start EPEL 10 around the same time. Until then, we have the opportunity to evaluate what we can improve in EPEL.

I am proposing a new workflow and structure for EPEL 10. The high level summary is for EPEL 10 to have unique branches, build targets, and repos for each minor version of RHEL 10, including CentOS Stream 10 as the upcoming minor version. This would be a significant change from how EPEL works today, but I think it would address several pain points for maintainers and users. I am opening this topic for discussion as early as possible before the EPEL 10 launch to gather feedback. Please note that this is currently just a proposal and has yet to be voted on by the EPEL Steering Committee.

Before getting into specific details, let’s recap how things currently work in EPEL.

The current EPEL model

In the past (EPEL 7 and earlier), EPEL only targeted the current minor version of each major version of RHEL. Once a new RHEL minor version was released, the corresponding EPEL buildroot would switch to using it more or less immediately.

branch built against dist tag repo
epel7 RHEL 7.x .el7 epel/7

New RHEL minor versions sometimes included library soname changes, which had the potential to cause EPEL packages to no longer install on the new minor version. One example of this was the rebase of ImageMagick in RHEL 7.8, which caused php-pecl-imagick, ripright, drawtiming, and others packages to no longer be installable until they could be rebuilt. Having affected packages installed could block system upgrades. Package maintainers were often caught off guard by these types of changes. Even if the maintainer knew the change was coming, there was nothing they could do to prepare their packages ahead of time. Thankfully these occurrences were rare overall.

This problem was exacerbated in EPEL 8 with the introduction of CentOS Stream. Now those library changes started happening in CentOS Stream 8 first, months before landing in the next RHEL 8 minor version. This resulted in some EPEL 8 packages being installable on RHEL 8 but not on CentOS Stream 8. EPEL 8 Next was created to solve this problem. It allowed maintainers to optionally build against CentOS Stream 8 and provide the resulting packages in a separate repo.

branch built against dist tag repo
epel8-next CentOS Stream 8 .el8.next epel/next/8
epel8 RHEL 8.x .el8 epel/8

This separate repo was not meant to be a complete duplication of EPEL. It was intended only for packages that needed a rebuild to work on CentOS Stream 8, so CentOS Stream users needed to enable both the EPEL and EPEL Next repos. Once the CentOS Stream 8 change landed in RHEL 8, the package would be rebuilt again with a higher release than both the existing EPEL and EPEL Next builds to facilitate a proper upgrade path for all users. This structure works well enough to allow maintainers to target both RHEL 8.x and RHEL 8.x+1 (CentOS Stream) when needed.

EPEL 9 eventually aligned to the same model as EPEL 8, but started off slightly differently. CentOS Stream 9 was available before RHEL 9. EPEL took advantage of this to launch early and give maintainers more time to prepare their packages. Packages were built against CentOS Stream 9 with an .el9 dist tag, and directly populated the main EPEL 9 repo.

branch built against dist tag repo
epel9 CentOS Stream 9 .el9 epel/9

After the RHEL 9.0 release, the EPEL 9 buildroot was reconfigured to build against actual RHEL 9. At this point things worked the same as EPEL 8, with EPEL 9 Next continuing to be built against CentOS Stream 9 and available to maintainers if they needed it.

branch built against dist tag repo
epel9-next CentOS Stream 9 .el9.next epel/next/9
epel9 RHEL 9.x .el9 epel/9

Packages built early against CentOS Stream 9 later worked as expected on RHEL 9. Launching early in this manner had a huge positive impact on package availability.

EPEL Next drawbacks

EPEL Next has solved real problems and been beneficial for maintainers and users. Unfortunately, it has also proven to be flawed in two primary ways.

The first is that it is easily misunderstood. Some maintainers try to treat it as a duplicate EPEL for CentOS Stream and build for EPEL and EPEL Next simultaneously when it is not necessary. Some maintainers only update their packages in EPEL Next and ignore EPEL. Even maintainers that understand the correct usage still have to think carefully about when they should use EPEL Next and when they shouldn’t, which has led to mistakes. We’ve tried to document this the best we can, but it is difficult for documentation to overcome the fact that the EPEL Next workflow is not intuitive.

The second is the problem of duplicate work. When a package needs to be rebuilt for a library change, it must first be rebuilt in EPEL Next, then months later rebuilt again in the main EPEL. There is no way for maintainers to reuse their EPEL Next builds in the main EPEL. Even if a way to do this was implemented, having packages with the .next dist tag suffix in the main EPEL repo would likely add to the confusion.

These flaws led me to wonder if we could improve things for EPEL 10. How could we allow maintainers to create builds before a RHEL minor release that work on that minor release on day one? How could we make the workflow more intuitive for maintainers?

Inspiration from Fedora

Since EPEL is part of the Fedora project, and EPEL maintainers are also Fedora maintainers, I looked to Fedora for ideas about how we could make the workflow more intuitive. Here is an overview of the Fedora branch structure.

branch built against dist tag repo
rawhide Fedora Rawhide .fc38 fedora/linux/development/rawhide
f37 Fedora 37 .fc37 fedora/linux/{releases,updates}/37
f36 Fedora 36 .fc36 fedora/linux/{releases,updates}/36
f35 Fedora 35 .fc35 archive/fedora/linux/{releases,updates}/35

Rawhide is leading branch for Fedora development. Rawhide builds have a dist tag that reflects what the next Fedora version will be. Fedora release branches are forked off of the rawhide branch. Maintainers are given the flexibility to do their work in as many branches are appropriate per the updates policy. Sometimes it makes sense to only make a change in rawhide and allow it to propagate into the next release once it’s branched. As Fedora releases reach their end of life date the corresponding branches are retired and the repos are archived.

EPEL 10 proposed branch structure

My proposal for EPEL 10 is to use unique branches, build targets, and repos for each minor version of RHEL 10, including CentOS Stream 10 as the upcoming minor version. The dist tag would include both the major and minor version that the package was built to be compatible with. This would facilitate reusing builds on RHEL that were originally built against CentOS Stream. Let’s walk through how this would be structured at different points of the EPEL 10 lifecycle.

branch built against dist tag repo
epel10 CentOS Stream 10 .el10_0 epel/10.0

CentOS Stream 10 is expected to be available in 2024, well before the RHEL 10 release. We can take advantage of this to launch EPEL 10 early and give maintainers lots of time to get their packages built. This is similar to what we did for EPEL 9, but this time the dist tag and repo directory would indicate the minor version currently being targeted.

branch built against dist tag repo
epel10 CentOS Stream 10 .el10_1 epel/10.1
epel10.0 RHEL 10.0 .el10_0 epel/10.0

Around the time of the RHEL 10.0 release, we would fork an epel10.0 branch from the epel10 branch. The epel10 dist tag would be advanced to the next minor version. Maintainers would focus their work in the epel10 branch, similar to how Fedora work starts in the rawhide branch. Maintainers would have the option to fast-forward merge or cherry-pick in older branches as appropriate. Some maintainers may choose to leave minor version branches alone unless a bug or security fix is necessary. Changes made only in the leading branch would be deferred for RHEL users until the next minor version, similar to how many updates work in RHEL itself. We would use two separate release packages, one for CentOS Stream and one for RHEL (and RHEL clones), to ensure users point at the correct repository.

branch built against dist tag repo
epel10 CentOS Stream 10 .el10_2 epel/10.2
epel10.1 RHEL 10.1 .el10_1 epel/10.1
epel10.0 RHEL 10.0 .el10_0 archive/epel/10.0

After the RHEL 10.1 release, we would retire the epel10.0 branch and archive the 10.0 repo. Users that choose to stick with 10.0 (either via RHEL EUS or by manually pinning) can continue to use the archived 10.0 repo and get compatible (but unmaintained) packages.

I believe this branching structure would be more intuitive for packagers due to its similarities to the Fedora branching structure. The best part of the current model is that it allowed prepopulating EPEL 9 with RHEL 9.0 compatible packages before the RHEL 9.0 release. This new model would extend that benefit to all RHEL 10 minor versions. After the end of life date for CentOS Stream 10, the epel10 branch would be retired and all work would take place directly on the epel10.10 branch.

FAQ

Note: I’ll expand this section as feedback comes in to address the most common questions.

Q: Does this mean I’ll be required to maintain my EPEL 10 packages in multiple branches?

A: No. Making changes in the minor version branches would be optional and will be left to the maintainers’ discretion. Many Fedora maintainers do a similar workflow for Rawhide.

Q: Right before RHEL 10.1 is released, CentOS Stream 10 will have already moved on to 10.2 content. How we will prevent 10.2 changes from affecting packages in the 10.1 repo?

A: We faced a similar challenge with the initial EPEL 9 launch. We can solve it the same way for each EPEL 10 minor version. In the 10.1/10.2 example, we would create a CentOS Stream 10 snapshot prior to any 10.2 changes landing, and temporarily building against that snapshot until the RHEL 10.1 release.

Q: Will this fix my QT/KDE upgrade problems right after a RHEL minor version release?

A: Yes. Maintainers will still have to rebuild packages for CentOS users when a library is rebased in CentOS Stream 10, but RHEL/clone 10 users will get the same builds automatically a few months later (at the next RHEL 10 minor version release) instead of having to wait for maintainers to repeat the rebuild work.

Q: What happens when CentOS Stream 10 reaches EOL, which is before the RHEL 10 EOL?

A: Once CentOS Stream 10 reaches EOL (projected 2030), the epel10 branches would be retired. At that point the only remaining branches would be epel10.10, which would remain for five more years through the RHEL 10 EOL (projected 2035).

Conclusion

RHEL has minor versions. The current EPEL structure mostly ignores that fact, which results in a subpar experience for EPEL users and maintainers. I believe this new structure would be a significant improvement over the current EPEL structure. Please share your feedback on this proposal to help refine it before the EPEL steering committee voting process.

9 Likes

I’d really appreciate the ability to stage builds for the next RHEL minor release! This would save time and make it so I didn’t have to scramble to rebuild ansible when a new RHEL minor release comes out. We’ve remedied this by relaxing the version constraints for ansible’s ansible-core dependency, but that’s a nonideal solution.

Even if the maintainer knew the change was coming, there was nothing they could do to prepare their packages ahead of time.

Yes, this is problematic and somewhat stressful. In addition to being able to prepare builds for the next RHEL minor release in advanced,there should be more visibility into when CentOS Stream changes will actually land. I watch the ansible-core Bugzilla component in RHEL to keep track of what’s coming done the pipe. There’s messages from the errata bot on when the build has been submitted, when the errata is RELEASE_PENDING in RHEL, and more, but there’s no notifications about which Stream compose it’ll be in either before or after it’s reached the mirrors.

Changes made only in the leading branch would be deferred for RHEL users until the next minor version, similar to how many updates work in RHEL itself.

I worry that this will be a quite confusing/jarring change for casual EPEL maintainers and maintainers who don’t care about CentOS Stream.

Around the time of the RHEL 10.0 release, we would fork an epel10.0 branch from the epel10 branch. The epel10 dist tag would be advanced to the next minor version.

Who will be responsible for this ongoing work? Will Infra/Releng get more staff/funding to take this on?

RHEL has minor versions. The current EPEL structure mostly ignores that fact, which results in a subpar experience for EPEL users and maintainers.

RHEL clones also mostly ignore this. This approach is somewhat of a middle ground. We’d have the ability to target specific minor versions, but there wouldn’t be any EUS. If we move forward with this, I think it’s important we make that clear to users.

This doesn’t solve the problem, and I don’t want to derail this conversation, but maybe the CentOS feeds could help a bit here in knowing when things land?

https://feeds.centos.org/

A more open and connected build system would obviously be more ideal, but in the meantime…

To be clear this proposal wouldn’t address this issue, but it would effectively limit the “blast radius” for EPEL packages to only CentOS Stream, rather than affecting CentOS Stream and RHEL and RHEL clones like it does today.

I agree this is a potential concern. But the stance of “caring about CentOS Stream is optional” is what we have today with EPEL Next, and I would argue that it’s not really working. Fedora maintainers aren’t really allowed to completely disregard Rawhide, and I feel the same mentality should apply to CentOS Stream.

It would indeed be a non-zero amount of additional work. That said, much of that work is quite similar to the work that the CPE Releng team already does for Fedora every six months, so I think modifying existing scripts and automation to handle this should be possible. It’s an open question whether the CPE Releng team would be able to get additional headcount to better handle it. Reallocating current CPE staff or granting more permissions to the CPE EPEL team are also possibilities that are being discussed. This is another reason why I wanted to open this for discussion so early. For now I’d like to focus on whether this is a good idea on it’s technical merits, and then allow CPE to make staffing and allocation decisions around that.

Yes, and also most users would be able to keep ignoring this because the release package would point them at a symlink to move them forward through minor versions. The difference is that at minor version release time they should get compatible packages right away instead of having upgrade issue as maintainer work to repeat their EPEL Next work.

What will the epel-release package’s .repo file point to? It seems to me that it should by default point to the latest released version number. So maybe there needs to be a “10.latest” tag or something like that. Maintainers will probably want to build most updates for that version, similar to what they do now. They should also build for the upcoming version. Although this is starting to sound a lot like epel9-next, isn’t it.

We will need two epel release packages. For now lets call them epel-release-centos and epel-release-rhel. The epel-release-centos repo files would point to a symlink that tracks the absolute latest minor version repo. The epel-release-rhel repo files would point to a symlink that track the current minor version of RHEL. I could see that looking something like this:

  • /pub/epel/10/pub/epel/10.1
  • /pub/epel/centos10/pub/epel/10.2

Yup, that’s pretty much how the current EPEL + EPEL Next structure works. This proposal would effectively flip that on it’s head. In the current structure building for the upcoming minor version is optional. In this proposed structure building for the current minor version is optional. The former results in a subpar experience for all users, but the latter would ensure compatible builds for CentOS users and also provide smoother upgrades for RHEL/RHEL clone users due to the staging mechanism of the repos.

1 Like

You didn’t mention -testing repos, but I assume they are implied for each release repo. Should there also be an easy way for epel-release-rhel users to enable installing (probably using --enablerepo) from the centos repos on request? Would such people be advised to additionally install epel-release-centos even though they are using rhel or a rhel clone?

Yes, there would be corresponding testing repos for each minor version.

An epel-release-rhel user could temporarily override $releasever to select a specific minor version repo to install from, either newer or older.

dnf --releasever 10.2 install foo

The resulting package might not work correctly since it was built to be compatible with a different minor version, but this could be useful for troubleshooting or to preview a package that a maintainer hasn’t built in an older minor version branch yet.

No, the release packages would conflict with each other and only one would be installed at a time.

I can see this improving a lot of the situations we’ve seen time and time again (with your examples of current branches, and no doubt EPEL 6 and 5 also had similar issues in the past, and especially with things like qt and the like for 8 and 9). And with stream’s existence, I can also see the EPEL 10 idea actually make it easier in the long run to be ready for RHEL’s minor release cadence. I also kind of think that “caring about CentOS Stream is optional” should be put to the side, especially with your example of Fedora rawhide in your later comments. IMO Stream should not be disregarded. It’s there and it should be used, especially if it can make things easier for EPEL as a whole.

My only concern is folks version pinning and then never updating/upgrading or doing something else wacky entirely. Though for EUS users, I can see the 10.X directories actually helping in some way, that way they’re not left out entirely of EPEL or they can’t install something because they’re version locked on certain libraries. As long as the policy is laid out and clear that the previous branches are not supported and aren’t updated (aka retired), then I think things will generally work out in the long run.

Though out of curiosity (and pardon my ignorance on the processes in general), let’s say this proposal is accepted and used for 10 in the coming years. Would there ever be an opportunity for this to be backported so to speak for EPEL 9? Though I imagine backporting this idea would probably create/cause more issues than solve since EPEL 9 has a process already, so maybe not. But it’s something that came to mind while reading and thinking through this idea, as I think it would help EPEL 9 too.

Generally, I like the idea :heart: To a highly experienced Fedora and RHEL+EPEL packager, it makes perfect sense.

I was afraid that casual EPEL contributors (who often simply branch and build everything from Fedora without looking left or right) might consider this overcomplicated, but OTOH if the official recommendation is “just build from the epel10 branch forever and the changes will go to c10s+EPEL and eventually land in the next RHEL+EPEL 10.x version as well” then this is actually way easier than the current EPEL 8/9 Next approach :+1:

Technicalities

(a.k.a. what color should we paint the bike shed)

What if the epel10.x branch was only created on demand by the packager who needs it?

I realize that creating a new branch is currently blocked on a human release engineer, which makes it uncomfortably slow, but AFAIK work is underway to unblock this and this is only needed for PDC and similar other complex systems that packagers do not care about.

Ideally, an EPEL packager should be able to create the branch by pushing it and build immediately. This could be accomplished by “creating” all the epel10.x branches in PDC when EPEL 10.x forks (or when a new epel10 branch is created), but not creating them in dist-git. That way, packagers who do not (need to) care, would still only see The One Branch and use only that one. Experienced EL packagers who want to backport a security bugfix to “z stream” etc. would know that they need to git switch -c epel10.x && ... && git push -u origin epel10.x to do it. Similarly, others would not see several EPEL 10 branches and would not be tempted to always build from all of them (like you say they do with the EPEL Next branches).

When a packager needs to fast-forward a commit from epel10 to epel10.x to build it, they could even avoid the branch entirely and override Koji’s target instead (build from the epel10 branch in the epel10.x target). (Not sure if this is actually a nice and tidy magic trick or a horrible hack, just mentioning the possibility.)

The branch space would also be less “clogged” by branches nobody uses.

$releasever and the epel.repo files

I suppose the RHEL 10 $releasever is 10, not 10.x. So we cannot have a repo file that uses $releasever to point to 10.x specifically. How do we ensure that users of RHEL 10.x-1 don’t accidentally get the repository for EPEL 10.x? You mentioned symbolic links, but I am not sure how is that going to work. Is there support for “minor” releasever in dnf variables?

1 Like

The worst impacted now are packages that are by necessity tied to packages in RHEL - either because they link against them, or they ship missing subpackages.

Right now it’s a day 0 scramble at best (and requiring needless rebuild of the package first in epelX-next and then in epelX) with uninstallable packages at worst.

Really looking forward to this being implemented!

People already pin to specific minor versions of RHEL rebuilds, in spite of regular warnings not to do so. I’m not sure if this proposal would make them more likely to do so, but even if that’s the case I don’t think it should stand in the way of EPEL improving. As you mentioned, this will be documented in EPEL policies.

This is an interesting idea. I think it’s worth considering, although I suspect changing EPEL 9 partway through its lifecycle would be more trouble than it’s worth. This level of change makes the most sense at a major version boundary. I think doing so would need three things:

  • demonstrated value of the new model in EPEL 10 over some period of time
  • significant demand from EPEL packagers to revamp the EPEL 9 workflow to match EPEL 10
  • a solid migration plan

I do think that will be one of the recommended workflows. I’m glad it appears to be easier, since that is the goal. Some folks will want their EPEL package updates in EPEL for RHEL sooner, so we will also want to highlight the workflow of building in multiple branches.

I’ve heard about this work, but I don’t want to plan against that before it comes to fruition.

The problem with this is that without a visible branch, maintainers that are not familiar with the changes will just see an epel10 branch, build out of it, create an update, and then start asking why their update isn’t visible to RHEL users. We don’t do “lazy” branch creation in Fedora, so I wouldn’t want to do it for EPEL either. Being tempted to build in all active branches is less of a problem in this model than it is for EPEL Next. I expect some maintainers will want to update in epel10 and epel10.X at the same time, the same way some maintainers update their rawhide and fXX packages at the same time.

How many packagers do you think are aware that this is possible? It’s an interesting trick, but I’d argue that it adds more confusion than it’s worth. I want the EPEL workflow to be as intuitive as possible. EPEL Next has shown us that many maintainers don’t read the documentation, and just do whatever they think is correct based on their experience.

Looking at an example package with EPEL branches, there are currently 8 active branches and 31 retired branches. Adding one more active branch and additional retired branches is not a concern to me at all. I’m not a git expert but I believe that branches are no more expensive (storage, cloning, etc) than a tag.

dnf doesn’t currently has any variables to reflect the minor version, which is why we would manage this with symlinks on the infrastructure side. RHEL and CentOS users would have separate release packages, with separate repo files, pointing to separate symlink directories managed in the infrastructure.

  • /pub/epel/10/pub/epel/10.1
  • /pub/epel/centos10/pub/epel/10.2

The former would be redefined to the next minor version when a new minor version of RHEL 10 is released. The latter would be redefined to the next minor version as soon as it exists (prepopulated with content from the previous minor release). Avoiding getting incompatible builds would just be a matter of installing the correct release package.

I do think that will be one of the recommended workflows. I’m glad it appears to be easier, since that is the goal. Some folks will want their EPEL package updates in EPEL for RHEL sooner, so we will also want to highlight the workflow of building in multiple branches.

I’m strongly in favor of this proposal, in part because it would make it considerably easier and more obvious to just do the right thing, especially for folks that aren’t deeply involved in packaging minutiae.

The problem with this is that without a visible branch, maintainers that are not familiar with the changes will just see an epel10 branch, build out of it, create an update, and then start asking why their update isn’t visible to RHEL users. We don’t do “lazy” branch creation in Fedora, so I wouldn’t want to do it for EPEL either. Being tempted to build in all active branches is less of a problem in this model than it is for EPEL Next. I expect some maintainers will want to update in epel10 and epel10.X at the same time, the same way some maintainers update their rawhide and fXX packages at the same time.

Agreed. I don’t think lazy branch creation is desirable, unless/until it becomes the default for all of Fedora. Having it just for EPEL would be likely just be confusing for maintainers.

How many packagers do you think are aware that this is possible? It’s an interesting trick, but I’d argue that it adds more confusion than it’s worth. I want the EPEL workflow to be as intuitive as possible. EPEL Next has shown us that many maintainers don’t read the documentation, and just do whatever they think is correct based on their experience.

Indeed. I think we should strive to minimize the amount of complexity that’s required in our workflows, and overriding targets is far too much magic for my taste.

So, in general I like this plan, but there’s some concerns/issues. :slight_smile:

We have avoided / said we will not support/use symlinks on the master mirrors. We tried long ago and people came out of the woodwork saying they synced to things that didn’t understand symlinks. We could of course try again, but we will likely drop some mirrors if we do.

I’m thinking the release packages dance is going to be pretty complex. Can you spell it out in more detail? Say I am a 10_0 user, 10_1 comes out. Do we update epel-release in the 10_0 branch to point to 10_1? To stick with 10_0 you have to modify repo files?
Or say I install 10_0, but never update, then a year or two later (when 10_2 is current) I update. Do I get 10_2? 10_1? Nothing without manually pointing to 10_2?

Do we have some way to Conflict or disallow mixing say epel 10_2 with a 10_0 el?

I fear also the casual user might only push to epel10 (stream) and not realize that they could / should push bugfixes to other el branches. I suppose bug reports would help there.

This will of course mean more releng work. We will have to mass branch for epel, but I guess thats doable, especially if some folks help out.

It also might be nice to have some easy way for a user to tell what the current state is… ie, epel10_1 is current, epel10_2 is pointing to stream, etc.

I might think of more things, but it’s a pretty interesting idea. It’s good that we have a lot of time to think it over/write it up and get it solid. :slight_smile:

I feel bad for those folks, but I would hate to block this proposal on something as trivial as symlinks. They’re not exactly a new technology. Classic CentOS, Alma, and Rocky all use symlinks this way (major version symlink pointing to the current major.minor directory). Push come to shove I have no problem dropping mirrors that can’t support symlinks.

I don’t think there will be as much dancing as you think. I don’t want the release packages to reflect the minor versions. It would be unnecessary update churn. I would like a layout as follows:

  • RHEL and RHEL clone users:
    • epel-rhel-release package
    • epel.repo file with repo=$releasever metalink
    • mirrormanager redirecting to $releasever (10) symlink on various mirrors
  • CentOS Stream users:
    • epel-centos-release package
    • epel.repo file with repo=centos$releasever metalink
    • mirrormanager redirecting to centos$releasever (centos10) symlink on various mirrors

Yup, this is another reason why I don’t want the release package to have minor versions. There isn’t a clean way to deliver the update that bumps users to the next minor version repo. Even if we figured something out, just running dnf update when RHEL has a minor release might not work, because you would need to upgrade your epel-rhel-release package before updating your packages from EPEL, in a separate transaction. It would be a mess and would hurt EPEL’s reputation. That’s why I want users’ systems to point to the symlinks, and then we move the symlinks on the infrastructure side. Pinning to an old minor version repo could be done by manually editing repo files, which is an appropriate opt-in to get unmaintained packages.

The separate release packages would conflict with each other. People could still manually configure custom repo files that enable multiple minor version repos at the same time, nothing we can do about that, but it would be generally harmless other than maybe having some packages that don’t install.

I think this will happen, and I think it will be fine. I’ve observed the same behavior in Fedora. A bug is fixed in Rawhide, then the user asks in the bugzilla if it can also be fixed in the current release. The maintainer either replies “yes” or “no, for update policy reasons”. I’ve also seen a similar process happen in RHEL itself. Worst case scenario, the fix is delayed until the next minor version. One of the benefits of running CentOS Stream is getting earlier access to bug fixes, and I’m in favor of extending that paradigm to EPEL packages.

Myself and @dherrera (CPE EPEL team) are ready and willing to help with this additional workload. My theory is that we can adapt the Fedora mass branch tooling to work for EPEL.

This would be self evident by looking in /pub/epel/ on dl.fpo. The centos10 symlink would always point to the latest minor version repo that exists. The 10 symlink would always point to the current minor version of RHEL 10, which users can determine from many places online, such as the RHEL life cycle page or the RHEL release dates page.

1 Like

I think one of the main reasons for not using symlinks was the sync to S3, but I just saw we do not sync to S3 directly any more for over two years now.

I am pretty sure MirrorManager does not care about symlinks and just ignore them completely. So if this proposal is implemented it probably also needs some support in MirrorManager to correctly handle the symlinks.

What happens when CentOS Stream 10 reaches it’s EOL, which is before RHEL’s EOL?

If MirrorManager ignores symlinks, then what support would need to be added? From that side it would just look like another directory. I was thinking something like this:

metalink params returned mirror path
arch=x86_64, repo=epel10 /10/Everything/x86_64/
arch=x86_64, repo=epel10-testing /testing/10/Everything/x86_64/
arch=x86_64, repo=epel10.0 /10.0/Everything/x86_64/
arch=x86_64, repo=epel10.0-testing /testing/10.0/Everything/x86_64/
arch=x86_64, repo=epel10.1 /10.1/Everything/x86_64/
arch=x86_64, repo=epel10.1-testing /testing/10.1/Everything/x86_64/

If I’m missing something regarding scanning or other nuances of how MirrorManager works I’d love to know more.

Once CentOS Stream 10 reaches EOL (projected 2030), the epel10 branches would be retired. At that point the only remaining branches would be epel10.10, which would remain for five more years through the RHEL 10 EOL (projected 2035).

This is somewhat similar to what we plan to do for EPEL Next branches in the current model. The epel8-next branches are planned to be mass retired once CentOS Stream 8 reaches EOL (2024-05-31), five years before the RHEL 8 EOL. The epel9-next branches are planned to be mass retired once CentOS Stream 9 reaches EOL (2027-05-31), five years before the RHEL 9 EOL. The branch names will differ in this new model, but the basic idea of retiring branches once their buildroot is EOL is the same.