Status update:
On 2023-02-08 the EPEL Steering Committee voted and agreed to the overall direction of this proposal, summarized as “per-minor-version EPEL 10 repos”. The exact implementation details are still being worked out. There are a few spin-off threads to focus on specific details:
Now that EPEL 9 is in full swing, I’d like to start planning ahead for what comes next. CentOS Stream 10 is expected to be available in 2024. We should be able to start EPEL 10 around the same time. Until then, we have the opportunity to evaluate what we can improve in EPEL.
I am proposing a new workflow and structure for EPEL 10. The high level summary is for EPEL 10 to have unique branches, build targets, and repos for each minor version of RHEL 10, including CentOS Stream 10 as the upcoming minor version. This would be a significant change from how EPEL works today, but I think it would address several pain points for maintainers and users. I am opening this topic for discussion as early as possible before the EPEL 10 launch to gather feedback. Please note that this is currently just a proposal and has yet to be voted on by the EPEL Steering Committee.
Before getting into specific details, let’s recap how things currently work in EPEL.
The current EPEL model
In the past (EPEL 7 and earlier), EPEL only targeted the current minor version of each major version of RHEL. Once a new RHEL minor version was released, the corresponding EPEL buildroot would switch to using it more or less immediately.
branch | built against | dist tag | repo |
---|---|---|---|
epel7 | RHEL 7.x | .el7 |
epel/7 |
New RHEL minor versions sometimes included library soname changes, which had the potential to cause EPEL packages to no longer install on the new minor version. One example of this was the rebase of ImageMagick in RHEL 7.8, which caused php-pecl-imagick, ripright, drawtiming, and others packages to no longer be installable until they could be rebuilt. Having affected packages installed could block system upgrades. Package maintainers were often caught off guard by these types of changes. Even if the maintainer knew the change was coming, there was nothing they could do to prepare their packages ahead of time. Thankfully these occurrences were rare overall.
This problem was exacerbated in EPEL 8 with the introduction of CentOS Stream. Now those library changes started happening in CentOS Stream 8 first, months before landing in the next RHEL 8 minor version. This resulted in some EPEL 8 packages being installable on RHEL 8 but not on CentOS Stream 8. EPEL 8 Next was created to solve this problem. It allowed maintainers to optionally build against CentOS Stream 8 and provide the resulting packages in a separate repo.
branch | built against | dist tag | repo |
---|---|---|---|
epel8-next | CentOS Stream 8 | .el8.next |
epel/next/8 |
epel8 | RHEL 8.x | .el8 |
epel/8 |
This separate repo was not meant to be a complete duplication of EPEL. It was intended only for packages that needed a rebuild to work on CentOS Stream 8, so CentOS Stream users needed to enable both the EPEL and EPEL Next repos. Once the CentOS Stream 8 change landed in RHEL 8, the package would be rebuilt again with a higher release than both the existing EPEL and EPEL Next builds to facilitate a proper upgrade path for all users. This structure works well enough to allow maintainers to target both RHEL 8.x and RHEL 8.x+1 (CentOS Stream) when needed.
EPEL 9 eventually aligned to the same model as EPEL 8, but started off slightly differently. CentOS Stream 9 was available before RHEL 9. EPEL took advantage of this to launch early and give maintainers more time to prepare their packages. Packages were built against CentOS Stream 9 with an .el9
dist tag, and directly populated the main EPEL 9 repo.
branch | built against | dist tag | repo |
---|---|---|---|
epel9 | CentOS Stream 9 | .el9 |
epel/9 |
After the RHEL 9.0 release, the EPEL 9 buildroot was reconfigured to build against actual RHEL 9. At this point things worked the same as EPEL 8, with EPEL 9 Next continuing to be built against CentOS Stream 9 and available to maintainers if they needed it.
branch | built against | dist tag | repo |
---|---|---|---|
epel9-next | CentOS Stream 9 | .el9.next |
epel/next/9 |
epel9 | RHEL 9.x | .el9 |
epel/9 |
Packages built early against CentOS Stream 9 later worked as expected on RHEL 9. Launching early in this manner had a huge positive impact on package availability.
EPEL Next drawbacks
EPEL Next has solved real problems and been beneficial for maintainers and users. Unfortunately, it has also proven to be flawed in two primary ways.
The first is that it is easily misunderstood. Some maintainers try to treat it as a duplicate EPEL for CentOS Stream and build for EPEL and EPEL Next simultaneously when it is not necessary. Some maintainers only update their packages in EPEL Next and ignore EPEL. Even maintainers that understand the correct usage still have to think carefully about when they should use EPEL Next and when they shouldn’t, which has led to mistakes. We’ve tried to document this the best we can, but it is difficult for documentation to overcome the fact that the EPEL Next workflow is not intuitive.
The second is the problem of duplicate work. When a package needs to be rebuilt for a library change, it must first be rebuilt in EPEL Next, then months later rebuilt again in the main EPEL. There is no way for maintainers to reuse their EPEL Next builds in the main EPEL. Even if a way to do this was implemented, having packages with the .next
dist tag suffix in the main EPEL repo would likely add to the confusion.
These flaws led me to wonder if we could improve things for EPEL 10. How could we allow maintainers to create builds before a RHEL minor release that work on that minor release on day one? How could we make the workflow more intuitive for maintainers?
Inspiration from Fedora
Since EPEL is part of the Fedora project, and EPEL maintainers are also Fedora maintainers, I looked to Fedora for ideas about how we could make the workflow more intuitive. Here is an overview of the Fedora branch structure.
branch | built against | dist tag | repo |
---|---|---|---|
rawhide | Fedora Rawhide | .fc38 |
fedora/linux/development/rawhide |
f37 | Fedora 37 | .fc37 |
fedora/linux/{releases,updates}/37 |
f36 | Fedora 36 | .fc36 |
fedora/linux/{releases,updates}/36 |
.fc35 |
archive/fedora/linux/{releases,updates}/35 |
Rawhide is leading branch for Fedora development. Rawhide builds have a dist tag that reflects what the next Fedora version will be. Fedora release branches are forked off of the rawhide branch. Maintainers are given the flexibility to do their work in as many branches are appropriate per the updates policy. Sometimes it makes sense to only make a change in rawhide and allow it to propagate into the next release once it’s branched. As Fedora releases reach their end of life date the corresponding branches are retired and the repos are archived.
EPEL 10 proposed branch structure
My proposal for EPEL 10 is to use unique branches, build targets, and repos for each minor version of RHEL 10, including CentOS Stream 10 as the upcoming minor version. The dist tag would include both the major and minor version that the package was built to be compatible with. This would facilitate reusing builds on RHEL that were originally built against CentOS Stream. Let’s walk through how this would be structured at different points of the EPEL 10 lifecycle.
branch | built against | dist tag | repo |
---|---|---|---|
epel10 | CentOS Stream 10 | .el10_0 |
epel/10.0 |
CentOS Stream 10 is expected to be available in 2024, well before the RHEL 10 release. We can take advantage of this to launch EPEL 10 early and give maintainers lots of time to get their packages built. This is similar to what we did for EPEL 9, but this time the dist tag and repo directory would indicate the minor version currently being targeted.
branch | built against | dist tag | repo |
---|---|---|---|
epel10 | CentOS Stream 10 | .el10_1 |
epel/10.1 |
epel10.0 | RHEL 10.0 | .el10_0 |
epel/10.0 |
Around the time of the RHEL 10.0 release, we would fork an epel10.0 branch from the epel10 branch. The epel10 dist tag would be advanced to the next minor version. Maintainers would focus their work in the epel10 branch, similar to how Fedora work starts in the rawhide branch. Maintainers would have the option to fast-forward merge or cherry-pick in older branches as appropriate. Some maintainers may choose to leave minor version branches alone unless a bug or security fix is necessary. Changes made only in the leading branch would be deferred for RHEL users until the next minor version, similar to how many updates work in RHEL itself. We would use two separate release packages, one for CentOS Stream and one for RHEL (and RHEL clones), to ensure users point at the correct repository.
branch | built against | dist tag | repo |
---|---|---|---|
epel10 | CentOS Stream 10 | .el10_2 |
epel/10.2 |
epel10.1 | RHEL 10.1 | .el10_1 |
epel/10.1 |
.el10_0 |
archive/epel/10.0 |
After the RHEL 10.1 release, we would retire the epel10.0 branch and archive the 10.0 repo. Users that choose to stick with 10.0 (either via RHEL EUS or by manually pinning) can continue to use the archived 10.0 repo and get compatible (but unmaintained) packages.
I believe this branching structure would be more intuitive for packagers due to its similarities to the Fedora branching structure. The best part of the current model is that it allowed prepopulating EPEL 9 with RHEL 9.0 compatible packages before the RHEL 9.0 release. This new model would extend that benefit to all RHEL 10 minor versions. After the end of life date for CentOS Stream 10, the epel10 branch would be retired and all work would take place directly on the epel10.10 branch.
FAQ
Note: I’ll expand this section as feedback comes in to address the most common questions.
Q: Does this mean I’ll be required to maintain my EPEL 10 packages in multiple branches?
A: No. Making changes in the minor version branches would be optional and will be left to the maintainers’ discretion. Many Fedora maintainers do a similar workflow for Rawhide.
Q: Right before RHEL 10.1 is released, CentOS Stream 10 will have already moved on to 10.2 content. How we will prevent 10.2 changes from affecting packages in the 10.1 repo?
A: We faced a similar challenge with the initial EPEL 9 launch. We can solve it the same way for each EPEL 10 minor version. In the 10.1/10.2 example, we would create a CentOS Stream 10 snapshot prior to any 10.2 changes landing, and temporarily building against that snapshot until the RHEL 10.1 release.
Q: Will this fix my QT/KDE upgrade problems right after a RHEL minor version release?
A: Yes. Maintainers will still have to rebuild packages for CentOS users when a library is rebased in CentOS Stream 10, but RHEL/clone 10 users will get the same builds automatically a few months later (at the next RHEL 10 minor version release) instead of having to wait for maintainers to repeat the rebuild work.
Q: What happens when CentOS Stream 10 reaches EOL, which is before the RHEL 10 EOL?
A: Once CentOS Stream 10 reaches EOL (projected 2030), the epel10 branches would be retired. At that point the only remaining branches would be epel10.10, which would remain for five more years through the RHEL 10 EOL (projected 2035).
Conclusion
RHEL has minor versions. The current EPEL structure mostly ignores that fact, which results in a subpar experience for EPEL users and maintainers. I believe this new structure would be a significant improvement over the current EPEL structure. Please share your feedback on this proposal to help refine it before the EPEL steering committee voting process.