EPEL 10 is expected to be launched in Q4 of 2024. You may start noticing parts of the infrastructure coming online sooner than that. Please do not start creating EPEL 10 builds until we announce that we are ready for packagers to start building.
I’m writing this post to share our planned timeline for EPEL 10, and to set expectations for the coming months. Early CentOS Stream 10 composes are already available, but it hasn’t been officially launched yet because the content set is still in flux. We expect to see the official launch announcement for CentOS Stream 10 later this year. We intend to align the EPEL 10 launch with that, which will give us the following approximate timeline.
August 2024: EPEL 10 hackfest at Flock (see below for more details)
Q4 2024: expected launch of CentOS Stream 10 and EPEL 10
Q2 2025: expected launch of RHEL 10 (three years after RHEL 9)
In the months ahead, you may notice parts of EPEL 10 infrastructure coming online, such as build targets in Koji and releases in Bodhi. We will need these infrastructure pieces in place to work out all of the details of integrating minor versions. We’ll send out an announcement when EPEL 10 is ready for package maintainers to start creating their builds. Please do not start creating EPEL 10 builds until we announce that we are ready for packagers to start building. Any builds created before this announcement may need to be discarded and rebuilt.
I will be running an EPEL 10 Hackfest at Flock in August (next month). The plan is to work on various infrastructure pieces related to EPEL 10. If we happen to have most of these pieces working prior to the hackfest, we can pivot our time to adding initial packages to EPEL 10. I hope to see you there!
The EPEL 10 hackfest at Flock took place yesterday. We were able to get enough of the infrastructure working prior to the hackfest to enable packagers in attendance to do a controlled trial run building packages. This also resulted in a few branch request bugzillas for dependencies, which I realize in hindsight was confusing for packagers not at the hackfest. I apologize for not setting that expectation ahead of time.
Here is a summary of what is and isn’t working at this point.
creating epel10.0 builds from epel10 branches (fedpkg build from an epel10 branch)
automatic bodhi updates, similar to rawhide
What has not been completed yet:
publish the epel/10 repo
mirror the epel/10 repo
create epel-release for epel10
epel10 configs in mock-core-configs
Fedora CI tasks for epel10 pull requests
enable the testing period in bodhi
epel10 versions in bugzilla
With no published repo, no mirroring, and no testing period (0 days to stable in bodhi), this is not a good time to start consuming EPEL 10. However, it is a good time for packagers to start building their packages. To be clear, this is the announcement to packagers that they can start building their packages. I’m hoping that in the next week or two we’ll be able to get the mock configs done to enable packagers to do local builds, but in the mean time koji scratch builds are a solid alternative.
We built about 70 packages during the allotted time of the hackfest, and as I write this we just passed 100 builds. Things seem to be working well. Please let me know if you notice any issues, aside from known items on the “not completed yet” list above.
I have a (rather modest) list of packages that I have personally
built for the (theoretical) EPEL10 for my personal consumption
that I have built in a COPR.
I am a little uncomfortable submitting that modest list of
packages as new bugzillas (please build … for EPEL10)
as the process still seems to be not fully ready, and I
really want to avoid annoying packagers before all the
ducks have been aligned. Maybe I am just being
too conservative.
Thanks for trying this one. We set up the side tag pattern for signing, but hadn’t tested that workflow yet. Can you file a releng issue so myself and others can investigate? Several of us are still traveling back from Flock.
Quick note for anyone watching this thread, please keep in mind that some pieces of the infrastructure will continue to be in flux until the official launch. For example, our plan was to turn on automatic bodhi updates until the official launch. That worked for a little while and builds were created and moved to stable, but then that stopped working once we tried to enable a bodhi compose. We’ve implemented a separate compose outside of bodhi, but we need to temporarily disable new epel10 builds and revert it to finish processing builds that are still in “pending->testing” and “pending->stable” states. I’ll post here again once the builds can resume.
We were able to get all of the in-between builds processed, and have re-enabled the new compose method. We’ve also re-enabled the build targets, and verified that a new build got an automatic update that moved to stable automatically, just like rawhide. Builds that are marked stable should be available in the buildroot as soon as the regen-repo task finishes, and then composed nightly. This compose pipeline should stay the same until the official EPEL 10 launch in Q4. That’s when we’ll switch to the standard EPEL pipeline, with manually created updates, a default one week testing period, and bodhi composes.
Packagers can resume building for epel10 at their leisure. Let us know if anything doesn’t seem to be working as expected.
It’s just about time for the official launch of EPEL 10. I’ve created a checklist of items we need to complete for this.
We’ll be turning off automatic bodhi updates soon as part of this checklist, hopefully this week. For maintainers, that means that when you create an EPEL 10 build you’ll need to create the bodhi update yourself, just like you would for EPEL 9 or EPEL 8. We’ll also go back to the default of seven days in the testing repo before updates move to stable. If you need to build closely related packages together, you still have side tags (preferred) and buildroot overrides at your disposal.
I also want to remind maintainers to check for bugs that have been reported against your EPEL 10 packages. We’ve identified several that were built and released but are not installable. Most of these have had fail-to-install bugs filed already, and I intend to file bugs for the rest of them. If you get one of these bugs, please give it your prompt attention so we can have the repo in a healthy state by the time of the launch announcement. If you don’t think you’ll be able to resolve the installation problem within the next 5-10 days, reply as such in the bug and I can untag the build to remove it from the repo. The branch won’t be retired, so as soon as you resolve the installation problem it can be added back to the repo with a new build.
With the new minor versions for EPEL, I was expecting to see something like /epel/10.0/ in the mirrors, but I see there’s only a /epel/10/. I assume this will be coming soon and minor versions will get their own paths? Or is that not part of the plan?
When RHEL-10.0 is out, would there be an /epel/10.1/ that /epel/10/ would be symlinked to? I’m asking because it would be nice if there was always an /epel/10.x/ for all minor releases, including the CentOS Stream one.
My slides from the EPEL 10 Hackfest at Flock give an overview of the publishing pipeline.
Currently the EPEL-10.0 bodhi release is being published to epel/10/. Once the EPEL-10.1 bodhi release is created, it will be publish to epel/10/, and EPEL-10.0 will switch to publishing to epel/10.0/. CentOS Stream users should always consume the epel/10/ repo, and RHEL users should always consume the minor version corresponding to their OS minor version.
This specific pipeline is the result of two years of planning, and factors in the capabilities and limitations of things like koji, bodhi, mirrormanager, and more. We intentionally aren’t using symlinks at the recommendation of the mirrormanager developers.