The code that scans the primary mirror completely disregards anything that is a link because it was not necessary until now. If you look at the existing EPEL tree there are links like 7Server and 7Client and that does not appear anywhere in MirrorManager.
It is not that it cannot work, it is just that it was not necessary until now.
I am not sure it should be treated as just another directory as that would lead to a lot of duplicated entries in the database.
My thinking is that we currently have support for what MirrorManager calls repository redirects. If you look today at the result for Fedora 38 you will be redirected to Rawhide. At this point setting up repository redirects is a manual step (SQL) in the database. Maybe symlinks could be a way to automatically create redirects. That would also make our lives a bit easier in Fedora if repository redirects can be influenced from the releng side without SQL commands.
That sounds like a great approach. Thanks for your insight on this. Please message me directly when you’re ready to start looking at the implementation for this.
Back when I ran mirrors, there were two situations:
ftp and http servers that hid the fact that something is a symlink from clients. This isn’t a problem for the mirror itself but can cause problems
For something trying to validate the site
For people mirroring from that mirror and not otherwise deduplicating
In cases where the symlink creates a loop, people getting caught in infinite directory trees (well, not people. Bots.
Not symlinks but hardlinks — AFS didn’t support them, so that was annoying. (Otherwise, AFS is awesome for mirrors, because you can mirror to a R/W volume and then do an atomic update to the RO representation of that volume.) But I bet no one is doing AFS anymore.
I think the first case might still be of some reasonable concern?
I have a concern about Users thinking that these older epel “releases” are still supported. Especially users of RHEL EUS.
Here is the scenario.
RHEL 10.5 is out.
A user is running RHEL 10.2 EUS, which is still under support.
KDE has had a critical security update, along with several normal updates corresponding with updates in RHEL 10.4
That user tries to install KDE and is unsuccessful.
Current Way : User contacts the KDE Sig and/or EPEL asking why their KDE packages won’t install. We tell them they need at least Version Y of Library X that came out in RHEL 10.4. They update Library X, or don’t install KDE. Either way is good because they don’t get that bad security issue.
New Way: User looks at dl.fpo and sees 10.2. Assuming that is supported they point their epel repos to it. They install KDE, that has a critical security issue in it. They are blissfully unaware until something critical gets damaged or hacked. Then they start complaining to EPEL and the KDE Sig.
Symlinks do not show on web pages. Go to the page you linked to and try to tell me if 5 / 5server / 5client are linked together or separate.
I’m not against this proposal. It would save me alot of work. Just trying to figure out a good way to not confuse people.
So what if we leave things as they currently are in /pub/epel. We have 10 and next/10. No extra point directories. But everything else flows the way you have proposed.
If we want point directories, maybe have them in next/. So you would have next/10.2 and next/10.3, and next/10 would point at the latest version.
And we would archive them just like we’ve done it before, over in archives with the point directories.
I haven’t had time to digest this and coherently lay out all my thoughts about this, but there are a couple of points I want to bring up:
I’d prefer epel10 to use .el10 DistTag like older EPEL branches do. For point release branches, I’d rather see something like .el10~10.x or similar, which makes it sort lower automatically. This makes the Fedora-style workflow of building for the top (in this case epel10 and merging downward (epel10.0, epel10.1, etc.) work flawlessly. It also somewhat mimics how RHEL package builds work, though their workflow is “bottom-to-top” rather than “top-to-bottom”.
In the given scenario, users would see 10.5 and 10.6 repos in the /pub/epel/ directory. They would have to navigate to /pub/archive/epel/ to see 10.2. The footer on that page even mentions that the content there is no longer maintained. Between the word “archive” in the path and the footer, it seems to me like there would be sufficient indication that the 10.2 repo is no longer maintained. We can also describe this in the documentation.
We wouldn’t be asking every user to determine the current state. All they would do is install the appropriate release package for either CentOS or RHEL, which will give them the appropriate repo. They don’t need to parse if a path is a symlink or a directory. If someone is wanting to parse this on dl.fp.o for whatever reason, it should be obvious that the higher 10.x repo is for CentOS and the lower 10.x repo is for RHEL. We can also describe this in the documentation.
Even though this workflow would be a significant change, I do believe it is more intuitive, which will work in our favor. Getting this new model documented will be important. We can also do some promotion and education to further the understanding of this, such as conference presentations and workshops.
This seems more complicated to me. I worry re-using the next directory would confuse people that EPEL 10 Next exists when it does not. Having point directories makes it clear to users which minor release the packages in that directory are intended to work with. It also makes archiving easier. The current releng process is flawed because it can result in an archive snapshot accidentally containing packages that were built against a newer minor version, e.g. an 8.5 archive repo that has packages built against 8.6 libraries. Having the minor versions be separate repos avoids this problem entirely.
This occurred to me, but then we would lose one of the benefits of this proposal. Embedding the specific version that a package was built against is valuable. It allows users to tell at a glance approximately how old a package is. It lets people who use EUS or manually pin releases know the likely minimum minor version they need to be on to use that package. It would also give a good indicator of why a package doesn’t install if a user accidentally enables the CentOS-targeted repo on RHEL or a RHEL clone.
What would be gained by removing this?
Using .el<major>_<minor> in all branches also sorts correctly automatically. The Fedora-style workflow of building for the top and merging downward also works in both. What is the benefit of using your suggested format instead?
That’s a neat concept, but I’m hesistant to have this proposal rely on a hypothetical feature that we don’t know will land in time for CentOS/RHEL 10.
By everyone else, I assume you mean other third party repos like RPM Fusion. How does the current proposal not scale for them? How would major/minor version dnf variables make it scale better?
I think the dnf developers already have their work cut out for them to get existing dnf functionality working in dnf5 in time for CentOS/RHEL 10.
I’d like to highlight another benefit of this proposal.
Per EPEL policy, packages that are added to RHEL must be retired from EPEL. By the spirit of this rule, we would retire EPEL packages if they’re added to CentOS as well. However, that would leave RHEL users without access to the package for up to six month. Instead, we allow a temporary overlap between EPEL and CentOS, and then when RHEL catches up the package is retired. Our current process involves filing a bug to give the maintainer a heads up that the EPEL package will need to be retired when the next RHEL minor release occurs. We’ve had multiple instances of maintainers retiring the EPEL package as soon as they see these bugs, instead of waiting for the next RHEL minor release as directed.
In this new model, maintainers would be able to retire the package in the leading branch while leaving it active in the minor versioned branch. To put it another way, they could retire the package for CentOS and RHEL users independently. They would only need to retire the package once, and the next minor version repo would inherit the lack of the package at the appropriate time. This would be similar to how a maintainer can retire a package in Rawhide, but leave it active in the current Fedora release branches.
There have been many s here and no real objections other than concerns about details (one could say bikeshedding and nitpicks). What are the next steps to make this happen?
If that comes to pass, I think the best course of action would be to switch epel10 to the epel/epel-next model that epel8 and epel9 are using today. That switch would involve the following:
lots and lots of announcements and communications about changing course
change the epel10 koji build target to build against RHEL 10 instead of CentOS 10 (assuming RHEL 10 is already released at that point)
create epel10-next koji build targets that build against CentOS 10
remove epel10.x koji build targets
retire all epel10.x branches
check existing epel10 packages to see if any need to be rebuilt against RHEL 10 and have a separate epel10-next build already
make necessary changes in MirrorManager
create an epel-release that obsoletes epel-release-rhel
create an epel-next-release that obsoletes epel-release-centos and requires epel-release
probably a dozen other small things
Basically rolling this back after deployed would be a good bit of work, but would be completely doable. I don’t think there would be any fundamental blockers to switching to the epel/epel-next model if we later decide that works better for us.
We discussed this at today’s EPEL Steering Committee meeting. We decided to limit the vote to the overall direction, summarized as “a repo per minor version of RHEL 10”, without committing to further implementation details. This passed with a vote of +5, 0, -0.
Unless dnf implements $release_major and $release_minor, I believe we’ll be blocked on symlink support in MirrorManager as described by @adrian earlier in the thread. That could be worked on now independent of other work. Actually building out the real thing in koji will have to wait until we have CentOS 10 content to build against. That can likely start with the early development composes (which I expect to be available late this year or early 2024), without having to wait until it’s declared as “launched”. I’ve also thought about trying to build out a demo of this in staging Koji but for EPEL 9, but I need to talk to Fedora RelEng about it more to see if that would be ok.
Thanks everyone for your feedback and support so far. I’m excited about where this is heading. Keep the questions and comments coming.
I thought about that, but as you said it would be fragile. I also think if anyone is creating those files, it should be the base distro. Managing packages outside of the base distro to own those files with the appropriate values sounds like a nightmare. This approach would also break the ability to temporarily switch to a different minor version repo with --releasever. Maybe this is where we eventually end up, but there are enough drawbacks that I think we should focus on the symlink approach first.
If this is done in redhat-release/centos-release, then it should be fine. That said, it’s probably worth poking the DNF team to get this feature on the DNF 5 roadmap.
I’d like to discuss the dist tag aspect further, but this thread is getting a bit long. I looked at splitting off relevant comments to a new thread, but several of those comments cover other things besides the dist tag, so I thought it best to just start a new thread with a summary of the discussion so far and link it here.
I’ve finally had time to catch up on all this. FWIW I’m +1 to the proposal.
There’s clearly implementation details that need sorting but in terms of structure, lifecycle, and tagging strategy; it seems like a great idea. The implementation details will get sorted, there’s plenty of super smart people involved and motivated so that’s not a concern of mine.
I am curious if this is going to inflate the size of the EPEL mirrors by 6x over time once we get to the epel10.10 point in the lifecycle since each previous minor version repo directory is preserved (or did I misunderstand that?)?
Thank you for all the hard work on this, I agree this will benefit both EPEL contributors and users. +1
There will be more storage space needed, but I don’t think it will be a 6x increase. The intent is that each minor version repo is moved to the archive once it is no longer the current version of RHEL 10. That means most of the time we’ll only have two EPEL 10 repos at a time. For a brief period right before each RHEL 10 minor version release, we’ll have a third repo. This may sound like a 2x or 3x increase, but in practice many packages will be duplicated between the repos, so I think mirrors using deduplication techniques shouldn’t see much increase at all.
s here and no real objections other than concerns about details (one could say bikeshedding and nitpicks). What are the next steps to make this happen?