EPEL 10.0 mass branching

Project Discussion
1

We are closing in on the point of doing our first EPEL 10 mass branching event. We have been modeling this after the Fedora mass branching. Similar to how Fedora just created f42 branches for every package with a rawhide branch, we are going to create epel10.0 branches for every package with an epel10 branch. However, while discussing this an interesting problem was noticed.

During the mass branching process, we’ll be creating the epel10.1 tag in koji by cloning the epel10.0 tag. As part of the tag cloning process, the latest build of each package tagged with epel10.0 will get the new epel10.1 tag added. This is similar to what happens in Fedora branching. However, in Fedora this process captures nearly all packages. For EPEL we’ll need to decide how to handle packages that are in pending or testing status in bodhi, which won’t yet have the epel10.0 tag.

One possible route would be to add the epel10.1 tag to all builds with the epel10.0-testing-pending, epel10.0-testing, or epel10.0-pending tags. This would have the effect of cutting short the testing period for these updates for CentOS users as the epel/10 repo is retargeted from 10.0 to 10.1. Even worse, we may end up tagging in builds that have negative karma in bodhi and do not work. This may still be the easiest option, and we can always go back and manually untag builds that are later discovered to not work.

Another possibility would be to force pending/testing updates to process through to stable if they have no negative karma, or unpush them if they do have negative karma. This would still have the timing short cut problem, but at least pending/testing updates with negative karma wouldn’t be included in the stable 10.1 composes. I also kinda like the idea of a periodic six-month cleanup of negative karma updates rather than letting them sit in testing forever.

We could also just do nothing, and leave it up to packagers to ensure a proper upgrade path. Affected packagers would just need to create new builds from the epel10 branch after the mass branching, which will use the EPEL 10.1 release instead of EPEL 10.0 release in bodhi. This would pretty straightforward for simple packages, but could be messy for multi-build updates from sidetags. We could address these on a case by case basis by manually tagging these builds for epel10.1 upon request.

What would EPEL packagers prefer? One of these options, or something else entirely I haven’t thought of? No matter how we handle this, we will likely need to have some form of a freeze period for new builds while we process the existing builds and create branches. The notice period for this will unfortunately be somewhat short, as we need to execute the mass branching relatively soon to ensure no EPEL 10.0 packages unintentionally build against RHEL 10.1 libraries that are landing in CentOS 10.

2

I think that we should force the pending/testing updates to process through to stable. This seems like the most logical way forward. That being said, I’m not sure what the cleanup process would look like. Is is something that is scriptable via api calls or something else?

3

I haven’t looked into it in depth, but my assumption is that we can query bodhi’s API to get all the pending/testing updates, filter out the ones with negative karma, then submit an API call to push the rest to stable. However, this may also require temporarily changing the “days to stable” setting.

What I meant by a cleanup process would be unpushing the pending/testing that have negative karma at the time of the mass branching, which should also be possible via the bodhi API.

4

I would expect any updates created before the fork that go stable after the fork to land in both 10.0 and 10.1. Can we do that?

  1. gather the list of pending/testing updates for 10.0
  2. fork
  3. watch the updates from (1), tag them to 10.1 when they go stable
1 Like
5

I like that idea… it would of course be more work for epel releng folks tho.

Also, there could be corner cases… like some update sits for a while, then someone pushes a new update for 10.1, it goes stable, then the old 10.0 one goes stable.
But this could be checked by a ‘is there a newer build already in 10.1’ check

6

Interesting idea…for reference, currently there are 99 pending and testing updates in bodhi for 10.0. That seems quite manageable, and I also expect that will be a much lower number with future minor versions. We could start with it being manual process, and work on automating it later.

7

On my side this proposal also seems to be the most logical.

Not sure if it actually works like this, but basically that would be something like this:

  • For all latest packages tagged epel10.0 → add tag epel10.1
  • For all package testing/pending, copy any tag epel10.0-whatever into epel10.1-whatever, so that during the promotion process it will be pushed to both 10.0 and 10.1.

This would be a one-off action, without any “watching” mecanism afterwards to promote an update to 10.1 when the update goes to stable, if ever. I am not sure a watching mecanism would always be reliable. However this means that never promoted updates will continue to sit in different testing/pending tags for ever, but apparently this is how fedora works, so not sure this is a problem. I don’t know what will be the support policies for point release on RHEL side, but you might decide to automatically unpush all pending updates targetting a given point release after it goes out of support on RHEL side, so eventually these will go away.

8

I think the problem with this approach would be that there is nothing to move those builds from -whatever into whereever they need to go. This would usually be handled by bodhi, but tagging the builds manually means bodhi is blissfully unaware that they should be moved out of those tags too.

In general, I think depending on the number of pending updates, it might just be feasible to notify affected packagers that they need to either 1) solicit more karma for those updates so they would get pushed before the mass branching happens, or 2) resubmit those builds from the epel10 branch after the mass branching is done, so they end up in both EPEL 10.0 and EPEL 10.1.

9

For the 10.0 branching, let’s go with the approach Miro suggested. We will be proceeding with the mass branching today, and before we start we’ll gather a list of the pending and testing updates for 10.0.

https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject.org/thread/7HPSCQDLFOZWNF4WACDDMULJCXZDVJRP/

10

We have completed mass branching for epel10.0.

https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject.org/message/IBGDGNEVXKWTLHBKHGICF4CKQWTV6477/

We’ve got a running list of the updates that were submitted to EPEL 10.0 before the branching, and will be reviewing these every couple of days to manually tag them for epel10.1 once they reach stable status and get the epel10.0 tag. If you have an update that is missing from EPEL 10.1 and you want it tagged sooner, please file a releng issue.

Also please note that a bug in fedpkg was discovered today that causes branch requests for epel10.0 to result in invalid requests for a corresponding repo
and branch in the modules namespace. These can be closed as invalid,
while the epel10.0 branch request in the rpms namespace should process
correctly. We hope to get this fixed in fedpkg soon.

3 Likes