[RFC] New Community Initiative - GitOps for Fedora Packaging

Project Discussion
,
1

Disclaimer

  • This is a PROPOSAL, not an announcement. This topic is a work in progress. We encourage you to do join this work in a constructive way.
  • The top post may be updated. If you comment on something specific - quote the text you are commenting.

This is a proposal for a new Community Initiative.

GitOps for Fedora Packagers

Overview

The goal of the initiative is to explore the possibility to use a Git Forge interface as a main driver for packaging tasks in Fedora. We will demonstrate the potential of this approach and gather packager feedback.

There are number of processes in the Fedora project which are traditionally implemented via different custom tools, for example:

  • Add a new package to the distribution → Bugzilla ticket for a Review Request
  • Notification of new upstream version → Bugzilla ticket
  • Create a new build → koji build
  • Add a new build to a buildroot → Bodhi update
  • Add a new build to a compose → Jenkins pipeline

If we implement each of those actions as pull requests in a Git Forge to a certain Git repository, we reduce the number of custom tools we maintain in the project and amount of onboarding a newcomer needs to start contributing.

Phase 1: Represent a Bodhi update as a Pull request to a Git repository

Idea

As a first exploratory step we propose to focus on a specific action in the Fedora Packager workflow: a bodhi update.

In this phase there will be no changes to the current workflow. Rather the goal is to create a mirror of the existing process.

  • We create a Git repository which represents a distribution. A branch in the repository is roughly equivalent to a target Koji tag of a Bodhi update.
  • For each new Bodhi update we create a Pull Request to the repository.
  • For each gating test, karma vote and comment we add CI vote and comment on that Pull Request
  • As soon as Bodhi update passes the gate the Pull Request will be merged.
  • [bonus] Each compose can be represented as a git tag on the commit, which represents the state of the koji tag which was used to build that compose.

Why

This first phase will allow us to iterate on the format of the pull request and the repository content using the real-world examples from the active Fedora packaging process. If, after a period of testing, we are satisfied with the implementation, we will be able to go to a phase two to completely replace Bodhi with the Git Forge-driven workflow.

While the first phase will not change anything in the existing workflow, it will already add visibility to the current state of the distribution:

  • a Git Forge user can watch the state of a pull request and get notifications about it
  • a Git user can compare the states of the distribution and easily get a diff between the branches or list of changes landing into a distribution in a last week.
  • downstream consumers of Fedora packages will be able to setup their own actions on the changes in the state of the repository.

This is technically doable with the existing infrastructure tooling, but it will become possible to do this using a standard Git Forge interface alone, without any knowledge of custom Fedora services, tool and concepts like Koji Event ID.

How

We need to run a service able to react to Fedora Messaging events and access the Git Forge APIs.

For that we need:

Design:

  • We need help from people with bodhi expertise to review the design and to support the work as SMEs.
  • We would benefit from the expertise of Packit team on dealing with Git Forge APIs.

Development:

  • We need to develop the core logic of the service: transform Fedora Messaging event into an action on GitForge.
  • We need to create deployment scripts for the service (Ansible playbooks, Helm charts,.., depending on the existing expertise and interest)

Ops:

  • Someone from CLE to help with deploying and running the service for at least a year.

Hardware and other resources:

  • We need a workspace at one of the Git Forges for the main repo. Most likely Codeberg (until we have our own Forgejo deployment)
  • We need a place to run a service - most likely a tenant at the one of the Openshift clusters
  • Any standard infrastructure to host the service and run deployment scripts (some GitForge + CI)
  • As the service will be a mostly stateless microservice translating one json into another, there is no need to discuss hardware requirements.

Communication and feedback loop:

  • Support for presenting the work at Flock and other relevant places and collecting feedback

Timeline

The rough possible timeline for the phase 1 is:

  • Apr 1, 2025 - Get the approval for the initiative
  • Apr 10, 2025 - Get the people and access to the infrastructure resources for them
  • May 1, 2025 - Run the initial implementation of the service reading Bodhi messages
  • June 1, 2025 - Implement Open PR/Merge PR functionality by Flock
  • July 1, 2025 - Add the Update the PR with karma votes and comments and CI results
  • August 1, 2025 - Add tags for some of the composes.
  • August 15, 2025 - Announce the work on Fedora Magazine and start the feedback gathering
  • February, FOSDEM 2026 - Collect feedback, present the result, and propose next steps at CentOS Connect or Distribution Devroom.

Please share you feedback in the comments below.

If you want to work on this project - reach out to @bookwar:fedora.im at the Fedora Council channel on Matrix.

4 Likes
5

  1. This is a good feedback, thank you for writing it.

  2. I think that we need CLI tools, no matter the way backend and interface are implemented. So the GitOps approach does not mean that we should get rid of fedpkg. The WebUI part may or may not be needed, but I would prefer it to be stateless and optional.

  3. I do not quite understand your point about karma. How is karma in Bodhi different from comments on Merge Requests in Git Forge? Why do you think karma comments need special treatment?

  4. When you say “updates are listed nicely” is there something specific in Bodhi listings which you like?

I am looking at Fedora Updates System and to me it does not look very different from a generic list of MRs like Merge requests · CentOS / Integration SIG / compose-tests · GitLab

And the update page FEDORA-2025-1cd2d082ff — unspecified update for ostree — Fedora Updates System looks relatively similar to Rebase to 3.11.9 (!22) · Merge requests · Red Hat / centos-stream / rpms / python3.11 · GitLab

There are comments, labels, test results, changelog, reviewers..

Except that on the Bodhi page I don’t immediately see whether update has been merged or not. Afaik it is controlled by the “testing” label, but it is not obvious.

Are there specific features which make it nice?

6

So, that would mean to rewrite all CLI tools (fedpkg is just one) for the switch from bodhi to gitops… seems a lot of work. Also, the WebUI could be optional for a tool targeted at RH developers, but to engage casual users in posting feedback this is highly recommended, in my POV.

Comments are one thing, karma is another. A negative karma in Bodhi usually means that automation to push the update to stable is deactivated. On the other side, positive karma posted by users is the key to make the update pushed to stable.
In Bodhi we also have time constraints for which an update needs to wait before being pushed to stable repositories.
Again, I see these are not priority for RH, since I suppose you just care about automated tests.

In the Bodhi list I can clearly see the build and the release the update is associated to and I can apply several filters (by release, by status - and several others that are not showed in the webUI selector, but are there).

In the Bodhi update page I can see at a first glance what release is associated, the list of builds, the configured thresholds, I have a warning when there is an ongoing freeze…
The git commit page just shows a scary (for a user which is not a developer) list of activities, mostly by bots which comment with test results, and I need to look at the changes details to show what it is about.
I really think a “normal” user would never understand such interface.

Finally, you didn’t reply about all other features Bodhi does: do we need another tool for running composes? What will take care of all information which Bodhi now takes care to serve after PDC decommissioning?

To summarize: gitops may be well fitted for a “closed” community of developers/workers, but IMO it does not fit at all when it comes to interact with end users.

7

Let me comment on one thing first - I am not acting on behalf of Red Hat here. Red Hat has its own approach to working with GitLab merge requests, and it is not clear if it make sense to try and copy that approach in Fedora.

It is fine to disagree with the proposal, it is not fine to dismiss my points because you (wrongly) assume that I am acting with RH priorities in mind.

Now back to the actual comments:

rewrite all CLI tools (fedpkg is just one) for the switch from bodhi to gitops… seems a lot of work

It is. Though we have to rewrite a lot of stuff for a new Git Forge too. Hopefully we can make it Git Forge agnostic as much as possible.

Also, the WebUI could be optional for a tool targeted at RH developers, but to engage casual users in posting feedback this is highly recommended
…
A negative karma in Bodhi usually means that automation to push the update to stable is deactivated.

Git Forge by design is a review system, thus it provides exactly the functionality for users to vote on a Merge Request. And votes can be made blocking or non blocking in the settings of the project. GitLab does it, Pagure does it. GitHub does it. Forgejo, Gerrit and Review Board do it as well.

The WebUI to approve a change is a page where a user can login with a FAS account and leave a comment and approve or dispprove the change. This is literally the same interface you expect for every Merge/Pull/Change Request page out there.

The only difference for a casual drive by contributor user is the name on the button and the location of the button on the page. And a drive by contributor is more likely to know how it looks on GitHub than in Bodhi UI.

In the Bodhi list I can clearly see the build and the release the update is associated to and I can apply several filters (by release, by status - and several others that are not showed in the webUI selector, but are there).

In GitOps approach release is the target branch of the merge request. The “Merged” status means the change is accepted. And the standard search capabilities are included in the interface of a Git Forge.

And your comment “several others that are not showed in the webUI selector” I think highlights the point that Bodhi interface is not very accessible for a new user.

Do we need another tool for running composes?

Same tool, another git repository maybe. And this is actually a key part: same tool for dist-git contributions, same tool for updates, same tool for composes, docs, design,..

gitops may be well fitted for a “closed” community of developers/workers

Funny, how we have actually completely opposite conclusions :slight_smile: I’d rather say that the custom tool is best suited for the audience which you can train to use that specific tool. For a wider community which comes to your project for the first time and then forgets about it next day, you can not expect that community to learn the custom tool. Thus you should figure out which widely adopted tools are already available.

But I think where we are getting with this is that we have rather different expectation on how an imaginary user will react on this or that UI.

I admit I do not have a good proof for the vision I have in mind. And this is why I want to do this exploratory project first. So that instead of comparing existing Bodhi page with a “theoretical implementation of the GitOps workflow” we rather can look at both interfaces side by side, and test them with real users.

8

I don’t think we should make GitOps the interface for most contributors. It’s a huge burden from a tooling and education point of view. A big part of why we have the “ClickOps” model is that it smooths out workflows of engagement and it makes it easy to onboard people.

We’d essentially alienate our testing community by doing this. As I said in another thread, if you want to have some kind of Git representation of things in the backend, that’s one thing. But Git as a frontend induces all kinds of pain that I don’t think we want.

One misunderstanding I think is playing out here is that there seems to be the idea that the testers and the packagers are the always the same people. That is the case a lot of the time, but not all the time. We have plenty of testers that are not engaged in the development process at all, and I don’t want to alienate them.

1 Like
9

This proposal can be summarized as “reimplement Bodhi interface on top of comments in a forge pull request and reimplement Bodhi business logic as some service somewhere in an Openshift cluster”.

Neal and Aleksandra disagree on whether the comment-in-a-pr interface is good for users. I think users would be able to adapt to it. We have prior art with release-blocker voting (example: Issue #1808: [anaconda] thinks Windows is installed, but it's macOS | rhbz#2354671 - blocker-review - Pagure.io). I think that interface is slightly worse than the dedicated website like Bodhi, but not catastrophic. For example, the status update is asynchronous and if you get the comment format wrong, there is really no indication. There is a bit of a learning curve and clunkiness, but most contributors are clearly able to use it.

OTOH, I don’t buy the argument that comments-in-a-pr is somehow more approachable. The user would need to go in some very specific place and perform some very specific action to provide karma. Opening a Bodhi page also requires knowledge about a specific place to go to, but otherwise is easier, just two clicks that you cannot get wrong.

If we didn’t have Bodhi and we needed to implement something from scratch, then the proposed approach with re/ab-using a git forge would be a reasonable proposal. But since Bodhi is there and our users are familiar with it and we have bookmarks and links to it, if we were to switch to something else, it’d need to be significantly better. But comments-in-a-pr interface is not significantly better. It’s at best of similar complexity for the users, but would require a lot of adaptation. We’d need to reimplement the Bodhi update logic in new code, running in a new environment. Overall, this seems a lot of work and risk to replace something that is not giving us problems with something unknown.

–

The Overview says “use a Git Forge interface as a main driver for packaging tasks in Fedora”. I think this is a very interesting direction, but the proposed “Phase 1” seems like the wrong place to start. It can at best deliver little benefit to users or packagers.

OTOH, packagers are already required to use git and do pushes to update dist-git. We could empower packagers by introducing workflows where the pushed commit indicates subsequent actions to be taken, for example creation of a build, and possibly even creation of a bodhi update if the build succeeds. This would save a lot of packager work, also by reducing the need to type duplicate information into the the git commit message and the bodhi update. We already have this kind of workflow for rawhide updates, just not configurable, so we know it can be done. This seems a much more promising area to start with a git ops workflow.