[RFC] New Community Initiative - GitOps for Fedora Packaging

Project Discussion
,
1

Disclaimer

  • This is a PROPOSAL, not an announcement. This topic is a work in progress. We encourage you to do join this work in a constructive way.
  • The top post may be updated. If you comment on something specific - quote the text you are commenting.

This is a proposal for a new Community Initiative.

GitOps for Fedora Packagers

Overview

The goal of the initiative is to explore the possibility to use a Git Forge interface as a main driver for packaging tasks in Fedora. We will demonstrate the potential of this approach and gather packager feedback.

There are number of processes in the Fedora project which are traditionally implemented via different custom tools, for example:

  • Add a new package to the distribution → Bugzilla ticket for a Review Request
  • Notification of new upstream version → Bugzilla ticket
  • Create a new build → koji build
  • Add a new build to a buildroot → Bodhi update
  • Add a new build to a compose → Jenkins pipeline

If we implement each of those actions as pull requests in a Git Forge to a certain Git repository, we reduce the number of custom tools we maintain in the project and amount of onboarding a newcomer needs to start contributing.

Phase 1: Represent a Bodhi update as a Pull request to a Git repository

Idea

As a first exploratory step we propose to focus on a specific action in the Fedora Packager workflow: a bodhi update.

In this phase there will be no changes to the current workflow. Rather the goal is to create a mirror of the existing process.

  • We create a Git repository which represents a distribution. A branch in the repository is roughly equivalent to a target Koji tag of a Bodhi update.
  • For each new Bodhi update we create a Pull Request to the repository.
  • For each gating test, karma vote and comment we add CI vote and comment on that Pull Request
  • As soon as Bodhi update passes the gate the Pull Request will be merged.
  • [bonus] Each compose can be represented as a git tag on the commit, which represents the state of the koji tag which was used to build that compose.

Why

This first phase will allow us to iterate on the format of the pull request and the repository content using the real-world examples from the active Fedora packaging process. If, after a period of testing, we are satisfied with the implementation, we will be able to go to a phase two to completely replace Bodhi with the Git Forge-driven workflow.

While the first phase will not change anything in the existing workflow, it will already add visibility to the current state of the distribution:

  • a Git Forge user can watch the state of a pull request and get notifications about it
  • a Git user can compare the states of the distribution and easily get a diff between the branches or list of changes landing into a distribution in a last week.
  • downstream consumers of Fedora packages will be able to setup their own actions on the changes in the state of the repository.

This is technically doable with the existing infrastructure tooling, but it will become possible to do this using a standard Git Forge interface alone, without any knowledge of custom Fedora services, tool and concepts like Koji Event ID.

How

We need to run a service able to react to Fedora Messaging events and access the Git Forge APIs.

For that we need:

Design:

  • We need help from people with bodhi expertise to review the design and to support the work as SMEs.
  • We would benefit from the expertise of Packit team on dealing with Git Forge APIs.

Development:

  • We need to develop the core logic of the service: transform Fedora Messaging event into an action on GitForge.
  • We need to create deployment scripts for the service (Ansible playbooks, Helm charts,.., depending on the existing expertise and interest)

Ops:

  • Someone from CLE to help with deploying and running the service for at least a year.

Hardware and other resources:

  • We need a workspace at one of the Git Forges for the main repo. Most likely Codeberg (until we have our own Forgejo deployment)
  • We need a place to run a service - most likely a tenant at the one of the Openshift clusters
  • Any standard infrastructure to host the service and run deployment scripts (some GitForge + CI)
  • As the service will be a mostly stateless microservice translating one json into another, there is no need to discuss hardware requirements.

Communication and feedback loop:

  • Feedback will be collected through an initiative-specific tag on Discourse.
  • Significant findings from investigation and feedback will be tracked in a Forgejo project
  • We also need support for presenting the work at Flock and other relevant places

Timeline

The rough possible timeline for the phase 1 is:

  • Apr 1, 2025 - Get the approval for the initiative
  • Apr 10, 2025 - Get the people and access to the infrastructure resources for them
  • May 1, 2025 - Run the initial implementation of the service reading Bodhi messages
  • June 1, 2025 - Implement Open PR/Merge PR functionality by Flock
  • July 1, 2025 - Add the Update the PR with karma votes and comments and CI results
  • August 1, 2025 - Add tags for some of the composes.
  • August 15, 2025 - Announce the work on Fedora Magazine and start gathering feedback from Fedora packagers and from Infrastructure, Release Engineering, and Quality Teams
  • February, FOSDEM 2026 - Collect feedback, present the result, and propose next steps at CentOS Connect or Distribution Devroom.

Please share you feedback in the comments below.

If you want to work on this project - reach out to @bookwar:fedora.im at the Fedora Council channel on Matrix.

4 Likes
5

  1. This is a good feedback, thank you for writing it.

  2. I think that we need CLI tools, no matter the way backend and interface are implemented. So the GitOps approach does not mean that we should get rid of fedpkg. The WebUI part may or may not be needed, but I would prefer it to be stateless and optional.

  3. I do not quite understand your point about karma. How is karma in Bodhi different from comments on Merge Requests in Git Forge? Why do you think karma comments need special treatment?

  4. When you say “updates are listed nicely” is there something specific in Bodhi listings which you like?

I am looking at Fedora Updates System and to me it does not look very different from a generic list of MRs like Merge requests · CentOS / Integration SIG / compose-tests · GitLab

And the update page FEDORA-2025-1cd2d082ff — unspecified update for ostree — Fedora Updates System looks relatively similar to Rebase to 3.11.9 (!22) · Merge requests · Red Hat / centos-stream / rpms / python3.11 · GitLab

There are comments, labels, test results, changelog, reviewers..

Except that on the Bodhi page I don’t immediately see whether update has been merged or not. Afaik it is controlled by the “testing” label, but it is not obvious.

Are there specific features which make it nice?

6

So, that would mean to rewrite all CLI tools (fedpkg is just one) for the switch from bodhi to gitops… seems a lot of work. Also, the WebUI could be optional for a tool targeted at RH developers, but to engage casual users in posting feedback this is highly recommended, in my POV.

Comments are one thing, karma is another. A negative karma in Bodhi usually means that automation to push the update to stable is deactivated. On the other side, positive karma posted by users is the key to make the update pushed to stable.
In Bodhi we also have time constraints for which an update needs to wait before being pushed to stable repositories.
Again, I see these are not priority for RH, since I suppose you just care about automated tests.

In the Bodhi list I can clearly see the build and the release the update is associated to and I can apply several filters (by release, by status - and several others that are not showed in the webUI selector, but are there).

In the Bodhi update page I can see at a first glance what release is associated, the list of builds, the configured thresholds, I have a warning when there is an ongoing freeze…
The git commit page just shows a scary (for a user which is not a developer) list of activities, mostly by bots which comment with test results, and I need to look at the changes details to show what it is about.
I really think a “normal” user would never understand such interface.

Finally, you didn’t reply about all other features Bodhi does: do we need another tool for running composes? What will take care of all information which Bodhi now takes care to serve after PDC decommissioning?

To summarize: gitops may be well fitted for a “closed” community of developers/workers, but IMO it does not fit at all when it comes to interact with end users.

7

Let me comment on one thing first - I am not acting on behalf of Red Hat here. Red Hat has its own approach to working with GitLab merge requests, and it is not clear if it make sense to try and copy that approach in Fedora.

It is fine to disagree with the proposal, it is not fine to dismiss my points because you (wrongly) assume that I am acting with RH priorities in mind.

Now back to the actual comments:

rewrite all CLI tools (fedpkg is just one) for the switch from bodhi to gitops… seems a lot of work

It is. Though we have to rewrite a lot of stuff for a new Git Forge too. Hopefully we can make it Git Forge agnostic as much as possible.

Also, the WebUI could be optional for a tool targeted at RH developers, but to engage casual users in posting feedback this is highly recommended
…
A negative karma in Bodhi usually means that automation to push the update to stable is deactivated.

Git Forge by design is a review system, thus it provides exactly the functionality for users to vote on a Merge Request. And votes can be made blocking or non blocking in the settings of the project. GitLab does it, Pagure does it. GitHub does it. Forgejo, Gerrit and Review Board do it as well.

The WebUI to approve a change is a page where a user can login with a FAS account and leave a comment and approve or dispprove the change. This is literally the same interface you expect for every Merge/Pull/Change Request page out there.

The only difference for a casual drive by contributor user is the name on the button and the location of the button on the page. And a drive by contributor is more likely to know how it looks on GitHub than in Bodhi UI.

In the Bodhi list I can clearly see the build and the release the update is associated to and I can apply several filters (by release, by status - and several others that are not showed in the webUI selector, but are there).

In GitOps approach release is the target branch of the merge request. The “Merged” status means the change is accepted. And the standard search capabilities are included in the interface of a Git Forge.

And your comment “several others that are not showed in the webUI selector” I think highlights the point that Bodhi interface is not very accessible for a new user.

Do we need another tool for running composes?

Same tool, another git repository maybe. And this is actually a key part: same tool for dist-git contributions, same tool for updates, same tool for composes, docs, design,..

gitops may be well fitted for a “closed” community of developers/workers

Funny, how we have actually completely opposite conclusions :slight_smile: I’d rather say that the custom tool is best suited for the audience which you can train to use that specific tool. For a wider community which comes to your project for the first time and then forgets about it next day, you can not expect that community to learn the custom tool. Thus you should figure out which widely adopted tools are already available.

But I think where we are getting with this is that we have rather different expectation on how an imaginary user will react on this or that UI.

I admit I do not have a good proof for the vision I have in mind. And this is why I want to do this exploratory project first. So that instead of comparing existing Bodhi page with a “theoretical implementation of the GitOps workflow” we rather can look at both interfaces side by side, and test them with real users.

8

I don’t think we should make GitOps the interface for most contributors. It’s a huge burden from a tooling and education point of view. A big part of why we have the “ClickOps” model is that it smooths out workflows of engagement and it makes it easy to onboard people.

We’d essentially alienate our testing community by doing this. As I said in another thread, if you want to have some kind of Git representation of things in the backend, that’s one thing. But Git as a frontend induces all kinds of pain that I don’t think we want.

One misunderstanding I think is playing out here is that there seems to be the idea that the testers and the packagers are the always the same people. That is the case a lot of the time, but not all the time. We have plenty of testers that are not engaged in the development process at all, and I don’t want to alienate them.

1 Like
9

This proposal can be summarized as “reimplement Bodhi interface on top of comments in a forge pull request and reimplement Bodhi business logic as some service somewhere in an Openshift cluster”.

Neal and Aleksandra disagree on whether the comment-in-a-pr interface is good for users. I think users would be able to adapt to it. We have prior art with release-blocker voting (example: Issue #1808: [anaconda] thinks Windows is installed, but it's macOS | rhbz#2354671 - blocker-review - Pagure.io). I think that interface is slightly worse than the dedicated website like Bodhi, but not catastrophic. For example, the status update is asynchronous and if you get the comment format wrong, there is really no indication. There is a bit of a learning curve and clunkiness, but most contributors are clearly able to use it.

OTOH, I don’t buy the argument that comments-in-a-pr is somehow more approachable. The user would need to go in some very specific place and perform some very specific action to provide karma. Opening a Bodhi page also requires knowledge about a specific place to go to, but otherwise is easier, just two clicks that you cannot get wrong.

If we didn’t have Bodhi and we needed to implement something from scratch, then the proposed approach with re/ab-using a git forge would be a reasonable proposal. But since Bodhi is there and our users are familiar with it and we have bookmarks and links to it, if we were to switch to something else, it’d need to be significantly better. But comments-in-a-pr interface is not significantly better. It’s at best of similar complexity for the users, but would require a lot of adaptation. We’d need to reimplement the Bodhi update logic in new code, running in a new environment. Overall, this seems a lot of work and risk to replace something that is not giving us problems with something unknown.

–

The Overview says “use a Git Forge interface as a main driver for packaging tasks in Fedora”. I think this is a very interesting direction, but the proposed “Phase 1” seems like the wrong place to start. It can at best deliver little benefit to users or packagers.

OTOH, packagers are already required to use git and do pushes to update dist-git. We could empower packagers by introducing workflows where the pushed commit indicates subsequent actions to be taken, for example creation of a build, and possibly even creation of a bodhi update if the build succeeds. This would save a lot of packager work, also by reducing the need to type duplicate information into the the git commit message and the bodhi update. We already have this kind of workflow for rawhide updates, just not configurable, so we know it can be done. This seems a much more promising area to start with a git ops workflow.

10

Thanks for your comment, it made me realize that the way I am explaining the effort is not really covering all the aspects I have in mind for it. But it is a hard thing to untangle.

So let me try to elaborate.

In previous comments we focused too much on replacing Bodhi. But it is not really the main driving force behind the initiative. It would be a nice outcome, in the end, if Git interface would be enough to cover the Bodhi functionality but it doesn’t necessarily have to.

Similarly, using Konflux is not the main goal either. From my perspective it is a (possible) implementation detail.

One of the core things which I want to try with this initiative is represent a change to a distribution as a merge request to a Git repo which describes the state of the distribution.

–

There are reasons why Fedora or RHEL CI efforts lead to very complicated infrastructure. Some of them of course are lack of resources to address the annoying usability problems, but some come from the very fundamental thing: we try to implement the distribution-level work on top of package-level merge request. And package-level merge request simply doesn’t have enough information about the state of the distribution, and enough scope to do multi-package and non-package changes.

It works for a leaf package with a straightforward release cycle, but creates too many corner cases which we then have to address with a fragile tooling.

We introduced Bodhi and Gating process to address this distribution integration, but for many packagers the integration done via Bodhi is still something “only Fedora QA knows”. And still mostly represents the queue of package updates, not the state of the distribution.

So the big idea I have is that by setting the foundation straight, acknowledging that distribution is not a set of dist-git repos but an integrated pool of components, and it has state and a version control, and approvals process, we make the whole process to be better understood for all, users, packagers, contributors and upstream devs.

–

Of course, switch from a dist-git to a compose-git is a big leap. And I am not myself completely confident it will work within Fedora and RHEL constraints and requirements. I have some vision, but it lacks details. That’s why I am trying to use the flow of Bodhi updates as way to test it, on the side, without interfering with the current process, but nevertheless getting real distribution changes as input.

This way we won’t be arguing about some theoretical concepts as I described above. We will be dealing with the full set of cases Fedora is dealing with.

11

OK, this explanation helps.

If the idea is to set up this service on the side, where it passively follows the state of koji and bodhi, then this should be OK as an experiment. I’m sure it’d be a useful learning experience. Though, to really make a full experiment, this service would also need to be active, i.e. drive the merging of updates. Doing this without interrupting koji and bodhi as they are now seems much harder.

12

Though, to really make a full experiment, this service would also need to be active, i.e. drive the merging of updates. Doing this without interrupting koji and bodhi as they are now seems much harder.

I think it should be possible via some CI pipelines logic, somewhat similar to how Packit manages to generate dist-git MRs from upstream activity. Manual MR to compose-git, not realted to any existing bodhi update, would generate a sequence of events (dist-git updates, koji builds, bodhi invocation..) which will eventually lead to the same bodhi update submission.

But that’s something for the future. For now I think one-directional sync would be good enough to start.

1 Like
13

That interface is only somewhat tolerable. It’s difficult for going back and identifying things later, as there’s essentially no real useful metadata to query on. Our dedicated Bodhi system lets you find updates, find contents of updates, what type they were, what stage they’re at, and “quality” via karma.

This is incredibly useful for people not deeply engaged in Fedora’s processes to be able to quickly zero in on things.

1 Like
14

Hi all,

First, I want to sincerely apologize for not sharing my feedback directly in this thread sooner. I know I was asked during the Council meeting to provide my specific suggestions here. I appreciate your patience.

After further reflection and considering the discussion during the Council meeting, I am formally changing my vote on this Community Initiative proposal from -1 to +1. I am enthusiastic about the potential of exploring a GitOps approach for Fedora packaging and fully support this investigatory work moving forward. I believe it could bring significant benefits, and I want to see this Community Initiative succeed.

My previous concerns were about ensuring we have a clear plan for communication and community engagement, particularly for potentially large shifts in workflow. Based on past experiences, I feel proactive communication planning is crucial, even for exploratory phases.

To provide transparency on my previous sticking points and offer constructive suggestions for strengthening the proposal as this work progresses, here’s a summary of the communication aspects I believe are important:

  • Stakeholder Engagement: Clearly identifying which specific community groups (e.g., packagers, developers, specific SIGs, users) we plan to engage with for feedback at different stages of the investigation.
  • Communication Milestones: Integrating specific, timeline-based milestones for when key communication efforts and feedback collection periods will occur.
  • Feedback Transparency: Defining how feedback will be gathered and made visible to the community throughout the process.

I believe incorporating more detail on these aspects as the initiative evolves will help ensure the community feels informed and involved, preventing the kind of “whiplash” that can happen with major changes. I appreciate @mattdm already incorporating some initial refinements based on the Council discussion and taking accountability as the sponsor.

Again, I apologize for my delay in bringing this feedback here directly. I’m excited to see this Community Initiative approved and look forward to seeing the results of the exploration.

15

FWIW I would like to +1 this proposal as an investigation or exploratory first step into what this kind of workflow could be like for Fedora if we adopt it in the future. Thank you @bookwar and the other folks involved, I know its a hard thing to do to challenge the status quo, but sometimes a lot of good can come from taking stock of current processes and investigating if there are in fact other, better ways of doing things. I acknowledge that this might not in fact work well, but I think its still worth the support you are asking for if you are willing to examine this for the project. +1 from me to this proposal.

1 Like