Konflux: What is the right time?

Project Discussion
, , ,
1

A few weeks back I started a discussion about accidental secrets with Konflux as an example. Now that FOSDEM has come and gone, I’d like to take the topic further. If you’re not familiar with it, in its own words, Konflux-ci “is an open source, cloud-native software factory focused on software supply chain security”, but for the sake of this discussion, it’s probably better to think of it as “the aspirational replacement for the myriad build and CI systems Red Hat uses in all its products”. Aspirational is key- we aren’t there yet… and it’s going to take a while to get there.

Despite many months of development, rpm support is just now being plumbed into the system (containers happened first). In fact, at this year’s CentOS Connect had Mike McLean, lead developer of koji, presented “Building RPMs with Konflux”. In his talk he related some of the details about the interim rpm approach, which injects builds into the koji after a build is complete. This seems weird, but it makes sense: When your goal is to eventually replace the full pipeline, but it’s going to take a long time, you have to write some throw-away code that bridges new to old.

Talking about replacing koji can evoke understandable feelings and skepticism: Fedora has used it since at least version 7, when Core & Extras merged- it’s effectively always been there. Also, Red Hat has previously started infrastructure replacement projects, then changed course. Also, its development isn’t far enough along that it could replace koji any time soon. Also, some parts are only now going into an upstream git forge (the outstanding work is to replace some hard-coded Red Hat internal values with a configuration system).

With all this in mind, the big question is: When and how is it the right time to officially bring up Konflux in the Fedora community context? If it happens too early, it won’t look credible or be useful. If it happens too late, there won’t be an opportunity for interested community members to meaningfully shape its development. So far, Red Hat’s development team has erred on the side of too-early, with presentations in 2024 at Flock and Devconf. Community feedback is valuable and showing up too late to accept it would be a loss.

Beyond presentations at conferences, the development team has created a Konflux + Fedora SIG, its own community mailing list, and even a matrix chat channel. The astute observer may note that some of the above URLs contain a combination of github.com and fedoraproject.org addresses. Similar to gcc using gnu.org and gnome using gnome.org, Konflux is meant to grow into a proper open source upstream, that many downstreams use, so public presence is not in Fedora alone.

As it matures, I expect Konflux to be part of the way we improve Fedora CI, to be part of what powers a more intuitive git-native workflow, and an easier onramp for people who don’t currently participate in Fedora to join in with less friction. These dreams may be a little way out, but they are worth pursuing as we bring Forgejo online and realize its potential. So, what are the next steps right now? Let’s talk about it.

(Note: This is cross-posted with the devel list)

2

I don’t have an answer here :slight_smile: But I agree that it still seems to be “too early”. When I learned at CentOS Connect that support for building RPMs was just only getting bolted on as an early experiment, that definitely felt like it’s “too early”, and what I heard made me a bit sceptical about the whole thing.

This was kind of confusing to me too. It’s a Konflux SIG for talking about Fedora, right? Not a Fedora SIG for talking about Konflux?

Side note: I would send plain-text to the devel list when you cross-post there. Getting HTML email from the list can be a bit jarring and is, to my knowledge, generally considered bad etiquette :smiley:

3

Yes, this is an understandable reaction. I explain it myself as follows: Which tooling is in greater need of improvement: koji or container builds? :grinning: - The nice thing, from my perspective, is that we can do incremental adoption on tooling. A 100% replacement for everything is far off, but it’s also possible that broad incremental utility for many is quite close. Konflux was used to create RHEL 10 beta’s containers, for instance. And while the RPM support is indeed experimental, it’s an experiment that select RHEL teams are incrementally joining into right now. I’m sure there will be some discoveries that need to be fixed as we go… but it’s where we’re going.

Yes, it’s odd from a Fedora-centric view and normal from a Konflux-centric view. I also suspect it’s a barrier to some for participating.

:man_facepalming:

4

I think you are not asking the right question here. This feels like a presupposition that Konflux is the right replacement for our existing infrastructure. But in the time I’ve heard about Konflux and seen stuff about it, it has bothered me that I haven’t heard anyone ask this question: “what do you as Fedora contributors need from our buildsystem that Koji isn’t doing now?”

Konflux being a skin on Tekton and leveraging OpenShift isn’t actually a valuable thing for us as Fedora contributors. The “git-native” workflow is not a real thing. Git is un-opinionated beyond “don’t store binaries in it”.

So I think the question first needs to be asked with the mind that Fedora contributors have experience with our existing build system and want to improve their workflows.

If I’m answering that question for myself, I would say that the biggest feature gap that we have from a modern build system like the Open Build Service is that we as packagers have to do dependency resolution for building groups of packages. Nobody likes manually sequencing packages for Koji chainbuilds. Nobody likes having to work through reverse dependencies and manually building them when a library has been upgraded. These are serious grunt work things that a computer should do for us. Before I started seriously contributing to openSUSE 10 years ago, I had never conceived it was possible, and now I want that for Fedora contributors.

It’s a big deal that we don’t have it: it means big stacks are hard and time-consuming to upgrade. SIGs like Python, Rust, and Go have to expend a lot of effort to sequence targeted mass builds. Desktops like KDE and GNOME have jalopies of scripts to try to automate build sequencing that are hard to generalize. But in openSUSE, it’s just “submit the packages and wait”. You don’t have to care about the order, it figures it out and bumps the release automatically to make the upgrade path work.

This is the kind of thing that would allow Fedora packagers to spend more time on higher value packaging work and contributing to upstream projects instead.

So if we’re presupposing a solution, why wouldn’t we consider the Open Build Service for Fedora? It has these features and we would massively benefit from them, in addition to OBS doing fully hermetic builds with actual ephemeral VMs rather than chroots or containers (which are not good enough for this task). It also has entry points for integrating whatever workflow you like. I ran a private OBS instance for 8 years and was able to integrate workflows for supporting working from Git, leveraging OBS features to have a persistent source cache and adding archive hooks to maintain historical binary builds. The build history database gives you information like “reasons” for why a build was made (dependency change? source change? project owner triggered it? etc.).

It offers a very rich experience compared to Konflux and Koji, because it’s actually optimized for being a build system, rather than being bolted onto a CI system (Konflux) or being bolted onto an artifact management system (Koji).

That’s not to say we couldn’t have these capabilities with either of those (though I personally think it would be easier to add to Koji than Konflux), but OBS is here and a proper open source project that is set up in a way that anyone can easily deploy.

5 Likes
5

Why would Fedora want to use Konflux instead of Koji? Why does Red Hat?

I’ve seen some talks about Konflux (from Flock I believe), but it reminds me of modularity in many ways. It’s presented as “the new cool thing,” yet I struggle to grasp the basic motivation. Call me old fashioned if you must – I’d appreciate an elevator pitch for “why should I want this (as a Fedora, EPEL and RHEL packager, as a Python maintainer, etc.)”.

4 Likes
6

Hi Miro,

A little more detail is included in the linked post (accidental secrets), but in brief: Red Hat has a lot of different products, and they’re all using different build systems. The effort to maintain all of them, enhance them, and have them work together to produce a coherent output is quite expensive. A secure software development pipeline is a problem that many large organizations have and Red Hat has chosen to invest in a solution that many organizations will find useful. All Red Hat products will eventually use Konflux, including RHEL. This is Red Hat’s interest, but it might not be Fedora’s interest.

For people who are looking for more information about motivation, you can learn more about it by, for example, listening to Ralph Bean’s talk at Devconf.cz 2024 (https://www.youtube.com/watch?v=2reMVZSDKC0) and Flock Rochester 2024 (https://www.youtube.com/watch?v=PnvN199qsS8).

Going back to my original question, should I infer that for you The Right Thing for Right Now is to share more about the features and motivation? I’m afraid the software stack is developing at too slow a pace to sustain people’s interest if we do a lot of that. Like NASA’s Crawler-transporter (Crawler-transporter - Wikipedia), this thing is moving a weighty collection of technologies, but it’s going to take a while to get to the launch pad.

1 Like
7

Right, so, where would you like to see that happen, how often, and what would a good feedback loop look like? Does it require a ticket queue? Does the ticket queue have to live inside fedoraproject.org?

Calling it a skin is like calling TCP a skin on top of IP, ne?

I do not understand why you are picking nits about “git-native workflow”.

That feels like one of several good next steps, sure.

I remember you bringing this up with Ralph at devconf.cz. Did you ever put the request into some written form? If not, perhaps one good next step would be to establish where to do that and doing it.

For my own part, I’m curious how common the problem is that you’re describing. As you are an voluminous contributor in the open source world, I know a lot of people don’t have to speak up because you’ve done it for them. This can have the effect of making it hard to tell how many people will benefit from a change. If a ticket were filed for this that had some sort of voting system then others could signal that it is something that matters to them, too.

It would be most productive to get the wanted capabilities into konflux.

8

Extremely common. Almost everyone that maintains more than a handful of packages ends up having to deal with chain builds. Everyone that maintains a library will have a one point to coordinate a mini mass rebuild of all the revdeps because of a soname bump (or equivalent incompatible change). This issue is massively multiplied in language ecosystems that encourage small components (e.g. in rust it’s not uncommon to have dependency trees of hundreds of crates).

7 Likes
9

Yes, but targeted at Fedora personas.

For example: Are Fedora packagers woried the building packages from distgit via Koji is insecure? I don’t think so. How does a secure software development pipeline map to what we do? Why do we need it?

I see what you mean. You don’t need to do a lot of that. Just a bit, to hook interested people up. You seem to genuinely want the Fedora community to get on board – but why should we (other than the fact that it would keep Fedora and Red Hat in sync – but that’s not always the case anyway, see e.g. gitlab vs forgejo, Jira etc.)

Thanks.

2 Likes
10

I’m picking at this because I wanted to head off any potential use of this phrasing as a benefit, because we already use a Git-based workflow for packaging. One that is designed around a loose coupling to our infrastructure, making it portable for multiple stakeholders (including RHEL and various Fedora Remixes).

I have requested this many times over several years to multiple people. The first time I did was back with modularity. I have attempted to move us in that direction many times and have been stymied by other people thinking it’s either impossible or too difficult to do (nevermind that SUSE has been doing it since the 90s with Autobuild, now OBS).

Why is that the case? What benefit do we get for taking on the load of the Konflux architecture? And keep in mind that Konflux also adds additional burdens for our packagers, systems administrators, downstreams, and so on because it adds OpenShift to the balance and introduces a lot more difficulty in understanding and replicating our processes. That’s a lot to ask for volunteers and side-along communities.

I get that you want to go from a service-oriented architecture that we have now to a more monolithic one with replacing Koji and Bodhi with Konflux, but I think it’s important to demonstrate the fitness of the architecture for the relevant stakeholders.

5 Likes
11

Yup. It can be hard to get RHEL-focused folks to understand this, because the solution to this problem for RHEL is usually “Fedora did it for us six months ago, so we don’t have to worry about it”.

4 Likes
12

Note that we’re making this a discussion topic for tomrrow’s @council meeting:

1 Like
13

git-native is not the right term I believe. Git Ops though is a real thing.

In short it means that any action in the infrastructure or a project is driven by a pull request to some Git repository.

There are number of processes in the Fedora project which are traditionally implemented via different custom tools, for example:

  1. Adding a new package to the distribution → Bugzilla ticket for a Review Request
  2. Creating a new build → koji build
  3. Adding a new build to a buildroot → Bodhi update
  4. Adding a new build to a compose → Jenkins?

If we implement each of those actions as pull requests in a Git Forge to a certain Git repository, we reduce the number of custom tooling we maintain in the project and amount of onboarding a newcomer needs so that they can start contributing to it.

I do believe that GitOps workflow is a good goal for the Fedora Project which we need to explore.

As I understand it, Konflux project does not on its own defines those GitOps processes. It tries to offer the tooling and infrastructure to support them, though.

So the one way how this collaboration between Fedora and Konflux may look like is that we (as Fedora) define the GitOps process, and then we ask Konflux folks whether they want or can support it. If yes - we get a benefit of something being done for us, if not, we figure out an alternative implementation ourselves.

6 Likes
14

I probably said it at some point.

15

I don’t think there is Fedora vs RHEL problem. RHEL on Konflux conversation has a lot of similarities with this one.

I think there is some sort of fundamental Distribution vs Application development problem. The difference between developing individual component, application, service… and developing an integrated platform of thousands of components which have circular dependency on each other.

But

  1. Koji is not free from it. Sidetags, chain builds and draft builds were not there in the original design of the build system. Dynamic sidetags feature in Koji for example has been driven partially by the RHEL CI initiative, and draft builds are a recent edition proposed by the “RHEL on GitLab” project.

  2. None of the CI systems (except Zuul CI :slight_smile: ) is free from it. And there is no one but us to describe the requirements for the Distribution Development. DevOps folks trained on the containerized pipelines will not figure it on their own.

So while Konflux folks currently work on the implementation of the secure reproducible and scalable RPM building backend we can create a certain vision of the architecture where we can go next. Especially for the build groups, chained builds and buildroot management.

1 Like
17

As a long-time old fogey ops person, I’m a firm believer that everything should move to ClickOps. The way it’s stored on the backend should never need to be exposed to people, and should merely exist as a way of maintaining records.

I don’t want a world where we have to send changes via Git for updates. It’s hugely regressive and not conducive for making things easier for people to work.

Note, that doesn’t mean that we couldn’t store records in Git, or a database, or whatever. But we should not increase the mental load for people to understand how to change things.

1 Like
18

I think too that moving something like Bodhi to a git based operated workflow is a bad idea.

Yes, a system like Bodhi is a pain to maintain and increases burden on an already thin layer of infrastructure maintainers. But having a purposed webUI, or CLI, for users / packagers to handle their update is much more easy than having to deal with a git based operated system.

  • Submitting an update now involves a user friendly webUI, or some commands from CLI. With a git based operation, that would mean to fork the main repo, edit some files, push a commit to the fork, create a pull request…
  • Giving feedback to an update (what is now called karma) would mean, I presume, to remember some obscure keyword to be posted as comment to the PR, instead of a simple click in the webUI or a command from CLI
  • Update info is nicely showed in Bodhi webUI, I don’t think the same readability would be obtainable in a git PR/commit

Other than that, Bodhi does a lot more than being an interface to post an update. Think at bodhi composer, getting rid of bodhi would mean develop yet another tool to manage composes. It also bring useful and easy to read searches and reports of updates by release.

IMO, switching all this to a git based solution, provided that would be possible, would bring significant hurdle to packagers and would move away testers from submit feedback.

As for Konflux: I understand this is a RH objective, but if we (Fedora) ever need to move away from Koji, why not join openSUSE in using OBS, so we don’t have to maintain our specific build tool on our own?

19

Let’s continue this subthread in [RFC] New Community Initiative - GitOps for Fedora Packaging

1 Like
20

OK, replied there about Bodhi.
But I would also have a reply here about OBS, if Fedora need to move away from Koji. Is there another reason other than “konflux is what RH wants, OBS is from competitors”? (if the word “competitors” have sense in Open Source)

1 Like
21

I don’t speak for Konflux project, but I don’t think that the goal has ever been - replace Koji with another implementation of Koji.

OBS has nice features, but it is essentially the same kind of tool: all-in-one system for building and storing built artifacts. With similar issues of large and relatively old codebase with a lot of legacy and opinionated functionality included, and not a lot of development expertise available.

It is of course hard to compare it with Konflux-which-doesn-t-really-exists-yet. But there should be something more we get out of it, if we ever adopt it.