Accidental Secrets

Last discussion I mentioned “new work” for CLE. Today I’m going to provide an example on the software side with a short story and a long secret.

Several years ago I was sitting in the audience at Red Hat Summit’s RHEL Roadmap session. I knew what was about to happen, and I was excited: It was 2019, we’d just launched RHEL 8, the culmination of 5 years of work that started with Fedora 28. Today the excitement was about something unusual: We were about to announce the move to a 3 year major release / 6 month minor release cadence.

This probably seems small since Fedora has been making 6 month releases forever. For RHEL it was a huge deal, because making the schedule work meant redesigning everything, from the approach to planning, the need for cross-functional teams, and even the flow of development and testing- everything had to be revisited. The past did not have to be the future. We couldn’t talk about it yet, not until it had been officially announced. The time was here! I live tweeted from the audience; followers retweeted- this cat was not going back in the bag.

The funny thing is that this was an unusual condition; Most of Red Hat’s engineering organization work isn’t secret. Usually we’re either working upstream on some enhancement or bug fix, tying together a solution. On occasion we get to create and share something new (e.g., podman). We have an open source development model, so “is this secret?” is rarely part of most people’s thought process. There’s this weird corollary though: “How do I make this open?” also isn’t something we spend a lot of time with, because most of what we do already is. This is where trouble starts.

If you’re an active community member, you probably have a decent understanding of the current events taking place in Fedora and the Open Source world. It takes a while to build this up, but once you’re in that flow, you want to stay there. Red Hat sometimes does this thing that breaks that flow: engineers create some new thing, and quite by accident, do not communicate The Big Why. Today I want to provide a little extra context to Konflux. The small parts have been talked about, but the greater context for what it means to Red Hat engineering hasn’t been explicitly described. And because of this, members of the Fedora community haven’t had a frame to decide what degree of engagement is necessary.

Red Hat has many products distributed in many ways: rpms, isos, containers, zip files (Java, sigh), and so forth. And we rely on many independent systems to support them. Evolved over decades, portions re-engineered, duct-taped, acquired, welded… Increasingly more time was being spent keeping the system running than providing real solutions to build, test, and distribution problems (e.g., Secure supply chain, onboarding new users, etc). Red Hat engineering decided it was time to make a system to replace it all so we could actually solve modern distribution engineering problems. This is Konflux.

Konflux is actually hundreds of different pieces, all engineered to work with each other, to replace all parts of Red Hat’s existing build, test, and distribution systems across all its products. The building blocks have been under development for years and it’s starting to become real. RHEL 10 Beta’s images were built with Konflux, and initial rpm support work is being developed now. If things go according to plan, over the next ~16 months most of the RHEL production pipeline will be running on Konflux. This is all RHEL. What about Fedora, what about community?

Konflux is not (intentionally) a secret, but it’s also not yet a major upstream project with a thriving community around it. In fact some parts are still being prepared for upstreaming, while other parts have yet to be written. At this point there have been Konflux talks at Flock and Devconf, the containers sig is talking about it on the regular and core developers are there looking for feedback.

How Konflux’s existence relates to Fedora is negotiable, but I hope we see extensive adoption as it solves real problems and has a realistic path to useful long term maintenance. As you probably know, there is already a test cluster available for Fedora users to play with, you just need the infra team to add you to the ACL list. Fedora Infrastructure is even now exploring its use to build images for the new Gitforge.

So, now what? So far the conversation has been open, but in niche places. I hope, with this broader context, we can have a more mainstream conversation about its use. While members of Release Engineering and Infrastructure are already involved in this topic, it has the potential to affect many other community members who rely on pieces of the pipeline that Konflux may eventually replace. Perhaps this thread is part of the conversation, but it probably merits discussion in or with FESCo sometime soon.

4 Likes

I like the practice of secrets spilled accidentally. I was reading some project issues related to bootc and discovered that Red Hat seems already commited to providing rpms for firefox and thunderbird in RHEL 10. Will Konflux be improving the software supply chain issues that big applications entail? I hope so. The potential is there.

Paging @ralph on the supply chain question.

Git commits and project issues are full of accidental spillage, though I would guess in most cases what’s actually been spilled isn’t actually meant to be secret in the first place. Often this leads to a Sherlock Holmes-style this-must-mean-that series of deductions, whose accuracy is… variable. It’s way better to be intentional, but that takes extra effort, and is often skipped.

Yeah, providing transparency on supply-chain security for applications either large or small is the main value proposition of konflux. The main feature to look into is our use of in-toto attestations (slsa post, konflux docs) and the way that we use those to gate artifacts with machine-readable policies (conforma, recently renamed). Those attestations and policy verification form the basis of the trust chain that we can then use to trust other aspects of the build process, like the sbom created as a byproduct of offline hermetic builds.

2 Likes

Great stuff. An outstanding value in using fedora/centos/redhat is the scrutiny made to the software supply chain along the way. Trying to add evaluations of each and every flatpak app and runtime and/or container to otherwise simple fedora/centos/redhat installs is onerous. Thanks for the links as I need to become more educated on what is actually inspected along the way. Projects that have amassed good levels of trust continue to need to guard against negative incidents and so what konflux can be leveraged to add will be welcomed.