[CentOS Connect Follow-up] Proposal for the Git interface to the Compose Gate

Our Neighbors CentOS
1

Intro

One of the main topics discussed at the CentOS Connect in Brussels last week was the ability for third-parties (for example RDO or IdM teams) to contribute test results to the Compose Gate of the CentOS Stream so that they can prevent regressions on CentOS Stream packages before they hit the public mirrors.

Currently we run the so called t_functional test suite in the Compose gate, which can be extended by contributing your own test cases (the TMT version of the test suite is being developed in CentOS / Integration SIG / compose-tests · GitLab)

But for larger projects like RDO (Openstack in RPMs) or IdM (Identity Management) test cases might be too complex or too specialized to be run in a generic test environment.

Thus we’d like to provide the ability for third-parties to get the latest nightly compose, test it on their own environment and contribute test results back to the CentOS RelEng team.

Technically speaking main pieces for this workflow are already available:
nightly composes are published in the open at Index of /production/latest-CentOS-Stream/compose and the message is sent to Fedora Message Bus or can be picked from the Testing Jenkins API, but it does not provide the visibility and requires custom tooling.

During the CentOS Integration SIG Meetup we discussed that we’d like to provide the Git interface to the Compose gate so that each new compose is represented as a new Merge Request to a Git repository.

Proposed Workflow

There is a new git repository (Compose Repo), in the Integration SIG space.

  • New compose is created by the CentOS Stream Infra
  • Automation is triggered by the new compose event. It generates new Merge Request to the Compose Repo (the content of the MR is TBD, but it can be for example a list of NVRs included in the compose)
  • Third-parties watch for the new merge request in the Compose Repo and trigger their own tests on the new compose
  • Once they are done with the testing, they report test results on the Merge Request via votes/labels/comments/TBD…
  • As soon as compose is verified and tested, we promote it to be published to the CentOS Mirrors
  • Automation recognizes the publishing event and merges the relevant Merge Request to the repository

Notes

On the first iteration we will use Git repo only as a representation of the existing pipeline which runs in the background. The Merge Request will be created slightly after the compose is built by the build system and merged slightly after the compose is pushed to the mirrors.

This way we can play with the idea on the side without disrupting the main compose workflow.
Once we figure out the format for Merge Request and Compose Repo content, we may eventually switch the workflow so that Merge Request becomes the main entry point for the compose pipeline.

What’s next

This is currently the outline of the idea.

We will track the work on this topic in the dedicated Epic in the CentOS Integration SIG GitLab space.

Please join us bringing your feedback, questions, suggestions and, of course, contributions :slight_smile:

3 Likes
2

From my perspective, a compose should be a set of lockfiles that determine which versions of which packages from a repository are bundled all together: build system dependency management / lockfile support · Issue #833 · rpm-software-management/dnf5 · GitHub

Maybe this is how we should represent things in this Git repo. This is what we use in Fedora CoreOS to lock the versions of packages in our builds.

In Fedora land, we (Fedora CoreOS) are looking at integrating into ResultsDB for test results.

3

This looks interesting, but there are 2 concerns:

1. deps file

If we put the entire koji tag in one file, it becomes the constant source of merge conflicts and generally a pain to read. So it would be nice to have some way to use /rpm/deps.d/* as an alternative

2. Builds vs rpms

This is a larger issue, which is honestly a big pain for the entire distro development process:

Our development pipeline, updates, koji tags, composes, etc operate on rpm builds not rpm packages. While dnf doesn’t have a concept of a build, as it operates on binary RPMs. Thus from the RPM/dnf side dependency trees for x86 or aarch64 are two different things, while from the pipeline perspective we don’t push update to mirrors if it fails on any of the architectures we support.

So in the proposal we need to track composes as sets of rpm builds. We can not treat them as individual arch-dependent binary rpm packages.

4

@bookwar Thank you so much for driving this effort. I really like where it is going.
There is an existing effort to standardize a rpm lockfile and implementing tools which takes a input file consists of list of bare rpms and outputs the lockfile. I’d suggest to take it into consideration when engineering this request.
I’ve also mentioned of the Epic that I’ve already implemented a sub-set of these effort with using the open source Renovate project.

5

Thanks for summarizing and sharing!

Some open questions:

  • What jobs could be blockers for the publishing events?
  • During the centos connect meeting we were discussing also about being able to create different combinations of CentOS repos for specific cases based on those definition files (i.e. latest content which passed RDO CI jobs). Will that be on the scope of this epic?

In RDO we use similar approach to manage our RPMs repos content and some other metadata (mostly used for tooling). I think it’s wothy to share some of our experience. The in-files db can be found in GitHub - redhat-openstack/rdoinfo: DO NOT send Pull Requests here, send reviews to https://review.rdoproject.org/r/#/q/project:rdoinfo

We use a python library ( GitHub - softwarefactory-project/distroinfo: python module for parsing, validating and querying distribution/packaging metadata stored in human readable and reviewable text/YAML files ) to parse and apply some logic to merge the content of the different files in rdoinfo . This allows us to have separate files for each independent repo (based on openstack and centos release, and promotion phase, see rdoinfo/buildsys-tags at master · redhat-openstack/rdoinfo · GitHub ).

It has been great to have a clean rebases without conflicts and without merge commits. Also we have rdopkg tool using distroinfo parser and rdoinfo repo.

We also use CI on reviews (gerrit based) to do proper CI before updating packages in repos etc… (however we don’t use pungi and composes but purely cbs tagging which is in post pipeline).

Some example reviews:

https://review.rdoproject.org/r/c/rdoinfo/+/52051
https://review.rdoproject.org/r/c/rdoinfo/+/51427

I hope this is useful.

1 Like