Looking for early feedback about the new artifact install approach in tmt / Testing Farm

lecris · November 10, 2025, 2:10pm

Cross-posting this here from the devel list. We should also make a formal change-proposal as well, but was considering doing it after getting some initial feedback, but not opposed to doing it earlier.

Hi!

Have you had issues with a tmt test not working due to a package installation issue? Maybe you had a big side-tag update not passing some gating test on bodhi or testing-farm? Good news, we are on our way to resolving these issues and your feedback would be greatly appreciated.

Recently we’ve started work on the new approach for installing tested artifacts (e.g. rpm packages) on the guest before the testing starts. We would like to gather early feedback on the proposed design as this is quite a significant change.

Here we’ll be demonstrating the approach on a koji build and included rpm packages but the same/similar might be applied for brew builds, copr builds, local rpms and other artifact sources. We also want to keep the implementation open for possible future extensions (e.g. container images).

What are the most significant changes?

Before: Testing Farm takes each requested koji build, downloads and installs all rpm packages included on the guest. There are some special hacks for the “exclude” key which allow handling conflicting rpms in a single build.

After: Testing Farm hands over the koji build id to tmt which installs from the koji build only those packages which are explicitly requested by the test or plan.

Which artifact sources will be supported?

Currently we have the following sources on the list:

koji/brew build (identified by build id, task id or nvr)
copr build (identified by :)
dnf repository (identified by baseurl of the repo)
dnf repository file (identified by url to the repo file)
rpm files (individual file paths or a directory, local or remote)

But more are expected in the future. See the linked issues for more details and the outlined implementation. Feel free to ask questions or suggest changes directly there.

How exactly will this work?

All packages from the koji build are downloaded on the guest. List of the exact package nvrs is stored. A dnf repository is created from the packages and enabled with a suitable priority. Packages mentioned in test require or recommend keys, or in the plan prepare step are installed. Versions of the installed packages are verified to make sure the right packages were installed.

Here’s an example prepare step config:

prepare:
  summary: Prepare repository with packages
  how: artifact
  stage: prepare
  provide:
    - koji.build:1234
    - koji.task:5678
    - repo-file:https://another.url/repofile
    - repo-url:https://another.url/repo

You will most likely not interact with it via the static files, instead testing-farm will take care of injecting this step via the command line:

tmt run --all prepare --insert --how artifact --provide koji.build:1234 --provide koji.task:5678

What are the benefits?

Consistent way to prepare artifacts for testing, e.g. when developing tests locally on your laptop you would get the very same result as when run in the Testing Farm. Simplified tmt reproducer line. This will also allow us to fully move to the new pipeline which supports multihost testing.

No problems when conflicting packages are included in the koji build as only the right subset of packages is installed on the guest. Possibility to test minimal install or various combinations of packages which can reveal bugs hidden by always installing all packages.

If a test needs to perform some advanced package handling actions (e.g. verifying that reinstalling a package is working as expected) it can make use of the dnf repository with the fresh packages to perform the desired dnf operations. No extra steps should be needed from either Testing Farm or the user.

What is expected from the users?

Make sure that all packages necessary for testing are included in the require or recommend key of the test or are explicitly installed using one of the prepare step plugins.

What do you think?

Does the proposed design look ok? Do you have any concerns? Are there any important use cases or corner cases we should keep in mind when working on the implementation? See the tracking issue for some more details about the design and the current progress:

github.com/teemtee/tmt

Implement the `artifact` install plugin

opened 09:24AM - 27 Aug 25 UTC

psss

step | prepare

This is the tracking issue for efforts related to implementation of the new `alm…ighty` install plugin. Here's a concise summary of the approach discussed in #2226: ### Prepare Artifacts The main goal is to set up repositories with packages to be tested with the correct priority set so that installing a package by its name will pick the right version from this repository and not from the default ones. * Download all rpms to the testing environment (selected methods only) * Create a dnf repository with all the rpms and enable (selected methods only) * Enable the repository and set the priority to override default system repositories * Extract package nvrs and store them for future use (double-checking versions) ### Install Artifacts This step takes care of gathering the list of required or recommended packages, installing them and verifying that the expected versions were actually installed on the system. * Create a list of files to be installed (with locked versions, i.e. lock file) * Create a list of packages to be verified (required vs. recommended) * Install packages from the lock file * Verify all packages were installed correctly ### Scope * For the first implementaion only `dnf` is considered, but keep the implementation extensible so that support for other package managers can be added later * Similarly, for now only `rpm` packages are considered as artifacts, but keep the approach open for future extensions (e.g. podman images) ### Notes * Multiple versions of the same package can be present in the artifacts repository (same package name, different nvr) * Single repository can include multiple arches of the same package * Plugin implementation always handles one guest at a time (artifacts are downloaded on each guest separately) * When doing package -> image mode conversion, the tool/script doing the conversion has to be adjusted to copy the local repo with packages into an image to make sure it remains available for the image mode system ### Examples The `prepare-artifacts` and `install-artifacts` phases should be configurable separately: ```yaml prepare: - summary: Prepare repository with packages how: almighty stage: prepare dnf-repository-file: ... - summary: Install required packages how: almighty stage: install package: [need-this, need-that] - summary: Install recommended packages how: almighty stage: install missing: skip package: [optional-this, optional-that] ``` ### Comparison In the current approach Testing Farm * provisions a guest * fetches the brew build * installs packages on the guest and instructs `tmt` to connect to the prepared guest: tmt run --all provision --how connect --guest IP --key KEY In the future approach Testing Farm will hand over data received through API and `tmt` will take care of provisioning and package installation: tmt run --all \ provision --how artemis --update-missing --compose COMPOSE \ prepare --insert --how almighty --brew-build BUILD_ID ### Links * #2226 * https://issues.redhat.com/browse/TT-224

We would be especially interested in suggestions regarding what would be the best way to set the correct priority for the repository so that it plays well with other defaults. Also, would you have any recommendations about what should happen if there are multiple nvrs for the same package included in the source repository?

Looking forward to your feedback.

psss