F42 Change Proposal: Deprecation of STI Tests (self-contained)

Project Discussion Change Proposals
,
1

Deprecation of STI tests

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Wiki
Announced

:link: Summary

Display a deprecation warning for Fedora 41 for all STI tests. Deprecate execution of STI tests in all CI pipelines for Fedora 42.

  • CI for bodhi updates
  • CI for dist-git pull requests

All users of STI tests will need to migrate to the new tmt format.

The change will affect 281 components. The list of components affected can be found via this sourcegraph query.

All references to STI will be removed from the Fedora CI documentation.

:link: Owner

:link: Detailed Description

For some time CI testing in Fedora can be defined using two different formats:

  • Standard Test Interface (STI)
  • Test Management Tool (tmt)

As tmt has matured to a state it can fully replace the STI functionality, we are proposing to drop STI as the supported format. STI format is not actively developed and is causing us a maintenance burden. More and more STI test failures are starting to appear as the Fedora packages evolve out of sync with the test scripts.

Tmt provides various advantages over STI that would make it easier to manage in the long term:

  • Better organization of tests and test environments (referred to as plans in tmt)
  • Local reproducible environment thanks to testing-farm reproducer script, allowing to thinker locally without pushing to dist-git
  • Tests can be defined in the local dist-git repo, inside the srpm archive, tests/* dist-git namespace, or in the upstream repo, allowing it to be reused more freely between packages and with upstream directly
  • Tmt tests are integrated with packit allowing it to be executed with upstream and better migrate to the new dist-git environment

Hopefully this list can help inspire some better ways of reorganizing the tests during the migration.

:link: Feedback

:link: Benefit to Fedora

Having two formats for executing the tests is an unnecessary duplication and causes confusion for the Fedora maintainers and community.

STI tests have limited functionality and are harder to develop and maintain, when compared to tmt.

:link: Scope

  • Proposal owners: Miroslav Vadkerti, Cristian Le

  • Other developers:

  • Release engineering: #Releng issue number

  • Policies and guidelines: N/A (not needed for this Change)

  • Trademark approval: N/A (not needed for this Change)

  • Alignment with the Fedora Strategy:

:link: Upgrade/compatibility impact

This is a change to the development experience, no changes to Fedora distribution are made.

:link: How To Migrate

To find out if you package has STI tests, check if tests/tests*.yml files are present in your dist-git repository.

Follow the sti to tmt migration guide. For the majority of tests involve just executing the already provided script you only need to:

  1. Initialize tmt structure using tmt init

  2. Create a minimal plan

/plan.fmf # Boilerplate to indicate tmt to run any tests files it finds in the repo discover: how: fmf execute: how: tmt # Prepare test environment, e.g. install packages # Note: the current packages in the spec file will already be installed by default prepare: how: install packages: - foo - bar

  1. Migrate individual tests

/tests/foo.fmf test: ./foo.sh

For more complex needs feel free to discuss in the #fedora-ci or #tmt matrix channels. You can also explore other project’s tmt tests (identified by having a .fmf folder) using tmt tests show and tmt plans show

:link: User Experience

For Fedora 41 users will be presented with a deprecation warning in the Testing Farm artifacts.

For Fedora 42 users who have setup gating on STI tests will be obligated to migrate to be able to push their packages to stable repositories.

:link: Dependencies

N/A

:link: Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
  • Contingency deadline: N/A (not a System Wide Change)
  • Blocks release? No (not a System Wide Change)

We will keep the STI support until all references to STI tests are gone in case all users have not migrated to tmt until Fedora 42. See the description for the list of the affected components.

:link: Documentation

:link: Release Notes

Last edited by @amoloney 2025-01-06T19:40:57Z

Last edited by @amoloney 2025-01-06T19:40:57Z

2

How do you feel about the proposal as written?

  • Strongly in favor
  • In favor, with reservations
  • Neutral
  • Opposed, but could be convinced
  • Strongly opposed
0 voters

If you are in favor but have reservations, or are opposed but something could change your mind, please explain in a reply.

We want everyone to be heard, but many posts repeating the same thing actually makes that harder. If you have something new to say, please say it. If, instead, you find someone has already covered what you’d like to express, please simply give that post a :heart: instead of reiterating. You can even do this by email, by replying with the heart emoji or just “+1”. This will make long topics easier to follow.

Please note that this is an advisory “straw poll” meant to gauge sentiment. It isn’t a vote or a scientific survey. See About the Change Proposals category for more about the Change Process and moderation policy.

3

I am confused. Is this a “deprecation”, or is it “removal”?

The change page says:

Display a deprecation warning for Fedora 41…

Right, but this is a Fedora 42 change.

Deprecate execution of STI tests in all CI pipelines for Fedora 42.

What is the difference between that and what it said for Fedora 41?

All users of STI tests will need to migrate to the new tmt format.

So, it is not a deprecation, but it will cease to function?

For Fedora 42 users who have setup gating on STI tests will be obligated to migrate to be able to push their packages to stable repositories.

This again seems like you are not proposing a deprecation, but disablement.

Please either call it deprecation and make it a deprecation, or call it retirement/removal/EOL and make it that. In either case, make it clear and don’t mix the two things.

4

The initial idea of the wording there is to migrate all F42 repos during this change, but still allow F41 branches to continue using the previous STI tests if the repo cannot have the changes easily cherry-picked, e.g. downstream uses tmt tests that are defined in the srpm, but these are only defined on a version later than the F41 release repo. So the runner would continue to provide the STI workflow, but:

  • F42 bodhi gaiting against STI tests would continue to run, but the tests results would be overwritten to show a failure as a further push for the migration (can of course be overwritten by waiving the tests gaiting)
  • F42 CI tests that are not gaiting would continue to run similarly with overwriting of the state to failure, but since they are not gaiting it has less impact. It could be used for PRs though to check equivalence during migration
  • F41 CI and others would just cause a warning message, but not alter the outcome

This is my understanding of the initial deprecation plan, but @mvadkert could share more details and motivation of these. Initially the change proposal was written for F41, but I suppose it is not possible to make a proposal for an already released Fedora branch.

As for how to word it and course of action, we’re open for the discussion, what approach you think would be most feasible? Some minimal actions that we want in this change proposal are:

  • Stop new repos from using STI tests
  • Have an issue tracker for the migration and post in the relevant repos to signal the migration
  • Announce in other ways possible (this cange proposal being one of them) in case the bugzilla issues do not get traction. Or have any other tool to nudge the maintainers towards some action
5

Good thing this is spelled out at least once, otherwise “STI Tests” would be … ambiguous :no_mouth:

3 Likes
6

@churchyard Thanks for the feedback! We made changes to the proposal according to your comments with @lecris. Can you please check again?

7

The proposal is called “Deprecation” yet it keeps saying STI tests will “stop running” and/or “keep running … until Fedora 42 release date” (and I suppose stop running after that) . I am still confused. The goal seems to be to stop running STI tests at some point in Fedora 42. Why do you call this deprecation?

8

Sorry, renamed to Disablement of STI tests.

@amoloney Howdy, I am unable to edit the transferred post. I wanted to rename the proposal name based on the discussion. It keeps asking me to set at least 3 tags, but I am unable to alter tags. Any idea?

9

Thanks.

Now when the intention is clear, I have a suggestion:

keep running STI tests for Fedora 42 dist-git pull requests until the release date and stop running them after the release.

This means that what was working suddenly stops at a rather late (and arbitrary) moment form the developer’s point of view – nothing should stop working this late in the development cycle. May I suggest we keep running STI tests on Fedora 42 forever, but stop running them on Rawhide when Fedora 42 branches? That would effectively make this a Fedora 43 change, but at least the boundary would make more sense and packagers would have the entire development cycle to adapt.

10

Nothing would “stop working” though, the tests would simply not run anymore, and since the dist-git PRs are not gating, it would just be a silent pass.

The only affected party are those who have gating on for STI tests. We couldn’t query them with sourcegraph.com so not sure how many there are. But they would not be able to gate on specific STI tests, since it is randomly generated afaics, so if they have any tmt test already setup, that would not be an issue either.

Edit: Oh, if you mean like the tests not running also means the workflow had stopped working. Indeed that could be the case. But, as per the giving time to adapt, we also want to make it a change as early as possible to tackle it as early as possible as well and give more time to work with.

11

To my understanding, that’s a contradiction.

I am an affected party even though I don’t use gating. I use Pull Requests CI.

Sure, just don’t do it this late in the release cycle, please.

12

This change proposal has now been submitted to FESCo with ticket #3338 for voting.

To find out more, please visit our Changes Policy documentation.