Fedora Quality scope reduction announcement and summary

Project Discussion
, , , , , , ,
1

Hi everyone, we have important updates regarding our Fedora Quality team and the scope of our operations for upcoming Fedora releases. This post serves as the main announcement with background info, and a central hub of all the listed changes (with links to individual posts, tickets, discussions, etc), which we can refer back to.

Why

Unfortunately, a significant portion of our internal team decided to switch teams or leave Red Hat [1], which means we ended up in a situation where we have many more tasks and responsibilities than our reduced team can handle. We still have our awesome Fedora community, of course, which provides tremendous support in tasks like reporting bugs and participating in Test Days, and we’re very grateful for it. However, there are areas in which the community is not involved as much, like some areas of release validation, or tool maintenance. Organizing all the activities and processing bug reports also requires a lot of time from us. And at this moment, we simply can’t keep doing all of it. That said, we decided to view it as an opportunity to re-evaluate everything that we did in the past, and try to figure out which activities are the most beneficial for the quality of Fedora as a whole, and which activities have a poor price/benefit ratio and are good candidates for scope reduction. Some of the changes could’ve been done a long time ago, we just lacked the right impetus.

[1] To prevent any misunderstanding or conspiracies, these were truly decisions of our colleagues, they were not pressured or fired. You might still be able to see most of them around, this time focusing on AI (which they found most appealing), or other tasks. In the past ~9 months, we’ve gradually said goodbye to tflink, frantisekz, jskladan, coremodule, lbrabec and sumantrom. We gained one person (kashyapc), so currently our team includes adamwill, kparal, lruzicka, lnie and kashyapc.

The changes

Below is a list of changes which we believe are needed. Many of them will be proposals, because a different entity (like FESCo) needs to approve them, or they need a wide stakeholder agreement (e.g. release criteria changes). Those can end up rejected, but since we don’t believe we have enough manpower for them, an alternative solution might be needed (e.g. some SIG involvement).

We also know that removing features is unpopular - there’s always someone affected. We expect it will be the same with these proposals. But in order for the Fedora project to stay healthy, adding more and more new features needs to be combined with trimming outdated or unnecessary ones, so that contributors are able to focus on the new ones, and not just maintain an evergrowing project. Please try to look at it through this lens.

The list below will get populated over time with links/details. We publish the main structure immediately, so that you can see that this is not targeted against some specific edition/architecture/feature/area/etc, but we’ve really done a wide evaluation of everything that we do.

Please don’t discuss items which have no links or concrete details yet, let us justify it first. Let’s use this topic for general discussion regarding Fedora Quality and its priorities, and linked discussions/proposals/etc for a specific discussion of a particular change.

Release criteria & validation testing changes

The list below affects how we validate new Fedora releases, what gets tested (and how much), and what constitutes a release-blocking bug.

A note for people not familiar with the Release Criteria process: Dropping a release-blocking status of something doesn’t mean dropping that thing completely from Fedora, or intentionally breaking it. It just means that the whole Fedora release wouldn’t stop, if an important bug was found in that area. The bug would be a “standard bug” that gets fixed on its own timeline.

Proposed changes to release criteria and test coverage:

We also feel that we need help from other teams in specific areas. Generally, we’d need more involvement from SIGs and WGs (Workstation, KDE, Server, Cloud, ARM, etc) when performing manual release-validation tests, debugging issues, and validating fixes. Here’s a breakdown of the current situation and needed help:

  • Installation - We have good automated coverage and Anaconda folks are setting up their own automated system that reports results to us, so no changes are needed here.
  • Workstation - Our automation covers over half of the required test cases, but there are still several that must be run manually and some of them are quite heavy, so a bit more help in performing manual tests of the GNOME desktop and apps would be appreciated.
  • KDE - Similar to Workstation, automation covers a chunk of the testing but a substantial manual chunk remains. The remaining RH team members have no day-to-day experience with a Plasma desktop and apps, so we really need more help from the KDE team here.
  • Server - Our tests are almost fully automated, which means we mostly need help just with debugging issues and testing fixes.
  • Cloud - Local testing is automated, but we strongly need Cloud folks assistance with testing the images in real-world cloud environments - either doing this manually (ideally earlier and more frequently) or automating it.
  • IoT & ARM - Virtualized testing is automated, but we need help with all manual testing, particularly desktop tests (especially if they are kept as release-blocking).
  • CoreOS - The CoreOS team handles all testing and fixes completely on their own, and they approach us when a blocker is found. (Thanks!). No changes necessary.

We will come up with a process to inform individual teams of areas/test cases which need testing assistance and make sure they are informed about the testing schedule.

Tools changes

We no longer have manpower to maintain the following tools:

  • Testcloud - This is used to test cloud images locally. Fortunately, the Fedora CI team uses it heavily as well, and they are taking over its maintenance. It’s already hosted in a new repository under the TMT umbrella.
  • Packager Dashboard (web) - This is well-loved by many Fedora packagers and we know it. It would be a pity to lose it. We might be able to find new maintainers within Red Hat. If that doesn’t work out, we’ll send an announcement asking for community maintenance.
  • Oraculum - This is the backbone of Packager Dashboard. It is also partially used by Fedora Easy Karma to speed up Bodhi queries, and by Testcloud. Its fate will likely mirror Packager Dashboard.
  • Fedora QA Landing page (web) - This is unmaintained and not very useful. We decided to kill it.

We will also orphan unneeded packages (to be clarified) owned by qa-tools-sig.

Why permanent changes?

You may wonder why we’re proposing major permanent changes instead of planning to bodge around the problem temporarily and hope to hire enough people back to keep things as-is for the future. We could do that, but we decided to look at this change as an opportunity to re-examine what’s really important to Fedora right now, and aim to use our resources as wisely as possible for the future. We’re not sure that spending a lot of time on manual testing of relatively less-widely-used functionality is the best thing the RH team could be doing, even if we had more people to continue doing it. We will be hiring in future (see below!), but we want to be able to consider the best possible way to spend all of the paid team’s time going forward, and perhaps spend it on other things that will be more valuable in the end, like automation or supporting individual teams. We also want to make sure the balance between the RH quality team, the quality community, and SIGs/WGs is at its best in all areas of validation testing.

Closing words & a hiring pitch

We know that that’s quite a lot of changes and some of them might affect your own systems. We tried to pick areas where the impact should be less severe, and where we can still save a lot of resources. Once again, please note that making something not release blocking doesn’t mean it will stop working and won’t get any fixes. It just gets less attention.

We will gradually populate the list with individual proposal and discussion links. Please wait and let’s discuss individual changes in their respective places, so that we can have a tidy discussion for each topic. Thank you.

Last but not least, we’re hiring. Please see our current job description (if the link is currently down, we’ve just closed the current round of interviews, but we’ll likely reopen it again in the future, please check again later!). Note that it is written for a Senior Engineer, but we’re also looking for a Junior/Medior with softer skill and experience requirements. So please, if you’re not sure if you’re good enough, but would be interested in working on the quality of Fedora, just try it and apply anyway! Or if you know somebody like that, please send them the link. You can also message @kparal or @adamwill and we’ll make sure that our Talent team won’t lose your application somewhere along the way. Or you ask us questions. Thanks for your interest!

11 Likes
2

I think your taking a good approach to the situation. In the long run I think it will serve the community well. As one who does manual bare metal testing I look forward to learning where to focus my efforts. Of course I’ll speak up if I think there is a problem with any of the proposals.

3

The proposed changes seem reasonable to me.

One question: does the resourcing situation in the Quality team have implications for the drive toward “Editions block on accessibility”, as prioritised in Strategy 2028?

4

While I can’t speak for the rest of Cloud SIG, from my perspective it makes sense for the Cloud SIG to own testing the public cloud images.

I’ll see about writing up a more detailed plan with the Cloud SIG in the next week or two.

4 Likes
5

I hope not, but it’s possible unfortunately. Ideally we’d be able to automate the required testing for that; the most obvious problem there is automating testing of screen reading functionality. openQA has a kinda clever feature for doing audio testing where it records a waveform and then runs its image comparison stuff on the waveform image, but I have never tried really using that feature in anger before, I don’t know whether it’s robust enough to reliably back an entire audio-based workflow. If we can’t find a way to automate that it’d be a significant chunk of manual testing, and we’d have to figure out some way to handle that.

6

Implementing the self-test stack for Azure would be a great gating tool for verification when we are making potentially breaking changes in the Cloud SIG would be a good idea. That would work well. Having some test runners with reporting is probably a good idea all around and we could implement something like that.

7

There’s a variety of speech-to-text tools out there - whisper is the one that comes to mind first, but there’s others - which I’ve spent a non-trivial amount of time messing with. If the waveform image approach is too flaky, I’d be interested in helping out with an approach that transcribes the audio.

1 Like
8

I thought IBM laid them off and moved people around?

This reads like IBM wants to fob off what was once done by the Red Hat team it laid off to the community.

9

None of this has anything to do with IBM or any layoffs. That’s why we specifically explained it in the text. Nobody was laid off. Not by Red Hat nor by IBM. Nobody was “moved around” by external forces - they chose to move, for various reasons (career opportunity, wanted to do less testing and more engineering, just wanted a change…)

3 Likes
10

As was discussed in our QA meeting yesterday, this does make sense.
Being in the Fedora KDE SIG as well, I do try to keep my focus there on Fedora and KDE testing.
This includes both x86 and aarch64. If there’s anything I am missing please let me know.

2 Likes
12

As one of the former Red Hat employed Fedora Quality folks mentioned, I can confirm that leaving Red Hat was my choice and I made that choice for my own reasons - I was not laid off nor was I pushed out by IBM or any other entity.

I’m still around in Fedora, I’m just involved in different areas and I’m spending a lower percentage of my work time on Fedora than I did in the past.

I’m not sure how you got to that conclusion, to be honest. There have always been things in Fedora Quality that have been done by Red Hat employees most (if not all) of the time for any number of reasons. I’ve thought of it as “if it was fun 100% of the time, they wouldn’t need to pay someone to do it” because not all things are fun and thus, not all tasks will have volunteers lining up to do them but that’s natural as far as I’m concerned - folks in the community are free to spend their time on whatever they choose to do or not do. I can guarantee you that nobody shows up for blocker review meetings because they’re fun.

It’s a fact that there are fewer folks employed to work solely on Fedora Quality than there were a year ago. I expect that to rebound but as @kparal said in the first post, it’s a natural time to reassess existing tasks to see if they’re as much of a priority as they used to be.

There are always going to be finite person hours available and new requests will always be coming in - it’s about balance and doing what people believe to be good in the long term :slight_smile:

edit: slight rewording for clarity

3 Likes
13

Sorry to prove you wrong. :rofl:

4 Likes
14

For all of these Fedora Quality proposals for release criteria, could we make an announcement once a decision is made on each of them to let folks know ahead of the release when these apply that there are changes to how testing will happen? Having one announcement that details all of the release criteria changes will be helpful if users have questions before and after the release when these apply.

15

(Joseph, I moved your comment to this topic).

That’s a good idea. Of course each individual proposal will get marked as approved or rejected, but I’ll also update the summary in this topic, to indicate what will be taking effect and what won’t.

1 Like
16

whisper is spooky good—I can transcribe podcasts on my modest x86 miniPC in near-real time using the small base.en model which uses just few gigabytes of memory

17

Update: The proposal Make Intel-based Macbook dual boot no longer release-blocking has now been put into action. See the summary above for links with details.

18

Update: The proposal Require basic functionality of selected apps equally on all release-blocking desktops, i.e. drop the Workstation x86_64 exception that covers all pre-installed apps has now been put into action. See the summary above for links with details.

19

Update: The proposal Make optical media boot no longer release-blocking has now been put into action. See the summary above for links with details.

20

Update: The proposal Reduce BIOS-based systems release blocking status from covering all scenarios (on parity with UEFI) to just limited scenarios has now been put into action. See the summary above for links with details.

21

FESCo was notified about the already-implemented changes here: