❌ Proposal: No longer block releases on ARM desktop

Project Discussion
, , ,
1

Update: This proposal was rejected, but ownership for selected parts of the Quality process was transferred. See more details in this post.

Hi everyone, this is a proposed change for Fedora Release Criteria. It is related to a wide set of changes that Fedora Quality team need to perform this cycle, which are summarized here, so please feel free to read that for more background information and general overview, thanks.

Proposal: Amend the release-blocking desktops definition at Basic_Release_Criteria#Basic_Release_Requirements to say “The current set of release-blocking desktops for x86_64 is GNOME and KDE. There are no release-blocking desktops for aarch64.”. As a consequence, mark all ARM desktop images non-blocking in https://docs.fedoraproject.org/en-US/releases/f43/blocking/ (when it exists), and mark all ARM desktop tests in the validation matrices as non-blocking.

Justification: We recognize that this is a radical proposal. However, desktop testing on ARM is a significant amount of work, due to the slowness and flakiness of the hardware, and the need to repeat testing on multiple desktops. Some testing is automated in openQA, but there is still significant manual testing required, and issues do quite often arise on hardware that do not appear in automated testing. The data we gathered indicate that there is very low usage of desktop Fedora on ARM platforms. The following graph shows architecture split for systems that identify as desktops:

Fedora desktops per architecture
Fedora desktops per architecture3208×1147 172 KB

There are a decent number of Fedora systems which don’t have a variant specified in /etc/os-release, possibly because they were installed using custom-made kickstarts (without including a desktop environment group) or have been around for a very long time. That changes the architecture split a bit, but we’re not sure what percentage of those variant-less systems are desktops. So even in the most optimistic unlikely scenario where all variant-less systems were desktops, the percentage of aarch64 desktops would be as shown below:

Fedora desktops + unspecified, per architecture
Fedora desktops + unspecified, per architecture3274×1147 182 KB

In reality the aarch64 numbers are likely somewhere between those graphs. The percentage of time that Quality spends on aarch64 systems testing and problem solving each cycle is far higher than its current usage share among Fedora users. This proposal (and related proposals [1] [2]) aims to rectify that, i.e. devote some time to aarch64, but avoid use cases which use up a lot of Quality time while being niche among our users.

Update: A FESCo ticket was created, proposing to drop Workstation aarch64 release-blocking status, while keeping it for KDE aarch64.

2

Or they just install using the network install thats about 1/3 the download size and always get you an updated install. Thats at last how I have done my installs for as long as I remember.

3

still, if you install a desktop environment that way - using the environment group, either by selecting it at install time, or doing something like dnf install workstation-product-environment (I didn’t check that name, might be a bit wrong) after install - it should check in as that edition, I believe.

1 Like
4

I would be disapointed if I cannot use Gnome and Plasma on aarch64.
I run Plasma and Gnome in VMs on Apple macOS for example and use it all the time.

The VMs on aarch64 I use to support Fedora on the forums and to do some of my Fedora development work.

There are also new arm laptops that are appearing at the moment that look like a great place to run Fedora.

3 Likes
5

It would be interesting to see those graphs for aarch64 only, as to better observe the trend of this architecture usage. But from what I can tell from the two graphs is that while still low in usage, aarch64 has increased several-fold over the analyzed period.

This is consistent with the growth of ARM market share on the desktop/laptop market overall. And Fedora’s VM usage on M series Macs (as opposed to Intel Macs) is only increasing.

So while the current architecture split might speak in favor of this proposal, the trend doesn’t.

(Written on Fedora Silverblue in a VM running on an M1 iMac host.)

6

There’s only a trend apparent in the second graph, which is due entirely to the contribution of “unspecified” systems. Basically, a bunch of those suddenly showed up. We have absolutely no idea what they are. They might be desktops, they might not.

The first graph shows the things that we know are desktop systems. There’s no recent increase trend apparent there.

7

When the data series of a specific line are significantly smaller than those of another data series, as it is the case of aarch64 in the first graph, the plot on the graph becomes visually relevant only for the higher data series. That’s why I was suggesting if it was possible to show the first graph, but for aarch64 only (i.e. without x86_64).

8

Even if there was an upwards trend, I believe Fedora gave desktop aarch64 a too high importance too soon. We should’ve waited for the real-world usage to be meaningful before having it as the top priority release-blocking deliverable. And the usage is still not there yet.

Regarding numbers, VMs on Macs probably constitute a large percentage of the user base. Because when we talk about aarch64 usage in general (not just desktops), the number one usage scenario there is podman on Mac.

3 Likes
9

A problem with having any installation media not be release-blocking is that the media may not be installable or have severe bugs.

Can such proposals be supplemented with the potential to officially respin releases at a later date for such situations?

10

It makes sense what you request, but I don’t want to include it as part of this proposal. The reason is that this, as written, is very simple to implement, we just change a definition on our wiki. However, official respins would require us to come up with a whole new process, which we currently don’t have, including design and probably adjustments in many toolings throughout Fedora. It wouldn’t need to target just ARM desktops, it would have to be generic, because it wouldn’t make sense to have it for e.g. ARM desktop, and not for x86_64 XFCE, or s390x Server, etc. Also, the work on preparing an official respin is not free, so there would have to be guidelines on when to trigger it. Making official respins process ready would probably take a lot of discussions and then time to implement, which we currently don’t have (this targets F43).

There are unofficial respins which are quite popular I think, and we even host them on Fedora’s main mirror:

However, those don’t include ARM images, just x86_64 Live images. But maybe they could, and it might be a faster bandaid before we have official respins process (I’m quite sure we’ll need it in future, we hit these scenarios regularly).

11

If the users aren’t there to justify having Arm support as being release blocking, then it is what it is. However, I do worry about what level of testing Arm images will receive if this is dropped.

That’s not to keep the onus on the Quality Team, but it begs the question - what offboarding can we provide to Fedora on Arm so that we can continue to move the Linux desktop further in this space? Do we need to bring the Arm people into this? Are there conversations we can have with vendors? How does Fedora Asahi play into this, if at all?

It would be cool to see a plan that says “Fedora on Arm is in this position today. We want it to be over here but we need x, y, and z to line up to make it happen.” Similar to my thoughts on what to do for Fedora IoT, having a plan like that would help so that contributors know where to start to solve the problems you’re dealing with, which today seems to be lack of use.

12

Interesting for me, since I have been contemplating the purchase of a Risc-V based laptop. I was holding back because Fedora on Arm still seems in the early stages. This does make sense (the proposal) since the project does have limited resources. Maybe we need more community testers involved on aarch64 systems testing. I know that when I do purchase a Risc-V based laptop, I will be installing Fedora onto it, and that will certainly increase my activity in this subject matter.

13

I agree to this, with a caveat.

You mentionned maybe adding arm isos to the respins; could we post those on the main website for the aarch64 builds? This way, if a crippling bug does happen, the iso can get fixed quickly? Or at least, can we setup a contingency where the iso on the website can be swapped for a good one in the case of a bug on arm?

14

I’m fine with all of that, but I’m not the one who can make that happen. This will probably require a FESCo ticket creation as the next step, so I can mention it in there as one of the possible contingency solutions.

What I can definitely do is to link to those unofficial respins (when arm ones actually exist) from Ask Fedora > Common Issues , if such a bug occurs. We do that even for x86_64, when necessary.

15

If I am reading the proposal right, this is only stopping the aarch64 images from being release blocking. Not from being created.
So all the GNOME / KDE / SPIN aarch64 images will still be created, they just won’t have the same testing that that x86_64 images have.
Thus, people’s worries that there won’t be any images are off. The images will still be there, and if people want to test and/or file bugs on their own, they will be free to do so.

Or have I read the proposal wrong?

16

Yes, that is correct. The images will see less testing by Fedora Quality, and won’t block the release, if they contain a major bug.

1 Like
17

Then I am totally fine with this proposal.

I, and the other KDE SIG members, can continue to test our aarch64 images on our personal machines like we have been. We’ll file any bugs we find in bugzilla/upstream like we usually do. We just won’t need to file the Fedora Quality forms.

(I know I haven’t done that for a couple months. My machine died and I just barely got a replacement.)

2 Likes
18

I don’t want to derail this conversation but…
I wonder if there is a way we can work with podman team so we can get better telemetry so we can sort them from other install types.

1 Like
19

I think the best way to say this is…

It’s likely that images to see less testing and there is an opportunity for self-interested contributors to stand up to take on more of the testing work to help make sure that doesn’t come to pass.

The truth is that available human QE resources have decreased. This may be the first time for this project that the QE resource changes have been very visible. That probably speaks to sort of hero effort work the QE team has been doing to hide ebb and flow of resources, but it also means as a community project we probably havent been doing the best job ensuring we are being honest about the workload and where the gaps are…and now there’s a big gap and we don’t have a process that has anticipated the shortfall.

And because of the nature of how QE work is structured this seems to be a rare event… unlike what happens with the ebb and flow of package maintainership…which has very visible churn and a process by which we make that churn visibility.. so possible self-interested contributors can stand up to do the work…

What the project needs long term is to figure out how to make the QE. contribution opportunities in the form of coverage gaps. Something in the same shape of the package orphan/retirement process.

This also means we need to re-examine what being an Edition means, because it seems like there’s so implicit qa stuff that goes along with that. We have to get away from a process that makes it feels like things are getting demoted. What we need is a process that self-interested groups can feel like they are doing the work to lift themselves up by meeting specific criteria.

1 Like
20

I don’t think KDE Plasma’s release blocking status on AArch64 should change. That is already not validated by Fedora QA, and done by us in the Fedora PSWG/KDE SIG. Whether resources go up or down with the staffed QA folks is irrelevant for Fedora KDE’s AArch64 release blocking status.

Instead of dropping release-blocking status, perhaps the other release-blocking desktops will need to move the same arrangement as we have with KDE.

Release blocking status is important to us in the PSWG because of how much work we are doing to support KDE Plasma on ARM systems.

1 Like