Proposal: Re-consider Edition status for IoT

Project Discussion
, , ,
1

Hi everyone, this is a suggestion to consider a change to the Fedora Editions. It is related to a wide set of changes that Fedora Quality team need to perform this cycle, which are summarized here, so please feel free to read that for more background information and general overview, thanks.

Proposal: Re-consider whether Fedora IoT merits Edition status.

Justification: When Fedora.next happened in 2014, we had three Editions: Workstation, Server, and Cloud. These clearly covered the three major use cases for Fedora at the time. Since then, we have kept on accumulating Editions over time. We now have six Editions - Workstation, Server, Cloud, IoT, CoreOS, and KDE Plasma Desktop - and one “emerging” Edition - Silverblue. This feels like too many editions.

IoT seems to have a degree of overlap with CoreOS. IoT has a clearer focus on hardware deployment via FDO (now, formerly it was via Zezere), but other than that, both are intended as minimal platforms for deploying container-based workloads. Of the two, IoT seems to have less maintainer support (we believe just two maintainers) and arguably less usage and mindshare. Looking at countme stats as of the end of July 2025, Workstation had about 200k active checkins weekly, KDE about 146k, CoreOS 107k, Server 49k. IoT had just under 4k, substantially less than any other edition, and about half of Silverblue (which is not an Edition).

Fedora variants usage
Fedora variants usage4800Ă—2700 482 KB

There is an interesting spike of IoT usage since Fedora 40, indicating there is some company that deployed it to at least a thousand systems. However, the absolute numbers are still very low compared to other Editions.

IoT usage
IoT usage1999Ă—1079 251 KB

Our primary concern as the Quality team is the amount of work it takes to maintain criteria and test cases, deal with the release process, and run manual tests and diagnose failures. For all these reasons we would like to see the number of Editions start going down, and IoT seems like the clearest candidate to be reconsidered. But we think this would also be beneficial overall - too many Editions also causes a burden on other teams, and is confusing for users and contributors.

4 Likes
2

Just a note on the procedure here - this would obviously need to be a Change proposal and/or Council ticket. We’re just floating it as a preliminary idea here; if we move forward with it we’d do it via the correct path.

1 Like
3

This will need to be requested to Council first, then submitted to FESCo as a Change linked to an approved Council ticket.

1 Like
4

It feels weird that the basis to decide on which edition to drop is popularity contest (I’m sure someone will bring up that countme isn’t particularly reliable for the type of systems Fedora IoT often gets deployed on) as it’s (often) used as something to build a product on top of; for example your deployment of a few thousand sensor boards. The numbers also seem to be very far apart from the deployments I know of, which have larger numbers of Fedora IoT instances deployed than the total in this graph. Are we sure these numbers are correct and can be used to base decisions on?

My idea was always that Fedora Editions serve specific use cases; that idea has been muddied a bit I guess with both Workstation and KDE Plasma and both Cloud and Server but comparing CoreOS and IoT also feels off, they have different boot processes support very different hardware, and are meant to be deployed in very different places. One is something you could take to build your own IoT product on top of, the other is something you deploy in your own datacenters.

Can you maybe give some more details on what it would mean, and how much work you would save, for the QA team if Fedora IoT gets demoted to a Spin? How much time are you currently spending on this edition versus the other editions?

For me personally I love IoT and have it running on all my SBCs, I feel it brings value to Fedora and its ecosystem and would hate to see it lose its Edition status.

2 Likes
5

I wouldn’t have any issues if IoT was downgraded to a spin.

6

Well, we had to pick something to propose, and those are the best numbers we know of. They’re not perfect, but I don’t think we have anything better.

You can also look at the iot and iot-wg tags here; they’re pretty stagnant. If you look for Fedora IoT discussion on e.g. reddit, there’s not much there either. By any metric we could find (all of which are imperfect), it felt like the least prominent of the Editions.

Can you maybe give some more details on what it would mean, and how much work you would save, for the QA team if Fedora IoT gets demoted to a Spin? How much time are you currently spending on this edition versus the other editions?

Well, the biggest issue is the hardware matrix - we rarely actually manage to fill that out, but by policy we’re supposed to. Even what we do manage to do is a time-sink similar to the ARM hardware desktop testing we’re also proposing changes to.

But beyond that, there’s also this - zezere was deprecated and there’s a replacement. We’ve killed the openQA zezere tests but haven’t written tests for the replacement yet. I’m hoping the WG is going to do that, but even if they do, it needs review and testing which will take some of my/@lruzicka 's time. And we have to write a wiki test case(s) and update the matrix, so we can track the results, and hook up the reporting for it.

And of course any time automated tests fail we need to investigate the failure and file a bug and follow up on it; that happens fairly regularly (often related to osbuild, since IoT is the only thing building with osbuild in Fedora). For release we have to check in with IoT team and make sure an IoT compose for release is nominated and ready to go.

7

How do we take you and @lruzicka out of this as bottlenecks so that those interested in working on the quality of IoT can do some/most of this work?

In a hindsight is 20/20 sort of way, I guess I feel Edition-ing was probably done a little backwards, if editions didn’t come with QA helpers attached to avoid exactly this problem of having QA resources out of balance with workloads created by having multiple Editions.

Given the state of things, what we need a way for Spins, to self-certify that they meet a bar for a specific release, and that needs to including active QA help. We’d need a process that allowed for that sort of rotating in of QA help though.

Worst case, we have no editions and everything is a spin. That would make for a very interesting release day.

1 Like
8

This may be slightly off topic, but if I’m not mistaken, most Fedora CoreOS platforms have recently switched to osbuild by default for building disk images.

9

Understood.

Right, and the Fedora ARM Minimal image.

As for the last bit, I think that’s mostly because Fedora IoT runs a separate Pungi instance to do composes with as opposed to being part of the main compose. Would it save some time from QA if it became part of the main compose instead?

10

The Fedora CoreOS people do a lot more support/work on it by themselves, so while they use the same build system(s) (at least partially) as IoT the experience for dealing with any issues is probably quite different from QA’s perspective :slight_smile:

11

yeah, CoreOS is basically invisible to us, except when we compare notes on failures. There is a very good integrated build pipeline for CoreOS which we more or less just don’t worry about.

Would it save some time from QA if it became part of the main compose instead?

Yeah, to some degree. Of course, Konflux is a wildcard for that whole area.

12

I am aware that FCOS handles most (if not all) testing on their own.

Since the biggest issue with testing Fedora IoT is the hardware matrix, I would like to participate, but unfortunately I don’t have any of the devices listed there. However, I have a Raspberry Pi 4B 8GB and although it is not included in the hardware matrix, I can test Fedora IoT on it if it would be useful.

13

Since I believe there’s also a proposal to reduce the ARM hardware Matrix I’d hold off on this a bit until it is clear what will stay required (maybe?).

As for an RPi4, that’s one of the most popular platforms and generally the one everyone has. The problem(s) are usually the RPi3 and/or RockChip devices and/or Nvidia Orin/Jetson that not everyone has.

14

Yeah, at least to me it’s a bit strange that the Fedora IoT hardware test matrix doesn’t contain a Raspberry Pi 4.

15

I think it’s basically just that nobody has updated it since it was written. For the last release we actually tested on a Pi 4 instead of Pi 3.

1 Like
16

Some unification of aarch64 supported platforms and testing is probably useful and can then be scoped to the most popular devices only, probably that alone drops a whole bunch of work :slight_smile:

1 Like
17

Right, which (if I recall) led to the partition tables being incompatible with the RPi3.

Separate from this topic; it would be really beneficial if there’s enough support to drop the RPi3 since it’s so out-of-spec compared to many of the more modern SBCs.

1 Like
18

Well, each edition brings some plates on the table and if not enough people sit around it, the ones who do sit there cannot eat all the food. Sometimes, there are overlaps, like some of the editions will have fries in it, so you do not have to eat all of it to know if they taste good.

Some time ago we would have people dedicated to a specific edition who would enjoy their share of the meal. For IoT, the person would be Geoff Marr and he was responsible to know if IoT is in a good shape. If we needed to know, we would have asked Geoff. However, Geoff is not the only one who decided to leave for greener grass - we have lost other dedicated people, too. The ARM guy, the community guy, the DevOps guy.

Now, we have two options. Either we only take a couple of bites from each of the plates, or we try to convince the waiter not to bring anything else on the table. By this, and other proposals, we try to limit the amount of food, so what we have could be eaten and digested properly. As you know, overeating might be great once here and there, you burp the aftertaste away, but doing it regularly leads to terrible consequences.

We simply need more people who commit to come to the table and help us eat. For openQA specifically, @adamwill and I could certainly help with the automation, e.g. write the tests, especially when it seems that we now have enough computation power to run them, but ideally the IoT community would say “what should be tested”, “how it should be tested”, they would debug the failures and report bugs to IoT developers (or to themselves). And they would test what is not automated and report the results to the test matrices.

Regarding the hardware, the community support is even more critical, especially when different boards might need a different approach to make Fedora running on them. You need to be a real “fan of the boards” to follow through. I confess not being one of them.

2 Likes
19

While I am seeing proposals for how to keep Fedora IoT as an edition, I think the fact that the edition has not been a notable part of the Fedora community is not a good sign. If things are in a bad spot today, I don’t know how long it would take for us to revitalize the Fedora IoT community enough to cover the responsibilities that the Quality Team would like to shed.

I think we can make Fedora IoT a spin while we work on building support back up. If there is interest then maybe after a sustained period it can be considered an edition again. If there isn’t interest then it is what it is. I don’t think we need to force Fedora IoT to be a thing we we’re not being asked for it and there aren’t contributors who are showing up to make it happen.

Also this doesn’t mean that Fedora IoT goes away, so I don’t think it would be a big change for users currently.

What I would say from an announcement perspective is that we couch the news with the opportunity for contributors to come to Fedora IoT and make it stronger. That takes pressure off us as a project and gives something actionable to those who care about the variant. If we can have like a starting point for new contributions and a the key problems to tackle ready for anyone who shows up, that would be better.

3 Likes
20

Could you provide examples of the kind of maintenance that Fedora QE has been doing?

The IoT team handles the release process and does most of the manual testing- many times all testing.

We’ve definitely had some help from QE, most notably Adam, but I don’t see a significant amount of time that would warrant the demotion.

2 Likes