Hi Paul, good to see you.
So, during F41 and earlier, we had Geoff (coremodule) focusing on ARM and IoT primarily. Here’s F41 RC IoT matrix with many tests executed by him. After he left, Lukas (lbrabec) took over during F42. Here’s F42 RC IoT matrix with some of his results.
I wasn’t very successful when looking for some Bugzilla examples, because I can’t search easily for “IoT-related” bugs, Geoff is not around, and he also served as the secretary in blocker bug meetings very often, so he touched lots and lots of bugs which are not relevant here. Here’s one bug I found, albeit quite old. Lukas was standing in for him just for one release, doesn’t remember anything IoT-related.
I would love if @adamwill could comment on the amount of work needed to implement automation, keep it running, and resolve issues. He’s on a long vacation now, unfortunately.
Honestly, I don’t think IoT is particularly buggy. We definitely struggle much more with desktops on ARMs, which is why we have a separate proposal for that. The CLI stuff mostly works. It’s just additional test cases we need to care about, bugs to read through and vote on, fixes to test if no-one else does it first. The small boards handling is time-consuming even for CLI testing, unless one has a dedicated setup without constant peripheral reconnecting, etc.
The general problem is just too many products and test cases we’re responsible for. If test cases are not done before a release, or blocker bugs are not discussed and determined in time, or fixes for blocker bugs are not verified in time, we’re the primary team to come to. Fedora kept on increasing its portfolio in the past years, and we feel we can’t fulfill our responsibilities anymore, especially in our current state. So we proposed reductions, and IoT seems like a natural choice of these discussions. It’s not personal, we understand IoT folks won’t be happy about it, and we’re sorry. We can of course also discuss different arrangements, e.g. whether IoT folks can handle everything on their own, etc. But if things stay as they are, we know we’ll not be able to keep up. Personally, I have doubts whether we can actually do just the regular set of x86_64 desktop testing, disregarding everything else. As described, we’ll need to rely on individual SIGs much more this time. There’s also a lot of stuff happening internally (have I mentioned we’re hiring?), so things are… hectic.
I would love to hear your thoughts on this. And also on the other proposal regarding limiting release-blocking ARM and IoT hardware, if you can. Thanks!