Make a Fedora Respins main method of get ISO images of Fedora

It allow to get fresh images of Fedora, and get a recent kernels and security/bugfix updates for users. It also increase of install time economy because user will install only small count of updates.

Iā€™m simply download Fedora Respins at here. But Iā€™m want WS Team to provide Respins (ISOs with all updates at image creation time) as main images in Fedora website and make this images officially.

A perfectly reasonable request, however, while respins get a bit of
testing, they do not get the full release testing that an actual release
gets. They are also made on a seperate infrastructure by the
respins-sig.

@adamwill may have some thoughts/more info here.

4 Likes

I think the question isnā€™t about making the existing respins ā€œmore officialā€, but about doing ā€œmore official respinsā€ - i.e. have us point the full firepower of QA and releng at doing periodic respins.

To which the answer is the same as it has always been, more or less: we could do that, but it would eat a lot of time and resources away from other work. We use the slower periods in the release cycle to build new stuff, fix longstanding problems, pay off tech debt, etc. For instance, right now (because weā€™re in the long ā€˜quiet periodā€™ after one release happens and before the next branches) Iā€™m overhauling a bunch of stuff in openQA scheduling and tests; if I had to be worried about constantly testing/qualifying regular respins, I wouldnā€™t necessarily have time to do that.

We could do a thing where we just automate the whole process of producing/testing/publishing respins, but then those would have to still be ā€˜less officialā€™ and probably not the thing we point you to first on the download page, because no human would be in the loop to check for problems that automation misses, which is always a possibility.

3 Likes

Iā€™m not want a custom ISO. Iā€™m want a official ISO with updated packages only. How Debian does it.

FYI Iā€™ve deleted some spam from this thread (including the AI-generated not-really-relevant-let-alone-correct post). Iā€™ll delete this message later too so itā€™s not distracting from the topic.

We do have these ā€” https://dl.fedoraproject.org/pub/alt/live-respins/ ā€” but theyā€™re unofficial.

@adamwill What would it take the do these, say, weekly for the current release and run through openqa? That still wouldnā€™t be official, but maybe a level of more confidence?

Iā€™m know. But Iā€™m want to link to Respins download link at least in ā€œalternative downloadsā€ with marking ā€œunofficialā€. Itā€™s make Respins is more popular than previously.

From the QA perspective, running it through openQA is free - it happens automatically (assuming the composes are generated by Pungi as usual). Iā€™d imagine itā€™d be a bit more immediate work for releng and websites folks. For QA the trickier bit would be that weā€™d kinda have to care about the results. Iā€™m going to see them, and if there are failures, then what? Do we follow the normal release process with blockers and reviews and all of that jazz? Do we just wing it? Do we just put the images up with a link to the results and say ā€˜this is the image, these are the tests results, do as you see fitā€™? Do we only publish images that pass all the tests, or a chosen subset?

edit: note, if we only generate live images, then a lot of openQA tests wouldnā€™t be run (unless I do rather a lot of work and reinvent something I just uninvented to make the scheduling code simpler) - most of the tests of various installer capabilities run on the Server DVD, not on live images. The tests for live images only do a basic install test, they are more focused on desktop functionality post-install.

1 Like

Well, as is, those problems hit users and end up in Ask Fedora. Thatā€™s not really better, more like the kind of better where I joke that we should send @kparal on vacation on release week so he stops finding bugs.

Our process (thank you, Adam and Kamil and all other heroes of QA) demonstrably increases quality over time, but still, after release day itā€™s a gamble as to whether applying updates will introduce exciting new problems along with fixes. If itā€™s relatively easy to expand the automation already in place to at least give us more information, that seems like a step in the right direction.

But I also wouldnā€™t want to put more load on the QA team and take away that between-releases (somewhat) calm. I definitely donā€™t think we should do anything like the release blocker process.

If we can do so reasonably, I think we should publish any images that pass the same set of tests the release-day images do. As a first pass, this would be more tested than the current respins, and also could be somewhere to look when people report issues.

As a next step, when there is a test failure, weā€™d automatically back out the updates that caused it. Not counting flakes with the test system itself, thatā€™s either:

  • a regression ā€” which should be fixed by the packager ā€” or
  • a big enough change that the test needs updating ā€” in which case the packager should help update the test, or given the updates policy, maybe the update should wait for the next release (and the test updated in Rawhide).

Ideally we would block just the offending update, but Iā€™m kind of leaning towards: itā€™s better than nothing to block the whole batch if thatā€™s where weā€™re at. Then packagers pushing those updates should (he said, full of naĆÆtivity and hope) work together to identify the specific problem.

Uh. Are you aware we do a lot of that already? We already automatically test all critical path updates, and they are gated on failures. If the tests fail the update canā€™t go stable.

We soft gate Rawhide these days, even - if a Rawhide update breaks stuff I either fix it before the next compose or get nirik to untag it. We just donā€™t push stuff that breaks tests any more, practically speaking.

The difference with respins is that updates can break the compose and install process. We do actually test this, but only minimally (openQA builds and does a basic install test with an Everything netinst, a KDE live and a GNOME live, but it doesnā€™t build anything else and for updates it doesnā€™t run the more extensive installer test suite we run on composes).

I donā€™t think itā€™s really right to say that updates are ā€œa gambleā€ these days, at least for core operations.

Also of note, we do actually test the current respins in openQA. Here are the test results for the last respin, for instance. The agreement between me and the respin maintainer is that the tests get run and itā€™s up to him what to do with the results; I do look at them and briefly mention what the cause of any failure is to him, if I get time to figure it out.

additional: as you can see there the compose test suite for a Workstation live image is 38 tests. For an update we run 14 Workstation tests (the test that builds a Workstation live, plus 13 tests from the 38 that we run on composes).

Oh, yes, definitely! Thatā€™s why Iā€™m hoping itā€™s not such a big stretch.

That is probably too harsh. Iā€™ve been spending a lot of time reading whatā€™s going wrong for people, which probably gives me a negative bias.

Oh, cool ā€” that I didnā€™t know.

I knew this for rawhide, because we talked about it. Although we really need to scale up the process ā€” it really needs to be on the person who pushed the update, and failing that a wider group. On general principle, let alone the siren song of the llama farm.

We talked about this too ā€” we probably need a better definition of ā€œcritical pathā€, and some broader layers. And, if some package is supposed to be outside of that path but yet breaks a testā€¦

Using the netinst will install the latest packages.

Just that I wish the options given in netinst will mirror that of the different editions / spins 100% .

Anyway. Fundamentally, if you want to do this, we can probably do it. Itā€™s just a question of defining the parameters and seeing if websites and releng are OK with it.

1 Like

Maybe this is wishful thinking, but my guess is that most of the things that might be caught in this way will hit us later one way or another, and if we can spread out the load beyond QA (and beyond you and Kevin!) thatā€™ll be net positive.

And we can finally make Linus Torvalds happy! (Iā€™d link to his comments, but I think they were on Google+ā€¦)

The category of things Iā€™m worried about is ā€˜updates that break something in the installer which we donā€™t test in the update testsā€™. Itā€™s not unmanageable, itā€™s just a thing that is likely to happen and which we would have to deal with.

1 Like

(lets email reply to two posts on discourse, will it work? lets see!)

Maybe this is wishful thinking, but my guess is that most of the things that
might be caught in this way will hit us later one way or another,
and if we can spread out the load beyond QA (and beyond you and Kevin!) thatā€™ll be net positive.

I fear that wouldnā€™t happen. I mean, it would be nice to spread the load
of rawhide and updates troubleshooting, but thats not happened yet?

And we can finally make Linus Torvalds happy! (Iā€™d link to his comments, but I think they were on Google+ā€¦)

Ha. Ahā€¦ memories!

The category of things Iā€™m worried about is
ā€˜updates that break something in the installer which we donā€™t test in the update testsā€™.
Itā€™s not unmanageable, itā€™s just a thing that is likely to happen and which we would have to deal with.

Yeah, so lets step back here (and sorry, this might be long):

What does this gain us? I see two things:

  1. Users donā€™t have to download an image and then download and apply a
    bunch of images. (ie, we are saving some download BW here).

  2. We may produce a image that has a newer kernel/bootloader/installer
    that enables some new hardare or fixes some common bug.

(did I miss any?)

The first one is nice and all, but once you install, you are still going
to be downloading a bunch of updates moving forward right?

The second one also could be nice, but since we release every 6 months
and the older release is still available, in practice I am not sure how
often that sort of thing happens.

Now, what would it take to do this:

  1. The respins sig is awesome and does great work, but they make images
    on a aws instance we provide on whatever schedule they like. If we were
    going to advertize these as our official images, I would really want
    them to be made in koji and have logs and tracking and such. So, that
    would take more cpu, more disk, etc.

  2. Process. I donā€™t like process for its own sake, but thereā€™s a reason
    we have process for releases. We would probibly need something for this.
    To answer questions like: Bug/security update X just appeared, should we
    redo this upcoming release? Or skip it? Bug XYZ isnā€™t going to be fixed
    quickly, do we cancel this release or delay it? Someone has to look for
    issues and propose them (I guess we could reuse the blocker bug setup,
    but would need to adjust for that). We would need to communicate and
    coordinate all the various groups.

  3. Maintainers of boot chain stuff would be ā€˜on the hookā€™ more.
    Right now anaconda/lorax/grub2 folks know they need to be available for
    release blocking bugs running up to releases. This could mean that they
    have to make sure and have cycles to handle those anytime.

Anyhow, I agree we could do it, but I think it would take a fair bit of
work and I am not sure the gain is enough. :slight_smile:

Iā€™m of course only speaking for myself hereā€¦ if fesco/council decided
this was something that was important to do, we could try and do it.