F42 Change Proposal: Koji uses Red Hat Image Builder locally (system-wide)

Project Discussion Change Proposals
,
1

Koji uses Red Hat Image Builder locally

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Wiki
Announced

:link: Summary

Switch the Red Hat Image Builder-built images in Koji to not build through a service, but locally.

:link: Owner

:link: Detailed Description

Several Fedora images are being built by the Red Hat Image Builder service (Fedora IoT, Fedora Minimal). Image Builder acts as a content generator in Koji. Images are built external to Fedora infrastructure as requested by Koji and then delivered back.

This has led to of issues over time. Amongst them instability and the inability of Fedora release engineering and infrastructure teams to intervene and the inability to freeze the external infrastructure during a release window.

I plan to replace the current approach in koji-osbuild with a local-only build. Image Builder machinery and definitions will still be used for the mentioned images but the build will run locally on Fedora infrastructure and builders. In the same way that kiwi or livemediacreator-images are built.

My plan is to start with adding these tasks to koji-osbuild, keeping them there for a cycle and if they are stable to upstream them into koji directly; deprecating koji-osbuild at that point in the future.

image-builder-cli built images can be easily rebuilt locally from their manifests uploaded in koji. The images can be easily built locally and support all end-user customizations available. Work is ongoing in the Image Builder stack to also move the definitions of the distribution to a declarative format.

:link: Feedback

This change proposal does not yet provide a way for the distribution definitions to be owned by, and live inside, Fedora infrastructure. I plan to address this in a later change proposal.

:link: Benefit to Fedora

Control over Image Builder on its own infrastructure, lessen remove the need to reach out to external parties when releases are blocked.

:link: Scope

:link: Proposal Owners

  • Submit the image-builder-cli package for inclusion in Fedora 42.

  • Request a image-builder-build comp group for the build requirements in koji roots.

  • Change the koji-osbuild plugin to provide a new task type that uses image-builder-cli.

  • Changes to pungi to be able to schedule these tasks.

  • Changes to pungi config for Fedora IoT.

  • Changes to pungi config for Fedora Minimal.

:link: Contingency Plan

Independent of where on the above tasks I get stuck images can continue to be built with the current setup.

:link: Release Notes

Last edited by @amoloney 2025-01-10T19:17:38Z

Last edited by @amoloney 2025-01-10T19:17:38Z

1 Like
2

How do you feel about the proposal as written?

  • Strongly in favor
  • In favor, with reservations
  • Neutral
  • Opposed, but could be convinced
  • Strongly opposed
0 voters

If you are in favor but have reservations, or are opposed but something could change your mind, please explain in a reply.

We want everyone to be heard, but many posts repeating the same thing actually makes that harder. If you have something new to say, please say it. If, instead, you find someone has already covered what you’d like to express, please simply give that post a :heart: instead of reiterating. You can even do this by email, by replying with the heart emoji or just “+1”. This will make long topics easier to follow.

Please note that this is an advisory “straw poll” meant to gauge sentiment. It isn’t a vote or a scientific survey. See About the Change Proposals category for more about the Change Process and moderation policy.

3

So, I think this is great overall.

I would ask a question I think other people might ask: Why not just move to using kiwi for these things? Might be good to add that into the change as I think people will ask it.

1 Like
4

I think Kiwi is capable of building both images though the Fedora IoT installer maybe not since it needs ostree/bootc bits and bobs.

There’s no good reason to prefer Image Builder over Kiwi other than that. It’s up to preference by variant maintainers.

5

From what I heard, Fedora IoT and Minimal maintainers are happy with Image Builder and don’t want to migrate to kiwi. The depedency on the image builder service was identified by them as a pain point, so we are trying to solve this issue for them.

If someone is willing to convince and help these folks to migrate to kiwi, that’s absolutely fine, we are just trying to improve what is there already. :slight_smile:

In the case of IoT, I would be slighly concerned about kiwi not having a great support for ostree yet. It might be actually better to stick to Image Builder, since CoreOS is also migrating to it.

disclaimer: I’m working on Image Builder, so I’m not bias-free. :slight_smile:

6

Nitpick but being accurate matters: Fedora CoreOS is currently migrating to using osbuild to build disk images and isos, not Image Builder (at least yet).

7

Kiwi can produce container images, but it cannot produce OSTrees. The effort to add rpm-ostree support has stalled out. In theory, kiwi should be able to produce images that bootc can consume (if you can use buildah or umoci to make images that bootc can consume, then kiwi can make them). However, I don’t know what the “rules” are for a correctly produced image that bootc can use. Without that information, it’s not possible to teach kiwi how to do it.

kiwi does already have a way to take container images as input and turn them into any other kind of image with the stackbuild plugin that just entered Fedora this morning, but it doesn’t force an immutable paradigm when doing it.

8

The question is about the bits where you need to mirror an ostree repository and/or container storage on installation media so that it can be consumed by Anaconda for installers not about the bits where an ostree commit or container image is created :slight_smile:

E.g. when building Fedora IoT installer we take the ostree commit and store it somewhere on the installation media after which we point at it in the kickstart(s); same goes for creating an ISO we take either a container image from either remote or local storage on the build host and copy it into the ISO so it can be used for (offline) installation.

While osbuild (and its higher layers in Image Builder) can build and create both ostree commits and container images it isn’t used by Fedora variants.

9

We can push images created in Koji to registries just fine, but I don’t think there’s network access to registries in the image build environment, so kiwi cannot pull down images and store them somewhere in the image for later use (kiwi does have the capability to do this, it just doesn’t work in the koji environment right now).

10

This change proposal has now been submitted to FESCo with ticket #3341 for voting.

To find out more, please visit our Changes Policy documentation.