Image Builder is a service that is part of the RHEL product pipeline and is on its way to becoming a part of the Fedora image build pipeline. In addition to building standard composes by integrating with Koji, it offers end-users a way of building customized operating system images for various architectures (x86, ARM, IBM) and targets (Bare metal, Private cloud, Public clouds). The way it exposes configurations and customizations is geared towards producing images that boot and are useful. Image Builder is available through console.redhat.com/insights/image-builder for anyone with a RHEL subscription.
While the service is currently a Red Hat offering, the team has been working to make it open source where it can. You can read our self-assessment here.
We want to deploy Image Builder (see below for details) as the first service of what we hope will eventually become console.fedoraproject.org. Image Builder is a great candidate, because itās fairly standalone (doesnāt depend on a lot of Red Hat Insights or other Red Hat specific components/services) and provides value without any controversial features that Insights comes with (e.g. the usage of insights-clients, the concept of subscribing, telemetry, etc)
In the first iteration, thereād only be a public API with basic authentication. It would look like this: API Catalog and allow select āpreviewā users to build customized Fedora images via this API.
Phase 2: An authenticated backend
In the second phase, we would try to tackle the authentication, specifically integrating with the Fedora Account System via OIDC.
Phase 3: A service with a frontend
In the third phase we would deploy the frontend for the service, so logged-in users could use it to see their previous builds etc.
Who is providing what
The Image Builder team will provide:
An Image Builder service and a Content Sources service (so end users can build images with custom content from e.g. COPR)
All operational support (best-effort dev team support during EMEA office hours, basic best-effort AppSRE support for the rest)
The service would be hosted on Red Hat clusters
Deployments, updates, etc would go through Red Hat infrastructure
Fedora will provide:
DNS
OIDC
How would users get support
We currently have three avenues:
GitHub issues upstream or issues in Red Hatās Jira
I think this is really cool. It provides a useful service for the Fedora community, and helps Red Hat service offerings like this act as active open source.
I am looking forward to use Image Builder for Fedora use cases. And Open Service is the most exciting concept for me.
I am a bit worried about the naming here and whether it sets the expectations right.
I would expect that services run under *.fedoraproject.org name are not just being deployed for Fedora or using Fedora, but they are also governed by Fedora. So how is it supposed to work here? Who is going to own the decision making on what merge and not merge in the service configuration?
In terms of naming weāre open to your suggestions and we definitely want to set the right expectations. You certainly have a better view of what the community might think/expect.
When proposing console.fedoraproject.org, my intention was to aim for the stars and inspire others currently running services on console.redhat.com to follow our lead.
On the topic of governance, I wanted to add that I see two ways for Fedora to influence and take ownership of the service:
The artifacts (i.e. images built) are based on image definitions which we would like to be owned by Fedora. E.g. when we build the official Fedora spins, we would expect that the owners of the spins would own what goes into those images.
The deployment logic currently goes through app-interface, but the deployment metadata/config is open (minus the secrets obviously, see e.g. https://github.com/osbuild/osbuild-composer/blob/main/templates/composer.yml) and the file describing the Fedora service could easily be put under Fedora governance.
This is a bigger topic that weāre thinking about a lot. Thereās a lot of ideas floating around about how to make these image definitions easier to work with for external contributors and maintainers. Currently, itās all included in the repository that Simon linked to and itās pretty much all Go code. We understand that, while this has certain benefits, it isnāt ideal for the situation weāre discussing here and weād like to work with you to find the best way forward.
It can be (maybe at first) a simple matter of bringing people into the project in its current state, making them code (co-)owners of the pkg/distro/fedora/ package. We also have ideas about restructuring the images to have less defined in code and support more in the external configuration, essentially enabling spin definitions to exist as build configurations.
Ultimately, the answer to your question will depend on where we (IB and the Fedora community) land on the above.
(I havenāt really formed my opinion so I am thinking out loud here)
Maybe consider the examples:
koji.fedoraproject.org - this is the most classical Fedora service. There is Koji as upstream project, where Koji developers make decisions what to merge and not to merge. And then there is CPE-owned deployment procedure to rollout changes and configure the Fedora instance of Koji.
But there is another service, like discussion.fedoraproject.org - it is provided to Fedora by a third-party according to a certain agreement. Within that agreement Fedora Project owns the internal configuration, but doesnāt have the merge power for the code. The governance is channeled through you, and the Fedora Council.
As far as I understand currently image-builder service would be less open and accessible than Koji but much more open than Discourse or Matrix.
I think in the end I am less interested in who has the actual practical permission to merge the code in the repo (if we have a merge request workflow where reviews happen collectively and in the open I am not too worried about it). Rather I think the question is - should we have some agreement about more high-level decision making, like enabling/disabling an arch, or managing access to the service or I donāt know, things like limiting the amount of builds per userā¦
Can we get CPE (or, more generally, Fedora Infra team) to be the āownersā of the service the same way how they own Matrix or Discourse now. And then watching after what is going on there and bringing things to attention of Fedora Council or FESCo if needed? Or should we setup a separate group for this?
I think this is a very interesting idea, and well worth persuingā¦
I think itās probibly fine for fedora infrastructure to help route requests to the right place, represent the users/consumers of the service and ask for fixes/changes if needed.
I think itās pretty similar to the matrix/discourse setupsā¦ although in this case it may be less possible for us to āself manageā some parts of the application (although I am not sure of that? can we have the ability to change arches/limits/etc on the fly without involving any of the console team?).
One other thing that might be nice is some kind of reporting? ie, can we look and see how many images were made, what kind or something? That might help adjusting the service and seeing that itās useful for community members.
Will this be accessible just to Fedora Infra/Spin owners, or to specialized groups as well? Iām in Quality and we often have a use case where we need to build an installer image with a certain package from updates-testing or Koji included (and then ask a community member to test it, for example). Currently we need to build it ourselves and then host it somewhere. It would be outstanding if we could log in to Image Builder, perform an installer image build with a particular set of package NVR overrides, and then receive a public URL to download it. Will that be possible?
I donāt know if this is possible now, but I think this should be a very high priority feature for entire range of use cases: Fedora CI, CentOS CI and RHEL CI also need that.
My dream is that for every MR/PR we would get an image built with the updated package and use it in local testing, Testing Farm, OpenQA and/or Zuul pipelines. It would simplify a lot of our pipelines and test scenarios which wonāt need to handle fetching and updating packages during the test run.
For things we actually rely on for the distribution to function, I would prefer us to be closer to Koji than Discourse in terms of services. Especially given the architecture of OSBuild and Composer, not having that ability means we can get in a lot of trouble with IB bugs during release development.
I certainly donāt want to get into the situation we ended up in for Module Build Service, where it was essential to our compose pipeline but no one believed they actually owned it or were responsible for making it work. Weād need some kind of strong agreement of expectations before using it in a release-blocking way.
For things we actually rely on for the distribution to function, I would prefer us to be closer to Koji than Discourse in terms of services.
I chose Discourse as an example for the very end of the spectrum, not as a role model
I agree that with Image Builder we should get much closer to the Koji setup.
When I say I donāt pay a lot of attention to who exactly merges things - thatās because I think that in code review workflows real decision is made before you hit a button to merge. The actual person who hits a merge button implements the decision made through the code review process.
As long as the code review process is open, and the owner of the merge button respects it, they may have it.
Like when you ship a change in Fedora you may not have access to a signing key, yet it is you who decide what to ship. The signing key owned by a certain infra engineer is the implementation detail here. And the power over that key is used to verify not to change the outcome.
I think that I am moving in the direction that we should deploy the staging instance and see how this works in practice, including the expectations from the infra accessibility (access to monitoring, debugging, logging) so that we can then make more explicit requests on what we should and should not do.
I think there may be some confusion here. (Or perhaps I am confused!
This thread is taking about providing imagebuilder for the fedora
community. ie, users, developers, whoever wants to make a custom image
for whatever reasons. Of course there will be limits, X images per Y
time to avoid resource problems/abuse, but otherwise (once itās deployed
in production) it will be available to anyone with a fedora account.
(please correct me if I am mistaken!)
note, you can always have the robot do this for you. openQA already builds live and installer images for many updates. if itās not in the critical path groups that would make it do that, we can easily manually schedule those tests on staging to get images built. I can show you how to do that if you like.
Indeed, thereās a related effort where weāre working with Fesco to make image builder part of the Fedora pipeline.
This proposal is purely for a community offering, where anyone with a Fedora account would be able to log in and build customized images.
