F42 Change Proposal: Distributing Kickstart Files as OCI Artifacts (Self-Contained)

Project Discussion Change Proposals
,
1

Distributing Kickstart Files as OCI Artifacts

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Wiki
Announced

:link: Summary

Fedora distributed as bootable container ships via OCI registry. Installation is typically done by conversion into a VM image or ISO installer via osbuild (image builder), however, booting from network is a useful workflow for bare-metal fleet deployments. Required files to perform such installation are not available in the OCI repository that could be fetched from registry in a similar manner as the bootable container.

As of today, files are only available in the Fedora RPM repository and the installation workflow would be cumbersome to find appropriate RPM repo version and extract needed files instead of fetching all the needed assets from the registry only. The change introduces a new OCI repository with the files in question for each Fedora stable version.

The change is complementary to the current distribution of kickstart, we are not proposing to stop distributing these files in dnf repositories.

:link: Owner

:link: Current status

  • Targeted release: Fedora Linux 42
  • Last updated: 2024-09-11
  • [Announced]
  • [ Discussion thread]
  • FESCo issue:
  • Tracker bug:
  • Release notes tracker:

:link: Detailed Description

Fedora bootable container is shipped via OCI registries without any supplementary files for automated kickstart installations. The files needed for this workflow are typically: bootloader, anaconda kernel, initramdisk and anaconda main image. These files can be found in regular Fedora RPM repository, for example in case of x86_64 architecture:

Some files are distributed unsigned in the images/ directory, others are signed and need to be extracted from RPM packages. A complete ISO ā€œnetbootā€ image is also available for network installations, the image can be customized using mkksiso tool found in Fedora.

The main goal of this change is to start publishing the mentioned files as OCI artifacts for each Fedora version and architecture. Buildah/Podman will be used for creating such manifest and pushing it to OCI registry and the process will be integrated into current or upcoming (Konflux) release processes.

There is currently no support for downloading OCI artifacts with podman but the feature is currently being discussed and worked on upstream. However, Fedora contains golang-oras tool which understands the OCI artifact format. This tool can already be used by Fedora users to consume the content:

$ oras pull quay.io/pulp/fedora-kickstart-artifacts:40-amd64
Downloading 8ea1dd040e97 initrd.img
Downloading 80c3fe2ae106 boot.iso
Downloading a3b7052d7b2f grubx64.efi
Downloaded  a3b7052d7b2f grubx64.efi
Downloading fff4b2feeef3 pxelinux.0
Downloaded  fff4b2feeef3 pxelinux.0
Downloading 4773d74d87c2 shimx64.efi
Downloaded  4773d74d87c2 shimx64.efi
Downloading 09cf5df01619 vmlinuz
Downloaded  80c3fe2ae106 boot.iso
Downloaded  09cf5df01619 vmlinuz
Downloaded  8ea1dd040e97 initrd.img
Restored    80c3fe2ae106 install.img
Pulled quay.io/pulp/fedora-kickstart-artifacts:40-amd64
Digest: sha256:0306e10fd556e12ce8c3674150bceb88c0917b74b63c37eecc17070b3b30003b

Alternatively, the content can be downloaded via skopeo tool with some scripting involving file renaming.

The proposed repository for the content is: quay.io/fedora/kickstart-artifacts and tag convention will be N where N is Fedora version with manifest index for all supported architectures pointing to tags in the form of N-arch. Only stable and N-1 Fedora versions will be kept for storage reasons and old artifacts will be regularly removed and garbage collected. For more info, read manifest specification.

Files are currently being published at a temporary space: quay.io/pulp/fedora-kickstart-artifacts and can be consumed from there. The pipeline currently lives on Fedoraā€™s gitlab.

:link: Benefit to Fedora

The change solves the situation for Fedora bootable containers users who currently need to find matching Fedora RPM repositories and use various tools like curl or rpm2cpio and cpio to download required files. This will significantly simplify provisioning workflows of Fedora systems en-masse via automation tools like Ansible or Foreman. All files will be also signed by Fedora GPG keys for increased security.

Users of regular (RPM) Fedora spin will benefit as well since bare-metal provisioning workflows, scripts or tools can be further simplified. Additionally, many provisioning systems (Beaker, Foreman) use one shim/grub for installing all OS versions which does not work reliably when SecureBoot is turned on. Published files can be easily downloaded for each OS version.

The newly published content is planned to be integrated with other open source projects: Foreman, Pulp and Ansible. This is out of scope for this change.

:link: Scope

  • Proposal owners: prepare CI/CD pipeline for fully automated build and push of kickstart artifacts, integrate the published repositories with related open-source project workflows Foreman and Pulp

  • Release engineering: create new repository in fedora namespace #12152 and assistance with integrating the new pipeline into the Fedora workflow

:link: Documentation

The newly created repository will be features in documentation of several upstream projects that will make use of it:

  • osbuild
  • foreman
  • pulp

:link: Release Notes

TBD

Last edited by @lzap 2024-09-12T17:54:12Z

Last edited by @lzap 2024-09-12T17:54:12Z

3 Likes
2

How do you feel about the proposal as written?

  • Strongly in favor
  • In favor, with reservations
  • Neutral
  • Opposed, but could be convinced
  • Strongly opposed
0 voters

If you are in favor but have reservations, or are opposed but something could change your mind, please explain in a reply.

We want everyone to be heard, but many posts repeating the same thing actually makes that harder. If you have something new to say, please say it. If, instead, you find someone has already covered what youā€™d like to express, please simply give that post a :heart: instead of reiterating. You can even do this by email, by replying with the heart emoji or just ā€œ+1ā€. This will make long topics easier to follow.

Please note that this is an advisory ā€œstraw pollā€ meant to gauge sentiment. It isnā€™t a vote or a scientific survey. See About the Change Proposals category for more about the Change Process and moderation policy.

1 Like
3

Added this statement:

The change is complementary to the current distribution of kickstart, we are not proposing to stop distributing these files in dnf repositories.

@amoloney thanks for taking this change, I replied to the announcement e-mail into announcement which I realized after I hit the sent button, feel free to ignore that email. Wrong place sorry.

3 Likes
4

These are OCI artifacts, right? Not containers with the files in them?

Are there plans for anaconda to be able to access these directly? (Like the current generic http support?)

1 Like
5

Yes, only the files.

No plans, in order to boot from network files must be exposed via either TFTP or HTTP(S) or manipulated manually for S/390. Only stage2 image could possibly be fetched directly, but since most of the distributed files need to be published this way I do not see huge benefit in there.

What I plan to do tho, and this is out of scope of the proposal itself, is to create a utility for Fedora for easy management of these files. I am thinking a CLI and/or an Ansible module that will help with downloading and setting up TFTP/HTTP for automated installations.

2 Likes
6

When performing Fedora installations (primarily CoreOS) on bare metal machines, I use a TFTP server in a container. Perhaps creating an official Fedota TFTP server container image, or at least documenting how to create one, would also be helpful.

2 Likes
7

Thanks for this, Iā€™m definitely in favor!

A good thing to teach how to consume this would be virt-install right? Maybe worth calling out and filing as an issue.

Letā€™s please cross-link the source and binary. Often when I see a container somewhere I want to know ā€œwhereā€™s the sourceā€. The current repo is https://gitlab.com/fedora/bootc/artifacts/kickstart-artifacts (right?). This also reveals that the artifact is missing some expected annotation keys such as org.opencontainers.image.source.

Thatā€™s the kind of stuff weā€™d work through and fix as part of productizing the buildsystem more (hopefully with konflux).

2 Likes
8

ā€¦snipā€¦

Some files are distributed unsigned in the images/ directory, others are signed and need to be extracted from RPM packages. A complete ISO ā€œnetbootā€ image is also available for network installations, the image can be customized using mkksiso tool found in Fedora.

So, one complexity hereā€¦ thereā€™s actually two boot.isoā€™s that get
produced. The ā€œEverythingā€ one as you note, but also the Server
deliverables have another one. Itā€™s just like the Everything one, but
itā€™s using different defaults (xfs instead of brtfs, and a few other
minor things).

So, I am not sure how to handle that. Perhaps there could be a ā€˜-serverā€™
one for those people who want to use the server defaults?
Or we could just stick to everything for this.

Everything else looks great to me, and is pretty cool that it can be
done. :slight_smile:

2 Likes
9

Are these just the Anaconda default kickstart values which can be overwritten by normal kickstart statements? Since the goal of all of this are advanced or fully automated installations it could be non-issue. I am assuming these will be used not by someone who just started with Fedora, but more likely an advanced user who knows those differences.

The boot.iso is actually not required for network installations which is the main goal for the change. I added it because there is this small tool named mkksiso which can be used to embed kickstart into an ISO so with this, one could write letā€™s say a simple Ansible doing some automated thing (generating such ISO, uploading it to VMWare or bare metal via Redfish etc).

Proposed solutions:

  • Remove the ISO from this change completely.
  • Add both Server and Everything which is technically possible it is just matter or a small update with an extra storage cost.
  • Select just one of the two and go with that.

I slightly lean towards shipping both of them, because quite frankly this change is all about publishing those files and waiting what our community starts doing with them. More options could generate more interesting workflows, tools and stories.

Thank you all for looking into this change, appreciated.

1 Like
10

Added to the todo, good idea.

Another one which is in the queue is to actually link the main Fedora repo and kickstart repo. While technically it would be possible to push the KS files into the main (bootc) Fedora repo, I think it is better to keep them separate. But an annotation in the main repo that would link kickstart files could be useful - tools could actually automatically detect and find the correct one.

I did not include this in the initial change because I do not have too many details on how the pipeline will be done. But if KS files can be generated together with bootc container (as a followup pipeline) automatically, I do not see a reason why not to do that.

1 Like
11

Proposed solutions:
Remove the ISO from this change completely.
Add both Server and Everything which is technically possible it is just matter or a small > update with an extra storage cost.
Select just one of the two and go with that.

I would be OK with any of these options.
But picking just one boot.iso is my favorite and hereā€™s why. Quite often as devs we ovethink how end user would use a feature, especially when we do not have a very clear vision of the combinations of the workflow path. I would include ā€˜bare minimumā€™ and just wait on feedback, this way weā€™ll know whatā€™s missing and can add things later, even retroactively, rather then shipping ā€˜everything possibleā€™ without knowing whether things are being used.

12

Speaking as the person who created the Server netinstall: I recommend just using the Everything netinstall for this.

The only difference in the Server netinstall here is the set of defaults, most notably the filesystem you would get if you chose the autopart directive. But if the plan here is to use kickstart to manually configure everything anyway, the choice of the ISO doesnā€™t matter.

2 Likes
13

I remain unconvinced that this is a good idea or even useful to do. Why wouldnā€™t someone just use a netinst ISO and just point it to a kickstart via http/https to fetch it? Itā€™s an order of magnitude simpler than all of this.

14

I hate to bikeshed, but using ā€˜kickstartā€™ to refer to this feels weird to me. This isnā€™t really about ā€˜kickstartā€™ deployments, itā€™s about PXE-style remote automated deployments, right? To me the ā€˜normalā€™ way to do a kickstart deployment is to boot an ISO and pass inst.ks. Doing a PXE install is a different thing again.

So itā€™s weird to me to keep reading ā€œkickstartā€ and ā€œkickstart fileā€ when reading the description of this Change, it kinda acts as an obstacle to understanding what itā€™s really about. (Also, to me, a ā€œkickstart fileā€ is literally a foobar.ks file).

15

The ISO file is not the main reason why we are doing this. It is all the other files, we believe shipping ISO alongside those is useful. It is being distributed in RPM kickstart repos too after all.

For bootable containers installation, we can take it a step further where the content is actually either distributed in the bootable container repository itself, or there is some annotation that links the two together. This means users have everything at one place. We are aiming at automated workflows (scripts, fleet management software), of course if you just want to do one installation then manually downloading a netboot ISO from the official web page is perhaps the way to go.

We are open to simply remove ISO from being distributed if that is a problem for anyone. We were told that there is enough space on quay.io for us as long as we will be deleting old content and performing garbage collection.

Yes, we really tried to find a good term and landed on this one. Thing is, this is mostly useful for network installations and most people still use PXE. This term, however, is only applicable for Intel platforms, there are other network boot protocols like BOOTC for POWER or even EFI-HTTP for Intel which is getting more adoption lately. So PXE was scratched during our elimitations. Kickstart just felt more neutral and inclusive.

After a lot of thinking and selecting, we have landed on this. ā€œBootableā€ repositories are known as ā€œkickstart repositoriesā€, at least this is how we call them in the Satellite group in Red Hat. Other names we were considering:

  • fedora-boot
  • fedora-installer
  • fedora-boot-installer
  • fedora-bootable-installer
  • fedora-instboot
  • fedora-bootinst
  • fedora-network-boot
  • fedora-netboot
  • fedora-netboot-iso
1 Like
16

@adamwill we specifically went with the plural of the noun so not kickstart file but kickstart fileS, so kickstart-artifacts in the OCI registry. As noted, naming is hard, in case you have a suggestion please bring it up!

3 Likes
17

This change proposal has now been submitted to FESCo with ticket #3274 for voting.

To find out more, please visit our Changes Policy documentation.

1 Like
18

The problem with this is that anything you publish becomes permanent API. You cannot delete anything because it will break users.

I am really concerned about this idea of stuffing all kinds of random things into OCI registries because aside from the performance implications, there are serious storage and availability issues too.

19

What is exactly the concern about?
On the contrary people nowadays take advantage of oci transport as delivery mechanism and they use registry as a storage. This approach helps build the infrastructure in a way that one needs only registry to talk to.

I would also argue that specifically this proposal ā€˜stuffs random thingsā€™. As described, each file that was selected and included into the manifest as OCI artifact has its value and purpose to be there and taken advantage of.

Can you provide an example? With rolling out zstd chunked things will only go better from now on.

Have not heard any storage concerns from folks running quay.io or PMs and looking at Quayā€™s SLA for pulls and pushes availability wonā€™t be an issue too.

20

Itā€™s not just about Fedoraā€™s usage of Quay, itā€™s also about how other people can use it too. Container registries giving storage and CDN access away for free is not going to continue forever and reaping stuff from the the registry is effectively not viable because it will break peopleā€™s setups. We already know this from Docker Hub, and I foresee it happening for Quay and every other public registry.

Thus weā€™re put in a situation where it grows forever and every revision takes up a huge chunk of space because every push has to be saved forever.

I have other concerns about performance, efficiency, and maintainability around this, but this is my biggest problem with it.