Annotations can apparently be added with podman-manifest-annotate — Podman documentation
Can someone elaborate on the chunk sizes, compared to OSTree? Afaik OSTree updates still use less bandwidth
Note that this change is not directly about the format used for Atomic Desktops, still upcoming, container images.
The support for zstd:chunked images in rpm-ostree/bootc/ostree-rs-ext is not yet capable of handling the chunked part of this format.
2 Likes
okay interesting, thanks!
I don’t think the proposed change makes sense basically because the 99% case for our base images is…well, using them as base images to make a new derived image.
And when you do e.g. FROM fedora and podman build, the compression for the derived image becomes whatever the engine default is (for docker and podman today: gzip)…so in the end we’re not really going to change much. Anyone who wants to opt-in to zstd:chunked today needs to either explicitly configure it or inherit the current global change in f41 to default to zstd:chunked…which is still going to cause a lot of problems until we get bootc support for it.
IMO the best approach here remains Add compression_format = inherit · Issue #5586 · containers/buildah · GitHub
1 Like
@walters Could you copy/paste that comment in F41 Change Proposal: Default podman created images to zstd:chunked (self-contained), which is the main discussion thread for the change, that will be looked at by FESCo? This is the thread that was created before the change was announced.
Do you mean that we should not use zstd:chunked for bootc images in Fedora 41?
1 Like
Should we close this thread then?
As I understood it, the compression usage depends on the client. Does dnf or bootc or rpm-ostree or whatever will be used in the future support zstd?
I understood that neither podman nor docker support it, and that it thus would not be directly usable. But if they would implement support, there would be benefits.
Added atomic-desktops-team
OSTree uses rpm-diff tooling so it only sees diffs in files, while zstd:chunked pulls any files that changed (Completely). So OSTree is more efficient, but harder to manage, since you could have multiple diffs, if you had image1.0 image 1.1, image 1.2 image1.3 you quickly start to grow image diffs exponentially.
Bottom line going forward we want to support zstd:chunked now and potentially rpmdiff functionality in the future.
1 Like
My understanding is that if a user pulls a zstd:chunked image from fedora-minimal and then pulls fedora image, they get take advantage. Similarly any non-layered image that contains the same content as a previously pulled image gets the benefit. In F41 we will switch people’s defaults to pushing zstd:chunked images, which means more an more images will be zstd:chunked by default, and everyone can take advantage.
1 Like
podman, docker and all current container engines support pulling zstd:chunked images. Only docker prior to march 2023 does not.
Bootc currently does not support using them, but this is being worked on. We need to get bootc to support zstd:chunked as quickly as possible, since it will be a main benifitiary of zstd:chunked.
2 Likes
Thanks! This clears things up.
Still, should these (now more) useful messages be moved to the main thread? Dont want to be too quick here, but fesco doesnt read them here.
Yes, that’s true. It feels hard to me to quantify the benefits there…
Yes, I think that’s the much bigger change and it’s a bit confusing to conflate them, really.
My take is at least until the bootc side is ready it’s confusing to try to make this switch, and we should just encourage people to opt in for their app containers today.
1 Like
No one opts in (Almost no one). If we don’t move forward on this, and force others to fix their code, then zstd:chunked will never take over.
Bootc images can stay as gzip for now, but we need to get bootc support ASAP.
1 Like
note on the image creation / publication side: it’s not quite true yet that cloud-image-uploader publishes the images. as of a couple of days ago it is publishing ELN images. everything else is still published by one of a set of bash scripts:
- Tree - releng - Pagure.io (pushes ‘normal’ container images)
- Tree - releng - Pagure.io (converts ostrees into containers and pushes those; we do not currently actually push the natively-built ostree containers that we build in the compose process out to registries)
- Tree - fedora-iot/pungi-iot - Pagure.io (pushes the bootc base images built in the IoT compose)
There might be a fourth one lurking somewhere but I can’t remember. These get called from various nightly compose scripts (the one for Rawhide, the one for Branched, the one for nightly stable release container composes (on f39 and f40 branches), and the one for IoT nightlies (on all branches IoT builds on)). These scripts use skopeo to push individual images and buildah to create and push manifests.
We are currently in the process of replacing all these with cloud-image-uploader, which - for hilarious reasons - builds its own manifests. It uses skopeo to push both images and manifests.
The container images are built by pungi as part of the compose process. “Regular” ones are built by Kiwi, which I think ultimately uses buildah (it also can use something called umoci, apparently, but I’m guessing for Fedora it uses buildah). ostree container images (including the Atomic desktop ones, and bootc ones) are built by a bit of pungi that runs rpm-ostree compose image in a Koji runroot, AFAICT. At a quick glance, neither of these appears to have the ability to customize the compression format used; they would just use whatever format the underlying tool defaulted to in the context in which they were run, as best as I can tell.
So unless I’m missing something…to implement this we would at least need to come up with mechanisms to specify the compression format in pungi config and for pungi to pass that through to the underlying tool in its ostree_container and kiwibuild phases. We would need to work out how to handle the metadata there, so we don’t try and build images with identical subvariant, type and format (which isn’t allowed). We would then need to update either the bash scripts or cloud-image-uploader to push all the images and produce and push manifests in the desired form, depending on whether we’d finished the migration to c-i-u by the time we get to that point (@jcline and I are hoping this will be done soon).
Note that container images are only compressed when pushed to a registry. So the only step that matters for the compression is who is pushing them. If cloud-image-uploader is doing the upload to a registry then adding support for specifying the compression format there will add it for everyone.
ahhh, okay, didn’t realize that part. that definitely makes it easier.
1 Like
I’m confused, hasn’t it been rejected on 2024-08-19 ?
It has been rejected indeed but the revert only landed recently: FEDORA-2024-5a61a2fa45 — security update for buildah, containers-common, & 1 more — Fedora Updates System
I would prefer to think of it as postponed until a later Fedora Release. The change is inevitable to at lease zstd.