Switch Fedora Container images to support zstd:chunked format by default

[dwalsh (Daniel J Walsh)]

For several years now Container images have been shipped with gzip format compression, first designed by Docker. But the OCI Container Image Format supports zstd and zstd:chunked format as well. Not only is this a better compression format, which should speed up image pulls, but zstd:chunked will allow container tools like Podman, Buildah and CRI-O to pull only the files that have changed in an image rather then the entire image. This should greatly reduce the time to pull an image.

We are switching the default for podman in Fedora 41 to push images in zstd:chunked format.

All current container tools support zstd:chunked format, including Podman, CRI-O, Buildah, Containerd, and Docker. Docker began supporting zstd:chunked in March of 2023.

All of the other container engines began supporting it back in 2020 or earlier.

One problem with changing this format for Fedora based images would be Docker versions older then 2023. If this is considered too big of an issue, there is a way to have images which include both the older gzip format and zstd:chunked format. Podman, Buildah, CRI-O will pull the newer format, and docker and containerd would pull the older gzip format, meaning these tools would not see the improvement with the new compression algorithm. One issue with supporting both formats is that image sizes will grow to slightly less then 2X times in size at the registry. This should not effect pulling for those using the traditional gzip format.

Thoughts.

[dwalsh (Daniel J Walsh)]

https://fedoraproject.org/wiki/Changes/zstd:chunked
Is the change request for F41.

[dwalsh (Daniel J Walsh)]

Pull container images faster with partial pulls

Describes zstd:chunked format.

[boredsquirrel]

I love how open source software becomes more sustainable and efficient over time!

Is there some kind of analysis what hosts run Fedora Containers? There would be absolutely use cases on Debian 11 or 12, or the old supported RHEL versions.

Would it be possible to rename the gzipped container images, and contact the distros that use them, to contact their users to switch the images?

People deploying Fedora containers should be the folks that respond to such messages and switch the containers.

That duplicated size doesn’t seem like a good option.

[hamrheadcorvette (hammerhead corvette)]

Wait a minute

Are you the Dan Walsh of the sandbox_t & container_t

Cool things with SELinux... Introducing sandbox -X

[dwalsh (Daniel J Walsh)]

Yes I am.

[castrojo (Jorge Castro)]

@dwalsh

there is a way to have images which include both the older gzip format and zstd:chunked format.

To the same tag? I wasn’t able to figure this out, is there an example somewhere?

(This would indeed be useful for a transitionary period!)

[dwalsh (Daniel J Walsh)]

You can associate the same image twice to a manifest list, and as long as the first one is gzip then older tools will pull that. Podman and friends is smart enough to look for the zstd:chunked version first, and then fall back to the gzip version.

[vrothberg (Valentin Rothberg)]

Yes, as Dan mentioned there is a way to include both. The OCI image spec mentions that “if multiple manifests match a client or runtime’s requirements, the first matching entry SHOULD be used”.

To remain backwards compatible, the “trick” is to have the image with gzip-compressed layers before the zstd one. This way, old clients will pick the gzip one while newer clients are smart enough to pick the zstd one.

[cverna (Clement Verna)]

IIUC it means that for example for fedora:latest we will have a manifest list with:

• x86 - gzip
• x86 - zstd
• aarch64 - gzip
• aarch64 - zstd
• ppc64le - gzip
• ppc64le - zstd
• s390x - gzip
• s390x - zstd

Also just to make it visible the images hosted on DockerHub won’t have the zstd support since we don’t push these images to the registry. Docker is building the images and pushing them themself.

[vrothberg (Valentin Rothberg)]

That’s exactly it. Thanks for visualizing the idea, Clement!

[dwalsh (Daniel J Walsh)]

So who is in charge of pushing these images? Would love to meet with them and attempt to create one image, that we could then test with Podman and Docker to make sure they both work. Even with an older version of Docker, perhaps on Ubuntu or something.

[kevin (Kevin Fenzi)]

The images are pushed by fedora release engineering.

Currently thats using cloud-image-uploader (yes, I know it has cloud in
the name, but it got expanded to handle container uploading too).

https://pagure.io/releng
and
https://pagure.io/cloud-image-uploader/

Can rhel8/9 podman handle zstd? Is it only older docker thats not able
to?

kevin

[gscrivano (Giuseppe Scrivano)]

our tools support zstd since at least 5 years, so we would have problems only with the Docker version on RHEL 7

[kevin (Kevin Fenzi)]

Which goes end of life on sunday.

[queeup]

Am I missing something?

$ rpm-ostree rebase --experimental ostree-unverified-registry:ghcr.io/queeup/silverblue:40
Pulling manifest: ostree-unverified-registry:ghcr.io/queeup/silverblue:40
Importing: ostree-unverified-registry:ghcr.io/queeup/silverblue:40 (digest: sha256:b893ef3c36fc6cf7fdc84a3313d0537d39f1d5482eb2efe5cee0d6c1bf51e4f7)
ostree chunk layers needed: 65 (2,2 GB)
custom layers needed: 3 (220,6 MB)
error: Importing: Unencapsulating base: Unhandled layer type: application/vnd.oci.image.layer.v1.tar+zstd

Without updating (rpm-ostree update) Silverblue 40 installer ISO (Fedora-Silverblue-ostree-x86_64-40-1.14.iso) there is no zstd support with ostree.

[vrothberg (Valentin Rothberg)]

We missed adding one important information. The zstd-compressed image must also be annotated in the OCI index/manifest list:

{
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
    "size": 772,
    "annotations": {
        "io.github.containers.compression.zstd": "true"
    },
    "platform": {
        "architecture": "amd64",
        "os": "linux"
    }
},

[queeup]

@vrothberg

The zstd-compressed image must also be annotated in the OCI index/manifest list

podman manifest push --add-compression zstd will add that annotation right?

Unfortunately redhat-actions/buildah-build@ github action is not supported annotations atm. You should add it manually with docker/metadata-action@.

- name: Container meta
  id: meta
  uses: docker/metadata-action@v5
  with:
    images: ${{ env.REGISTRY }}/${{ env.GITHUB_USERNAME }}/${{ env.IMAGE_NAME }}
    tags: |
    ...
    annotations: |
      io.github.containers.compression.zstd=true

---

@castrojo, Is it what you are looking?
From podman manifest push docs:

--compression-format gzip --add-compression zstd will push a manifest list with each instance being compressed with gzip plus an additional variant of each instance being compressed with zstd.

[siosm (Timothée Ravier)]

We’ve now submitted the change and it should be announced (and then voted on by FESCo) soon.

I did not know that we could make a manifest with both gzip & zstd images. If folks can create some examples with podman/buildah that would be great.

Thanks