Bootc-installer fails with "malformed pax extension" after Anaconda deploy

Ask Fedora
,
1

Hello everyone,

I’ve been trying to build a customized Fedora IoT Bootc image using Fedora 44 and package it into an ISO using image-builder (bootc-installer).

I managed to work around several build-time osbuild errors, but I have hit a wall during the actual Anaconda installation phase with a “malformed pax extension” error, and I’m hoping someone can point me in the right direction.

The Blocker

I successfully generate the ISO. It boots into Anaconda. However, right as the deployment starts (ostree container image deploy), Anaconda crashes with this error:

pyanaconda.modules.common.errors.installation.PayloadInstallationError: The command 'ostree container image deploy ...' exited with the code 1: error: Performing deployment: Importing: Parsing layer blob ...: Processing tar: Filtering tar stream: Copying entry: malformed pax extension

Here’s an image:

image(3)
image(3)1332×931 143 KB

From what I got from Gemini, this is likely due to modern OCI/Podman layer metadata (PAX headers) that the installer’s ostree or libarchive cannot parse. To fix this, I tried using the CoreOS Chunkah to normalize the layers, but the installation still fails.

My Setup

Here is my payload Containerfile (using the chunkah trick):

FROM quay.io/fedora/fedora-bootc:44 as rootfs-builder

# Idk, why but `ssh-key-dir/d` is not found, is this a bug?
RUN find /usr/share/doc/bootc-base-imagectl/manifests/ -type f -exec sed -i '/ssh-key-dir/d' {} +

# Use Fedora IoT manifest to build rootfs.
# https://docs.fedoraproject.org/en-US/iot/fedora-iot-bootc-pmachine-example/
RUN /usr/libexec/bootc-base-imagectl build-rootfs --manifest=fedora-iot /target-rootfs

FROM scratch as fedora-iot
COPY --from=rootfs-builder /target-rootfs/ /

LABEL containers.bootc 1
STOPSIGNAL SIGRTMIN+3
ENV fedora_version=44
CMD ["/sbin/init"]


FROM fedora-iot as fedora-iot-opinionated
# Enable faster downloads & DNF5 optimizations
RUN echo -e "[main]\nfastestmirror=True\nmax_parallel_downloads=10" > /etc/dnf/dnf.conf

# Add `machinectl` for Quadlets.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf5 -y install systemd-container

# Set up SSH Server to only accept SSH key logins.
ADD config/override_pubkey_authentication.conf /etc/ssh/sshd_config.d/override_pubkey_authentication.conf

# Change shell to Nushell.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf5 -y install 'dnf5-command(copr)' && \
    dnf5 -y copr enable atim/nushell && \
    dnf5 -y install nushell
RUN echo "/usr/bin/nu" >> /etc/shells
ADD config/useradd /etc/default/useradd

# Install utilities.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf5 -y install b3sum zoxide p7zip
#
# Add Zoxide Nushell integration.
RUN mkdir -p /usr/share/nushell/vendor/autoload/env && \
    zoxide init nushell > /usr/share/nushell/vendor/autoload/zoxide.nu

# Add Terra Repository.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf5 -y install --nogpgcheck --repofrompath 'terra,https://repos.fyralabs.com/terra$releasever' terra-release && \
    curl -fsSL https://github.com/terrapkg/subatomic-repos/raw/main/terra.repo | tee /etc/yum.repos.d/terra.repo

# Enable Free and Non-Free Repositories.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf5 -y install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-44.noarch.rpm && \
    dnf5 -y install https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-44.noarch.rpm

# Override `KillUserProcesses` to `no`.
ADD config/override_kill_user_processes.conf /etc/systemd/logind.conf.d/override_kill_user_processes.conf

# Add network configuration.
ADD config/<redacted> /etc/NetworkManager/system-connections/<redacted>.nmconnection
RUN chmod 600 /etc/NetworkManager/system-connections/<redacted>.nmconnection

# Add Carapace
ADD config/fury.repo /etc/yum.repos.d/fury.repo
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf5 -y install carapace

RUN carapace _carapace nushell > /usr/share/nushell/vendor/autoload/carapace.nu && \
    echo '$env.CARAPACE_BRIDGES = "zsh,fish,bash,inshellisense"' \
        > /usr/share/nushell/vendor/autoload/env/carapace_bridges.nu

# Install `sbctl`.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf5 -y copr enable chenxiaolong/sbctl && \
    dnf5 -y install sbctl

# Install Tuned, Tuned-PPD
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf5 -y install tuned tuned-ppd && \
    systemctl enable tuned tuned-ppd

# Install ncurses for clear command.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf5 -y install ncurses


FROM fedora-iot-opinionated as ben-server
RUN rm /var/{log,cache,lib}/* -rf
RUN bootc container lint
RUN dnf5 -y upgrade

# Normalize layers with Chunkah.
FROM quay.io/coreos/chunkah:latest AS chunkah
RUN --mount=from=ben-server,src=/,target=/chunkah,ro \
    --mount=type=bind,target=/run/src,rw \
        chunkah build --prune /sysroot/ --max-layers 128 --label ostree.commit- --label ostree.final-diffid- > /run/src/out.ociarchive


FROM oci-archive:out.ociarchive
LABEL containers.bootc=1
LABEL ostree.bootable 1
ENV container=oci
STOPSIGNAL SIGRTMIN+3
ENV fedora_version=44
CMD ["/sbin/init"]

And my dedicated Installer image (Containerfile.installer):

FROM quay.io/fedora/fedora-bootc:44

RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf install -y \
        anaconda \
        anaconda-install-env-deps \
        anaconda-dracut \
        dracut-config-generic \
        dracut-network \
        net-tools \
        squashfs-tools \
        grub2-efi-x64-cdboot \
        python3-mako \
        lorax-templates-* \
        biosdevname \
        prefixdevname \
        && dnf clean all

RUN mkdir -p /boot/efi && cp -ra /usr/lib/efi/*/*/EFI /boot/efi
RUN mkdir /var/mnt

I build them, and then generate the ISO with:

sudo image-builder build \
  --bootc-ref localhost/anaconda:44 \
  --bootc-installer-payload-ref localhost/ben-server:44 \
  --bootc-default-fs ext4 \
  --blueprint config.toml \
  bootc-installer

Here’s the command I use to build:

distrobox-host-exec sudo image-builder build --bootc-ref localhost/anaconda:44 --bootc-installer-payload-ref localhost/ben-server:44 --bootc-default-fs ext4 --blueprint config.toml bootc-installer

Yes, I am in a Distrobox, but it’s using the host’s Podman in rootful mode so it shouldn’t affect my build (I think).

And the thing is, when I use --type qcow2, it works:

image(2)
image(2)1332×931 151 KB

Questions

  1. Is the malformed pax extension error a known bug in F44’s ostree/libarchive?
  2. Is my usage of chunkah in the multi-stage build incorrect, or is there another way I should force F44 Podman to write ostree-compatible layers?
  3. Am I missing something fundamental about building F44 IoT bootc installers?

Any help or pointers would be immensely appreciated.
Thanks for reading!

2

I found the fix. Turns out I have to use the bootc-base-imagectl tool.

Here’s my build script:

set -euo pipefail
                                                                                                                                                                                                                    
echo "Build ben-server OCI image..."
distrobox-host-exec sudo podman build -f Containerfile.main -t localhost/ben-server:dev-unchunked . --security-opt=label=type:container_runtime_t --device /dev/fuse --cap-add=all
                                                                                                                                                                                                                    
echo -e "\n"
                                                                                                                                                                                                                    
echo "Build anaconda OCI image..."
distrobox-host-exec sudo podman build -f Containerfile.installer -t localhost/anaconda:44 . --security-opt=label=type:container_runtime_t --device /dev/fuse --cap-add=all
                                                                                                                                                                                                                    
echo -e "\n"

# This is what fixed it, it's a must then.                                                                                                                                                                                                                    
echo "Rechunking ben-server OCI image..."
distrobox-host-exec sudo podman run --rm --privileged -v /var/lib/containers:/var/lib/containers \
  quay.io/fedora/fedora-bootc:44 \
  /usr/libexec/bootc-base-imagectl rechunk \
  localhost/ben-server:dev-unchunked \
  localhost/ben-server:44

But why the Fedora Documentation doesn’t list as such? I have been scratching my head and digging around but no documentation lists so.

I would like to know, any feedback is appreciated.

Thanks for reading.