Proposal for shared layers for bootc systems (future CoreOS, Silverblue, Kinoite, IoT, etc)

Project Discussion
, , , , ,
1

Going forward in the bootc world, I’d like to see something like this:

Layer 0: Really Minimal

FROM fedora:rawhide AS helper
RUN dnf install --installroot /var/tmp/installroot --use-host-config --setopt install_weak_deps=false -y filesystem \
 && dnf --installroot /var/tmp/installroot clean all

FROM scratch AS ring0
COPY --from=helper /var/tmp/installroot/. /

4.19 mb. Just as tiny as we can make it, while still being nominally Fedora. [1]

This is kind of frivolous, but it provides a starting point and a pattern for other (presumably more useful) minimal containers).

Layer 1: add DNF

FROM fedora:rawhide AS helper
COPY --from=localhost/fedora/ring0:rawhide /. /var/tmp/installroot
RUN dnf install --installroot /var/tmp/installroot --use-host-config --setopt install_weak_deps=false -y dnf \
 && dnf --installroot /var/tmp/installroot clean all

FROM scratch AS ring1
COPY --from=helper /var/tmp/installroot/. /

92 mb. For the convenience of having DNF in the image, so you don’t have to do the helper pattern thing.

Layer 2: Make it Bootable

FROM localhost/fedora/ring1:rawhide
RUN dnf install --setopt install_weak_deps=false -y kernel-core selinux-policy-targeted bootc \
 && dnf clean all
LABEL containers.bootc 1

419 mb. Contains useful stuff like systemd and podman, too. (And selinux — it’d be 393 mb without that.) At this stage, we’d do whatever magic is necessary to make this into a bootc base container.[2]

Layer 3: Common Consensus Core

FROM localhost/fedora/ring2:rawhide
RUN dnf install --setopt install_weak_deps=false -y [whatever we agree on] \
 && dnf clean all
LABEL containers.bootc 1

The rule for this layer is simple: it contains all packages that every Fedora Edition has in common, and no packages that aren’t in every edition. That is, every Edition gets veto power.[3] Not in a hostile way — the intention is completely the opposite: there’s no need to debate or vote or anything like that.

Above That…

Each Edition would start with FROM fedora/ring3 and add in what they need.

Ideally, we would have more shared layers at this level too — like graphics stack support, shared by all of the desktop environments. (And sound. fedora/desktop-common or something.)

So…

Part of the problem with minimization and standardization previously is that we didn’t have a clear definition for what “minimal” or “base” really meant.[4] With this, we have a clear target, and can focus on the lower layers for:

  • Minimization
  • Testing
  • Gating (if something breaks at a lower layer, it breaks everything!)
  • Security review

These things can focus first on layer 0, and then move up as we have capacity and confidence.

These also provide more clear domain separation — if you’re working on a spin starting from layer 3, you should have confidence that everything up until there should just work, and be able to focus on your changes. On the other side, I can imagine a renewed “Base Working Group” with ownership of the lower layers.[5]

  1. I notice that fedora-gpg-keys is taking up a relatively large amount of that; probably keys for releases more than the last three could be moved to a -old subpackage. ↩︎

  2. The bootc project’s intention is to make the above just work, but currently there’s some complication. ↩︎

  3. Non-Edition deliverables can start from this point as well, or if they want something more minimal, from Layer 2 instead. ↩︎

  4. See Base - Fedora Project Wiki from a decade ago. ↩︎

  5. I would expect that to also include members of Edition working groups who care about the end-to-end experience. ↩︎

8 Likes
2

This is meant to be a conversation starter and straw-person proposal. Call it “strong opinions lightly held”. What are your thoughts?

1 Like
3

I’ll repeat what I’ve said elsewhere about this…

I don’t think this is the panacea you think it is. In practice, we already have layering through composition groups (which are tested and verified through both dedicated comps tests and tests through image build CI). Almost every Fedora deliverable derives from the same basic set of components, but that doesn’t change the fact that the integration of those components is what changes things. You can’t actually test them in isolation and expect those test results to propagate up to changed environments.

The top down test model makes more sense for integration than a bottom-up one like you propose. Your suggested model might actually make things worse because the assumption of quality is not validated and enforced at the upper layers where the user experience exists.

A simple example: the Raspberry Pi 4 series graphics driver works totally fine on its own in a simple case, but integrate it with the GNOME deliverable or the KDE Plasma deliverable, and you get two distinct sets of seriously broken results. We have spent the past several weeks chasing bugs that only exist when you stack everything together. These make up the bulk of our problems every cycle.

“Common core” is functionally not testable in a useful way for issues like this, and even more generally, they are a very rare source of bugs and problems. Thus, this is an example of optimizing for a problem we don’t have.

In addition to this, the layering effect can cause problems with package transactions. We use conditional and weak dependencies a lot throughout the stack these days in order to optimize the installed package set for the target deliverable. This layering model completely breaks that since weak dependencies are not re-evaluated as the conditions change down the road. Some conditional dependencies are re-evaluated, but only if they are strong dependencies. This means that you create new unexpected bugs by doing this because the total package set can be wrong because the segmented installs can result in undesirable selections or packages getting skipped altogether.

It also affects storage: bootable containers are not storage efficient. Having stacks of images means that there’s additional overhead and extra space taken up by these archive files.

5 Likes
4

Ring3 for user’s custom layer, maybe ? E.g., not everyone needs libvirt or OpenVPN, but for those who needs it, ring3 is where user customizations goes…

1 Like
6

Please stop saying that I think it is a “panacea”. I never said anything like that.

I do, however, think that these are useful places to make separations, because I can see useful derivations branching at each point. The goal is to centralize what can be shared, while also decentralizing at higher levels, giving more flexibility to Spin and Edition teams.[1] It also gives more options to downstream projects like https://universal-blue.org/.

I understand that full integration testing and validation is crucial. See the last footnote above. However, going to a shared base will reduce current complexity in openqa, and there is value in testing in expanding rings (just as integration testing doesn’t mean unit tests aren’t important). Testing Layers 1, 2, 3 doesn’t mean the Spin-level layer won’t expose problems that go back down — but problems in lower layers will affect everything above, so a lot of time could be saved by failing early.

Common layers will also reduce build times, and in many cases allow us to “re-spin” deliverables without re-doing the core. And, it’s my understanding that with the composefs approach, stacked layers aren’t a significant problem. In any case, it’s a soluble one.

Finally, it’s objectively the case that the comps group approach has not succeeded in minimizing core packages, despite several efforts. That’s still an important goal. Of course this won’t magically solve that, but it’s a lot easier to conceptualize, and is measurable by default (as each layer is a deliverable, with the size right there).

  1. whereas the comps approach keeps everything centralized but not quite shared. ↩︎

7

My initial sketch had several more layers meant specifically for people wanting to do that kind of thing, but I decided to simplify.

I put DNF at Layer 1 because that’s what people expect from a container. I think what you need really requires a branch, where Layer 1 and Layer 2 are swapped. (And then you’d build on Layer 2, ignoring Layer 3.)

1 Like