F41 Change Proposal: Enabling composefs by default for Atomic Desktops, CoreOS and IoT (Self-Contained)

Project Discussion Change Proposals
,
1

Enabling composefs by default for Atomic Desktops, CoreOS and IoT

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Wiki
Announced

:link: Summary

We want to enable composefs by default for Fedora Atomic Desktops, Fedora CoreOS and Fedora IoT. This makes the root mount of the system (/) a truly read only filesystem, increasing the system integrity and robustness. This is the first step toward a full at runtime verification of filesystem integrity.

This change will be enabled only for the Bootable Container images of Fedora Atomic Desktops and not the classic ostree ones.

:link: Owner

:link: Detailed Description

Ostree based systems currently have /usr mounted as read-only and managed by ostree/rpm-ostree. The integrity of the content of /usr is only validated by ostree/rpm-ostree during updates and deployment operations, but not at “runtime”. If a file is corrupted on disk (maliciously or not), it will only be detected if a full check is performed using ostree fsck.

On those systems, the runtime root (/) of the system is currently mounted as read-write but with the immutable bit set (chattr +i /) to prevent accidental modifications.

composefs is a new project that combines several existing filesystems (overlayfs, EROFS) to provide a very flexible mechanism to support read-only mountable filesystem trees, stacking on top of an underlying “lower” Linux filesystem.

Using composefs, it will no longer be possible to mutate the underlying file content that is part of the system (/usr) nor the layout of the root directory. It will result in I/O errors at the kernel level.

The content in /etc and /var will remain writable as it is today.

This change is part of the Fedora Bootable Containers Initiative. The bootc container images already enable composefs, thus this change is to align existing variants to the new Bootable Containers defaults.

It is tracked in:

This is the first step toward a full boot chain integrity, that will require signing the composefs metadata during composes and using Unified Kernel Images (UKI). See: document and support creating "sealed" base images (#14) · Issues · fedora / bootc / Issue Tracker · GitLab

As podman also uses composefs to store container layers, this enables deduplication of files between containers and host. This will result in less disk usage but also faster container startup and less memory use. See Add shared library/tool for managing backing store files · Issue #125 · containers/composefs · GitHub

:link: Feedback

Nothing specific so far.

We have the following “known issues”:

:link: Benefit to Fedora

This will increase the robustness of image based Fedora systems and prepare them for future increased security guarantees.

This will align the existing image based variants of Fedora (Atomic Desktops, CoreOS, IoT) with the work that is done as part of the Bootable Containers Initiative.

:link: Scope

  • Proposal owners:

    • Enable composefs in Atomic Desktops (bootable containers only)
    • Enable composefs in CoreOS
    • Enable composefs in IoT

  • Other developers:

    • Applications doing disk-full checks on / will have to be updated to look at other places as / will be small (a few MB) and full (100% used).

  • Release engineering: N/A (not needed for this Change)

  • Policies and guidelines: N/A (not needed for this Change)

  • Trademark approval: N/A (not needed for this Change)

  • Alignment with the Fedora Strategy 2028:

    • Aligns with the goal: “Immutable variants are the majority of Fedora Linux in use”

:link: Upgrade/compatibility impact

To be fleshed out

:link: Early Testing (Optional)

Do you require ‘QA Blueprint’ support? N

:link: How To Test

:link: User Experience

The main visible change will be that the root filesystem (/) is now small and full (a few MB, 100% used). The real root is mounted in /sysroot and most of the data is stored in /var.

:link: Dependencies

For the Atomic Desktops, this change depends on:

CoreOS and IoT already do not depends on ostree-grub2.

:link: Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) Undo the change. It’s a single line change in a configuration file.
  • Contingency deadline: Beta Freeze / Release Freeze
  • Blocks release? No

:link: Documentation

To be written.

:link: Release Notes

To be written once the change is accepted.

Last edited by @boredsquirrel 2024-06-27T23:23:17Z

3 Likes
2

How do you feel about the proposal as written?

  • Strongly in favor
  • In favor, with reservations
  • Neutral
  • Opposed, but could be convinced
  • Strongly opposed
0 voters

If you are in favor but have reservations, or are opposed but something could change your mind, please explain in a reply.

We want everyone to be heard, but many posts repeating the same thing actually makes that harder. If you have something new to say, please say it. If, instead, you find someone has already covered what you’d like to express, please simply giving that post a :heart: instead of reiterating. You can even do this by email, by replying with the heart emoji or just “+1”. This will make long topics easier to follow.

Please note that this is an advisory “straw poll” meant to gauge sentiment. It isn’t a vote or a scientific survey. See About the Change Proposals category for more about the Change Process and moderation policy.

3

I was so free and corrected a few spellos.

Awesome change! This solves so many things at once, container host/guest deduplication is brilliant too.

Do the currently existing Fedora container images on quay (that uBlue uses) use that system already?

Would it be possible to skip the bootupd step and directly switch to the bootc variants?

Thanks!

4

thanks for the contribution !

container host/guest deduplication is brilliant too.

The deduplication of data between the host filesystem and container storage requires the unification of ostree and containers/storage, the implementation details are still discussed.

So it’s not going to come automatically with composeFS, but it’s the required cornerstone to make it happen.
See Add shared library/tool for managing backing store files · Issue #125 · containers/composefs · GitHub for more discussion.

Do the currently existing Fedora container images on quay (that uBlue uses) use that system already?

Not yet, this change targets exactly that. See the Scope section.

Regarding bootupd, it’s shipped in bootc images yes, but the issue is that auto-updates are not enabled by default, which must be done first to make sure we don’t break old installs upgrading to this

1 Like
5

Ok so I test this now (uBlue kinoite-main, using the OCI container images) and in KDE got an error message “warning, storage in / is low, 0B left”. Which is expected, but all such tools will need a workaround here.

6

Linking reported issue: Issue #539: Kinoite, composefs (F41 change proposal): low storage warning - SIG - Pagure.io

1 Like
7

I also encountered another possibly related issue: system deployments after reboot are not possible.

No rebase, reset, update, deploy.

Still on ublue kinoite-main, they had this signing issue. I thought it was just because of that and tried rebasing to Fedoras quay image of Kinoite.

Went well, everything fine, the official image showing up in rpm-ostree status as pending deployment.

Then, after reboot, it is gone and I am still on the old one. So I cannot update, rebase etc.

I suppose this is because of the removal of the grub integration, how do I bypass that? I cannot upgrade right now.

Am I missing something? Are the images ready? I would appreciate more detailed “how to test” instructions

8

[moved from above thread]

so i’m testing again just running the first rebase, and i’m getting a filelight notification saying that my root partition is running out of space

image

edit: ok, apparently it’s a known issue https://pagure.io/fedora-kde/SIG/issue/539

1 Like
9

Okay as my updates were still broken (luckily already had the openssh update) I reinstalled ostree-grub2 and disabled composefs.

sudo ostree config set ex-integrity.composefs no
rpm-ostree override reset ostree-grub2
rpm-ostree --reboot update

The update still did not apply.

Using an a bit outdated uBlue kinoite-main, and the bootloader was updated manually a short time ago.

10

@boredsquirrel you are likely hitting grub2-mkconfig fail with composefs enabled · Issue #3198 · ostreedev/ostree · GitHub

What’s the output of journalctl -u ostree-finalize-staged -b -1 ?

1 Like
11

This is likely Use composefs by default for Bootable Containers (#35) · Issues · fedora / Fedora Atomic Desktops / SIG Issue Tracker · GitLab.

You need to make sure that your bootloader is updated and BLS capable before switching to composefs right now.

1 Like
12

I updated it like 2 weeks ago following your guide. But that may not be recent enough.

I updated it again, re-enabled composefs and kept ostree-grub2 uninstalled, lets see

13

This change proposal has now been submitted to FESCo with ticket #3240 for voting.

To find out more, please visit our Changes Policy documentation.

3 Likes
14

I am still not getting updates to stage, my bootloader files were updated yesterday, I tried an update still with ostree-grub2 removed.

The issue may be that I am using a ublue image from 29.6. So over a week outdated.

I am now trying to add back ostree-grub2 and disable it. But I fear that without ostree-grub2 in the used image, updates will not be staged.

Happy about help.

@jbtrystram

Jul 08 15:58:15 PC systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Jul 09 01:43:51 PC systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Jul 09 01:43:51 PC ostree[77428]: Finalizing staged deployment
Jul 09 01:43:52 PC ostree[77428]: Copying /etc changes: 42 modified, 11 removed, 264 added
Jul 09 01:43:52 PC ostree[77428]: Copying /etc changes: 42 modified, 11 removed, 264 added
Jul 09 01:43:52 PC ostree[77428]: Refreshing SELinux policy
Jul 09 01:43:53 PC ostree[77428]: Refreshed SELinux policy in 1126 ms
Jul 09 01:43:53 PC ostree[77428]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jul 09 01:43:53 PC ostree[77428]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jul 09 01:43:53 PC ostree[77428]: Finalized deployment
Jul 09 01:43:54 PC ostree[77428]: bootfs is sufficient for calculated new size: 156,1 MB
Jul 09 01:43:54 PC ostree[77428]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jul 09 01:43:54 PC ostree[77428]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jul 09 01:43:54 PC ostree[77428]: error: Bootloader write config: grub2-mkconfig: Der Kindprozess wurde mit Status 1 beendet
Jul 09 01:43:54 PC systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Jul 09 01:43:54 PC systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Jul 09 01:43:54 PC systemd[1]: Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Jul 09 01:43:54 PC systemd[1]: ostree-finalize-staged.service: Consumed 1.949s CPU time.

So this may be a grub2-mkconfig issue.

After a reboot, adding back ostree-grub2, the update could still not be applied. On boot, journalctl shows an error of the OSTree finish steps

15

This is what I wrote in Use composefs by default for Bootable Containers (#35) · Issues · fedora / Fedora Atomic Desktops / SIG Issue Tracker · GitLab that I linked earlier. Unfortunately removing ostree-grub2 is not enough. You also have to set sudo ostree config set sysroot.bootloader none, but before that, you have to make sure that your bootloader is updated first or it may result in a non-bootable system.

It’s not well tested yet and the transition part for non-container Atomic Desktops is not part of this change for Fedora. We will have to figure that out independently for Universal Blue users.

2 Likes
16

Thanks, sorry didnt read that.

@jbtrystram can you update the “how to test” instructions accordingly, and please also add a big warning that this may break a system?

I will try, as I was close to reinstalling anyways. Thanks for the help!

Update: okay this command seems to have broken GRUB.

17

Update: this broke my install and now throws me into a grub shell.

I have a Fedora Kinoite 40 install on a second SSD, and did Timothees trick with the copying of the boot files but from that ones location, to the mounted /boot/efi of the broken install

So the bootloader should really be updated now to the latest possible, still the grub shell.

18

Did you disable BLS entries in the past ? This was the fix for having duplicated entries in GRUB.

I encountered the same issue as you, and ended up having to reinstall fedora. It’s on my TODO list to create a known reproducer for this, so we can put up more documentation and more failsafes, but yeah, that bootloader issue you have is a known one.

1 Like
19

Oooh, yes this may be it!

No actually I enabled BLS back then.

So this also means that BLS was already working a year ago?

20

This is tracked in Fedora 41 change: composefs enabled by default · Issue #608 · ublue-os/main · GitHub on the Universal Blue side of things.