Embedding containers into custom Fedora IoT systems

This discussion came up on IRC, I’m going to paste my response here so I can link to it later:

walters: what would be the recommended way to embed and autostart podman containers in a cosa build ?

16:32 older discussion on that here Re: Deployment when having separate development streams

16:32 basically, put the container rootfs in /usr/share/mycontainers and use podman --root or systemd-nspawn -D or bwrap etc. launched via a systemd unit

16:33 here the container rootfs is lifecycle bound to the host, ostree’s automatic dedup is nice if the container root has some of the same content (bash, libc) as the host

16:33 at some point maybe rpm-ostree compose tree will learn an opinionated way to do this, accepting container images direclty

16:35 it won’t work to put it in the default podman/docker storage of /var because that is explicitly separate from ostree updates

16:37 today rpm-ostree is fairly oriented to shipping mostly RPMs, but see the ostree-layers bit in rpm-ostree/treefile.md at master · coreos/rpm-ostree · GitHub - you can commit arbitrary content (e.g. from git, or unpack container images) into ostree commits, and then those can get merged on top of the base RPM content

16:38 this is how the overlay.d directory works in fedora-coreos-config/overlay.d at testing-devel · coreos/fedora-coreos-config · GitHub

16:38 coreos-assembler knows to commit all the files in there into ostree refs and pass them to rpm-ostree

16:40 so if you just want to prototype this today, use e.g. mkdir overlay.d/mycontainer && cd mycontainer && podman export [quay.io/myuser/mycontainer](http://quay.io/myuser/mycontainer) | tar xf - and also make a systemd unit, cosa will commit it

16:40 obviously this could be made very opinionated and streamlined but…73.2% of my time is focused on OpenShift, I am hopeful though we can make some progress on a cosa/osbuild merge and flesh out some of these things

16:45 (coreos-assembler doesn’t support pre-committing containers because for CoreOS the goal is those run on a separate lifecycle from the host)

16:47 you can also of course take the ostree commit generated by rpm-ostree compose tree and layer more stuff in whatever format with whatever buildsystem you want on top via pure ostree

16:49 though there’s some downsides to this model, like rpm-ostree and coreos-assembler emit a lot of useful metadata in the ostree commit, such as the git hash of the config repo, metadata about the input rpm-md repos (names and timestamps), rpm-ostree also has built-in change detection and automatic version number handling; ostree is a low level tool that allows you to build that stuff