How to deploy CoreOS on IncusOS?

Ask Fedora
, ,
1

use case

I tried installing Fedora CoreOS as a VM to then run a container inside it with podman on IncusOS (aka Incus as a fork of LXD, but similar to Proxmox as a hypervisor specifically focused on security and reliability).

resources/related posts/research

I tried looking at these posts:

The last thing is most interesting:

This is about the mechanism we would use to pass the Ignition config to the instance. In this case we would likely use the same option used to pass cloud-init configs: How to use cloud-init - Incus documentation

The link looks interesting, so one would need to use cloud-init

However:

Maybe the OpenStack image would work as is as it has support for cloud-init drives from memory.

So it seems only(?) this image includes cloud-init for it to work, actually? I used the qemu ones (see below), so using different ones just for cloud-init kinda feels like “misusing” the images. :see_no_evil_monkey:

what I tried so far

You can see my progress in this repo:

Basically, currently I just use the qemu images, package them as a tar.xz and push them to the CoreOS host. And this works flawlessly.
This follows this awesome blog post by ignisc that already explains a lot:
https://agnomi.net/posts/incus/fedora-coreos/

problems

I do not (only) run Incus, but IncusOS. That means, I have no access to the file system and cannot run terminal commands on that host.

The blog post linked above, however, then tries to pass the Ignition config as a qemu variable:

incus config set fcos02 raw.qemu=" -fw_cfg name=opt/com.coreos/config,file=/var/lib/incus/ignition/fcos02.ign"

Obviously, this does not work on a remote IncusOS instance/server. You also cannot place the file “on the server” to adjust the path.

Questions

  • Is there any way to pass an init file like the mentioned one to the server?

  • What do you think of using the OpenStack image, as indicated? (And then cloud-init? Somehow?)

    • So if cloud-init is used, how to do that actually? I read the docs, but I found no way of placing a file somewhere (the Ignition file) to later be picked up by cloud-init.

  • The blog post says, at the end:

    Find a way not to have to scp the ignition files (maybe using a volume containing them all ?)

    This confuses me a bit, because I do not actually see scp mentioned in the guide before at all and how it should work.

Cross-posted to the Incus community forum.

2

Incus has a web interface, and you can pass instance configuration through it, so you shouldn’t necessarily need filesystem access to the IncusOS host.

The important distinction here seems to be between not having shell/filesystem access to the host and not being able to configure the VM through Incus.

For the Ignition part specifically, I’d check whether the Incus API/web UI exposes the configuration mechanism you need rather than trying to place the Ignition file directly on the server.

Also, regarding the concern about accessing the system: have you checked the official IncusOS documentation?

I’m not using Incus at the moment, so I may be misunderstanding the exact limitation you’re hitting, but I don’t think the lack of direct filesystem access necessarily prevents you from configuring the VM.