Network config needed to pull ignition

Moving this discussion from New Package Request: nmstate-libs and nmstate · Issue #1175 · coreos/fedora-coreos-tracker · GitHub since it was off-topic there.

The concern that was raised dealt with the fact that there is no way to configure the network environment in the ramdisk when Ignition is initially running.[0] This means if you want to use certain network configurations like static IPs or bonds that can’t run in a degraded state Ignition won’t be able to reach any network configurations that are need to complete configuration.

In the GH issue I was pointed to Host Network Configuration :: Fedora Docs , but right on that page there is a note that states: “If you need networking connectivity to pull your Ignition configuration, or if your Ignition has remote references, you won’t be able to provide your networking configuration via Ignition.” This is the exact scenario I’m describing.

Ideally, what we’d like to be able to do is provide a bootstrap Ignition config in a config-drive that would provide just the network config for Ignition to reach the remote sources. I realize this isn’t how Ignition works today, but these networking scenarios are very common on baremetal deployments and it’s quite painful to deal with today (in some cases we have to customize the image deployed to every node to bake network config in). There are some more details on how we’re getting around this limitation in enhancements/baremetal-ipi-network-configuration.md at master · openshift/enhancements · GitHub

This does work, but because the implementation is based on baremetal-specific services it makes it impossible to provide a cross-platform network configuration mechanism, which is something we also need quite badly (for reasons which are a whole can of worms themselves).

Hopefully that provides a reasonable intro to the problems we’re facing. I’m writing this somewhat off-the-cuff because I hadn’t planned to start this conversation yet, but since it came up I thought it might be worth discussing now.

0: This isn’t strictly true - you could use kernel arguments, but those are somewhat limited and can’t be used for many network configurations. As a result, you end up having to maintain one set of config for Ignition and another for the final OS, which is error-prone and tedious.

I think the most powerful, and most general solution to this is listed on the page already, but could use more elaboration:

Basically you can inject a small bit of Ignition into the Live ISO that configures networking and such and that affects the “real root” of the live environment. From there you can fetch the full Ignition and pass it to coreos-installer.

Another way to look at this is that the Live ISO is like the “initramfs” except it’s fully featured. You can even pull container images to compute the Ignition to pass to the installed system, etc.

Colin is correct; coreos-installer {iso,pxe} customize --network-keyfile is the intended solution to this problem. One small correction: under the hood, that option doesn’t use Ignition at all; it’s a parallel mechanism to embed NM keyfiles that take effect in both the live ISO initrd and the real root. In addition, if you then run coreos-installer with --copy-network (or if you have {iso,pxe} customize arrange to do it for you), those keyfiles will also take effect in both the initrd and the real root of the installed system.

(coreos-installer is in Rust, so we could potentially add a --network-nmstate that accepts NMState configs and renders them into keyfiles for embedding.)

That section also links to a demo of the mechanism in action.

No key file generation is needed if nmstatectl is provided with FCOS, from what I see nmstate API is way superior thant NetworkManager keyfiles.

If coreos-installer {iso,pxe} customize deferred keyfile generation until OS boot, we’d need to ship nmstatectl in the initramfs, and I think that might be the only use case for which we’d need to do so. So it may make more sense to handle the keyfile generation during live image customization.

Customizing the live ISO to inject network config and then using --copy-network for the install is what we ended up implementing for baremetal IPI. However, I’m not aware of a general mechanism for customizing the image in IPI deployments so that solution ended up being baremetal-specific because it uses a new component in the baremetal-operator to do the customization. I assume we could just write a cross-platform image customization layer, but it seemed like this would be a common enough use case that it might be worth supporting natively in Ignition.

Being able to provide NMState configs and have them applied would be even better because it provides better error handling (for example NMState will roll back a bad network config, which in some cases might ease debugging of a problem). I suppose that’s a discussion for the original GitHub issue though.

Many of the Fedora CoreOS network configuration features have been implemented specifically for bare-metal installs. Aside from bare metal and certain VMware deployments, we’ve generally assumed that autoconfiguration will get us far enough to perform Ignition provisioning.

Rolling back a bad network config isn’t really important for initial provisioning. Ignition’s semantics are generally that if provisioning fails for whatever reason, the machine refuses to boot.