Network files propagation when loading rootfs from network

Server, Cloud, and IoT CoreOS
#1

Greetings
Recently I came with an issue, with configuring network using nmconnection files, in combination with loading rootfs remotely.
What I’m doing is that I have an small ISO image, with just the ramdisk, and I load rootfs from network. My kernel arguments look similar to: /proc/cmdline: BOOT_IMAGE=/images/pxeboot/vmlinuz initrd=/images/pxeboot/initrd.img,/images/ignition_ramdisk ignition.firstboot ignition.platform.id=metal ignition.config.url=http://192.168.112.199/ignition_config coreos.live.rootfs_url=http://192.168.112.199/rootfs.img
See that in initrd, I load two ramdisks.

The second ramdisk simply injects a file into /etc/NetworkManager/system-connections/eno1.nmconnection . I can see that file being copied, before the rootfs is loaded
When rootfs starts to be copied, I enter into that directory and see that the i injected into /etc/NetworkManager/system-connections has been removed. As a consequence it doesn’t propagate my network configuration.

So I wanted to raise the issue, looks as a bug to me. Has any one else hit this issue, or is able to reproduce it? What could be the workarounds?
Thanks.

#2

Is there anything in the system journal that looks relevant?

#3

journalctl -u coreos-teardown-initramfs.service
– Logs begin at Wed 2020-11-25 12:20:05 UTC, end at Thu 2021-01-07 10:46:25 UTC. –
Nov 25 12:23:19 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: taking down network device: eno1
Nov 25 12:23:19 singlenode.cluster.testing coreos-teardown-initramfs[1222]: RTNETLINK answers: Operation not supported
Nov 25 12:23:19 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: taking down network device: eno2
Nov 25 12:23:19 singlenode.cluster.testing coreos-teardown-initramfs[1222]: RTNETLINK answers: Operation not supported
Nov 25 12:23:19 singlenode.cluster.testing systemd[1]: Stopping CoreOS Tear down initramfs…
Nov 25 12:23:19 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: taking down network device: eno3
Nov 25 12:23:19 singlenode.cluster.testing coreos-teardown-initramfs[1222]: RTNETLINK answers: Operation not supported
Nov 25 12:23:20 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: taking down network device: eno4
Nov 25 12:23:20 singlenode.cluster.testing coreos-teardown-initramfs[1222]: RTNETLINK answers: Operation not supported
Nov 25 12:23:22 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: flushing all routing
Nov 25 12:23:22 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: hostname is defined in the real root
Nov 25 12:23:22 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: will not attempt to propagate initramfs hostname
Nov 25 12:23:22 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: no networking config is defined in the real root
Nov 25 12:23:22 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: propagating initramfs networking config to the real root
Nov 25 12:23:22 singlenode.cluster.testing coreos-teardown-initramfs[1222]: /usr/bin/coreos-relabel
Nov 25 12:23:22 singlenode.cluster.testing coreos-teardown-initramfs[1222]: Relabeled /sysroot//etc/NetworkManager/system-connections/default_connection.nmconnection from (null) to system_u:object_r:NetworkManager_etc_rw_t:s0
Nov 25 12:23:22 singlenode.cluster.testing coreos-teardown-initramfs[1222]: info: no initramfs automatic multipath configuration to propagate
Nov 25 12:23:22 singlenode.cluster.testing systemd[1]: Stopped CoreOS Tear down initramfs.

#4

Nothing in that service’s logs seems relevant here. Anything in the journal generally?

#5

No, nothing that i could see.

#6

The network propagation bit is primarily meant to propagate automatically-generated NM configs based on network-related kernel arguments (i.e as translated by nm-initrd-generator). Those are in /run/NetworkManager/system-connections, so it doesn’t look at /etc at all.

So one approach here is to use the equivalent network kargs instead if that’s possible. Another approach is to also include the same NM config as part of your Ignition config.

That said, I think we could look at expanding the propagation logic so it also looks at /etc/NetworkManager/system-connections. (I’m not a big fan of those network kargs myself.)

#7
#8

The second ramdisk simply injects a file into /etc/NetworkManager/system-connections/eno1.nmconnection . I can see that file being copied, before the rootfs is loaded
When rootfs starts to be copied, I enter into that directory and see that the i injected into /etc/NetworkManager/system-connections has been removed.

I tried to reproduce this behavior on FCOS 33.20210104.2.0, and the injected file was still there immediately before exiting the initramfs. @jlebon’s points still stand, but the initramfs itself doesn’t seem to be deleting NM configs for me.