Issue persisting podman networks when PXE booting live images

Project Discussion
1

Hi,

I am trying to achieve a (mostly) “stateless” raspberry pi server setup and currently have the following working:

  • Raspberry Pi firmware is set to network boots and loads EDK2 UEFI from TFTP
  • EDK2 then PXE boots CoreOS over TFTP
  • CoreOS downloads an ignition file
  • this sets up a portainer edge agent container, that connects to a central portainer instance that then deploys a docker compose stack
  • I also have /var setup as persistent via a USB drive to store the container image files

The issue comes when I try to reboot the server. On the second boot it fails to launch the containers because the docker-compose stack created a network that no longer exists.

Turns out these exist in /etc/containers/networksas configured by /usr/share/containers/containers.conf and this directory gets wiped as its running on tempfs in RAM when the server reboots. I have tried using ignition to modify containers.conf but since this lives on /usr it is read-only and ignition fails to run.

Does anyone have any ideas on how I might work around this issue? It seems like something that should be supported. The documentation discusses this usecase of running via PXE and persisting containers here: Installing CoreOS on Bare Metal :: Fedora Docs and Running Fedora CoreOS directly from RAM :: Fedora Docs

1 Like
2

you should be able to override anything set in /usr/share/containers/containers.conf. The big hammer approach is to just get a copy of it, tweak, and then put it in /etc/containers/containers.conf using Ignition.

See podman — Podman documentation

1 Like
3

Ah! So simple, thank you! This solved my problem.

In case anyone else ever comes across this problem. This is what I needed in my butane file in order for networks to persist and existing containers to start on boot:

storage:
    disks:
      - device: /dev/sda
        wipe_table: true
        partitions:
          - label: var
    filesystems:
      - device: /dev/disk/by-partlabel/var
        label: var
        format: xfs
        wipe_filesystem: false
        path: /var
        with_mount_unit: true
    directories:
      # Store persistent networks
      - path: /var/lib/containers/networks
        overwrite: false
    files:
      # Make podman store its network config in our persistent directory
      - path: /etc/containers/containers.conf
        mode: 0644
        contents:
          inline: |
            [network]
            network_config_dir = "/var/lib/containers/networks"
  
  systemd:
    units:  
      # At boot, start containers with `--reset=always`
      - name: podman-restart.service
        enabled: true