Recommended pattern for moving container storage to a separate LUKS data disk on CoreOS?

Ask Fedora
1

Hi,

I am looking for the recommended Fedora CoreOS pattern for separating container storage/application data from the OS disk.

My goal is:

  • Keep the FCOS root disk small
  • Put container images, writable layers, volumes, and other growing application data on a separate virtual disk
  • Encrypt both disks with LUKS + Clevis/Tang
  • Have the secondary data disk be independently expandable and growable live

My first approach was to mount the second LUKS volume directly on: /var/lib because this is where container runtimes normally store their data.

This mostly works during normal operation, but I noticed issues on the first reboot immediately following an rpm-ostree update. The new deployment boot does not always complete correctly (including LUKS auto-unlock behavior for the secondary disk). A subsequent reboot usually works.

This made me suspect that replacing /var/lib introduces ordering/dependency issues during the first boot into a new OSTree deployment.

Is mounting a separate filesystem over /var/lib considered unsupported or discouraged on FCOS?

Specifically:

  1. What is the recommended location for a separate application data disk?
  2. What is the recommended way to relocate container storage on FCOS?
  3. For an encrypted secondary disk unlocked with Clevis/Tang, what is the recommended systemd ordering pattern?
  4. Are there existing examples or recommended Ignition/Butane snippets for this layout?

I would like to follow the intended FCOS design rather than fighting the immutable OS model.

Thanks!

2

Note there is currently a bug in systemd that affects mounting over /var (and i assume also /var/lib) from a luks encrypted volume. We are waiting on a fix to be made.

See 2487442 – fail to boot when / and /var are encrypted

3

Thanks for the heads-up and the Bugzilla link! That’s an absolute lifesaver. I am tracking this now. Appreciate it!

4

After quite a bit of troubleshooting trying to mount a secondary network-encrypted (Clevis/Tang) volume directly onto /var or /var/lib, I repeatedly hit unresolvable systemd ordering cycles during boot (tracked via journalctl -b 0 | grep -i "cycle").

The Root Cause

Mounting a network-encrypted drive anywhere under /var creates a conflict with systemd’s default lifecycle assumptions. By design, systemd treats /var subdirectories as local OS targets (local-fs.target), meaning it tries to mount them before initializing network services.

However, because a Tang-backed volume explicitly relies on NetworkManager and gssproxy to fetch decryption keys, systemd hits an unresolvable circular dependency loop: it cannot start the network stack until the local filesystems are mounted, but it cannot mount the filesystem until the network stack provides the key.

The Solution

Instead of battling systemd’s local OS tracking constraints, the cleanest approach is to move the mount location out of the early boot tracks entirely (e.g., to /var/mnt/data), configure it as a remote device using _netdev, and tell Docker and Containerd to look at the new path instead.

To achieve this cleanly, you must also drop Butane’s automatic mount unit generation (with_mount_unit: false) so that Butane doesn’t force it back into local-fs.target.

Here is the finalized working Butane configuration for anyone else fighting storage dependency loops:

---
variant: fcos
version: 1.7.0

storage:
  disks:
    - device: /dev/sdb
      wipe_table: false
      partitions:
        - label: data
          number: 1
          size_mib: 0
          resize: true

  luks:
    - name: root_crypt
      device: /dev/disk/by-partlabel/root
      discard: true
      wipe_volume: true
      clevis:
        custom:
          pin: tang
          # Thumbprint extracted using `tang-show-keys 8080` on tang_server
          config: '{"url": "<url>", "thp": "<thp>"}'
          needs_network: true
    - name: data_crypt
      device: /dev/disk/by-partlabel/data
      discard: true
      clevis:
        custom:
          pin: tang
          config: '{"url": "<url>", "thp": "<thp>"}'
          needs_network: true

  filesystems:
    - device: /dev/mapper/root_crypt
      format: ext4
      label: root
      wipe_filesystem: true
    - device: /dev/mapper/data_crypt
      path: /var/mnt/data
      format: ext4
      label: data
      with_mount_unit: false # Must be false to override the default systemd boot target

  directories:
    - path: /etc/docker
    - path: /etc/containerd

  files:
    - path: /etc/zincati/config.d/90-disable-auto-updates.toml
      contents:
        inline: |
          [updates]
          enabled = false

    - path: /etc/docker/daemon.json
      contents:
        inline: |
          {
            "data-root": "/var/mnt/data/docker"
          }

    - path: /etc/containerd/config.toml
      overwrite: true
      contents:
        inline: |
          version = 2
          root = "/var/mnt/data/containerd"
          state = "/run/containerd"

          [plugins]
            [plugins."io.containerd.grpc.v1.cri"]
              [plugins."io.containerd.grpc.v1.cri".cni]
                bin_dir = "/usr/libexec/cni/"
                conf_dir = "/etc/cni/net.d"
            [plugins."io.containerd.internal.v1.opt"]
              path = "/var/mnt/data/containerd/opt"          

systemd:
  units:
    - name: zincati.service
      mask: true
      enabled: false
    - name: console-login-helper-messages-gensnippet-os-release.service
      mask: true
      enabled: false
    - name: console-login-helper-messages-gensnippet-ssh-keys.service
      mask: true
      enabled: false
    - name: bootloader-update.service
      dropins:
        - name: 10-wait-for-boot.conf
          contents: |
            [Unit]
            RequiresMountsFor=/boot

    - name: var-mnt-data.mount
      enabled: true
      contents: |
        [Unit]
        Description=Network Decrypted Container Storage Volume
        DefaultDependencies=no
        Requires=systemd-fsck@dev-mapper-data_crypt.service
        After=systemd-fsck@dev-mapper-data_crypt.service remote-fs-pre.target network-online.target
        Wants=network-online.target
        Conflicts=umount.target

        [Mount]
        Where=/var/mnt/data
        What=/dev/mapper/data_crypt
        Type=ext4
        Options=_netdev,rootcontext=system_u:object_r:container_file_t:s0

        [Install]
        WantedBy=remote-fs.target # Re-binds to late-stage network storage boot targets

    - name: docker.service
      dropins:
        - name: 10-storage-delay.conf
          contents: |
            [Unit]
            Wants=var-mnt-data.mount
            After=var-mnt-data.mount

    - name: containerd.service
      dropins:
        - name: 10-storage-delay.conf
          contents: |
            [Unit]
            Wants=var-mnt-data.mount
            After=var-mnt-data.mount
5

Once the proposed fix in systemd is back ported you will not need a work around any more.