FCOS 39: Ignition Won't Fully Config Users Home With BTRFS Subvolumes

Hey guys. I’m having an issue with ignition and btrfs subvolumes. I know there is some interest on this issue here at https://github.com/coreos/ignition/issues/890. I’m testing a live iso environment with a persistent /var but for some reason, after booting the iso, /var/home is empty. I can see in the journalctl logs that my users get created from my ignition file but before my systemd.mount units for /var/ and /var/home with the respective Options=subvol= mount option. The btrfs subvolumes for the mountpoints are both toplevel, not nested on a single btrfs partition.

Here is the butane file:

variant: fcos
version: 1.5.0
# Setup our core user to autologin after boot.
systemd:
  units:
    - name: getty@tty1.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override ExecStart in main unit file.
          ExecStart=
          # Add new ExecStart with "-" prefix to ignore failure.
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
    - name: var.mount
      contents: |
        [Unit]
        Description=Mount disk to /var.

        [Mount]
        What=/dev/disk/by-id/$DISKID
        Where=/var
        Type=btrfs
        Options=rw,relatime,subvol=st_var

        [Install]
        WantedBy=multi-user.target
      enabled: true
    - name: var-home.mount
      contents: |
        [Unit]
        Description=Mount disk to /var/home.

        [Mount]
        What=/dev/disk/by-id/$DISKID
        Where=/var/home
        Type=btrfs
        Options=rw,relatime,subvol=st_var_home

        [Install]
        WantedBy=multi-user.target
      enabled: true
passwd:
  users:
    - name: core
      password_hash: ...
      ssh_authorized_keys:
        - ssh-ecdsa AAA...
    - name: pihole
      ssh_authorized_keys:
        - ssh-ecdsa AAA...
storage:
  directories:
    # Create necessary directories for user 'pihole'.
    - path: /home/pihole/.config
      mode: 0700
      user:
        name: pihole
      group:
        name: pihole
    - path: /home/pihole/.config/containers
      mode: 0700
      user:
        name: pihole
      group:
        name: pihole
    - path: /home/pihole/.config/containers/systemd
      mode: 0700
      user:
        name: pihole
      group:
        name: pihole
  files:
    # Set our 'KEYMAP' to 'us'.
    - path: /etc/vconsole.conf
      mode: 0644
      contents:
        inline: KEYMAP=us
    # Create a podman-quadlet file for our 'pihole' user
    # to start our pihole service.
    - path: /home/pihole/.config/containers/systemd/pihole-ftl.container
      mode: 0644
      contents:
        inline: |
          ...
      user:
        name: pihole
      group:
        name: pihole
    - path: /var/lib/systemd/linger/pihole
      mode: 0644

EDIT: Following this guide https://docs.fedoraproject.org/en-US/fedora-coreos/live-booting/#_using_persistent_state instead of the systemd.mount units (using format: btrfs), the iso boots fine with expected behavior.

I’m thinking either:

  • ignition is exectuing the mount units after user creation and if the btrfs subvolume is empty on first real boot of the system, the empty subvolume for /var/home replaces the system’s already created /var/home OR
  • there is a permission issue using manually created btrfs subvolumes from a separate machine.
    Whatever logic there is to get user’s home directory to /var/home isn’t allowed due to some permission issue.

Or maybe I’m missing something here.

EDIT #2: It seems like buggy behavior. Having the same device: /dev/disk/by-id/$DISKID line for both - path: /var and - path: /var/home as recommended here creates the error:

error at $.storage.filesystems.1, line 149 col 7: duplicate entry defined
Error translating config: config generated was invalid

where line 149 is the second path - path: /var/home. When I change it to:

filesystems:
  - path: /var
    device: /dev/disk/by-id/$DISKID (symlink to /dev/sda)
    ...
  - path: /var
    device: /dev/sda
    ...

it works as expected. It won’t work if I have more the 2 subvolumes though.

I think the reason the filesystems aren’t gettting mounted in the initramfs (i.e. before ignition-files gets run and users get created) is because they aren’t specified in the storage/filesystems section of the Ignition config. Instead of defining the systemd mount units directly try something like this and see if it works:

variant: fcos
version: 1.5.0
systemd:
  units:
    - name: getty@tty1.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override ExecStart in main unit file.
          ExecStart=
          # Add new ExecStart with "-" prefix to ignore failure.
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
passwd:
  users:
    - name: core
      password_hash: ...
      ssh_authorized_keys:
        - ssh-ecdsa AAA...
storage:
  filesystems:
    - path: /var
      device: /dev/disk/by-id/$DISKID
      format: btrfs
      mount_options: rw,relatime,subvol=st_var
      wipe_filesystem: false
      with_mount_unit: true
    - path: /var/home
      device: /dev/disk/by-id/$DISKID
      format: btrfs
      mount_options: rw,relatime,subvol=st_var_home
      wipe_filesystem: false
      with_mount_unit: true
1 Like

Couldn’t get butane to build to ignition as I get with your config:

error at $.storage.filesystems.1, line 149 col 7: duplicate entry defined
Error translating config: config generated was invalid

Line 149 is - path: /var/home but I believe it might be complaining about the device: /dev/disk/by-id/$DISKID line right after (line 150).

This might be a bug.

ok yeah I had mount_options as just a string where it is actually a list of strings. so mount_options: [rw,relatime,subvol=st_var] is what we need.

Though I’m hitting another problem. It looks like we can’t define a filesytem mount for the same device (which makes sense for anything but btrfs with subvolumes I guess). Maybe you can use /dev/disk/by-id/ for one of the entires and /dev/disk/by-uuid for the other?

variant: fcos
version: 1.5.0
systemd:
  units:
    - name: getty@tty1.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override ExecStart in main unit file.
          ExecStart=
          # Add new ExecStart with "-" prefix to ignore failure.
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
passwd:
  users:
    - name: core
      password_hash: ...
      ssh_authorized_keys:
        - ssh-ecdsa AAA...
storage:
  filesystems:
    - path: /var
      device: /dev/disk/by-id/$DISKID
      format: btrfs
      mount_options: [rw,relatime,subvol=st_var]
      wipe_filesystem: false
      with_mount_unit: true
    - path: /var/home
      device: /dev/disk/by-uuid/$DISKUUID
      format: btrfs
      mount_options: [rw,relatime,subvol=st_var_home]
      wipe_filesystem: false
      with_mount_unit: true
1 Like

Yea that’s what I figured to do after seeing the duplicate error. It’s in EDIT #2 in the original post. Thanks for the guidance on this. Is this something that can maybe be added in ignition? To work with the same device with different btrfs subvolumes?

@arey1027 I found this open feature request: Support creating and mounting Btrfs subvolumes · Issue #890 · coreos/ignition · GitHub so I guess add your input and follow there.

by the way if using /dev/disk/by-id/name and then /dev/disk/by-uuid/uuid works then can you report over in the issue that as a workaround ?

Yea no problem. Will do. Thanks Dusty for helping with this!