Ignition fails at raid creation

Ask Fedora
1

Hi there,

first time FCOS user here. I’m currently trying to compile an ignition file for a test bare metal machine that is supposed to become my home server once it is all in place.

But it fails when setting up the raid.
Excerpt from my config:

    disks:
        - device: /dev/sdb
          wipe_table: true
        - device: /dev/sdc
          wipe_table: true
        - device: /dev/sdd
          wipe_table: true
        - device: /dev/sde
          wipe_table: true
    raid:
        - name: jbod
          level: linear
          devices:
            - /dev/sdb
            - /dev/sdc
            - /dev/sdd
            - /dev/sde

Result in rdsosreport.txt:

[    8.639138] localhost ignition[1085]: disks: createRaids: created device alias for "/dev/sdb": "/run/ignition/dev_aliases/dev/sdb" -> "/dev/sdb"
[    8.639154] localhost ignition[1085]: disks: createRaids: created device alias for "/dev/sdc": "/run/ignition/dev_aliases/dev/sdc" -> "/dev/sdc"
[    8.639168] localhost ignition[1085]: disks: createRaids: created device alias for "/dev/sdd": "/run/ignition/dev_aliases/dev/sdd" -> "/dev/sdd"
[    8.639182] localhost ignition[1085]: disks: createRaids: created device alias for "/dev/sde": "/run/ignition/dev_aliases/dev/sde" -> "/dev/sde"
[    8.639201] localhost ignition[1085]: disks: createRaids: op(13): [started]  creating "jbod"
[    8.639211] localhost ignition[1085]: disks: createRaids: op(13): executing: "mdadm" "--create" "jbod" "--force" "--run" "--homehost" "any" "--level" "linear" "--raid-devices" "4" "/run/ignition/dev_aliases/dev/sdb" "/run/ignition/dev_aliases/dev/sdc" "/run/ignition/dev_aliases/dev/sdd" "/run/ignition/dev_aliases/dev/sde"
[    8.652823] localhost 55-scsi-sg3_id.rules[1485]: WARNING: SCSI device md127 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[    8.789042] localhost systemd-journald[371]: Missed 83 kernel messages
[    8.789042] localhost kernel: md: personality for level -1 is not loaded!
[    8.790014] localhost kernel: md: md127 stopped.
[    8.828408] localhost ignition[1085]: disks: createRaids: op(13): [failed]   creating "jbod": exit status 1: Cmd: "mdadm" "--create" "jbod" "--force" "--run" "--homehost" "any" "--level" "linear" "--raid-devices" "4" "/run/ignition/dev_aliases/dev/sdb" "/run/ignition/dev_aliases/dev/sdc" "/run/ignition/dev_aliases/dev/sdd" "/run/ignition/dev_aliases/dev/sde" Stdout: "mdadm: Defaulting to version 1.2 metadata\n" Stderr: "mdadm: RUN_ARRAY failed: Invalid argument\n"
[    8.829732] localhost ignition[1085]: disks failed

It looks like a kernel module for my selected raid level is missing.
I tried to add different systemd units to work around this (e.g. using “add_drivers+=linear” in some dracut config file or running modprobe in a oneshot unit) but nothing worked or had any impact on the result in rdsosreport.txt.

Any hints what I could be missing?

2

Hello @banzai2004 and welcome to :fedora: !

Would you please post the entire Butane config file?

You can also check out this post to see if it would be helpful.

3

Sure, here’s the full config, slightly redacted:

variant: fcos
version: 1.5.0
passwd:
    users:
        - name: core
          password_hash: $xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
          ssh_authorized_keys:
            - ssh-ed25519 Axxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
          groups:
            - docker
            - sudo
storage:
    directories:
        - path: /var/data
          mode: 0660
          user:
            name: root
          group:
            name: core
    files:
        - path: /etc/hostname
          mode: 0644
          contents:
            inline: frxxxxxxxxxxx
        - path: /etc/vconsole.conf
          mode: 0644
          contents:
            inline: KEYMAP=de
        - path: /etc/zincati/config.d/55-updates-strategy.toml
          contents:
            inline: |
              [updates]
              strategy = "periodic"
              [[updates.periodic.window]]
              days = [ "Mon", "Tue", "Wed" ]
              start_time = "01:00"
              length_minutes = 120
        - path: /root/samba.docker-compose.yml
          contents:
            inline: |
              services:
                samba:
                  image: dockurr/samba
                  container_name: my_samba
                  environment:
                    NAME: "Data"
                    USER: "samba"
                    PASS: "secret"
                  ports:
                    - 445:445
                  volumes:
                    - /home/example:/storage
                    - /root/smb.conf:/etc/samba/smb.conf
                    - /root/samba_users.conf:/etc/samba/users.conf
        - path: /root/samba_users.conf
          contents:
            inline: |
              #username:UID:groupname:GID:password
              bka:1001:p_users:103:bka_secret
              sxxxxxxx:1002:p_users:103:sxxxxxxxxxxxx_secret
        - path: /root/smb.conf
          contents:
            local: smb.conf
        - path: /etc/dracut.conf.d/linear.conf
          mode: 0644
          contents:
            inline: |
              # Dracut configuration to load linear RAID module
              add_drivers+="linear"
    disks:
        # - device: /dev/sda
          # wipe_table: true
          # partitions:
            # - label: system
              # wipe_partition_entry: true
        - device: /dev/sdb
          wipe_table: true
#          partitions:
#            - label: data
#              wipe_partition_entry: true
        - device: /dev/sdc
          wipe_table: true
#          partitions:
#            - label: data
#              wipe_partition_entry: true
        - device: /dev/sdd
          wipe_table: true
#          partitions:
#            - label: data
#              wipe_partition_entry: true
        - device: /dev/sde
          wipe_table: true
#          partitions:
#            - label: data
#              wipe_partition_entry: true
    raid:
        - name: jbod
          level: linear
          devices:
            - /dev/sdb
            - /dev/sdc
            - /dev/sdd
            - /dev/sde
    filesystems:
        - device: /dev/sda1
          format: ext4
          label: system
        - path: /var/data
          device: /dev/md/jbod
          format: xfs
          label: data
          with_mount_unit: true
systemd:
  units:
    # - name: load-linear-module.service
      # enabled: true
      # contents: |
        # [Unit]
        # Description=Load Linear RAID Kernel Module
        # After=coreos-installer-pre.service
        # Before=coreos-installer.service
        # RequiredBy=coreos-installer.service
        
        # [Service]
        # Type=oneshot
        # ExecStart=/usr/sbin/modprobe linear
        # RemainAfterExit=yes
        
        # [Install]
        # WantedBy=multi-user.target
    # Installing docker-compose as a layered package with rpm-ostree
    - name: rpm-ostree-install-docker-compose.service
      enabled: true
      contents: |
        [Unit]
        Description=Layer docker-compose with rpm-ostree
        Wants=network-online.target
        After=network-online.target
        # We run before `zincati.service` to avoid conflicting rpm-ostree
        # transactions.
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp
        
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        # `--allow-inactive` ensures that rpm-ostree does not return an error
        # if the package is already installed. This is useful if the package is
        # added to the root image in a future Fedora CoreOS release as it will
        # prevent the service from failing.
        ExecStart=/usr/bin/rpm-ostree install -y --allow-inactive docker-compose
        ExecStart=/bin/touch /var/lib/%N.stamp
        ExecStart=/bin/systemctl --no-block reboot
        
        [Install]
        WantedBy=multi-user.target
    - name: samba.service
      enabled: true
      contents: |
        [Unit]
        Description=Samba FileServer
        After=network-online.target
        Wants=network-online.target
        
        [Service]
        TimeoutStartSec=0
        ExecStartPre=-/bin/docker kill my_samba
        ExecStartPre=-/bin/podman rm my_samba
        ExecStartPre=/bin/docker pull dockurr/samba
        ExecStart=/bin/docker-compose -f /root/samba.docker-compose.yml up
        
        [Install]
        WantedBy=multi-user.target

the commandline I used to install:
sudo coreos-installer install /dev/sda --ignition-url http://local-pc:8000/transpiled_config.ign --insecure-ignition

4

As far as I can see, you want to make a RAID array of /dev/sdb, /dev/sdc, /dev/sdd and /dev/sde, but not /dev/sda which is the boot device, and set up the array as a separate /var mount. Is this correct?

5

Yes, correct

6

Digging into this I found that some md personalities are deprecated from the upstream Kernel, see md: mark some personalities as deprecated · torvalds/linux@608f52e · GitHub. Perhaps we should consider removing the explicit mention of the linear array redundancy level from the Butane and Ignition docs.

Not sure if it would be suitable for your use case, but here is a simplified Butane config for RAID 0 level that you can use as a reference.

Butane config file 
variant: fcos
version: 1.6.0

# Authentication
passwd:
  users:
    - name: core
      ssh_authorized_keys: [ssh-ed25519 AAAA...]

storage:
  disks:
    # The link to the block device the OS was booted from.
  - device: /dev/disk/by-id/coreos-boot-disk
    # We do not want to wipe the partition table since this is the primary device.
    wipe_table: false
    
    partitions:
    
    - number: 4
      label: root
      # Allocate at least 8 GiB to the rootfs. See NOTE above about this.
      size_mib: 8192
      resize: true
      
  # This defines two partitions, each on its own disk.
  - device: /dev/vdb
    wipe_table: true
    partitions:
      # Each partition gets a human-readable label.
    - label: "raid-linear-1"
      # Each partition is placed at the beginning of the disk.
      number: 1
      size_mib: 0
      start_mib: 0
      
  - device: /dev/vdc
    wipe_table: true
    partitions:
    - label: "raid-linear-2"
      number: 1
      size_mib: 0
      start_mib: 0

  directories:
  - path: /var/data

  # We use the previously defined partitions as devices in a Linear RAID md array.
  raid:
    - name: md-data
      level: raid0
      devices:
      - /dev/disk/by-partlabel/raid-linear-1
      - /dev/disk/by-partlabel/raid-linear-2

  # Configuring the filesystems.
  filesystems:
    
    # Changing the root filesystem to ext4.
    - device: /dev/disk/by-partlabel/root
      wipe_filesystem: true
      format: ext4
      label: root
      
    # The resulting md array is used to create an ext4 filesystem.
    - device: /dev/md/md-data
      path: /var/data
      label: DATA
      format: ext4
      with_mount_unit: true

I tested it on a virtual machine.

core@localhost:~$ sudo lsblk -pfa 
core@localhost:~$ sudo lsblk -pfa
NAME           FSTYPE            FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
/dev/loop0     erofs                                                                                   
/dev/vda                                                                                               
├─/dev/vda1                                                                                            
├─/dev/vda2    vfat              FAT16 EFI-SYSTEM  7B77-95E7                                           
├─/dev/vda3    ext4              1.0   boot        bba81ae6-b86f-495d-95bb-186c11dfa660  210.1M    33% /boot
└─/dev/vda4    ext4              1.0   root        42265ff8-bccb-4908-b6c6-e76f3c1960d6    5.8G    20% /var
                                                                                                       /sysroot/ostree/deploy/fedora-coreos/var
                                                                                                       /etc
                                                                                                       /sysroot
/dev/vdb                                                                                               
└─/dev/vdb1    linux_raid_member 1.2   any:md-data bc928f4d-f41b-2584-c619-682a904bbbe0                
  └─/dev/md127 ext4              1.0   DATA        f4c66015-b15c-42ba-bb15-8f4a5bbb153b   18.5G     0% /var/data
/dev/vdc                                                                                               
└─/dev/vdc1    linux_raid_member 1.2   any:md-data bc928f4d-f41b-2584-c619-682a904bbbe0                
  └─/dev/md127 ext4              1.0   DATA        f4c66015-b15c-42ba-bb15-8f4a5bbb153b   18.5G     0% /var/data
1 Like
7

Thanks a lot for digging into this.
Your example with raid0 instead of linear works fine for me. I will stick to raid0 for now and first finish the rest of my config before checking how I could use device mapper to get the same behavior as “linear”.

Have a nice christmas time and thanks again!

1 Like