Hi guys,
first of all thanks for the efforts and recent progress on providing RAID1 support in ignition.
After reading the following discussions:
https://discussion.fedoraproject.org/t/diskless-and-raid/24422
https://discussion.fedoraproject.org/t/trying-to-setup-full-raid1-installation/18708
\-- https://github.com/coreos/fedora-coreos-tracker/issues/94
\-- https://github.com/coreos/fedora-coreos-config/pull/503
\-- https://github.com/coreos/coreos-installer/pull/285
\-- https://github.com/coreos/coreos-installer/pull/358
\-- https://github.com/coreos/fedora-coreos-tracker/issues/581
We ended up trying the RAID-only Ignition setup described at 20201103-full-disk-raid.md by @bgilbert and whose PR has been aproved but not merged yet on a VirtualBox VM (4GB RAM, 3vCores, 2x8GB VDI).
The testing
and next
streams give the same results:
- Ignition partitions the disks and creates the RAID arrays correctly :
sda 8:0 0 8G 0 disk
|-sda1 8:1 0 1M 0 part
|-sda2 8:2 0 127M 0 part
| `-md126 9:126 0 126.9M 0 raid1
|-sda3 8:3 0 384M 0 part
| `-md127 9:127 0 383.9M 0 raid1
`-sda4 8:4 0 7.5G 0 part
`-md125 9:125 0 7.5G 0 raid1 /sysroot/sysroot/sysroot
sdb 8:16 0 8G 0 disk
|-sdb1 8:17 0 1M 0 part
|-sdb2 8:18 0 127M 0 part
| `-md126 9:126 0 126.9M 0 raid1
|-sdb3 8:19 0 384M 0 part
| `-md127 9:127 0 383.9M 0 raid1
`-sdb4 8:20 0 7.5G 0 part
`-md125 9:125 0 7.5G 0 raid1 /sysroot/sysroot/sysroot
- the disks are labeled correctly :
:/root# ls -Al /dev/disk/by-label
lrwxrwxrwx 1 root root 11 Nov 17 10:03 EFI-SYSTEM -> ../../md126
lrwxrwxrwx 1 root root 11 Nov 17 10:03 boot -> ../../md127
lrwxrwxrwx 1 root root 11 Nov 17 10:03 root -> ../../md125
- But the Ignition fails :
[ 25.593790] ignition[578]: disks: createRaids: op(a): [finished] waiting for devices [/dev/disk/by-partlabel/boot-1 /dev/disk/by-partlabel/boot-2 /dev/disk/by-partlabel/esp-1 /dev/disk/by-partlabel/esp-2 /dev/disk/by-partlabel/root-1 /dev/disk/by-partlabel/root-2]
[ ... ]
[ 26.748324] ignition[578]: disks: disks passed
[ 26.749002] ignition[578]: Ignition finished successfully
[ ... ]
[ 41.477423] systemd[1]: Finished Ignition OSTree: restore rootfs.
[ ... ]
[ 46.855771] systemd[1]: Starting CoreOS Inject Rootmap...
[ 46.858883] systemd[1]: Starting Reload Configuration from the Real Root...
[ ... ]
[ 46.933737] EXT4-fs (md127): mounted filesystem with ordered data mode. Opts:
[ 46.935718] ext4 filesystem being mounted at /tmp/coreos-installer-0pMo8K supports timestamps until 2038 (0x7fffffff)
[ 47.010615] rdcore[1550]: Error: appending rootmap kargs
[ 47.011686] rdcore[1550]: Caused by: reading directory /tmp/coreos-installer-0pMo8K/loader/entries
[ 47.013651] rdcore[1550]: Caused by: No such file or directory (os error 2)
[ ... ]
[ 47.041047] systemd[1]: coreos-inject-rootmap.service: Failed with result 'exit-code'.
[ 47.044809] systemd[1]: Dependency failed for Ignition Boot Disk Setup.
[ 47.046839] systemd[1]: Dependency failed for Ignition Complete.
[ 47.049289] systemd[1]: Dependency failed for Initrd Default Target.
[ ... ]
[ 52.926748] systemd[1]: Finished Ignition OSTree: Regenerate filesystem UUID (boot).
[ 53.749299] systemd[1]: Starting Ignition OSTree: save rootfs...
[ 53.764849] systemd[1]: Finished dracut pre-mount hook.
[ 53.770446] xfs filesystem being mounted at /sysroot supports timestamps until 2038 (0x7fffffff)
[ 53.774371] ignition-ostree-dracut-rootfs[1884]: mount: /sysroot: /dev/md125 already mounted on /sysroot.
[ 53.789607] systemd[1]: ignition-ostree-rootfs-save.service: Main process exited, code=exited, status=32/n/a
[ 53.801558] systemd[1]: ignition-ostree-rootfs-save.service: Failed with result 'exit-code'.
[ 53.810935] systemd[1]: Failed to start Ignition OSTree: save rootfs.
And if I check in the emergency console, the filesystem on /dev/md127
is indeed empty :
:/root# mkdir -p /tmp/coreos-installer-0pMo8K
:/root# mount /dev/md127 /tmp/coreos-installer-0pMo8K
:/root# ls -al /tmp/coreos-installer-0pMo8K
total 13
drwxr-xr-x 3 root root 1024 Nov 17 10:03 .
drwxrwxr-x 3 root root 60 Nov 18 09:45 ..
drwx------ 2 root root 12288 Nov 17 10:03 lost+found
Hence a few questions :
- what should be written on
md127
? - which disk image was used to get that setup working ?
- is there any documentation on how to bootstrap a full FCOS system manually, skipping the ignition ?
It would greatly help to know how to properly chroot into thesysroot
, configure things and just reboot to test. - what can we do to help you to make progress on this ?
Thanks a lot,
The OKDebuggers