Testing initial full-disk RAID proposal

Project Discussion
1

Hi guys,
first of all thanks for the efforts and recent progress on providing RAID1 support in ignition.

After reading the following discussions:

https://discussion.fedoraproject.org/t/diskless-and-raid/24422  
https://discussion.fedoraproject.org/t/trying-to-setup-full-raid1-installation/18708  
\-- https://github.com/coreos/fedora-coreos-tracker/issues/94  
\-- https://github.com/coreos/fedora-coreos-config/pull/503  
    \-- https://github.com/coreos/coreos-installer/pull/285  
    \-- https://github.com/coreos/coreos-installer/pull/358  
    \-- https://github.com/coreos/fedora-coreos-tracker/issues/581

We ended up trying the RAID-only Ignition setup described at 20201103-full-disk-raid.md by @bgilbert and whose PR has been aproved but not merged yet on a VirtualBox VM (4GB RAM, 3vCores, 2x8GB VDI).

The testing and next streams give the same results:

  • Ignition partitions the disks and creates the RAID arrays correctly :
sda         8:0    0     8G  0 disk  
|-sda1      8:1    0     1M  0 part  
|-sda2      8:2    0   127M  0 part  
| `-md126   9:126  0 126.9M  0 raid1 
|-sda3      8:3    0   384M  0 part  
| `-md127   9:127  0 383.9M  0 raid1 
`-sda4      8:4    0   7.5G  0 part  
  `-md125   9:125  0   7.5G  0 raid1 /sysroot/sysroot/sysroot

sdb         8:16   0     8G  0 disk  
|-sdb1      8:17   0     1M  0 part  
|-sdb2      8:18   0   127M  0 part  
| `-md126   9:126  0 126.9M  0 raid1 
|-sdb3      8:19   0   384M  0 part  
| `-md127   9:127  0 383.9M  0 raid1 
`-sdb4      8:20   0   7.5G  0 part  
  `-md125   9:125  0   7.5G  0 raid1 /sysroot/sysroot/sysroot
  • the disks are labeled correctly :
:/root# ls -Al /dev/disk/by-label
lrwxrwxrwx 1 root root  11 Nov 17 10:03 EFI-SYSTEM -> ../../md126
lrwxrwxrwx 1 root root  11 Nov 17 10:03 boot -> ../../md127
lrwxrwxrwx 1 root root  11 Nov 17 10:03 root -> ../../md125
  • But the Ignition fails :
[   25.593790] ignition[578]: disks: createRaids: op(a): [finished] waiting for devices [/dev/disk/by-partlabel/boot-1 /dev/disk/by-partlabel/boot-2 /dev/disk/by-partlabel/esp-1 /dev/disk/by-partlabel/esp-2 /dev/disk/by-partlabel/root-1 /dev/disk/by-partlabel/root-2]
[ ... ]
[   26.748324] ignition[578]: disks: disks passed
[   26.749002] ignition[578]: Ignition finished successfully
[ ... ]
[   41.477423] systemd[1]: Finished Ignition OSTree: restore rootfs.
[ ... ]
[   46.855771] systemd[1]: Starting CoreOS Inject Rootmap...
[   46.858883] systemd[1]: Starting Reload Configuration from the Real Root...
[ ... ]
[   46.933737] EXT4-fs (md127): mounted filesystem with ordered data mode. Opts: 
[   46.935718] ext4 filesystem being mounted at /tmp/coreos-installer-0pMo8K supports timestamps until 2038 (0x7fffffff)
[   47.010615] rdcore[1550]: Error: appending rootmap kargs
[   47.011686] rdcore[1550]: Caused by: reading directory /tmp/coreos-installer-0pMo8K/loader/entries
[   47.013651] rdcore[1550]: Caused by: No such file or directory (os error 2)
[ ... ]
[   47.041047] systemd[1]: coreos-inject-rootmap.service: Failed with result 'exit-code'.
[   47.044809] systemd[1]: Dependency failed for Ignition Boot Disk Setup.
[   47.046839] systemd[1]: Dependency failed for Ignition Complete.
[   47.049289] systemd[1]: Dependency failed for Initrd Default Target.
[ ... ]
[   52.926748] systemd[1]: Finished Ignition OSTree: Regenerate filesystem UUID (boot).
[   53.749299] systemd[1]: Starting Ignition OSTree: save rootfs...
[   53.764849] systemd[1]: Finished dracut pre-mount hook.
[   53.770446] xfs filesystem being mounted at /sysroot supports timestamps until 2038 (0x7fffffff)
[   53.774371] ignition-ostree-dracut-rootfs[1884]: mount: /sysroot: /dev/md125 already mounted on /sysroot.
[   53.789607] systemd[1]: ignition-ostree-rootfs-save.service: Main process exited, code=exited, status=32/n/a
[   53.801558] systemd[1]: ignition-ostree-rootfs-save.service: Failed with result 'exit-code'.
[   53.810935] systemd[1]: Failed to start Ignition OSTree: save rootfs.

And if I check in the emergency console, the filesystem on /dev/md127 is indeed empty :

:/root# mkdir -p /tmp/coreos-installer-0pMo8K
:/root# mount /dev/md127 /tmp/coreos-installer-0pMo8K
:/root# ls -al /tmp/coreos-installer-0pMo8K
total 13
drwxr-xr-x 3 root root  1024 Nov 17 10:03 .
drwxrwxr-x 3 root root    60 Nov 18 09:45 ..
drwx------ 2 root root 12288 Nov 17 10:03 lost+found

Hence a few questions :

  • what should be written on md127 ?
  • which disk image was used to get that setup working ?
  • is there any documentation on how to bootstrap a full FCOS system manually, skipping the ignition ?
    It would greatly help to know how to properly chroot into the sysroot, configure things and just reboot to test.
  • what can we do to help you to make progress on this ?

Thanks a lot,
The OKDebuggers

2

There’s still work outstanding to wire it through. Notably, you would need to build FCOS with Support mirroring /boot, ESP, BIOS bootloader on first boot by bgilbert · Pull Request #718 · coreos/fedora-coreos-config · GitHub.

3

Thank you for your answer. Do you have any idea when we can expect to have this wired and packed into the Next image ?

We’re still interested in the proper way to chroot into the installed system after running the installer with --preserve-on-error : right now we would like to complete manually what Ignition fails to do but we could not find any documentation on that (something like the debootstrap documentation in Debian for instance :wink: )

All the best,
The OKDebuggers

4

The fedora-coreos-config PR still needs some refactoring which depends on code that hasn’t been written yet, so the PR won’t land for a little while yet. Meanwhile, if you’d like to help with testing, you can build your own FCOS image as @jlebon suggested.

--preserve-on-error won’t help here. coreos-installer isn’t responsible for any part of RAID setup; that happens later, when Ignition runs. And in general, we don’t encourage manually configuring a machine or manually fixing up a machine after an Ignition failure, since we recommend that all provisioning be performed via the Ignition config.

5

–preserve-on-error won’t help here. coreos-installer isn’t responsible
for any part of RAID setup; that happens later, when Ignition runs.

Ah thanks, I thought that was telling Ignition not to wipe partitions on failure.

we don’t encourage manually configuring a machine or manually fixing up
a machine after an Ignition failure, since we recommend that
all provisioning be performed via the Ignition config.

I clearly appreciate the logic behind it and agree that fully automating the provisionning is the optimal way but from a sysadmin perspective having an alternative is always a nice option. It’s hard to gather knowledge and get familiar with a system without interacting with it. When something is going wrong in a production environment that’s when you need that knowledge, so having a way to learn and practice how to get things working manually is really valuable in those situations.

Meanwhile, if you’d like to help with testing, you can build
your own FCOS image as @jlebon suggested.

We definitely want to help with testing, so if this is the way we would definitely want to know how to do this.

I found this but it seems to be outdated : Red Hat Customer Portal - Access to 24x7 support and knowledge

There is also this but I’m not familiar with Jenkins : GitHub - coreos/fedora-coreos-pipeline: Build pipeline for Fedora CoreOS

Could you point me to the most appropriate documentation that I should follow to build an FCOS image ?

Thank you !
The OKDebuggers

6

The instructions to build Fedora CoreOS: Building Fedora CoreOS - coreos/coreos-assembler

See also Working with CoreOS Assembler - coreos/coreos-assembler for building FCOS with selected packages or files overrides. In this case you might want to include an Ignition RPM rebuild with the changes and then rebuild an FCOS image with this override and potential changes to the fedora-coreos-config repo. You should then be able to test that in QEMU with the CoreOS Assembler (cosa).

7

Thanks a lot, we will try that !

All the best,
The OKDebuggers

8

Alright we were able to build fedora-coreos-33.20201119.dev.0-qemu.x86_64.qcow2 and test our ignition files in there.

The guide is pretty straightforward and the build system is very well thought.

Now our next step would be to build an image that includes Support mirroring /boot, ESP, BIOS bootloader on first boot by bgilbert · Pull Request #718 · coreos/fedora-coreos-config · GitHub in order to test those changes.

How should we do that ?

Good,
The OKDebuggers

9

All you need to do is get a checkout of that commit in src/config. For example:

cd src/config
git remote add bgilbert https://github.com/bgilbert/fedora-coreos-config
git fetch bgilbert
git checkout bgilbert/raid

Then just rerun cosa build.

10

Thanks a lot, that worked like a charm !

We can confirm that this PR solves our Ignition RAID setup for the system and properly mounts and boots from there.

Are you referring to the FCC sugar ? In our case we generate our Ignition separately on a per-node basis so we only need CoreOS Ignition to support RAID as it is in the PR. If I understand correctly we won’t be able to upgrade our OKD nodes if we install them with a custom image.

Do you think there is a way to have at least the Ignition RAID support in the upcoming FCOS33-stable so that we can deploy OKD nodes with the stable image and RAID sysroot ?

All the best,
The OKDebuggers