Fedora Server RAID1 on NVMe – install-time setup for root on mdadm?

Hey all, long time no seen…

I’ve been working on a Fedora Server 42 setup, running Cockpit, with a focus on resilience: software mirrored NVMe drives (RAID1 with mdadm), bootable and fully functional even if one drive dies. After quite a bit of hands-on work — supported by guides, documentation, and ChatGPT for step-by-step checks — I got the system running with:

  • / on LVM on top of mdadm RAID1 (/dev/md0)
  • Booting via UEFI (shimx64.efi)
  • GRUB seeing both old and new VGs
  • Degraded RAID shown in Cockpit (1 disk listed, but both are physically present)
  • And a slightly messy boot menu with entries pointing to both the old and new root
  • muspell is the host name

It does boot and function, but it’s not working as intended.

Tried:

  • Creating a RAID1 array (mdadm)
  • Creating a volume group (vg_mirror)
  • Putting a logical volume (root) on it
  • Copying the system over
  • Editing /etc/fstab to point to the new root
  • Trying to update the GRUB config and initramfs

What I want to do now:
Reinstall cleanly and set everything up during install. No post-install fixes, no mdadm tweaking in rescue mode. My goal is:

  • Clean RAID1 / mirror root setup at install time,
  • GRUB and EFI fully aware of the RAID mirror
  • Reliable dracut/initramfs config for boot
  • No manual intervention after reboots or updates

Questions:

  1. Is this achievable purely within Anaconda’s custom partitioning - cli or btrfs?
  2. Any tricks for initramfs or bootloader configs during install?
  3. Anyone succeeded with this setup and willing to share their flow?

The system is Supermicro-based (EPYC), with two NVMe drives on a SuperMicro AOC-SLG3-2M2 PCI card. I’ve gone through at least four reinstalls trying to make this clean — and while the exercise has been educational (and ChatGPT has been helpful to not drown in detail), I’m now looking for best practice.

All of the above is so easy to do with zfs in TrueNAS Scale and Proxmox, but I would like to stick with Fedora.

Thanks in advance

I don’t understand this, I expect you have only one VG on top of the raid.

Can you share the output of lsblk -f?

Here goes

NAME                    FSTYPE            FSVER    LABEL     UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                                                
sdb                                                                                                                
sdc                                                                                                                
sdd                                                                                                                
sde                                                                                                                
sdf                                                                                                                
zram0                   swap              1        zram0     df9a3dd7-83f5-433c-9831-919412f4c7e4                  [SWAP]
nvme0n1                                                                                                            
├─nvme0n1p1             vfat              FAT32              E019-2416                               591.3M     1% /boot/efi
├─nvme0n1p2             xfs                                  d4c3c467-c61f-4bac-b2a8-0ba8af5b35ee    449.9M    41% /boot
└─nvme0n1p3             LVM2_member       LVM2 001           fev0kw-mZCk-Hd65-rBHC-wQh4-KALM-PgQkeI                
  └─fedora_muspell-root xfs                                  949a7bb2-dfef-4655-8b0c-2dde9fb5d205     11.5G    22% /
nvme1n1                                                                                                            
├─nvme1n1p1             vfat              FAT32              3448-9BC4                               591.3M     1% /mnt/boot/efi
├─nvme1n1p2             xfs                                  26b1ff0c-54f9-41af-b2c7-0cd76828671c      448M    42% /mnt/boot
└─nvme1n1p3             linux_raid_member 1.2      muspell:0 4b0bdba1-7823-3f4f-5a44-77d28e7cac9b                  
  └─md0                 LVM2_member       LVM2 001           6b1pdq-SXPL-vI08-3Udf-l6fN-KKec-uDSH7F                
    └─vg_mirror-root    xfs                                  b1166264-cac1-414f-8849-04175cc043e6     27.9G    12% /mnt

Should perhaps mention that
sda
sdb
sdc
sdd
sde
sdf
are not used for the moment and not intended to be used for this.

You don’t have a RAID across the 2 nvme drives.

The steps I would use are:

In bios setup the two nvme as a raid.
Install lvm so you can add and remove partitions on the whole raid
Add the partitions you want in the lvm.
Complete the install.

Thanks for the reply, but I think there’s a misunderstanding of what I’m trying to do here…

I’m not using BIOS or motherboard RAID. That NVME card is not a harware RAID card.

This setup is based on what I think should work with mdadm software RAID1 (level 1 mirror), specifically targeting two NVMe drives. The goal is to mirror the root filesystem across them using mdadm and LVM, while also ensuring the system can boot reliably from either disk.

Which can be done very easily with Proxmox or TrueNas Scale, so why not in Fedora?

What I’ve been working through (with help and references from guides and GPT-assisted troubleshooting) is:

  • Creating a RAID1 array manually (mdadm) for the root
  • Building an LVM volume on top of that array
  • Migrating an existing Fedora install onto it
  • Configuring grub, initramfs (via dracut), and fstab accordingly
  • Dealing with bootloader and EFI entry quirks to allow failover booting

My main issue is Fedora’s installer (Anaconda) doesn’t provide native support for this kind of layout with NVMe drives during install — or at least, not in a straightforward way. That’s what I’m trying to improve, or at least document to get past this hurdle.

Open to tips on setting this up during install, or improving bootloader reliability post-migration. Thanks again.

The BIOS RAID is the same as mdadm RAID.

By setting up in the BIOS you can boot from a RAID’ed /boot.
Once Linux kernel comes up it takes over the RAID from the UEFI BIOS.

The spec for the RAID comes from Intel and it’s fully support in the linux kernel. The tech is known as The Intel® Rapid Storage Technology (Intel® RST) .

Thanks, but I’m not using Intel RST or a chipset that supports it. This is an AMD EPYC platform with NVMe drives and a Linux-native mdadm RAID1 setup. My goal is to boot Fedora Server from a mirrored root using mdadm — not firmware RAID. I’ve already created the array, copied the system, and updated grub and initramfs, but it doesn’t boot reliably. That’s what I’m trying to solve.

I suspect that what you are trying to do is relatively easy with the installer.
You would need to do a custom install.

Since you are not using bios raid, the efi and /boot partitions would not be set as raid, and should be on only one drive.
Then create the raid on both drives in the remaining space.
Finally, since you also seem to be using LVM you would need to designate that raid device as the PV, then create the VG on that raid device and add the 2 LVs for / and /home onto that VG.

Finally, after doing all the partitions and LVM management you could do the installation onto the predefined partitions. (the LVM management can likely be done during the installation, but the raid would probably need to be done before)

It may be necessary to do all the partitioning manually from the install media before starting the actual installation, which is relatively simple.

1 Like

I was curious and went looking.
It seems that there is a AMD BIOS RAID feature as well that linux supports.

What is your motherboard?

This would be the easiest option if there’s nothing important on those root disks. The Anaconda installer (at least the old one, I haven’t used the new one) is quite powerful when it comes to partitioning, and very easy. I have two 500GB SSDs in raid 1 for Fedora Server. Then I use raid 6 for 4x16TB HDDs which I use for storage, which is unimaginatively mounted as /storage on my home server. All of that was done in Anaconda using the GUI pre-install.

2 Likes

@barryascott

@computersavvy What I have tried is using manual partitioning in btrfs, but not getting it quite right. The image is not what I did, but only to show where:


Maybe I just messed up the different parts of it all.

@brianallenby Yup, that would seem to be right, care to elaborate on how you solved this?

1 Like

I use the Advanced Custom (Blivet-GUI) option, rather than the Custom option. Give me a minute to spin up a VM.

Ok, under the following Blivet screen…

…when you load in there, it should list all your disks on the left hand side. Click one of the disks you want to use for your OS/root RAID. Then select the plus sign under Logical View. It will list available devices and device type. That’s where you make the choice of disks you want to use for your OS/root RAID. Then select your filesystem, label it and for mount point just set it as /.

You can do the same thing for any other disk pairs/raid configurations you want to do the exact same way. As I mentioned in my other post, when I installed my home server, I did my OS/root RAID, and then did the exact same thing for my storage pool RAID, just instead of selecting two drives, I selected the four disks from the available devices.

I might be mis-remembering some bits since it’s been a while since I did it on bare metal (apologies for the VM screens), so just post a screenshot of any part you get stuck.

The Wiki has a bare bones piece on Blivet that features a screenshot of what it’s more likely to look like for you when you start building RAID’s.

Edit: I had intended a more indepth guide but Anaconda seems to crap the bed in virt-manager with multiple virtual disks and partitioning. Never had that problem on bare metal. ¯\_(ツ)_/¯

And then I see this thread about ZFS, Upgrade from fedora 41 to fedora42 - root on ZFS , and then this: Overview — ZFSBootMenu 3.0.1 documentation

Hokay. Feels unsupported, unofficial and not something I would want to do to keep release integrity.

Firing up a new setup with Blivet and see how it goes. Will paste screenies.

Starting off here

Cleaning out what I have:

Here we have some interesting options, please note SOFTWARE RAID:

Please note, NOT done yet. Will copy/post working conf when I have a working conf.

After juggling a bit back and forth I spent three hours trying to get anaconda to accept the configuration and start installing and never got there.

Then I lost what was almost a working config by accident, and had to start over and was unable to find my way back to the right place to get the stuff working again.

Fedora on a Software Raid1 OS drive system it clearly not something that can be done in a reasonable way…

Maybe in yet another few years I may try again to get some fedora machines running…

Maybe boot a live USB image and setup the partitions from that environment.

Then in the anconda installer tell it to use the existing partitions?

1 Like

Under available devices you select both drives you want to RAID. That opens up the RAID level option, from which you select raid 1. For size I select 1 GiB, set the filesystem to efi (if my system is efi), set the label to efi, name it efi, and set the mount point as /boot/efi. Then select OK. You should now have a 1 GiB partition for efi under logical view.

Under logical view will be the efi partition you created, and free space. Select the free space, press the plus sign and once again repeat the process above and a create a 1 GiB partition for /boot.

Every thing else I put on logical volumes. So the third partition I create I set the filesystem as physical volume for LVM and use all the space and on that I put swap, root and home.

I am booting via OOB / IPMI so that is the way, but yes, it might be worth seeing it from a better interface than Anaconda and Blivet.

Had to take out the drives and wipe them, several attempts and unsuccessful cleaning by Blivet lead to many residual data issues, so even if I had not done any changes it still felt there were sector changes and meta data lying about…

Not even booting a Gparted was able to run through

sudo mdadm --stop /dev/md127
sudo mdadm --remove /dev/md127
sudo mdadm --zero-superblock /dev/nvme0n1p3
sudo mdadm --zero-superblock /dev/nvme1n1p3
sudo wipefs -a /dev/nvme0n1p3
sudo wipefs -a /dev/nvme1n1p3

properly since somehow the raid partition was marked as busy.

How can mdadm still be running after a reboot and loading gparted from an ISO?

Ah well…

The wipefs command does not necessarily erase everything when using only the -a option.
This from the man wipefs data

OPTIONS
       -a, --all
           Erase all available signatures. The set of erased signatures can be restricted with the -t option.

       -b, --backup[=dir]
           Create a signature backup to the file wipefs-<devname>-<offset>.bak in $HOME or the directory specified as the optional argument.
           For more details see the EXAMPLE section.

       -f, --force
           Force erasure, even if the filesystem is mounted. This is required in order to erase a partition-table signature on a block device.

I think you would need to repeat that with sudo wipefs -a -f /dev/nvme0n1 to wipe everything from the drive and not from the partition. The same for nvme1n1.

Once that is done then repeat the installation steps from the beginning. The raid metadata is mostly stored in the partition table at the beginning of the drive and not exclusively within the partition itself. The -f option gives details.

The method you used seems to be able to create 3 raid devices on the 2 drives, one for each file system. I think you intended to create one raid device with 3 contained partitions.

An additional factor is that mdadm is not active during initial boot so the efi partition (and maybe the /boot partition) would not be accessible to the bios for booting when mirroring the devices.