I just upgraded a server to F39. I have encountered the strangest error I have ever seen. I have a raid6 mdraid array with luks encryption applied to the /dev/md device. After upgrading the array did not come online and one of the disks showed failed. The strange part is the failed disk, after the upgrade, now has an exact duplicate of the Luks header from the md device applied to it.
I did
mdadm /dev/md123 --remove /dev/sde1
fdisk /dev/sde
removed the partition and recreated
for good measure I ran mkfs.ext4 /dev/sde1
ran cryptsetup luksDump /dev/sde1 and verified there was no luks header on the disk
then mdadm /dev/md123 --add /dev/sde1
confirmed the array rebuild started
cryptsetup luksDump /dev/sde1 now has a duplicate header as cryptsetup luksDump /dev/md123
even the uuid’s in the luks header match
After the array rebuilds I can open the /dev/md123 luks device and mount the filesystem and everything works as expected until I reboot.
If I reboot at this point luks will try to activate /dev/sde1 instead of /dev/md123 so the array will fail and I have to delete / recreate the partitions and rebuild the array again to get rid of the luks headers. Also before the reboot blkid shows /dev/sde1 has the correct uuid as all the other member devices of the array. After the reboot /dev/sde1 now has the same uuid as the /dev/md123 luks device.
I have never seen where mdadm writes luks headers to disk. Its like there is a bug where fedora is joining the disk to the array but at the same time is mirroring the mdraid devices headers to the newly admitted device.