I hate to raise another question about failed filesystems right on top of the last one but I trying to get my previous workstation running to replace the failed LVM filesystem. This was a 3 disk RAID 5 filesystem that I have never had the time until now to try and get running but after a full day I;m still as baffled as I was last year. Here;s what I have done and the results so far.
As background, /dev/sdb failed and to replace the disk I just shut everything down. I issued the fail and stop commands to mdadm --manage, installed a new disk and tried to restart the array. Thatâs where I am stuck now.
mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Fri Apr 6 03:07:32 2018
Raid Level : raid5
Used Dev Size : 244192256 (232.88 GiB 250.05 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Sat Jan 22 11:49:52 2022
State : active, degraded, Not Started
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Name : wkstn04.iliffe.ca:root
UUID : 623b22cb:827bd418:c7148f3b:d3165770
Events : 141421
Number Major Minor RaidDevice State
- 0 0 0 removed
1 8 33 1 active sync /dev/sdc1
3 8 49 2 active sync /dev/sdd1
4 8 16 - spare /dev/sdb
cat /etc/mdadm.conf
MAILADDR root
AUTO +imsm +l.x -all
ARRAY /dev/md/root level=raid5 num-devices=3 uuid=623b22cb:827bd418:c7148f3b:d31
65770
mdadm --assemble --scan
mdadm --create /dev/md127 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sd
d1
mdadm: cannot open /dev/sdb1: Device or resource busy
fdisk /dev/sdb
Welcome to fdisk (util-linux 2.30.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 03440085-D9F6-46A4-8CE6-F5BEB6DA3CA9
Device Start End Sectors Size Type
/dev/sdb1 2048 1953525134 1953523087 931.5G Linux RAID
Command (m for help):
mdadm --manage --re-add /dev/sdb1 /dev/md127
mdadm: /dev/sdb1 does not appear to be an md device
I have also tried to start the array i n degraded mode without success and tried a lot of suggestions from searching various help sources that went nowhere and are not pasted here.
The very first command showed that there were 3 devices as part of md127, 2 active, 3 working, 1 spare. The array showed as active, degraded, Not Started.
In other words. the array had all the devices but had not been started so it could not rebuild.
The command cat /proc/mdstat should have given info about the status as well
From what you said above I would assume that you had marked the device you replaced as failed with mdadm --fail then removed it with mdadm --removebefore you physically removed and replaced the device.
Once the device had been replaced then it should have been added back into the array with mdadm --add and it should have immediately started the array rebuilding the data.
This shows what devices are seen and can be worked with.
Number Major Minor RaidDevice State
- 0 0 0 removed
1 8 33 1 active sync /dev/sdc1
3 8 49 2 active sync /dev/sdd1
4 8 16 - spare /dev/sdb
First I would start the array by removing /dev/sdb, and doing an assembly of the remaining devices to bring it active with only 2 devices. Once the array is active then using gdisk remove the partition table on /dev/sdb (create a new table) and write that, following which one would create the new partition /dev/sdb1, making sure it is the same size as the /dev/sdc1 partition. If the new partition on sdb is smaller than either sdc1 or sdd1 then adding it wonât work.
Finally add /dev/sdb1 back into the array and it should start rebuilding.
I had already tried that without success but here is another attempt with the results:
First, a starting point, note that sdb is no longer showing as a member of the RAID cluster (I removed it to try your suggestions)
/dev/md127:
Version : 1.2
Creation Time : Fri Apr 6 03:07:32 2018
Raid Level : raid5
Used Dev Size : 244192256 (232.88 GiB 250.05 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sat Jan 22 11:49:52 2022
State : active, degraded, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : wkstn04.iliffe.ca:root
UUID : 623b22cb:827bd418:c7148f3b:d3165770
Events : 141421
Number Major Minor RaidDevice State
- 0 0 0 removed
1 8 33 1 active sync /dev/sdc1
3 8 49 2 active sync /dev/sdd1
Then I tried to start it:
mdadm --manage --run /dev/md127
and here is the result:
/dev/md127:
Version : 1.2
Creation Time : Fri Apr 6 03:07:32 2018
Raid Level : raid5
Used Dev Size : 244192256 (232.88 GiB 250.05 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sat Jan 22 11:49:52 2022
State : active, degraded, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : wkstn04.iliffe.ca:root
UUID : 623b22cb:827bd418:c7148f3b:d3165770
Events : 141421
Number Major Minor RaidDevice State
- 0 0 0 removed
1 8 33 1 active sync /dev/sdc1
3 8 49 2 active sync /dev/sdd1
So it still fails to start. As for /dev/sdb, I created a new partition table and a single partition (same size as the others) and trying to do anything with it gives the error message â/dev/sdb1 does not appear to be an md deviceâ
FYI, in case it matters, since the computer wonât start at the moment Iâm running from a Linux Live DVD. Do I have to mount anything to see the correct â/â partition and /etc files? ie is everything I am doing being saved elsewhere?
Jeff: I just answered my own question re if there is a difference with the live disk sort of. I rebooted into the recovery function on the existing installation and while the reformatting of /dev/sdb seems to have worked and been saved, the mdadm details are different: (only the different part typed here, ssh is not supported on the recovery kernel)
Number Major Minor RaidDevice State
This would never be done for a device being added to a raid array.
Simply create the partition and leave it blank
Use mdadm --assemble --run <array name> <device name> <device name> ...to bring the 2 member drives active
Use mdadm add to add the new partition into the array (after it is active). The array itself does the formatting, etc. while rebuilding the array.
The question I must ask is whether the failed drive was removed using mdadm then physically removed? Or was it physically removed without using mdadm to mark it failed and remove it first?
What I see in your comments and the latest info that the system cannot start dirty degraded array implies the device may have been physically removed so the array is seen as dirty and degraded. If it was removed using mdadm then it should be clean but degraded.
Do you still have the old drive? Could it be reconnected?
The array must be gotten out of the dirty state before the degraded array can be started.
Answer to Gregory: I think I already did that but since I canât be sure I just did it again. Fails with âdevice busyâ errors on both sdc1 and sdd1. Iâm using the system emergency shell at the moment, not the Live DVD.
Answer to Jeff: I was following a procedure that I found online so may have been done in either order. Iâll try reinstalling the old disk as it is still on my workbench. Iâll do that in the morning. What is missing from the man pages is a succinct map to follow when a disk dies. I also issued a âstopâ command to the array and that is exactly what I should not have done as it turns out.
As I recall I think I issued fail, remove, stop in that order then replaced the disk. Also, there is no fdisk or gdisk on the emergency kernel so you are left with a new disk and no way to get it a partition table. I used the Live DVD to do that when I finally thought of it. This is the original install DVD which is Fedora 27.
More info to Jeff: I reconnected the old sdb disc and the array status has changed to âactive failed not startedâ and /dev/sdd1 shows as âspare rebuildingâ and /dev/sdc1 shows as âsync /dev/sdc1â . This has been going on for about half an hour now and Iâm not going to do anything else to the machine until this finishes.
Then, assuming I understand your previous post, I will fail and remove the old disc, replace it with the new one, and see if everything starts as expected. I am assuming that at this point a new assemble command may be needed but I will wait an see what it does.
Right, waiting for the array to stabalize is mandatory.
Once that is completed and the array can be started then the status can be checked.
One does not need the assemble command if the array is active, even with a failed device.
If active and displaying a failed device then the steps of
fail with mdadm
remove with mdadm
remove physically and replace
add with mdadm
Allow the rebuild to complete.
One should never interrupt at any time a rebuild process. All drives will be active with both reads and writes as the array is rebuilding. The time required for a rebuild is dependent upon the number of devices and the size of the array devices, as well as the amount of data stored there.
There is about 3 Gb of data including the operating system and after 7 hours the rebuild is still running according to mdadm --detail. Looks like it is trying to resync the defective disk.
8 49 0 spare rebuilding /dev/sdd1
8 33 1 sync /dev/sdc1
Is there any SAFE way to interrupt this or to find out if it is doing anything at all? smartctl isnât on rescue system so I canât check that the disks are where I think they are (/dev/sdb should still be mapped as the defective one). Before all this started BOTH disks sdc and sdd were shown as status âsyncâ.
Attempting to start the array gives error ânot enough operational devices (2/3 failed)â
No, there is no way to interrupt an array in the rebuild process without potentially destroying it.
It may be the drive you removed was not actually the one that failed.
It is I believe an incorrect statement when one says there is about 3 GB of data âincluding the operating systemâ since the OS itself is about 10 GB or more after installation.
7 hours is not unheard of. I have had an array where I replaced a 1 TB drive and it took more than 24 hours to do the rebuild. In fact, from what you showed above each device should be 250 GB, so a raid 5 array with 3 devices would be ~500 GB and it requires to rebuild the entire 500 GB array on 3 devices (750 GB of drive space) and not just the data on it.
Please be patient, do not continue running random commands, and also post the actual data from cat /proc/mdstat
It does seem as though you removed the wrong drive. If you are lucky, it may be able to recover now that all three drives are reconnected.
For future reference, something like smartctl -i <failed device> can be used to get the serial number of the failed device and then you can find that number on the drive you remove when youâve shut off your PC to be sure you have the right hard drive (or failing that, you can use the same command to get the numbers for the working drives and use the process of elimination to be sure you have the right drive).
Reply to Jeff: yes, I did understand your note, I was just surprised at the result. As of right now, 27 about hours since I reconnected /dev/sdb it is still in the same state, âactive, FAILED, not startedâ. I didnât realize that the whole disk space would be involved and so I had expected that it would have finished faster. Also, with none of the usual utilities available it is impossible to see if anything is happening at all.
As for disks, I know I originally removed the correct failed /dev/sdb because I had the serial number from smartctl which had already reported it as a read error problem. This time, with no way to find the correct disk, I had to look at the manufactured dates and assume the replacement was the one with the newest date. That was surprising when the first activity that occurred was to rebuild the existing working disk (/dev/sdd) when I reinstalled the failed drive. Just didnât seem logical.
As for why did I try to start the array; everything I have read about RAID says that you can use the disks normally while they are being re generated so I was trying to get some information that I needed.
cat /proc/mdstat:
Personalities : [raid6] [raid5][raid4]
md127 : inactive sdd1[0] sdc1[1]
488384512 blocks super 1.2
unused devices : none the angle brackets caused the content to disappear here
This looks like the original sdb was not connected, and that sdd appears failed since it showed as spare rebuilding. It is quite possible that the devices were renamed when adding another drive to the system and that the one now seen as sdd is the one that was reconnected. If that is the case then the 3rd drive is not seen and the array cannot be rebuilt with only one of the functional array members attached and attempting to rebuild an array with one good drive in a raid 5 array.
One needs to check and verify that all drives that should be in the array are correctly attached and seen with ls /dev/sd*
So it appears the system sees sdb, sdc & sdd. Yet sdb is not seen as part of the array.
One may try to assemble and start the array with mdadm --assemble --force md127 /dev/sdb1 /dev/sdc1 /dev/sdd1
If that does not work then sdb1 may need to be either added or readded to the array with mdadm --manage --add /dev/sdb1 md127 which will do a re-add if the device was already part of the array.
Note that for now the attempt is merely to get the array functional even if it has a failed member. Once the array is functional then removing and replacing the failed member is possible.
One may carefully use the information in man mdadm to decide exactly what to try and what may or may not work.
This seemed to indicate that the device was removed without failing it so the metadata for the array should still be on it. That also shows the array as active at that time.
Removing it from the array after physical removal means other steps must be done to reconnect it properly.
Be aware also that it is quite possible (even likely) that what was /dev/sdb, /dev/sdc, etc. before you swapped the drives will be different after you swap the drives. Since you appear to be applying the RAID to partitions rather than whole disks, you could use ls -al /dev/disk/by-partuuid to keep straight which drive is which (the partition uuids will not change even though the drive lettering may).
e.g.:
$ ls -al /dev/disk/by-partuuid/
total 0
drwxr-xr-x. 2 root root 160 Aug 8 08:26 .
drwxr-xr-x. 9 root root 180 Aug 8 08:26 ..
lrwxrwxrwx. 1 root root 10 Aug 8 08:26 3276647c-b0eb-4789-abfa-8d524b83e0d7 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Aug 8 08:26 33b4c550-a5db-4fad-a44a-dbc6eb917922 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Aug 8 08:26 55a7d9a6-5e04-4de4-995d-d7e14483a192 -> ../../sdc1
lrwxrwxrwx. 1 root root 10 Aug 8 08:26 863ec17d-6677-4830-9b4a-b5340b4eddea -> ../../sdb2
lrwxrwxrwx. 1 root root 10 Aug 8 08:26 88b650ea-b765-4328-bd18-0d5874f47dc5 -> ../../sdb1
lrwxrwxrwx. 1 root root 10 Aug 8 08:26 ddc9e8ae-9aee-4781-8a91-5413e094be72 -> ../../sdd1
OK, Iâm wildly out of my depth here. Looking at your sample uuid list I noticed that while I got ascending uuid numbers as you did, the disk partition identifiers start with /dev/sdb1, then go to a1,a3,c1,a4,d1,a2. Does this mean that the partitions have been remapped? I donât have the original list to compare to since I have never used âby-partuuidâ before. (didnât know it existed). I used partitions when I installed because this was what anaconda did. Not a conscious choice on my part.
Noting Jeffâs comments about using --force, the rebuild is still running and it is coming up to two days on that now. Also noting previous comments on not interrupting build in progress I donât want to make a bigger mess than I have. Would doing an assemble or an add while the build is running break anything that isnât already broken?
Where is the information on what is in the array coming from and can the missed âremoveâ be done by deleting or commenting out a line there? The man pages are not currently available so the best I could do would be look them up on google and hope the version I get applies to this O/S version. Even âtopâ is not available and ps -ef only shows 1 RAID function active (raid5wq) which seems to have started before the system crashed (lower pid # than the start dracut-emergency.service). No mdadm running.
It seems quite obvious that the rebuild can never complete since it only sees one of 2 devices that were apparently functional earlier and is now trying to rebuild with one good and one spare which cannot work.
At this point one would have no choice but to interrupt the already failed rebuild.
Note that you are giving us your interpretation of what you see when running commands and we are giving you actual samples from our system. Please start providing the full details so we may have actual data to review in return instead of depending upon your self-admitted âout of my depthâ interpretation of our suggestions.
Start with the actual text you see when running each command, such as the suggested ls -al /dev/disk/by-partuuid/ so we may have info that is pertinent instead of the totally unhelpful disk partition identifiers start with /dev/sdb1, then go to a1,a3,c1,a4,d1,a2.
We are showing you the command as it is entered and the returned text as copied from the screen â Please do the same with your replies.
You did note that you saw info about sdb1, c1, & d1. What makes you question anything other than that might be the desired and needed info?. The actual text you saw would help us answer since at that point we are looking at exactly what you see on the screen.
Thanks Jeff. Yes, iâm sorry about that. The usb ports on the workstation are inactive, and ssh isnât available so I have to type what I see. In the case at hand:
I canât stop the rebuild, interrupting with ctl-c doesnât do anything as it is running in the background and I donât know the pid so I canât issue a âkillâ. Should I just issue a systemctl reboot?? or is there a better way?