Best practice to increase / and /boot from MDADM array

Hi all, after about 12 years of upgrading my server continually with no issues, I have recently been struggling with my /boot and / partitions getting 100% full. I guess they have served me well over the years but I need to resize them somehow.

I have two disks, but they are partitioned using “regular” RAID 1 with no underlying volume manager (so no LVM). Although I started with 3 partitions, to free space I created another so basically I have:

md0: /
md1:swap
md2:/boot
md3: /home
md4:/opt

So these are all RAID 1 partitions, but I need to increase md0 and md2.

I searched for walkthroughs but really struggled to find anything that explained how to do this and I obviously don’t want to risk actually trashing my system inadvertently.

Can anyone point me towards a good method to do this, ideally without needing to do a full re-install then a recovery.

I was contemplating something like:

  • failing one disk out of the array
  • creating completely new re-sized partitions on this disk
  • boot into a live image from USB
  • using mdadm to mirror from old to new (so all the filesystems will clearly be smaller than the partitions)
  • use resize2fs to increase /boot and /

Am I missing a trick or are there any other recommendations?
thanks
Mark

1 Like

That sounds like how I would do it. If I were really worried about losing something though, I’d probably swap in a new drive instead of repartitioning one of the current drives. Then you could just put the old drive back in in case of a catastrophe.

Because you have a Raid1, the first I will do is to break it up.

So that your system is bootable using only one disk, and the second disk can be reformated.

I will use btrfs in a single partition

Subvolumes
/boot
/
/home
/opt

(no swap partition needed)

After using rsync to sync contents to the btrfs disk, use LiveUSB to fix bootloader.

Once the btrfs disk is bootable, then I can add the old disk to form a btrfs raid1.

There might be one thing to be aware of if you are considering switching to btrfs raid1. I’ve heard that the system will not boot off of just one drive with btrfs raid1. MDADM will function/boot when only one drive is working. This might have been fixed in a very recent version of btrfs, but if so, I haven’t seen any mention of it.

1 Like

Thanks for the hints, I’ll be doing more reading over the next couple of days!

@glb is correct that booting btrfs raid1 in degraded mode is still problematic.

In order to enjoy the efficient use of disk spacez you might just want to build a btrfs raid1 for data only.

You can move large size folder into btrfs subvol and mount it back to the original location.

Adding a disk to the array, is that an option for you?

Didn’t do this myself yet, however in theory this should be your best bet.

One thing to add, before that it is suggested to write the partition table from an old drive to the new drive, like

sfdisk /dev/sde -d | sfdisk /dev/sdf

While sdf would be the new disk.

I am not convinced that will allow me to resize / or /boot though. The more I look into it the more I am thinking of just removing one disk from the array, then using a live USB to configure that disk exactly how I want (possibly brtfs), probably even with a fresh install. Then I can just use the live USB again to copy everything from the “old” disk to the “new” completely overwriting everything (possibly with the exception of /etc/fstab). Then add the old disk as a mirror.

1 Like

https://gparted.org/news.php

You can see if the gparted iso is helpful or not.

Since you have a raid 1, maybe not :wink: :sweat_smile:

You may use tools like dd, to copy your data.

However, if you would expand your raid, you may consider something like this:

But I’d try to make a backup in advance.

The other idea which I think would work would be to:

  • fail the second disk, remove from array
  • create new partition table with larger sizes on the second disk
  • use mdadm to mirror again (so initially all filesystems would be the same size BUT in larger partitions on the second disk).
  • fail the first disk
  • reboot using live image
  • extend2fs the “new” disks filesystems to maximum size (based on the new partition sizes).
  • then just copy partition table from 2nd to 1st disk and re-mirror back.

Any gotchas in this as this outwardly looks fairly straightforward.

I have about 24 hours to ponder while my server does as S3 backup :grinning: just in case.

1 Like

You might want to be careful about using backup/restore tools to copy the partition table from one disk to the other. With the older MBR tables, this was fine and even the recommended way to do it. The newer GPT format, however, assigns a unique id to every partition. One way you can view these partition UUIDs is by long-listing the contents of /dev/disk/by-partuuid:

$ ls -al /dev/disk/by-partuuid/
total 0
drwxr-xr-x. 2 root root 120 May 11 11:25 .
drwxr-xr-x. 8 root root 160 May 11 11:25 ..
lrwxrwxrwx. 1 root root  10 May 11 11:25 55a7d9a6-5e04-4de4-995d-d7e14483a192 -> ../../sdb2
lrwxrwxrwx. 1 root root  10 May 11 11:25 6096a2e8-618a-4355-ba05-f84752c5252c -> ../../sda1
lrwxrwxrwx. 1 root root  10 May 11 11:25 8735d258-f7dc-4f35-a904-32e2435bfa1c -> ../../sdb1
lrwxrwxrwx. 1 root root  10 May 11 11:25 ddc9e8ae-9aee-4781-8a91-5413e094be72 -> ../../sda2

If you do a backup/restore of a GPT partition table to the 2nd drive, then the system will end up with “duplicate” unique IDs for the partitions and that is not supposed to happen. You might get away with it, but it might also confuse some tools (the long-listing of /dev/disk/by-partuuid, for example, will only list half of the partitions).

You might be better off recreating the partitions on the second drive by hand rather than restoring a backup of the first drive. Alternatively, the sgdisk tool has an option to (re)randomize a partition’s UUID.

1 Like

Great hint thanks, I’d have probably fallen foul of this.

As the original poster, I thought I would just post a final update. I can confirm upgrading / and /boot effectively all online did work for me. The process was pretty much as I outlined although I did tweak a few things. Just in case it helps others this is just to document at a high level the steps.

  • As my RAID 1 disks had no spare capacity I had to find some space so firstly I just reduced the size of /home to free up enough space for the planned increase in / and /boot. This was done with resize2fs and mdadm --grow (making sure mdadm final size is larger than the filesystem itself of course).
  • Failed and removed the 2nd disk from the array. Done online.
  • Created a new partition table (using sfdisk dumped from the first disk as the basis). Obviously I had to increase the sizes for /boot and / but reducing the extended partition size. Done online.
  • Mirrored back from 1st disk to 2nd disk (i.e. with same filesystem size as on the 1st disk initially
  • Then reboot test to make sure all ok
  • Failed and removed 1st disk from the array. Done online.
  • Dumped the partition table from the 2nd disk with sfdisk and applied to the first disk. Done online.
  • Re-mirrored back. Done online.
  • Then needed to grow the partitions to full size and increase the filesystems. I had read some warnings about write intent bitmap so was cautious and did this be disabling the bitmap (with mdadm) for the partitions which had the bitmap) then just did a mdadm --grow --size-max followed by a resize2fs on / /boot and /home. All done online.
  • final reboot just to make sure the array was OK.

I didn’t need to do anything with the boot sector as these disks were not replaced so boot sectors etc were all OK.

So I didn’t even need to use recovery mode which was great.

Thanks for the suggestions/pointers, happy for my server to continue now for another 10 years! I increased the boot and / by 10-fold should things grow more than expected.

regards
Mark

2 Likes