Hi all, after about 12 years of upgrading my server continually with no issues, I have recently been struggling with my /boot and / partitions getting 100% full. I guess they have served me well over the years but I need to resize them somehow.
I have two disks, but they are partitioned using “regular” RAID 1 with no underlying volume manager (so no LVM). Although I started with 3 partitions, to free space I created another so basically I have:
So these are all RAID 1 partitions, but I need to increase md0 and md2.
I searched for walkthroughs but really struggled to find anything that explained how to do this and I obviously don’t want to risk actually trashing my system inadvertently.
Can anyone point me towards a good method to do this, ideally without needing to do a full re-install then a recovery.
I was contemplating something like:
failing one disk out of the array
creating completely new re-sized partitions on this disk
boot into a live image from USB
using mdadm to mirror from old to new (so all the filesystems will clearly be smaller than the partitions)
use resize2fs to increase /boot and /
Am I missing a trick or are there any other recommendations?
That sounds like how I would do it. If I were really worried about losing something though, I’d probably swap in a new drive instead of repartitioning one of the current drives. Then you could just put the old drive back in in case of a catastrophe.
There might be one thing to be aware of if you are considering switching to btrfs raid1. I’ve heard that the system will not boot off of just one drive with btrfs raid1. MDADM will function/boot when only one drive is working. This might have been fixed in a very recent version of btrfs, but if so, I haven’t seen any mention of it.
I am not convinced that will allow me to resize / or /boot though. The more I look into it the more I am thinking of just removing one disk from the array, then using a live USB to configure that disk exactly how I want (possibly brtfs), probably even with a fresh install. Then I can just use the live USB again to copy everything from the “old” disk to the “new” completely overwriting everything (possibly with the exception of /etc/fstab). Then add the old disk as a mirror.
You might want to be careful about using backup/restore tools to copy the partition table from one disk to the other. With the older MBR tables, this was fine and even the recommended way to do it. The newer GPT format, however, assigns a unique id to every partition. One way you can view these partition UUIDs is by long-listing the contents of /dev/disk/by-partuuid:
$ ls -al /dev/disk/by-partuuid/
drwxr-xr-x. 2 root root 120 May 11 11:25 .
drwxr-xr-x. 8 root root 160 May 11 11:25 ..
lrwxrwxrwx. 1 root root 10 May 11 11:25 55a7d9a6-5e04-4de4-995d-d7e14483a192 -> ../../sdb2
lrwxrwxrwx. 1 root root 10 May 11 11:25 6096a2e8-618a-4355-ba05-f84752c5252c -> ../../sda1
lrwxrwxrwx. 1 root root 10 May 11 11:25 8735d258-f7dc-4f35-a904-32e2435bfa1c -> ../../sdb1
lrwxrwxrwx. 1 root root 10 May 11 11:25 ddc9e8ae-9aee-4781-8a91-5413e094be72 -> ../../sda2
If you do a backup/restore of a GPT partition table to the 2nd drive, then the system will end up with “duplicate” unique IDs for the partitions and that is not supposed to happen. You might get away with it, but it might also confuse some tools (the long-listing of /dev/disk/by-partuuid, for example, will only list half of the partitions).
You might be better off recreating the partitions on the second drive by hand rather than restoring a backup of the first drive. Alternatively, the sgdisk tool has an option to (re)randomize a partition’s UUID.
As the original poster, I thought I would just post a final update. I can confirm upgrading / and /boot effectively all online did work for me. The process was pretty much as I outlined although I did tweak a few things. Just in case it helps others this is just to document at a high level the steps.
As my RAID 1 disks had no spare capacity I had to find some space so firstly I just reduced the size of /home to free up enough space for the planned increase in / and /boot. This was done with resize2fs and mdadm --grow (making sure mdadm final size is larger than the filesystem itself of course).
Failed and removed the 2nd disk from the array. Done online.
Created a new partition table (using sfdisk dumped from the first disk as the basis). Obviously I had to increase the sizes for /boot and / but reducing the extended partition size. Done online.
Mirrored back from 1st disk to 2nd disk (i.e. with same filesystem size as on the 1st disk initially
Then reboot test to make sure all ok
Failed and removed 1st disk from the array. Done online.
Dumped the partition table from the 2nd disk with sfdisk and applied to the first disk. Done online.
Re-mirrored back. Done online.
Then needed to grow the partitions to full size and increase the filesystems. I had read some warnings about write intent bitmap so was cautious and did this be disabling the bitmap (with mdadm) for the partitions which had the bitmap) then just did a mdadm --grow --size-max followed by a resize2fs on / /boot and /home. All done online.
final reboot just to make sure the array was OK.
I didn’t need to do anything with the boot sector as these disks were not replaced so boot sectors etc were all OK.
So I didn’t even need to use recovery mode which was great.
Thanks for the suggestions/pointers, happy for my server to continue now for another 10 years! I increased the boot and / by 10-fold should things grow more than expected.