When I did the initial copy … to copy the data out of the BTRFS system … I used cp -a [ORG] [DEST]. I forgot about rsync =(
I need to get used to rsync. I’m used to the old DOS days with xcopy. With that, it’s probably still a good task to run touch /.autorelabel and fixfiles onboot
Once my hard drives settle down, I’ll reboot and run dracut and dnf reinstall kernels to make sure that something didn’t get touched when the BTRFS partition burped. Presently, the mirrored partitions are syncing.
This saga starts with a mix of technologies coming together and repeatedly
barfing. These technologies for me are … VirtualBox, Wine, and BTRFS
file systems … I’m not sure why, but whenever these 3 technologies
come together the BTRFS file system becomes corrupted. But, let’s set
aside why it barfs and focus on
What to do when it barfs.
If you want to move from a BTRFS to an EXT4 partition.
If you just want to move your linux operating system to a new hard
drive, because you need more drive space.
Note: where ever possible, I will try to provide the search terms needed
to allow you to find the right command to perform the function on your
version / distro of Linux.
!!! ALWAYS USE THE LATEST LIVE CD WHEN TROUBLESHOOTING BTRFS PARTITIONS !!!
!!! ALWAYS USE A LIVE CD THAT MATCHES YOUR CURRENT OS WHEN MANIPULATING INITRAMFS AND GRUB !!!
In this tutorial I will use LiveCD and terminal for most of the operations
to be executed here. Whenever you see " > ", assume that I’m refering
to a terminal command, and I’m logged in as a super user.
> su
> whoami
root
Note: LiveCD does not require a password when logged in as a super user.
However, if your drive is encrypted, you will need to unlock it before
performing any operations on it.
The first thing we want to do is find out where all our drives and
partitions have been mapped to. `blkid`, and `ls` will tell us what
the drive UUIDs are, for future reference, while the `lsblk` command
tells where the drives / partitions have been mounted.
> blkid | grep /dev/sd
total 0
drwxr-xr-x. 2 root root 180 May 4 11:40 .
drwxr-xr-x. 9 root root 180 May 4 11:39 …
lrwxrwxrwx. 1 root root 10 May 4 2024 0ae2646a-8a22-4ab9-b4e9-1300348a52e3 → …/…/dm-1
lrwxrwxrwx. 1 root root 10 May 4 2024 d8755717-901c-47d0-9a72-d584f93eeb5c → …/…/sda1
lrwxrwxrwx. 1 root root 10 May 4 2024 6bec82ae-4d47-4704-adfa-fd51dab23cc8 → …/…/sda2
lrwxrwxrwx. 1 root root 10 May 4 2024 8d30ac74-d8f5-472d-8fb2-c45988c4e89a → …/…/sda3
lrwxrwxrwx. 1 root root 9 May 4 11:40 bf087e2f-5dea-4f19-90bb-bbc17406b717 → …/…/md0
lrwxrwxrwx. 1 root root 10 May 4 2024 2023-04-13-22-14-25-00 → …/…/sdf1
lrwxrwxrwx. 1 root root 10 May 4 2024 EBE8-26A4 → …/…/sdf2
[liveuser@localhost-live ~]$
> lsblk -l
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS LABEL
sda 8:0 0 1.8T 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:3 0 1.8T 0 part /
sdb 8:16 0 1.8T 0 disk
└─sdb1 8:17 0 1.8T 0 part
└─md0 9:0 0 1.8T 0 raid1 /Data
sdf 8:0 0 1G 0 disk
└─sdf1 8:1 0 1G 0 part /LiveCD
sdg 8:0 0 2T 0 disk
├─sdg1 8:1 0 1G 0 part /mnt/boot GRUB
├─sdg2 8:1 0 0.8T 0 part /mnt/root Root
└─sdg3 8:1 0 1T 0 part /mnt/root/home Home
OR
sdg 8:0 0 2T 0 disk
└─sdg1 8:1 0 1T 0 part /mnt/sdg1 Backup
Let's review the partions before here
/dev/sda1 .. contains grub
/dev/sda2 .. contains the BTRFS with /root and /home subvolumes
/dev/sdb1 .. is a raid array with sdb, sdc, and sdd
/dev/sdf1 .. is the LiveCD
/dev/sdg .. is a backup drive where root and home will be copied to
/dev/sdg1 .. is the boot partition containing grub
/dev/sdg2 .. is the root partition containing the OS
/dev/sdg3 .. is the home partition containing the user directories
=== Preparing a Replacement / Backup Drive ===
It far faster and easier to use GParted GUI to partition, label, and
format a new drive. However, if you choose, use the commands here to
walk through the process manually.
> parted --list
> parted /dev/sdg
> mklabel msdos # or gpt
> print
> mkpart primary EXT4 0 2T
> print
> set 1 boot off
> quit
To label the drive
> e4label /dev/sdg1 Backup
to format the drive
> mkfs.ext4 /dev/sdg
To create a mount point
> mkdir -p /mnt/sdg1
To mount the drive
> mount /dev/sdg1 /mnt/sdg1
To view drive/partition sizes
> df -hT
=== Cloning the Drive ===
If you don't feel comfortable doing a destructive operation on your
hard drive, then perform a block level copy of the partition to
another drive first.
To duplicate the partition to a new drive, use `dd`. Short for
Disk Duplicate
> dd if=/dev/sda1 of=/dev/sdg bs=64K conv=noerror,sync
if .. means the input partition
of .. means the output partition
bs .. is the size of the chucks you want to read and write
larger chucks mean faster service
smaller chucks means better quality data
conv ..
=== Recover your BTRFS file system ===
!!! ALWAYS USE THE LATEST LIVE CD WHEN TROUBLESHOOTING BTRFS PARTITIONS !!!
Initially, GParted GUI can be used to recover the BTRFS partition.
Where possible, use the latest LiveCD OS when asking GParted to
recover a BTRFS partition, as it has the latest methods to reconstruct
the BTRFS tree.
If possible, store any reports that BTRFS generates to a seperate
drive. GParted, used to ask you to send these reports to them, but
it seems that they are no longer interested.
If GParted can't restore your hard drive to a usable state, switch
to terminal and attempt the recovery.
=====
Confirm that the drive has no physical issues by using `badblocks`.
The command below will only do a verify, not a destructive
write / read operation.
> badblocks -v [PARTITION]
> badblocks -v /dev/sda2
When `badblocks` is finished, we can attempt to recover the BTRFS
partition, with `btrfs check`, and `btrfs check --repair`.
Use `btrfs check` initially for a non-destructive report on what has
happened.
> btrfs check [PARTITION]
> btrfs check /dev/sda2
Use `btrfs check --repair` for a destructive repair. This will touch
the drive and attempt to repair it. (If you don't feel comfortable
with this option yet, then perform a block level copy of the
partition to another drive first.)
> btrfs check --repair /dev/sda2
Once the BTRFS tree has been fixed, you can mount the partition
and move the contents to a backup device.
=== Copying Data Out ===
For my purposes, I did not use /dev/sdg as a replacement drive. I used
it as a backup drive. For that, I only had /dev/sdg1 which contained
sub directories /root and /home. I chose this option, as I didn't
want to disturb grub in /dev/sda1.
In the Linux distro that I use, the root and home folders are
seperated into 2 sub volumes in the BTRFS partition. What this
means, or how to think about this from a directory perspective,
is that the BTRFS has created two folders .. root and home ..
However, be careful with this perspective, since linux requires
these directories .. root and home .. to be either merged into 1 tree
or copied to seperate partitions when migrated to EXT4.
For the backup process, I choose to create 1 partition on /dev/sdg1 to
backup all the data to. When that was complete, I choose then to
create 2 partitions on /dev/sda, labeled "root" and "home",
/dev/sda2 .. root
/dev/sda3 .. home
to restore the backed up folders to.
The other option for the restore is to create 1 large partition, and
the restore the contents of the /mnt/sdg1/root to /mnt/sda2 first, then
restore /mnt/sdg1/home to /mnt/sda2/home
For this process, we will use `rsync`
Note: you can use copy `cp -a [SRC] [DEST]`, but you will lose the
SELinux labels, and it won't give you a progress report.
> rsync --progress [SRC] [DEST]
> rsync --info=progress2 [SRC] [DEST]
> mkdir -p /mnt/sda2
> mkdir -p /mnt/sdg1
To copy root and home as seperate directories to sdg1
> mount /dev/sda2 /mnt/sda2
> mount /dev/sdg1 /mnt/sdg1
If mount /dev/sda2 throughs an error use `dmesg` to see what happened.
Unfortuantely, I don't remember what my BTRFS threw, but with a bit
of quick searching I was able to sort it out.
> dmesg
> rsync --progress /mnt/sda2/root /mnt/sdg1
> rsync --progress /mnt/sda2/home /mnt/sdg1
To copy root and home to seperate partitions
> mount /dev/sda2 /mnt/sda2
> mount /dev/sdg1 /mnt/sdg1
> mount /dev/sdg2 /mnt/sdg2
> rsync --progress /mnt/sda2/root/* /mnt/sdg1
> rsync --progress /mnt/sda2/home/* /mnt/sdg2
Once the contents have been copied out, you can swap the drives
or reformat the existing drive with a set of EXT4 partitions
for root and home, and copy the data back.
=== Updating Grub ===
!!! ALWAYS USE A LIVE CD THAT MATCHES YOUR CURRENT OS WHEN MANIPULATING INITRAMFS AND GRUB !!!
Once the partitions have the data, you are ready to tell Grub
where to look to start the OS. This is known as updating Grub and
Updating the `initramfs`.
One mistake that most people make, is mounting /dev/sda1 and making
edits to grub directly. !!! DONT DO THIS !!! Grub will make the
necessary changes to /dev/sda1.
Since all the drive and parition UUID's have changed, you will need
to relist them gain.
> blkid | grep /dev/sd
> ls -lha /dev/disk/by-uuid
> lsblk -l
If you followed any of the steps in copying the data out of a BTRFS
partiton, make sure that /mnt is empty.
> cd /mnt
> rm { sda2, sdg1, sdg2 }
The next steps map the partitions to the /mnt directory and prepare
it for OS manipulation.
> cd /
> mount /dev/sda2 /mnt
> mount -t proc proc /mnt/proc
> mount -t sysfs sys /mnt/sys
> mount -o bind /dev /mnt/dev
> mount --bind /run /mnt/run
> mount /dev/sda1 /mnt/boot
To configure your network while in chroot
> cp -L /etc/resolv.conf /mnt/etc/resolv.conf
Enter `chroot`
=== ENTER Change Root ===
Do not exit `chroot` until you are explicitly told to in this script
otherwise, the commands you issue will affect LiveCD.
> chroot /mnt /bin/bash
Confirm that the `mtab` file to has all drives listed
> grep -v rootfs /proc/mounts
> cat /etc/mtab
Or
> grep -v rootfs /proc/mounts > /etc/mtab
Make sure the `fstab` file has the drives that should be auto-mounted
`vi` is a text editor
> cat /etc/fstab
> vi /etc/fstab
List the `initramfs` found in the boot partition. You may need the
versions of the OS, if you have to run dracut on a different
PC to regenerate the initramfs.
> ls /boot | grep initramfs
initramfs-6.8.4-100.fc38.x86_64.img
Regenerate your image file with `dracut`. The first command makes
sure you are hitting the right images. The second command will
overwrite them.
> dracut --regenerate-all
> dracut -f --regenerate-all
Use `lsinitrd` to make sure the files are correct. It's not a necessary
step, but I was trying to see if `fstab` had been copied into the
image, and if so, did it contain the drive mappings.
> lsinitrd /boot/initramfs-6.8.4-100.fc38.x86_64.img
!!! DO NOT EXIT chroot, just yet !!!
=== Update Grub ===
Since drive / partition UUID's changed, you will need to run grub2's
list and update commands make sure grub matches `fstab`
> cat /etc/fstab
Use Grub to list the boot environments in the system
> grub2-editenv - list
Note: if you only see this (boot_success=1), you exited `chroot`.
You can re-enter `chroot` by issueing the command above.
boot_success=1
Now issue the update 1 of the 2 commands below.
They do the same thing ...
> update-grub2
> grub2-mkconfig -o /boot/grub2/grub.cfg
Now confirm that the grub command actually updated the configuration
file.
> cd /boot/loader/entries
> ls -al
-rw-r–r–. 1 root root 326 May 4 11:50 a6a84281584b4fb69829b970ecb6d26b-6.8.4-100.fc38.x86_64.conf
All entries should have today's date
> cat a6a84281584b4fb69829b970ecb6d26b-6.8.4-100.fc38.x86_64.conf
title Fedora Linux (6.8.4-100.fc38.x86_64) 38 (MATE-Compiz)
version 6.8.4-100.fc38.x86_64
linux /vmlinuz-6.8.4-100.fc38.x86_64
initrd /initramfs-6.8.4-100.fc38.x86_64.img
options root=UUID=6bec82ae-4d47-4704-adfa-fd51dab23cc8 ro video=HDMI-0:D video=VGA-1-1:d
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
Make sure the UUID listed as root, matches the device partiton
/dev/sda1's UUID.
If you see quiet in the output, re-run update-grub2 and turn off quiet.
You will want to see any and all errors during the boot process to
trouble shoot them.
One such error that can occur, is the system won't boot if the JournalCtl
fails to start. This happens due to a SELinux issue.
!!! DO NOT EXIT chroot, just yet !!!
=== Updating SELinux ===
SELinux is the enhanced security system Linux uses to make sure
roge programs can't touch files they aren't supposed to. Sometimes
turning off the system can help you trouble shoot boot issues.
See the current status of the OS
> sestatus
See if SE Linux is being enforced or not
> getenforce
Check SE Linux's configuration file
> cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# grubby --update-kernel ALL --args selinux=0
# To revert back to SELinux enabled:
# grubby --update-kernel ALL --remove-args selinux
SELINUX=enforcing
SELINUXTYPE=targeted
To fix the files for SE Linux so that they are all labeled correctly
use `fixfiles` or `touch /.autorelabel`
> fixfiles onboot
> touch /.autorelabel
=== End Change Root ===
You can now safely exit `chroot`
> exit
Dismount all the drive and partition mappings from above
> umount /mnt/boot
> umount /mnt/run
> umount /mnt/{proc,sys,dev}
> umount /mnt
Issue mount to determine if all the mounts have been dismounted
> mount
> lsblk -l
Shutdown the pc, either from CLI or from the LiveCD desktop
> reboot -p
=== Reboot ===
As your computer reboots it will run into errors. Remember, though,
one of the operations running in the background is `fixfiles`.
Let this operation complete before you begin any trouble shooting.
If you see that the JournalCtl has failed, this is becuase `fixfiles`
has not completed its job, and JournalCtl cant access its files
due to SELinux.
If the problem persists, turn off SELinux as instructed above.
Another reported issue that could come up is the `machine-id`
mismatch between /etc/machine-id and /var/lib/dbus/machine-id.
You could try resetting them, but as I found out, the ID's contained
in these files match those in Grub, and many other locations. So,
avoid them until you cant.
> systemd-firstboot --root=/mnt --setup-machine-id
or
> systemd-machine-id-setup
If problems persist, and you need to use the LiveCD to go back
into the OS to fix things, follow the `Enter Change Root` and
`Exit Change Root` steps.
=== Post Clean Up ===
Once your OS is up and running, you should update it immediately
and verify that all packages are functional.
> dnf update
> rpm -Va
If `dnf` doesn't update the kernal, then issue `dnf reinstall` and
execute `dracut` again. You won't need to use `chroot` as you are
in the system you want to change.
> dnf reinstall
> dracut --regenerate-all
> dracut -f --regenerate-all
After the update reboot
> reboot -p
The next thing I did was remove VirtualBox
> dnf remove VirtualBox-\*
> dnf list installed | grep -i VirtualBox
> dnf list installed | grep -i vbox
Make sure `systemctl` says that anything vbox or VirtualBox is
stopped.
> systemctl | grep -i VirtualBox
> systemctl | grep -i vbox
Reboot
> reboot -p
You can re-install VirtualBox if needed at this point.
> dnf install VirtualBox