Btrfs create snapshot and send | receive to backup

bennyisaiah · January 26, 2022, 10:04am

Thanks a lot. So I should create a snapshot first then I can move it with btrfs send ?

Specifically what I would like to do as a test would be to move one directory to my HDD with this technique aforementioned.

So the data I am copying is at this path

/home/solomon/Documents/Tax

And I want to move it to the sdb drive.

[solomon@fedora Tax abatement strategies]$ lsblk
NAME                    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                       8:0    0 232.9G  0 disk  
└─sda1                    8:1    0 232.9G  0 part  
sdb                       8:16   0   2.7T  0 disk  
└─sdb1                    8:17   0   2.7T  0 part  /run/media/solomon/3TB WD Red
sdc                       8:32   0 223.6G  0 disk  
├─sdc1                    8:33   0   600M  0 part  /boot/efi
├─sdc2                    8:34   0     1G  0 part  /boot
└─sdc3                    8:35   0   222G  0 part  
  └─luks-2aafcb54-d261-46b8-8423-2ccc7ddc45cb
                        253:0    0   222G  0 crypt /home
                                                   /
sdd                       8:48   1     0B  0 disk  
sde                       8:64   1     0B  0 disk  
sr0                      11:0    1  1024M  0 rom   
zram0                   252:0    0     8G  0 disk  [SWAP]
[solomon@fedora Tax abatement strategies]$

so according to the path on the partition it would be

/run/media/solomon/3TB WD Red

So would I enter

sudo btrfs send "path of the location where the snap shot is" | sudo btrfs receive /run/media/solomon/3TB WD Red

I dont seem to have a snapshot directory anywhere in the system though.

I also must ensure the receiving disc drive has a btrfs filesystem which is does

dalto · January 26, 2022, 1:01pm

You can only use btrfs send|receive with entire subvolumes.

So if you want to do that, you would need to first create a subvolume, move the data into it and then mount it at /home/solomon/Documents/Tax. If that is the only folder you want a copy of, this is pretty reasonable.

However, if you want to use btrfs send|receive on arbitrary folders, it doesn’t really work that way.

chrismurphy · January 26, 2022, 6:02pm

As @dalto mentions, you need a subvolume to make a snapshot, and it needs to be a read-only snapshot to do btrfs send. Both source and destination need the same snapshot to do an incremental send (using -p option).

Note that btrfs snapshots are not recursive, i.e. it will exclude nested subvolumes (snapshots are also subvolumes). Instead of creating a directory you can create a subvolume and it’ll mostly behave like a directory, except you can snapshot it. For example:

cd ~/Documents
btrfs sub create Tax2
mv Tax/* Tax2
rm -rf Tax
mv Tax2 Tax

And now you can snapshot it, and send/receive.

btrfs sub snap -r Tax Tax.20220126
sudo btrfs send Tax.20220126 | sudo btrfs receive  /run/media/solomon/backup

The snapshot can be anywhere on the same Btrfs filesystem, it doesn’t have to be in ~/Documents, that’s up to you. The flexibility can be confusing because there are many conventions. Likewise, the commands can be shortened, like I’m doing above, so long as they aren’t ambiguous. This is built-in, not special configuration needed. btrfs fi sh is the same as btrfs filesystem show and each user figures out their own shortening convention.

To do an incremental backup, first make some changes inside of Tax/, then:

btrfs sub snap -r Tax Tax.20220126-2
btrfs send -p Tax.20220126 Tax.20220126-2 | sudo btrfs receive  /run/media/solomon/backup

Make more changes to Tax, do an incremental backup…

btrfs sub snap -r Tax Tax.20220126-3
btrfs send -p Tax.20220126-2 Tax.20220126-3 | sudo btrfs receive  /run/media/solomon/backup

You can choose to keep or discard older snapshots, but you probably want to keep the most recent snapshot on both btrfs file sytsems, so that you can continue to do incremental backups. The cost is quite small, a couple hundred KiB of metadata plus whatever exclusive data is in the snapshot.

Let’s say you replace ~/Documents directory with a ~/Documents subvolume. And further say you’ve got LOT of stuff in ~/Documents, but it’s not changing much over time. This is a great use case for send/receive because Btrfs is already tracking the changes, so it doesn’t need to do deep traversal of either source or destination to know what files have chanced since the last snapshot.

Advanced:
You can see this in action with btrfs subvolume find-new - seeman btrfs subvolume and check out the find-new and list commands. Point the find-new command at the subvolume you’re actively using, but use the generation number from a previous snapshot of that subvolume. You’ll see a file listing of all files added/modified since the previous snapshot. This even works without snapshots, just use a lower generation number than the current value for the subvolume you’re looking at. Generations don’t represent a fixed amount of time, they can get updated quickly if the workload involves frequent fsync, otherwise the generation is incremented once every 30 seconds.

bennyisaiah · January 26, 2022, 8:43pm

Right, thanks so im still trying to get a grasp on the concept of a subvolume

Is the subvolume created when make a snapshot or is it automatically setup this way by default when we opt for a btrfs file system on the hard drive.

btrfs-and-snapper-the-next-steps-from-pure-filesystem-features-to-integration-and-compliance-15-638

dalto · January 26, 2022, 8:48pm

The installer does create some subvolumes. You can see them with:

sudo btrfs subvolume list /

That being said, you can create more if you want.

It depends what you mean. If you try to take a snapshot of something that isn’t currently a subvolume, it will not create one for you. However, snapshots are a type of subvolume so the question is somewhat hard to answer.

bennyisaiah · January 26, 2022, 9:01pm

Thanks very much for the instructions. I now have a much better handle on how to execute this. I’ll let you know how I go but I decided I may as well backup the entire Documents directory as a snapshot backup as Dalto recommended, since it’s much faster this way

Although here is my concern, from my experience with timeshift which used snapshots I was unable to restore backups on two occasions and it was a huge data loss of important files.

I decided to avoid snapshots as i preffered to copy/duplicate/clone the raw files instead.

I’m not experienced with snapshots enough to trust them enough. Are snapshots compressed like zip or tar files?

Very scared to lose data again if I depend on the snapshot.

bennyisaiah · January 26, 2022, 9:16pm

Okay so subvolume is the way the snapshot is organised when you extract it.

I should say the method of preference I had was more of a robocopy style without subvolumes, and without snapshots and rsync.

I think for my use case using the terminal to use copy command should be good for smaller directories, can this be done across different drives easily.

The issues I’m having are mentioned in this video hence my search for a Linux style robocopy. Sorry to reference windows but this method may be quicker for my use case, what do you think.

dalto · January 26, 2022, 9:25pm

It is called rsync. I know you don’t believe us, but rsync is the Linux alternative of robocopy.

bennyisaiah · January 26, 2022, 10:04pm

Okay okay sorry sorry I’m converted…Im taking the plunge now into the world of rsync and will report back today how I went. Thanks for your patience

chrismurphy · January 26, 2022, 10:42pm

A subvolume is more than a directory but less than a file system. Btrfs gets the name from b-trees, which you can read about on Wikipedia and elsewhere, but Btrfs has many btrees: fs tree, chunk tree, device tree, uuid tree, extent tree, root tree. And files tree, which is what we call subvolumes and subvolume snapshots. You can make snapshots of snapshots. A snapshot is just a prepopulated subvolume. Subvolumes share all the other trees on a Btrfs file system.

chrismurphy · January 26, 2022, 11:06pm

You can use regular tools like Nautilus (Files), Dolphin, ls and cp, to copy files out of your snapshots. So you should never be separated from your data.

Even if you use something like btrbk to help automate the creation of btrfs snaphots, and send/receive to another file system, you can still just navigate to the snapshot as if it’s a regular directory and copy files out. Including using rsync.

bennyisaiah · January 27, 2022, 4:02am

Okay I understand everything here

cd ~/Documents
btrfs sub create Tax2
mv Tax/* Tax2
rm -rf Tax
mv Tax2 Tax

and here

btrfs sub snap -r Tax Tax.20220126
sudo btrfs send Tax.20220126 | sudo btrfs receive  /run/media/solomon/backup

Although I dont understand why we need to do this process first

btrfs sub create Tax2
mv Tax/* Tax2
rm -rf Tax
mv Tax2 Tax

So we are creating a sub-volume here ?

bennyisaiah · January 27, 2022, 4:44am

It seems a bit tedious to start with taking snapshots of my Tax directory so instead I decided its easier to just backup to backup and copy the entire Documents directory.

This is what I have so far

[solomon@fedora Documents]$ cd
[solomon@fedora ~]$ mkdir .snapshot
[solomon@fedora ~]$ ls
Applications    Desktop    Fedora-Workstation-Live-x86_64-35_Beta-1.2.iso  pCloudDrive  SoftMaker  xmrig
cpuminer-multi  Documents  Monero                                          Pictures     Templates
debug.out       Downloads  Music                                           Public       Videos
[solomon@fedora ~]$ ls -ad .*
.              .bash_logout   .dbus        .librewolf    .pcloud    .steampath            .wget-hsts
..             .bash_profile  .fontconfig  .local        .pki       .steampid             .zohoassist
.android       .bashrc        .gnome       .mozilla      .qt        .var                  .zoom
.anydesk       .cache         .gnupg       .nx           .snapshot  .vnc
.bash_history  .config        .lesshst     .openshot_qt  .steam     .wallpaperdownloader
[solomon@fedora ~]$ pwd
/home/solomon
[solomon@fedora ~]$ cd Documents
[solomon@fedora Documents]$ pwd
/home/solomon/Documents
[solomon@fedora Documents]$ sudo btrfs subvolume snapshot -r /home/solomon/Documents /.snapshot/home-backupJan2022
ERROR: Not a Btrfs subvolume: Invalid argument
[solomon@fedora Documents]$

chrismurphy · January 27, 2022, 5:21am

Yes. It’s initial setup. Create a subvolume, move everything that’s in the directory into the subvolume, delete the directory (it should be rmdir because it should be empty), rename the subvolume.

bennyisaiah · January 27, 2022, 6:19am

Thanks. Is there a GUI method to the btrfs snapshot?

[solomon@fedora Documents]$ btrfs sub create Tax2
Create subvolume './Tax2'
[solomon@fedora Documents]$ cd Tax
[solomon@fedora Tax]$ mv home/solomon/Documents/Tax /Tax2
mv: cannot stat 'home/solomon/Documents/Tax': No such file or directory

[solomon@fedora Tax]$ pwd
/home/solomon/Documents/Tax
[solomon@fedora Tax]$

Not sure whats happening here, i know its obvious just cant see it…

bennyisaiah · January 27, 2022, 12:42pm

Ah I see it worked…!

So now I am cooking with gas, I will move the snapshot subvolume with aformentioned command

[solomon@fedora Documents]$ ls -d -- */
 Business/   Personal/   Tax2/          'To Do/'
 Keepass/    Tax/        Tax.20220126/  'Typora /'
[solomon@fedora Documents]$ sudo btrfs send Tax.20220126 | sudo btrfs receive  /run/media/solomon/backup
[sudo] password for solomon: 
Sorry, try again.
[sudo] password for solomon: 
ERROR: realpath(/run/media/solomon/backup) failed: No such file or directory
At subvol Tax.20220126
[solomon@fedora Documents]$ lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                         8:0    0 232.9G  0 disk  
└─sda1                      8:1    0 232.9G  0 part  
sdb                         8:16   0   2.7T  0 disk  
└─sdb1                      8:17   0   2.7T  0 part  /run/media/solomon/3TB WD Red
sdc                         8:32   0 223.6G  0 disk  
├─sdc1                      8:33   0   600M  0 part  /boot/efi
├─sdc2                      8:34   0     1G  0 part  /boot
└─sdc3                      8:35   0   222G  0 part  
  └─luks-2aafcb54-d261-46b8-8423-2ccc7ddc45cb
                          253:0    0   222G  0 crypt /home
                                                     /
sdd                         8:48   1     0B  0 disk  
sde                         8:64   1   7.4G  0 disk  
├─sde1                      8:65   1   2.9G  0 part  /run/media/solomon/Ubuntu 21.10 amd64
├─sde2                      8:66   1   4.1M  0 part  
└─sde3                      8:67   1   300K  0 part  
sr0                        11:0    1  1024M  0 rom   
zram0                     252:0    0     8G  0 disk  [SWAP]
[solomon@fedora Documents]$ sudo btrfs send Tax.20220126 | sudo btrfs receive /run/media/solomon 3TB WD Red
btrfs receive: exactly 1 argument expected, 4 given
usage: btrfs receive [At subvol Tax.20220126
options] <mount>
       btrfs receive --dump [options]

    Receive subvolumes from a stream

    Receives one or more subvolumes that were previously
    sent with btrfs send. The received subvolumes are stored
    into MOUNT.
    The receive will fail in case the receiving subvolume
    already exists. It will also fail in case a previously
    received subvolume has been changed after it was received.
    After receiving a subvolume, it is immediately set to
    read-only.

    -q|--quiet       suppress all messages, except errors
    -f FILE          read the stream from FILE instead of stdin
    -e               terminate after receiving an <end cmd> marker in the stream.
                     Without this option the receiver side terminates only in case
                     of an error on end of file.
    -C|--chroot      confine the process to <mount> using chroot
    -E|--max-errors NERR
                     terminate as soon as NERR errors occur while
                     stream processing commands from the stream.
                     Default value is 1. A value of 0 means no limit.
    -m ROOTMOUNT     the root mount point of the destination filesystem.
                     If /proc is not accessible, use this to tell us where
                     this file system is mounted.
    --dump           dump stream metadata, one line per operation,
                     does not require the MOUNT parameter
    -v               deprecated, alias for global -v option
    
    Global options:
    -v|--verbose       increase output verbosity
    -q|--quiet         print only errors

[solomon@fedora Documents]$

Although I get this response

dalto · January 27, 2022, 1:19pm

It is the spaces. You need to escape them.

/run/media/solomon\ 3TB\ WD\ Red

bennyisaiah · January 27, 2022, 1:50pm

ah yes thanks I should have known that

[solomon@fedora Documents]$ sudo btrfs send Tax.20220126 | sudo btrfs receive /run/media/solomon\ 3TB\ WD\ Red
[sudo] password for solomon: 
ERROR: realpath(/run/media/solomon 3TB WD Red) failed: No such file or directory
At subvol Tax.20220126
[solomon@fedora Documents]$

Do I need to specify a blank folder on the drive to remove this error:

dalto · January 27, 2022, 2:03pm

Can we see the output of findmnt | grep solomon

bennyisaiah · January 27, 2022, 2:07pm

Certainly,

[solomon@fedora ~]$ findmnt | grep solomon
│ └─/run/media/solomon/3TB WD Red /dev/sdb1                                                              btrfs                                                                       rw,nosuid,nodev,relatime,seclabel,space_cache,subvolid=5,subvol=/
│ └─/home/solomon/pCloudDrive     pCloud.fs                                                              fuse                                                                        rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
[solomon@fedora ~]$