Couldnt wait until tonight and just did a little test, with ~/VirtualBox VMs and .local/share/libvirt/images excluded from the backup. Went to drink a coffee and when I came back, maybe 10 minutes after, the backup was done.
So, in essence, the issue are the large VMs that are transferred entirely every time…