How to sync two folders in one go?

Ask Fedora
1

Two Folder trees: Source and Target
Source contains Common + ExtraS (items in Source but not in Target)
Target contains Common + ExtraT (items in Target but not in Source)

After running one pass of rsync, I want

Source = Target = Common + ExtraS + ExtraT

(that is, never remove any files, and add missing files)

If one pass is not possible, what is the solution for a two pass rsync?

1 Like
2

rsync only copies one way at a time, so I think you would have to do it in two passes. rsync does not delete anything unless you specify a deletion flag like --delete. So I think just rsync -a /a/ /b && rsync -a /b/ /a should do it (maybe test that with a couple small directories first to be sure).

P.S. Beware that you may need more that just -a if you want to preserve some types of special files like hard links or sparse files.

4 Likes
3

Add this, otherwise the result is unreliable due to a race condition:
https://man.cx/rsync/#:~:text=--update

1 Like
4

After one pass,

sent 48.32G bytes  received 346.63K bytes  42.82M bytes/sec
total size is 339.06G  speedup is 7.02
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]

What options I can use to log which file failed so that I can further one by one? Or just retry those failed files?

5

What should I add?

1st pass:
rsync --archive -hh --info=stats1,progress2 --progress --modify-window=1 source/. target/

2nd pass:
rsync --archive -hh --info=stats1,progress2 --progress --modify-window=1 target/. source/

6

Use the verbose flag and then grep for “failed”.
E.g. here on a ext4 filesystem I always see "/media/lost+found": Operation not permitted "

1 Like
7

Sorry, I expected the Scroll-To-Text-Fragment feature to work cross browser.

1 Like
8

No worries.

Thanks for help. Yes, I want -u as well.

1 Like
9

Please help me to understand the difference and how to memorize it:

In the treeS, there are three folders
/treeS

  • aS
  • bS
  • cS

Under current directory, there are existing files and folders, but no aS/bS/cS .

  1. what are the difference of:
    a. rsync -a /treeS/cS ./
    b. rsync -a /treeS/cS/ ./
    c. rsync -a /treeS/cS/. ./
    or even
    d. rsync -a /treeS/cS/* ./

I just want to sync /treeS/cS so that it will result in pwd/cS . Which of a/b/c I should use?

10

rsync is unique and confusing about how it handles paths. I find it difficult to remember as well. I think c is equal to a and d is equal to b. I usually just use the a and b forms (and I always leave the / off the end of the target path). There might be a gotcha (a trick) to watch out for with form d in that it will not match dotfiles unless the dotglob shell option is set (it normally isn’t).

There is also “/./” to be aware of. It allows you to work with relative paths. I don’t think I can do a better job at explaining it than the man page. When I have a large operation to do or when I’m writing a script, I try to test things carefully with a small example.

Sorry that I’m not able to help more. I don’t know any mnemonics.

I think you want to use rsync -a /treeS/cS . to duplicate cS under the current working directory.

2 Likes
11

Any experience sharing is valuable and helpful!

What I find problematic is with the source path syntax is, with bash path completion (by pressing tab), the result is usually not what I wanted.

1 Like
12

For rsync, the destination doesn’t matter so much, but for the source the trailing slash is very important. If you don’t specify one, then you are copying the directory. If you do, then you are copying the directory contents.

So b and c are the same, and d is almost the same. For d, the * is expanded by the shell, which does not expand hidden files, so you would get a few less files that way.

If you are uncertain, pass --verbose / -v and also --dry-run/-n to see what it will do.

4 Likes
13

rsync -avhP --ignore-existing /source /Destination

Example:
rsync -avhP /mnt/TempSeagate/Jim1 /mnt/fourTB_RaidZ/Jim/

rysync options explained

a= recursive, copy links, preserve permissions,preserve modification time,preserve group, preserve owner, preserve Device files
v= verbose output
h= human readable
P= keep partially transfered files and show progress

#note that using the -z option to compress may be useful to limit bandwidth over a network, however if using rsync from one dir to another on the same local host, z is uneeded and will slow down the transfer speed to compress the dataz.

Perhaps, helpful? It is what I use but I have to do it one command process at a time. You can line up a bunch in chron jobs if you want. Fire them off a few seconds apart. Someone correct me if I am wrong? Rsync can run multiple insistence? I now I am doing that on FreeBSD server.

Fedora is my daily workstation workhorse. Thanks to any community devs reading this post. :wink:

2 Likes