It seems, that I misled you by the formulation of my previous reply. While the context
might be different for each file, I need to change it only for the directories…
… because I am in fact using “cp -a” for everything else.
Anyway, in the meantime I re-wrote the program in Python. The memory usage is now
in a “decent” area (less then 1/2 of the Perl version) and I actually managed to run
almost a full backup. I got stuck at ca. 40 GB, because one of the files has an “a Umlaut”
(the German a with 2 dots on top) in it’s name. So apart from that small detail, I am now
happy.
I can’t comment on rsync, because, while I studied the man page, I never used it.
As far as “cp” goes, YMMV, but in my experience it doesn’t have a problem. Where
the problems come in, is that it is used at shell level, and you have to escape it properly
so that the command knows where to look. (Remember, we are doing a backup, so
we are not creating new names. If the OS gives us a name, you should expect that
a copy command can take it. But don’t let me get onto the soap box talking about the
filenames you encounter nowadays.)
In fact, the issue was not with the cp, but with printing the name (I protocol the changes).
Python prints in UTF-8, and an ISO text stops it dead.
Anyway, now I have a clean version. Python is slightly, but not significantly, faster, but
the real difference is in the memory usage. The Python version uses ca. 14 MB, Perl
(thanks to the fact, that it doesn’t give anything back) uses ca. 3,500 MB.
One significant difference between the functioning of rsync and cp.
cp will not copy anything hidden, such as the many dot files and folders within ones home directory unless it is explicity named. It copies only that which is seen by ls (without options) by default.
rsync will by default copy everything existing in the directory including the hidden dot files.
Thus rsync makes a backup simpler as it does a complete copy instead of a selective copy.
There is actually nothing in this thread which would lead us to this
comparison, but since we got there, let’s see:
Once you read the man pages, you will find that the above has
to be rephrased as follows:
Neither cp nor rdist will copy anything you do not explicitely
tell it to copy. If you tell it to copy “*”, it will not copy
any dot files or directories.
Both cp and rdist, once given the “-a” option, will copy everything
existing in the directory you ask them to copy, including the dot
files and directories.
If you don’t believe it, look at the examples at the end.
You can NOT talk about “cp” doing a selective copy and “rsync”
not doing one. The only selection is done by the shell, which is
interpreting your request, and it is done the same way for both
cp and rsync.
But perhaps we should look at the original purpose of these commands:
“cp” was written to copy files or directories from one place
to another.
“rsync” was written to synchronise (keep identical) data collections
on separate machines.
You can copy with both of them, you can synchronise data collections
with both of them, you can use both of them for doing backups. rsync
might make your life a bit easier, if you are moving between two
different machines.
So - as always - your mileage will vary depending on what you want
to do, how you do it, and how deep your knowledge of (and experience
with) the different commands is.
And - again - none of this has got anything to do with the problem
described in the title of this thread.
And here are the examples (I am leaving off the “ls -a” used for
displaying the contents of the resulting directories):
[test]$ mkdir src
[test]$ touch src/a
[test]$ touch src/b
[test]$ touch src/.dot
src:
. .. a b .dot
[test]$ cp src desCa
cp: -r not specified; omitting directory 'src'
[test]$ rsync src desRa
skipping directory src
neither desCa nor desRa exists
[test]$ cp -a src desCb
[test]$ rsync -a src desRb
desCb:
. .. a b .dot
desRb:
. .. src
desRb/src:
. .. a b .dot
[test]$ cd src/
[src]$ cp * ../desCc
cp: target '../desCc' is not a directory
[src]$ mkdir ../desCc
[src]$ cp * ../desCc
[src]$ cp -a * ../desCd
cp: target '../desCd' is not a directory
[src]$ mkdir ../desCd
[src]$ cp -a * ../desCd
desCc:
. .. a b
desCd:
. .. a b
[src]$ rsync * ../desRc
[src]$ rsync -a * ../desRd
desRc:
. .. a b
desRd:
. .. a b
[src]$ cp . ../desCe
cp: -r not specified; omitting directory '.'
[src]$ rsync . ../desRe
skipping directory .
neither desCe nor desRe exists
[src]$ cp -a . ../desCf
[src]$ rsync -a . ../desRf
desCf:
. .. a b .dot
desRf:
. .. a b .dot