I tried to copy files from my laptop to my phone (specifically to /storage/emulated/0/Android/data). These were around 500 files in a folder whose total size is 50 MB. But then, I found that its speed is less than 1 KB???!!! And yet when I compress the folder into a tar.gz it is instantly copied! And given that I cannot decompress it on my phone due to the limitations of android, I tried doing that on my pc in the same data folder, but again, the speed of decompression was 10KB/S), so I had to delete the tar.gz, go to the folder, and copy each 50 files to the destination. The speed range became between 60-500 KB/S and I was able to successfully move it.
What I wanna now ask is, why is it that copying multiple files is taking so slow despite the fact that I only copied 500 (which isn’t many) to my phone through a perfectly functional usb? I tried to use a different phone alongside a different usb and copying was also too slow.
Note: moving/copying thousands of files from my phone to my laptop happens much much faster even if they are tens of GBs in size, and yet moving/copying files from the laptop to my phone is way too slow, so the problem happens in only one direction which is:
are you copying to/from and SD/micro SD card inserted into your phone?
most internal mobile phone storage memory is actually the same as SD memory cards.
is your USB cable to specifications?
And now a couple of answers:
SD cards / phone memory usually can be read faster then they can be written to. This is a limitation of the technology
when selecting multiple files to write to an SD card / phone memory, you need to remember that you are still limited to the write speed the SD card / phone memory will allow.
when you compress a file, like when you tgz’d it, the file size is going to be considerably smaller (say 1GB vs 4GB) so, of course it will take less time to write to your phone
when you write in random access mode vs stream, the writes will be even slower.
As an example, some of the fastest Sandisk SD memory can be written at about 60Mbps while the read speed is over 150Mbps. Generally speaking, the cheaper the memory/phone the slower it will perform. Same for the cable in some cases.
When I want to move data between my phone and computer, I usually do it using a known fast removable micro SD card and a card reader on my computer.
The problem was that most of the files were in kilobytes, so that is why the speed was slow, but then the speed becomes faster once it startes copying the larger files. Windows seems to also behave the same
As someone who recently had to move 77 million (relatively) small PDF files (average 23KB) over a network, even relatively large files of 23 KB can be a monumental pain in the butt to deal with when there are enough of them. Took multiple weeks of robocopy running 24/7 to move them
Small files, and lots of them, is basically the worst case scenario for copying anything from A to B. You spend way more time setting up and tearing down the overhead of the file move than actually pumping the data, so the throughput sucks.
As someone who recently had to move 77 million (relatively) small PDF files
Would it help to archive the small files into one big tgz file, then send that file over the network? Unpack the fat tgz file on the destination machine. That way the details of all the separate files is only handled inside each of the local machines which tends to be faster.
Not in this case - the corpus of documents was constantly being updated - older documents were sometimes removed after GDPR requests, or aged out, and new documents were being added.
In addition that would require additional disk space to hold the archive as it was created and produce an archive which was immediately out of date.
This is why I used robocopy to perform the transfer - rsync or rclone would have been just as good, but this was a Windows box to Windows box transfer and then onto a NAS drive so it was actually 3 transfers to get the corpus to the destination, which added a few several weeks on to the process… plus the reading of each file contents to produce a manifest file containing the name, filesize and SHA256 of the content.