Take the 2-minute tour ×

Running multiple scp threads simultaneously:


I'm often finding myself mirroring a set of server files a lot, and included in these server files are thousands of little 1kb-3kb files. All the servers are connected to 1Gbps ports, generally spread out in a variety of data-centers.


SCP transfers these little files, ONE by ONE, and it takes ages, and I feel like I'm wasting the beautiful network resources I have.


I had an idea; Creating a script, which divides the files up into equal amounts, and starts up 5-6 scp threads, which theoretically would then get done 5-6 times faster, no? But I don't have any linux scripting experience!


  • Is there a better solution to the mentioned problem?
  • Is there something like this that exists already?
  • If not, is there someone who would give me a start, or help me out?
  • If not to 2, or 3, where would be a good place to start looking to learn linux scripting? Like bash, or other.
share|improve this question
possible duplicate of How to copy a large number of files quickly between two servers –  Scott Pack May 22 '12 at 19:52
add comment

4 Answers

I would do it like this:
tar -cf - /manyfiles | ssh dest.server 'tar -xf - -C /manyfiles'

Depending on the files you are transferring it can make sense to enable compression in the tar commands:
tar -czf - /manyfiles | ssh dest.server 'tar -xzf - -C /manyfiles'

It may also make sense that you choose a CPU friendlier cipher for the ssh command (like arcfour): tar -cf - /manyfiles | ssh -c arcfour dest.server 'tar -xf - -C /manyfiles'

Or combine both of them, but it really depends on what your bottleneck is.
Obviously rsync will be a lot faster if you are doing incremental syncs.

share|improve this answer
add comment

Use rsync instead of scp. You can use rsync over ssh as easily as scp, and it supports "pipelining of file transfers to minimize latency costs".

One tip: If the data is compressible, enable compression. If it's not, disable it.

share|improve this answer
I just did one transfer last night with scp and am doing another similar transfer with rsync -- it seems a lot faster. However, it still seems to be transferring one file at a time — any idea how to make this do multiple threads (Beyond --include'ing and --exclude'ing a bunch of directories via script; see: sun3.org/archives/280) –  aendrew Jul 2 '12 at 10:45
add comment

I was about to suggest GNO Parallel (which still requires some scripting work on your part), but then I found pscp (which is part of pssh). That may just fit your need.

share|improve this answer
It seems pssh operates concurrently to multiple machines. I don't think it implements file-level parallelism. –  David Schwartz Nov 1 '11 at 0:05
I probably should be specific - I meant pscp. –  Rilindo Nov 1 '11 at 0:16
add comment

Possibly unrelated, but if you want something more real time you could try GlusterFS. Works well, but requires some tuning if you're wanting to efficiently read small files.

share|improve this answer
add comment

Your Answer


