title: Efficient copy method in Hadoop
body: <p>Is there a faster or more efficient way of copying files across HDFS other than <code>distcp</code>. I tried both the regular <code>hadoop fs -cp</code> as well as <code>distcp</code> and both seem to be giving the same transfer rate, around 50 MBPS.</p>
  
  <p>I have 5TB of data split into smaller files of 500GB each which I have to copy to a new location on HDFS. Any thoughts?</p>
  
  <p>Edit:
  The original <code>distcp</code> is only spawning 1 mapper so I added <code>-m100</code> option to increase the mappers</p>
  
  <pre><code>hadoop distcp -D mapred.job.name="Gigafiles distcp" -pb -i -m100 "/user/abc/file1" "/xyz/aaa/file1"
  </code></pre>
  
  <p>But still it is spawning only 1 and not 100 mappers. Am I missing something here?</p>