title: Apache Spark to S3 upload performance Issue
body: <p>I'm seeing a major performance issue when Apache Spark uploads its results to S3. As per my understanding it goes these steps...</p>
  
  <ol>
  <li><p>Output of final stage is written to <code>_temp/</code> table in HDFS and the same is moved into <code>"_temporary"</code> folder inside the specific <code>S3</code> folder.</p></li>
  <li><p>Once the whole process is done - Apache spark completes the <code>saveAsTextFile</code> stage and then files inside <code>"_temporary"</code> folder in <code>S3</code> are moved into the main folder. This is actually taking a long time [ approximately 1 min per file (average size : 600 MB BZ2) ]. This part is not getting logged in the usual <code>stderr</code> log.</p></li>
  </ol>
  
  <p>I'm using Apache <code>Spark 1.0.1</code> with <code>Hadoop 2.2</code> on AWS EMR.</p>
  
  <p>Has anyone encountered this issue ?</p>
  
  <p><strong>Update 1</strong></p>
  
  <p>How can I increase the number of threads that does this move process ?</p>
  
  <p>Any suggestion is highly appreciated...</p>
  
  <p>Thanks</p>