title: Out of memory error in Mapreduce shuffle phase
body: <p>I am getting strange errors while running a <strong>wordcount-like</strong> mapreduce program. I have a hadoop cluster with 20 slaves, each having 4 GB RAM. I configured my map tasks to have a heap of 300MB and my reduce task slots get 1GB. I have 2 map slots and 1 reduce slot per node. Everything goes well until the first round of map tasks finishes. Then there progress remains at 100%. I suppose then the <strong>copy phase</strong> is taking place. Each map task generates something like:</p>
  
  <pre><code>Map output bytes    4,164,335,564
  Map output materialized bytes   608,800,675
  </code></pre>
  
  <p>(I am using SnappyCodec for compression)</p>
  
  <p>After <strong>stalling</strong> for about an hour the reduce tasks crach with the following exception: </p>
  
  <pre><code>    Error: java.lang.OutOfMemoryError: Java heap space at  
  org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1703) at
  org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1563) at
  org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401) at
  org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333
  </code></pre>
  
  <p>I was googling and found this link but I don't really know what to make of it:
  <a href="http://hadoop-common.472056.n3.nabble.com/Shuffle-In-Memory-OutOfMemoryError-td433197.html">hadoop common link</a></p>
  
  <p>I don't understand why hadoop would experience any problems in copying and merging if it is able to perform a terasort benchmark. It cannot be that all map output should fit into the RAM of the reducer thread. So what is going on here?</p>
  
  <p>In the link provided above they have a discussion about tuning the following parameters:</p>
  
  <pre><code>mapreduce.reduce.shuffle.input.buffer.percent = 0.7
  mapreduce.reduce.shuffle.memory.limit.percent = 0.25
  mapreduce.reduce.shuffle.parallelcopies = 5
  </code></pre>
  
  <p>They claim that the fact that the product of the parameters is >1 allows for <strong>heapsize errors</strong>.
  EDIT: Note that 5*1.25*0.7 is still &lt;1 so focus om my second solution post!)
  Before restarting this intensive simulation I would be very happy to hear about someone's opinion concerning the problem I am facing since it is bothering for almost a week now. I also seem to not completely understand what is happening in this copy phase, I'd expect a merge sort on disk not to require much heap size?</p>
  
  <p><strong>Thanks a lot</strong> in advance for any helpful comments and answers!</p>