title: Controlling reducer shuffle merge memory in Hadoop 2
body: <p>I want understand how memory is used in the reduce phase of a MapReduce Job, so I can control the settings in the designated way.</p>
  
  <p>If I understand correctly, the reducer first <strong>fetches</strong> its map output and leaves them in memory up to a certain threshold. The settings to control this are:</p>
  
  <ul>
  <li><a href="https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml" rel="nofollow">mapreduce.reduce.shuffle.merge.percent</a>: The usage threshold at which an in-memory merge will be initiated, expressed as a percentage of the total memory allocated to storing in-memory map outputs, as defined by mapreduce.reduce.shuffle.input.buffer.percent.</li>
  <li><a href="https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml" rel="nofollow">mapreduce.reduce.input.buffer.percent</a>: The percentage of memory- relative to the maximum heap size- to retain <strong>map outputs</strong> during the reduce. When the shuffle is concluded, any remaining map outputs in memory must consume less than this threshold before the reduce can begin.</li>
  </ul>
  
  <p>Next, these spilled blocks are <strong>merged</strong>. It seems the following option controls how much memory is used for the shuffle:</p>
  
  <ul>
  <li><a href="https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml" rel="nofollow">mapreduce.reduce.shuffle.input.buffer.percent</a>: The percentage of memory to be allocated from the maximum heap size to storing map outputs during the shuffle.</li>
  </ul>
  
  <p>But then, there is the setting:</p>
  
  <ul>
  <li><a href="https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml" rel="nofollow">mapreduce.reduce.shuffle.memory.limit.percent</a>: Maximum percentage of the in-memory limit that a single shuffle can consume. </li>
  </ul>
  
  <p>But it is not clear to what value this percentage applies. <strong>Is there more information available regarding these values, i.e. what they control and how they differ?</strong></p>
  
  <p>Finally, after the merge completes, the reduce process is ran on the inputs. In the [Hadoop book][1], I found that the final merge-step directly feeds the reducers. But, the default value for mapreduce.reduce.input.buffer.percent=0 contradicts this, indicating that everything is spilled to disk BEFORE the reducers start. <strong>Is there any reference on which one of these explanations is correct?</strong></p>
  
  <p>[1]: Hadoop, The definitive guide, Fourth edition, p. 200</p>