title: Hadoop error in shuffle in fetcher#1
body: <p>I'm running a parsing job in hadoop, the source is a 11GB map file with about 900,000 binary records each representing an HTML file, the map extract links and write them to the context.
  I have no reducer written for this job.</p>
  
  <ul>
  <li>When I run it on smaller files, of about 5GB with about 500,000 records it works ok.</li>
  <li>This is a single machine cluser</li>
  <li>The output has about 100 Million records, TEXT</li>
  <li>It failed after 11 maps tasks out of 200 planned.</li>
  <li>I'm running with Hadoop 0.22.0</li>
  </ul>
  
  <p>I'm getting the following error:</p>
  
  <blockquote>
    <p>org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error
    in shuffle in fetcher#1 at
    org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
    org.apache.hadoop.mapred.Child$4.run(Child.java:223) at
    java.security.AccessController.doPrivileged(Native Method) at
    javax.security.auth.Subject.doAs(Subject.java:396) at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
    at org.apache.hadoop.mapred.Child.main(Child.java:217) Caused by:
    java.lang.OutOfMemoryError: Java heap space at
    org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:58)
    at
    org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:45)
    at
    org.apache.hadoop.mapreduce.task.reduce.MapOutput.(MapOutput.java:104)
    at
    org.apache.hadoop.mapreduce.task.reduce.MergeManager.unconditionalReserve(MergeManager.java:267)</p>
  </blockquote>
  
  <p>This is my mapreduce-site.xml:</p>
  
  <pre><code>&lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.job.tracker&lt;/name&gt;
    &lt;value&gt;Hadp01:8012&lt;/value&gt;
    &lt;description&gt;The host and port that the MapReduce job tracker runs
    at.  If "local", then jobs are run in-process as a single map
    and reduce task.
    &lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.local.dir&lt;/name&gt;
    &lt;value&gt;/BigData1/MapReduce,/BigData2/MapReduce&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.child.java.opts&lt;/name&gt;
    &lt;value&gt;-Xmx1536m&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
          &lt;name&gt;dfs.datanode.max.xcievers&lt;/name&gt;
          &lt;value&gt;2048&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;mapreduce.task.io.sort.mb&lt;/name&gt;
      &lt;value&gt;300&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;io.sort.mb&lt;/name&gt;
      &lt;value&gt;300&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;mapreduce.task.io.sort.factor&lt;/name&gt;
      &lt;value&gt;100&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;io.sort.factor&lt;/name&gt;
      &lt;value&gt;100&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;tasktracker.http.threads&lt;/name&gt;
      &lt;value&gt;80&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p>Anyone has any idea how to fix it?
  Thank you!</p>