title: why hadoop capacity scheduler uses 200% of Capacity
body: <p>I  encountered the same problem on our cluster and  returned to my pc to do some simple experiments hoping to figure it out.I configured hadoop in Pseudo-distributed mode and used the default capacity-scheduler.xml and configured the  mapred-site.xml as the following:</p>
  
  <pre><code>&lt;!-- Put site-specific property overrides in this file. --&gt;
  &lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;io.sort.mb&lt;/name&gt;
    &lt;value&gt;5&lt;/value&gt;
  &lt;/property&gt;
   &lt;property&gt;
  &lt;name&gt;mapred.job.tracker&lt;/name&gt;
   &lt;value&gt;localhost:9001&lt;/value&gt;
   &lt;/property&gt;
  &lt;property&gt;
   &lt;name&gt;mapred.child.java.opts&lt;/name&gt;
   &lt;value&gt;-Xmx10m&lt;/value&gt;
   &lt;/property&gt;
  &lt;property&gt;
  &lt;name&gt;mapred.jobtracker.taskScheduler&lt;/name&gt;
  &lt;value&gt;org.apache.hadoop.mapred.CapacityTaskScheduler&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
  &lt;name&gt;mapred.queue.names&lt;/name&gt;
  &lt;value&gt;default&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
  &lt;name&gt;mapred.cluster.map.memory.mb&lt;/name&gt;
  &lt;value&gt;100&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
  &lt;name&gt;mapred.cluster.max.map.memory.mb&lt;/name&gt;
  &lt;value&gt;200&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt; 
  </code></pre>
  
  <p>The web UI looks like this :</p>
  
  <pre><code>Queue Name  default      
  Scheduling Information
  Queue configurationfatal
  Capacity Percentage: 100.0%
  User Limit: 100%
  Priority Supported: NO
  -------------
  Map tasks
  Capacity: 2 slots
  Used capacity: 2 (100.0% of Capacity)
  Running tasks: 1
  Active users:
  User 'luo': 2 (100.0% of used capacity)
  -------------
  Reduce tasks
  Capacity: 2 slots
  Used capacity: 0 (0.0% of Capacity)
  Running tasks: 0
  -------------
  Job info
  Number of Waiting Jobs: 0
  Number of users who have submitted jobs: 1
  </code></pre>
  
  <p>Actually , it did work without anything wrong when I submitted a streaming job with one map task which occupies 2 slots and no reduce task.The streaming script is rather simple</p>
  
  <p>~/hadoop/hadoop-0.20.2/bin/hadoop  jar Streaming_blat.jar -D mapred.job.map.memory.mb=199 -D mapred.job.name='memory alloc' -D mapred.map.tasks=1 -input file://<code>pwd</code>/input/  -mapper ' /home/luo/hadoop/hadoop-0.20.2/bin/a.out' -output file://<code>pwd</code>/output/ -reducer NONE</p>
  
  <p>a.out is just a C program simply outputting the pid and ppid to a specified file.</p>
  
  <p>And problems came when I set mapred.map.tasks=3. The web UI showed</p>
  
  <pre><code>Map tasks
  Capacity: 2 slots
  Used capacity: 4 (200.0% of Capacity)
  Running tasks: 2
  Active users:
  User 'luo': 4 (100.0% of used capacity)
  </code></pre>
  
  <p>which means it already exceeds the limit of map slots I set in mapred-site.xml. As a result, it prompted something like this again and again</p>
  
  <pre><code>Killing one of the least progress tasks - attempt_201210121915_0012_m_000000_0, as the cumulative memory usage of all the tasks on the TaskTracker exceeds virtual memory limit 207618048.
  </code></pre>
  
  <p>What I want it to do is suspend the map task until there are available slots without exceeding the capacity.So what's wrong have I done ? Could any one provide some solutions? Thanks a lot.</p>