title: Hadoop task schedulers: Capacity vs Fair sharing or something else?
body: <h2>Background</h2>
  
  <p>My employer is progressively shifting our resource intensive ETL and backend processing logic from MySQL to Hadoop ( dfs &amp; hive ).  At the moment everything is still somewhat small and manageable ( 20 TB over 10 nodes ) but we intend to progressively increase the cluster size.</p>
  
  <p>Now that hadoop is being shifted into production use, its becoming a bigger issue of batch scheduling and sharing the cluster between ad-hoc user hive queries, hourly M/R processes, and I believe eventually some usage of hbase.  The fear is that a naive query will be made by a user that could potentially run for an unreasonable amount of time ( say 4 hours ) clogging up the task queue and producing potential infrastructure load instabilities.</p>
  
  <h2>Question</h2>
  
  <p>Another section of my company has already been burned by Flume's immaturity, so my question is, how stable are the two known schedulers ( Capacity &amp; Fair ) and besides usage in their sponsoring companies ( Yahoo &amp; Facebook ) are they used elsewhere?</p>
  
  <h2>Edit:  Background info</h2>
  
  <p><a href="http://www.cloudera.com/blog/2008/11/job-scheduling-in-hadoop/" rel="nofollow noreferrer">http://www.cloudera.com/blog/2008/11/job-scheduling-in-hadoop/</a></p>
  
  <p><a href="http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html" rel="nofollow noreferrer">http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html</a></p>
  
  <p><a href="http://hadoop.apache.org/mapreduce/docs/r0.21.0/capacity_scheduler.html" rel="nofollow noreferrer">http://hadoop.apache.org/mapreduce/docs/r0.21.0/capacity_scheduler.html</a></p>