title: Hadoop MapReduce vs MPI (vs Spark vs Mahout vs Mesos) - When to use one over the other?
body: <p>I am new to parallel computing and just starting to try out MPI and Hadoop+MapReduce on Amazon AWS. But I am confused about when to use one over the other.</p>
  
  <p>For example, one common rule of thumb advice I see can be summarized as...</p>
  
  <ul>
  <li>Big data, non-iterative, fault tolerant => MapReduce </li>
  <li>Speed, small data, iterative, non-Mapper-Reducer type => MPI</li>
  </ul>
  
  <p>But then, I also see implementation of MapReduce on MPI (<a href="http://mapreduce.sandia.gov/" rel="noreferrer">MR-MPI</a>) which does not provide fault tolerance but <a href="http://dx.doi.org/10.1016/j.parco.2011.02.004" rel="noreferrer">seems to be</a> more efficient on some benchmarks than MapReduce on Hadoop, and seems to handle big data using out-of-core memory.</p>
  
  <p>Conversely, there are also MPI implementations (<a href="https://github.com/alibaba/mpich2-yarn" rel="noreferrer">MPICH2-YARN</a>) on new generation Hadoop Yarn with its distributed file system (HDFS).</p>
  
  <p>Besides, there seems to be provisions within MPI (Scatter-Gather, <a href="http://fault-tolerance.org/2014/11/15/tutorial-sc14-fault-tolerance-for-hpc-theory-and-practice/" rel="noreferrer">Checkpoint-Restart</a>, <a href="http://www.anl.gov/events/fault-tolerant-mapreduce-mpi-using-user-level-failure-mitigation" rel="noreferrer">ULFM</a> and other <a href="https://www.open-mpi.org/faq/?category=ft" rel="noreferrer">fault tolerance</a>) that mimic several features of MapReduce paradigm.</p>
  
  <p>And how does Mahout, Mesos and Spark fit in all this? </p>
  
  <p>What criteria can be used when deciding between (or a combo of) Hadoop MapReduce, MPI, Mesos, Spark and Mahout?</p>