title: Is spark more efficient than map-reduce for running a bunch of sequential processing tasks?
body: <p>I have a MapReduce job which works perfectly fine. I wonder can I benefit by rewriting it to Spark and will it be significant difference, like twice of the previous performance for example? My MapReduce job doesn't have a reducer it just transforms data, there is no sophisticated computations, there is no iterative operations on the same input data, so it seems like Spark in-memory features are useless in my case. </p>
  
  <p>What also worries me is that with MapReduce I don't need to set the amount of memory and the number of executors needed for job to be done, they will be calculated from the size of input data. While in case with Spark I will have to use <code>spark.executor.memory</code> and <code>num-executors</code>, dynamic allocation has some troubles on our cluster.</p>