title: Which AWS product to use to run batch jobs?
body: <p>I have a program written in C++11.  On the current input it takes too long to run.  Luckily, the data can be safely split into chunks for parallel processing, which makes it a good candidate for, say, a Map/Reduce service.</p>
  
  <p>AWS EMR could be a possible solution.  However, since my code uses many modern libraries, it's quite a pain to compile it on the instances that are assigned for Apache Hadoop clusters.  For example, I want to use soci (not available at all), boost 1.58+ (1.53 is there), etc etc.  I also need a modern C++ compiler.</p>
  
  <p>Obviously, all libraries and compilers can be manually upgraded (and the process scripted), but this sounds like a lot of manual work.  And what about slave nodes - will they get all the libraries?  Somehow I'm not sure.  And the whole process of initializing the environment can now take very long time - thus killing a lot of performance advantage that distributing the jobs was supposed to bring in to begin with.</p>
  
  <p>On the other hand, I don't really need all the advanced functionality that Apache Hadoop provides.  And I don't want to set up a personal permanent cluster with my own installation of Hadoop or similar, because I will need to run the tasks only periodically and most of the time the servers will be idle, wasting money.</p>
  
  <p>So, what would be the best product (or overall strategy) that could do the following:</p>
  
  <ul>
  <li>Grab the given binaries + set of input files</li>
  <li>Run the binaries on a predefined number of instances, using a recent Linux, ideally Ubuntu 15.10</li>
  <li>Put the resulting files in a predefined location (S3 bucket?)</li>
  <li>Shut everything down</li>
  </ul>
  
  <p>I am sure I could write a number of scripts using the aws tool to achieve that manually, but I really don't want to reinvent the wheel.  Any thoughts?</p>
  
  <p>Thanks in advance!</p>