title: Using Hadoop for Parallel Processing rather than Big Data
body: <p>I manage a small team of developers and at any given time we have several on going (one-off) data projects that could be considered "<a href="http://en.wikipedia.org/wiki/Embarrassingly_parallel">Embarrassingly parallel</a>" - These generally involve running a single script on a single computer for several days, a classic example would be processing several thousand PDF files to extract some key text and place into a CSV file for later insertion into a database.</p>
  
  <p>We are now doing enough of these type of tasks that I started to investigate developing a simple job queue system using RabbitMQ with a few spare servers (with an eye to use Amazon SQS/S3/EC2 for projects that needed larger scaling)</p>
  
  <p>In searching for examples of others doing this I keep coming across the classic Hadoop New York Times example:</p>
  
  <blockquote>
    <p>The New York Times used 100 Amazon EC2 instances and a Hadoop application to process 4 TB of raw image TIFF data (stored in S3) into 11 million finished PDFs in the space of 24 hours at a computation cost of about $240 (not including bandwidth)</p>
  </blockquote>
  
  <p>Which sounds perfect? So I researched Hadoop and Map/Reduce.</p>
  
  <p>But what I can't work out is how they did it? Or why they did it?</p>
  
  <p>Converting TIFF's in PDF's is not a Map/Reduce problem surely? Wouldn't a simple Job Queue have been better?</p>
  
  <p>The other classic Hadoop example is the "wordcount" from the <a href="http://developer.yahoo.com/hadoop/tutorial/">Yahoo Hadoop Tutorial</a> seems a perfect fit for Map/Reduce, and I can see why it is such a powerful tool for Big Data. </p>
  
  <p>I don't understand how these "Embarrassingly parallel" tasks are put into the Map/Reduce pattern?</p>
  
  <p>TL;DR</p>
  
  <p>This is very much a conceptual question, basically I want to know how would I fit a task of "processing several thousand PDF files to extract some key text and place into a CSV file" into a Map/Reduce pattern?</p>
  
  <p>If you know of any examples that would be perfect, I'm not asking you to write it for me.</p>
  
  <p>(Notes: We have code to process the PDF's, I'm not asking for that - it's just an example, it could be any task. I'm asking about putting that processes like that into the Hadoop Map/Reduce pattern - when there is no clear "Map" or "Reduce" elements to a task.)</p>
  
  <p>Cheers!</p>