title: Objects from memory as input for Hadoop/MapReduce?
body: <p>I am working on the parallelization an algorithm, which roughly does the following:</p>
  
  <ol>
  <li>Read several text documents with a total of 10k words.</li>
  <li>Create an objects for every word in the text corpus.</li>
  <li>Create a pair between all word-objects (yes, O(n)). And return the most frequent pairs.</li>
  </ol>
  
  <p>I would like to parallelize the 3. step by creating the pairs between the first 1000 word-objects the rest on the fist machine, the second 1000 word-objects on the next machine, etc.</p>
  
  <p>My question is how to pass the objects created in the 2. step to the Mapper? As far as I am aware I would require input files for this and hence would need to serialize the objects (though haven't worked with this before). Is there a direct way to pass the objects to the Mapper?</p>
  
  <p>Thanks in advance for the help</p>
  
  <p>Evgeni</p>
  
  <p><strong>UPDATE</strong>
  Thank you for reading my question before. Serialization seems to be the best way to solve this (see java.io.Serializable). Furthermore, I have found this tutorial useful to read data from serialized objects into hadoop: <a href="http://www.cs.brown.edu/~pavlo/hadoop/" rel="nofollow">http://www.cs.brown.edu/~pavlo/hadoop/</a>).</p>