title: Hadoop Controlling Split Size for Similar Sections of Input Data
body: <p>I've got a situation where my input data looks something like the following.</p>
  
  <pre><code>AA1
  AA2
  AA3
  AA4
  BB1
  BB2
  BB3
  CC1
  CC2
  CC3
  CC4
  CC5
  CC6
  </code></pre>
  
  <p>What I want to do is split the data up into InputSplits where each split covers a section of the strings that begin with certain leading letters. For example the 1st input split would be all strings that start with "AA", the 2nd split would be those that start with "BB", etc. </p>
  
  <p>I want to do it this way because my data needs to be together like that in order for the reduce phase to correctly operate. </p>
  
  <p>What I've been playing around with so far is writing my own <code>InputFormat</code> and <code>RecordReader</code> classes to do this, however I see in some examples (http://developer.yahoo.com/hadoop/tutorial/module5.html#fileformat) that the splits are already created by the time the reader gets to them. I believe that I run into danger of having splits not align correctly with the boundaries between strings. </p>
  
  <p>In order to make this work fully, do I have to implement my own version of <code>InputFormat</code> <code>getSplits</code> function? If I do this, do I run a risk of distributing my splits across machines in a fashion that does not take advantage of machine locality? Finally, is there a better way to this in general?</p>
  
  <p>Any help is appreciated. Thanks,</p>
  
  <p>mj</p>
  
  <p>EDIT 0</p>
  
  <p>I'm including more information per the request of several commentators. </p>
  
  <p>The objective of my program is to compare strings that belong in groups to find the overlap between those strings and record which strings together share that overlap. Consider the following example. </p>
  
  <pre><code>AAAA
  AAAB
  AAAC
  AAB
  BAAA
  </code></pre>
  
  <p>All the strings that share an "A" at the beginning have some overlap that is common between them. The one that starts with "B" obviously doesn't. When it comes to actually discovering what the specific overlap is, and building those groups, if I'm looking at "AAAA",  I need to compare all the way down to "BAAA" and no further. My concern is that the InputSplits will chop up my data such that certain strings won't be compared against and I will have missing/incomplete groups. I was hoping to use the Map step (or reading of data) to split the problem into these groups and then allow the Reduce step to calculate the groups and return results. </p>
  
  <p>I've got millions of strings like this and it takes a while on a single machine. I've logically implemented a ton of "tricks" to streamline the process and make it run fast. I was hoping Hadoop could step in and help and make it even faster. </p>
  
  <p>Joe K - to answer your question, I don't know the extent of the overlap between all strings. Overlap can differ, for example AA1 can overlap 2 characters all the way through AA4, but if AB5 were present, only 1 character would overlap. The strings can vary in length greatly so you may get huge overlap in other instances. Also, detecting what exactly the overlap is is what I wanted to do in the reduce phase. That's what my whole objective was. </p>
  
  <p>I don't know if the shuffle/sort phase will correctly distribute the adjacent strings to the same reducer or if there will be breaks. My ignorance of the process unfortunately is tripping me up here. </p>
  
  <p>EDIT 0 END</p>