title: Processing paraphragraphs in text files as single records with Hadoop
body: <p>Simplifying my problem a bit, I have a set of text files with &quot;records&quot; that are delimited by double newline characters. Like</p>
  <blockquote>
  <p>'multiline text'</p>
  <p>'empty line'</p>
  <p>'multiline text'</p>
  <p>'empty line'</p>
  </blockquote>
  <p>and so forth.</p>
  <p>I need to transform each multiline unit separately and then perform mapreduce on them.</p>
  <p>However, I am aware that with the default wordcount setting in the hadoop code boilerplate, the input to the <code>value</code> variable in the following function is just a single line and there are no guarantees that the input is contiguous with the previous input line.</p>
  <pre><code>public void map(LongWritable key, Text value, 
                  OutputCollector&lt;Text, IntWritable&gt; output, 
                  Reporter reporter) throws IOException ;
  </code></pre>
  <p>And I need it to be that the input <code>value</code> is actually one unit of the double newline delimited multiline text.</p>
  <p>Some searching turned up a <code>RecordReader</code> class and a <code>getSplits</code> method but no simple code examples that I could wrap my head around.</p>
  <p>An alternative solution is to just replace all newline characters in the multiline text with space characters and be done with it. I'd rather not do this because there's quite a bit of text and it's time consuming in terms of runtime. I also have to modify a lot of code if I do this so dealing with it through hadoop would be most attractive for me.</p>