title: Using negative lookbehind regex as record delimiter to read Hadoop File in Spark
body: <p>I have a lot of large tsv files that I am processing using Apache Spark. These files have newlines as record delimiters but every few thousand records, some escape sequence of newline preceded by \ is added within the column. </p>
  
  <p>Here is a sample dataset </p>
  
  <pre><code>a   b   c
  a   b   c
  a   b   c
  a   b   c
  a   b   c
  a   b   c\
  \   \   \
  \   \   d
  a   b   c
  a   b   c
  </code></pre>
  
  <p>Whenever the newline is preceded with \, it means this is not end of record and to continue till it hits a newline without . I am using negative lookbehind assertion to pick the escaped string as one record and remaining lines as a record.</p>
  
  <pre><code>a   b   c\
  \   \   \
  \   \   d
  </code></pre>
  
  <p>Here is my code </p>
  
  <pre><code>import org.apache.hadoop.io.LongWritable
  import org.apache.hadoop.io.Text
  import org.apache.hadoop.conf.Configuration
  import org.apache.hadoop.mapreduce.lib.input.TextInputFormat
  
  val LINE_DELIMITER = "(?&lt;!\\\\)\n"
  
  @transient val conf = new Configuration
  conf.set("textinputformat.record.delimiter", LINE_DELIMITER)
  val raw_file = sc.newAPIHadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text], conf).map(_._2.toString)
  </code></pre>
  
  <p>Instead of getting 8 records, I get the whole string. Here is the console output</p>
  
  <pre><code>scala&gt; raw_file.collect
  res12: Array[String] = 
  Array(a b   c
  a   b   c
  a   b   c
  a   b   c
  a   b   c
  a   b   c\
  \   \   \
  \   \   d
  a   b   c
  a   b   c)
  </code></pre>
  
  <p>If I take this string and split it with Line Delimiter defined above, I get an array of 8 strings. So there does not seem to be a problem with the regular expression itself. Actual dataset is 500+ columns, 4-5G file each so using wholeTextFiles and using split later is not an option.</p>
  
  <p>Is there a difference between how regex is applied in split function compared to how data is read in newAPIHadoopFile? Should I use a different regex? Or use some other mechanism to do this?</p>