title: Spring Batch - Reading multiple line log message
body: <p>I am facing a problem to read multi-line log message as a single message in our spring batch application configured with spring integration, this application has to read multiline log message (example exception stack trace) as a single message, later it has to process and classify the message for further indexing.
  Each line is identified by its timestamp (pattern mentioned above i.e. DATE_PATTERN) and it may continue mutltiple lines, I am trying to continue reading a message until I see another timestamp by overriding 
  isEndOfRecord method from SimpleRecordSeparatorPolicy when second line reaches in preProcess method I am returning true for isEndOfRecord but this is not working as expected, could any one help me to read the mentioned log file by identifying the timestamp pattern?</p>
  
  <p>I am using org.springframework.batch.item.file.FlatFileItemReader, and org.springframework.batch.item.file.mapping.PassThroughLineMapper as mapper.</p>
  
  <p>Please see for complete message,</p>
  
  <p>1) log message file :sample-message-test.log</p>
  
  <pre><code>2013-10-19 07:05:32.253 [My First Class..] LOG LEVEl  first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 
  first-message-line-2 first-message-line-2 first-message-line-2 
  first-message-line-3 first-message-line-3 first-message-line-3 
  first-message-line-4 first-message-line-4 first-message-line-4 
  first-message-line-5 first-message-line-5 
  first-message-line-6 
  2013-10-19 07:05:32.257 [My Second Class..] LOG LEVEl  second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 
  second-message-line-2 second-message-line-2 second-message-line-2 
  second-message-line-3 second-message-line-3 second-message-line-3 
  second-message-line-4 second-message-line-4 second-message-line-4 
  second-message-line-5 second-message-line-5 
  second-message-line-6
  2013-10-19 07:05:32.259 [My Third Class..] LOG LEVEl  third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 
  third-message-line-2 third-message-line-2 third-message-line-2 
  third-message-line-3 third-message-line-3 third-message-line-3 
  third-message-line-4 third-message-line-4 third-message-line-4 
  third-message-line-5 third-message-line-5 
  third-message-line-6
  </code></pre>
  
  <p>2) Batch Configuration file</p>
  
  <pre><code>&lt;batch:job id="fileReadingJob"&gt;
          &lt;batch:step id="flatFileReadingStep"&gt;
              &lt;batch:tasklet &gt;
                  &lt;batch:chunk reader="reader" writer="writer" commit-interval="10" /&gt;
              &lt;/batch:tasklet&gt;
          &lt;/batch:step&gt;
      &lt;/batch:job&gt;
  
      &lt;bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader"  scope="step"&gt;
          &lt;property name="lineMapper"&gt;
              &lt;bean class="org.springframework.batch.item.file.mapping.PassThroughLineMapper"/&gt;
          &lt;/property&gt;
          &lt;property name="bufferedReaderFactory"&gt;
              &lt;bean class="org.springframework.batch.item.file.DefaultBufferedReaderFactory"/&gt;
          &lt;/property&gt;
          &lt;property name="recordSeparatorPolicy" &gt;
              &lt;bean class="com.batchlog.explorer.batchio.FlatFileRecordSeperationPolicy"/&gt;
          &lt;/property&gt;
          &lt;property name="resource" value="file:///#{systemProperties['logfolder']}/#{jobParameters['inputfile']}" /&gt;
      &lt;/bean&gt;
      &lt;bean id="writer" class="com.batchlog.explorer.batchio.FlatFileWriter" scope="step"/&gt;
  ........
  </code></pre>
  
  <p>3) </p>
  
  <pre><code>public class FlatFileRecordSeperationPolicy extends SimpleRecordSeparatorPolicy {
  
      public static final String STARTING_OF_THE_LINE = "-STARTING_OF_THE_LINE-";
      public static final String CONTINUATION_OF_THE_FILE  = "-CONTINUATION_OF_THE_FILE-";
      public static final String END_OF_THE_LINE = "-END_OF_THE_LINE-";
      public static final String END_OF_THE_LINE_CHARACER = " \n ";
      public static final String DATE_PATTERN ="^(?&gt;\\d\\d){1,2}-(?:0?[1-9]|1[0-2])-(\\s)?(?:2[0123]|[01][0-9]):? (?:[0-5][0-9])(?::?(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?))?(?:Z|[+-](?:2[0123]|[01][0-9])(?::?(?:[0-5][0-9])))?.*?";
  
  
      @Override
          public boolean isEndOfRecord(String line) {
              if(line.matches(DATE_PATTERN) || line.startsWith(STARTING_OF_THE_LINE)
                              || line.contains(CONTINUATION_OF_THE_FILE) || line.startsWith(END_OF_THE_LINE)){
                  if(isNextLineStarts(line) || line.startsWith(END_OF_THE_LINE)){
                      return true;//to break line
                  }
          }
          return false; //to conitnue line
  
      private boolean isNextLineStarts(String preProcessOfLine){
              if(preProcessOfLine.contains(CONTINUATION_OF_THE_FILE) &amp;&amp; !preProcessOfLine.endsWith(CONTINUATION_OF_THE_FILE)){
                  String[] lines = preProcessOfLine.split(CONTINUATION_OF_THE_FILE);
                  if(lines[1].trim().matches(DATE_PATTERN)){
                      return true;
                  }
              }
              return false;
      }
      @Override
          public String preProcess(String line) {
              if(line.matches(DATE_PATTERN) &amp;&amp; !line.contains(CONTINUATION_OF_THE_FILE)){
                  line = new StringBuilder(STARTING_OF_THE_LINE).append(line).toString();
              }else if(line.startsWith(STARTING_OF_THE_LINE) &amp;&amp; !line.contains(CONTINUATION_OF_THE_FILE)){
                  line =  new StringBuilder(line.substring(STARTING_OF_THE_LINE.length())).append(CONTINUATION_OF_THE_FILE).toString();
              }else if(line.contains(CONTINUATION_OF_THE_FILE) &amp;&amp; !line.endsWith(CONTINUATION_OF_THE_FILE)){
                  String[] lines = line.split(CONTINUATION_OF_THE_FILE);
                  if(lines[1].trim().matches(DATE_PATTERN)){
                      line = new StringBuilder(END_OF_THE_LINE).append(lines[0]).toString();//.append(lines[1]).toString();
                  }else{
                      line = new StringBuilder(lines[0]).append(lines[1]).append(CONTINUATION_OF_THE_FILE).toString();
                  }
              }
                  return super.preProcess(line);
      }
      @Override
          public String postProcess(String record) {
              if(record.startsWith(END_OF_THE_LINE)){
                  record = new StringBuilder(record.substring(END_OF_THE_LINE.length())).toString();
              }else if(record.contains(CONTINUATION_OF_THE_FILE) &amp;&amp; !record.endsWith(CONTINUATION_OF_THE_FILE)){
                  String[] lines = record.split(CONTINUATION_OF_THE_FILE);
                  if(lines[1].trim().matches(DATE_PATTERN)){
                      record = new StringBuilder(END_OF_THE_LINE).append(lines[0]).toString();
                  }else{
                      record = new StringBuilder(lines[0]).append(lines[1]).toString();
                  }
              }
              return super.postProcess(record);
      }
  </code></pre>