title: Match lines between two patterns in Python with regular expressions
body: <p>I am parsing log files that include lines regarding events by many jobs, identified by a job id. I am trying to get all lines in a log file between two patterns in Python. </p>
  
  <p>I have read this very useful post <a href="https://stackoverflow.com/questions/38972736/how-to-select-lines-between-two-patterns">How to select lines between two patterns?</a> and had already solved the problem with <code>awk</code> like so:</p>
  
  <pre><code>awk '/pattern1/,/pattern2/' file
  </code></pre>
  
  <p>Since I am processing the log information in a Python script, I am using <code>subprocess.Popen()</code> to execute that awk command. My program works, but I would like to solve this using Python alone.</p>
  
  <p>I know of the <code>re</code> module, but don't quite understand how to use it. The log files have already been compressed to bz2, so this is my code to open the .bz2 files and find the lines between the two patterns:</p>
  
  <pre><code>import bz2
  import re
  
  logfile = '/some/log/file.bz2'
  
  PATTERN = r"/{0}/,/{1}/".format('pattern1', 'pattern2')
  # example: PATTERN = r"/0001.server;Considering job to run/,/0040;pbs_sched;Job;0001.server/"
  re.compile(PATTERN)
  
  with bz2.BZ2File(logfile) as fh:
      match = re.findall(PATTERN, fh.read())
  </code></pre>
  
  <p>However, <code>match</code> is empty (<code>fh.read()</code> is not!). Using <code>re.findall(PATTERN, fh.read(), re.MULTILINE)</code> has no effect.
  Using <code>re.DEBUG</code> after <code>re.compile()</code> shows many lines with</p>
  
  <pre><code>literal 47
  literal 50
  literal 48
  literal 49
  literal 57
  </code></pre>
  
  <p>and two say</p>
  
  <pre><code>any None
  </code></pre>
  
  <p>I could solve the problem with loops like here <a href="https://stackoverflow.com/questions/51077894/python-print-between-two-patterns-including-lines-containing-patterns">python print between two patterns, including lines containing patterns</a> but I avoid nested for-if loops as much as I can. I belive the <code>re</code> module can yield the result I want but I am no expert in how to use it.</p>
  
  <p>I am using Python 2.7.9.</p>