title: Python: split a large text file by many headers
body: <p>I have a large text file, which looks like this:</p>
  
  <pre><code>lat lon altitude pressure
  3 lines data group bsas
  2.3 4.5 45.0 875
  5.6 6.5 46.2 676
  3.4 3.4 48.2 565
  6 lines data group sdad
  3.4 4.5 56.1 535
  5.6 6.5 46.2 676    
  3.4 4.5 56.1 535
  2.3 4.5 45.0 875
  5.6 6.5 46.2 676
  3.4 3.4 48.2 565
  50 lines data group asdasd
  5.5 6.6 44.5 343
  ...
  3.7 8.4 56.5 456
  ... and so on
  </code></pre>
  
  <p>I want to split whole text file to separate data group, each data group will be stored in a 2d array. Till now I have tried 2 ways to do that.</p>
  
  <p>The first way is going through each line and get data as follows:</p>
  
  <pre><code># define an object class called Wave here
  # each object has 4 attributes: lat, lon, altitude, pressure
  wave_list = []
  with open(filename, 'r') as f:
      next(f) # skip the header
      wave = Wave()
      for i, line in enumerate(f):
          if 'data' in line:
              if wave is not empty:
                  wave_list.append(wave)
              wave = Wave()
          else:
              wave.lat.append(line.split()[0])
              wave.lon.append(line.split()[1])
              wave.altitude.append(line.split()[2])
              wave.pressure.append(line.split()[3])
          wave_list.append(wave)
  return wave_list
  </code></pre>
  
  <p>The second approach is using numpy loadtext:</p>
  
  <pre><code>f = open(filename, 'r')
  txt = f.read()
  # split by "data", remove the first element
  raw_chunks = txt.split("data")[1:]
  # define a new list to store results
  wave_list = []
  # go through each chunk
  for rc in raw_chunks:
      # find the fisrt index of "\n"
      first_id = rc.find("\n")
      # find the last index of "\n"
      last_id = rc.rfind("\n")
      # temporary chunk
      temp_chunk = rc[first_id:last_id]
      # load data using loadtxt
      data = np.loadtxt(StringIO(temp_chunk)          
      wave = Wave()
      wave.lat = data.T[0]
      wave.lon = data.T[1]
      wave.altitude = data.T[2]
      wave.pressure = data.T[3]
      wave_list.append(wave)
  return wave_list
  </code></pre>
  
  <p>However, both approaches are quite slow. I have a look at pandas documentation but cannot find the way to avoid the headers in the middle of the file. I also have a look at different questions for examples:</p>
  
  <p><a href="https://stackoverflow.com/questions/19791024/splitting-a-file-based-on-text-in-python">Splitting a file based on text in Python</a></p>
  
  <p><a href="https://stackoverflow.com/questions/13622326/split-the-text-file-in-python">Split the text file in python</a></p>
  
  <p><a href="https://stackoverflow.com/questions/15602848/how-to-split-and-parse-a-big-text-file-in-python-in-a-memory-efficient-way?rq=1">How to split and parse a big text file in python in a memory-efficient way?</a></p>
  
  <p>but none of them solves my problem. Is there any faster way to read this kind of text file. Thank you in advance.</p>