title: Using Pandas to read data and skip metadata
body: <p><strong>Background</strong></p>
  
  <p>I have data files which consist of two parts: data in CSV format, and Metadata. I can use the method given here <a href="https://stackoverflow.com/questions/45722949/how-to-skip-header-and-footer-data-in-pandas-dataframe">1</a> and here <a href="https://stackoverflow.com/questions/54481859/how-to-deal-with-metadata-lines-in-pandas-read-csv">2</a> to manually skip the Metadata portion by specifying the location/line number of the beginning of the Metadata.</p>
  
  <p>Following is the sample of the data file:</p>
  
  <p><a href="https://i.stack.imgur.com/qE5D9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qE5D9.png" alt="3"></a></p>
  
  <p>Here, you can see that I can specify the line number (420) manually and use the following code to skip the Metadata:</p>
  
  <pre><code>with open('data.csv', 'r') as f:
      metadata_location = [i for i, x in enumerate(f.readlines()) if 'Metadata' in x]
  with open('data.csv', 'r') as f:
      flat_data = pd.read_csv(f, index_col=False, skiprows=lambda x: x &gt;= metadata_location[0])
  
  with open('data.csv') as f:
      df = pd.read_csv(f, index_col=False)
  df = df[:420]
  </code></pre>
  
  <p><strong>Question</strong></p>
  
  <p>How can I scan the file to capture the Metadata and then skip reading it? (I will need to process multiple such files, hence, I wish to write such a code)</p>