title: Divide .csv file into chunks with Python
body: <p>I have a large .csv file that is well over 300 gb.  I would like to chunk it into smaller files of 100,000,000 rows each (each row has approximately 55-60 bytes).</p>
  
  <p>I wrote the following code:</p>
  
  <pre><code>import pandas as pd
  df = pd.read_csv('/path/to/really/big.csv',header=None,chunksize=100000000)
  count = 1
  for chunk in df:
      name = '/output/to/this/directory/file_%s.csv' %s count
      chunk.to_csv(name,header=None,index=None)
      print(count)
      count+=1
  </code></pre>
  
  <p>This code works fine, and I have plenty of memory on disk to store the approximate 5.5-6 gb at a time, but it's <em>slow</em>.</p>
  
  <p>Is there a better way?</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>I have written the following iterative solution:</p>
  
  <pre><code>with open('/path/to/really/big.csv', 'r') as csvfile:
      read_rows = csv.reader(csvfile)
      file_count = 1
      row_count = 1
      f = open('/output/to/this/directory/file_%s.csv' %s count,'w')
      for row in read_rows:
          f.write(''.join(row))
          row_count+=1
          if row_count % 100000000 == 0:
              f.close()
              file_count += 1
              f = open('/output/to/this/directory/file_%s.csv' %s count,'w')
  </code></pre>
  
  <p><strong>EDIT 2</strong></p>
  
  <p>I would like to call attention to Vor's comment about using a Unix/Linux split command, this is the fastest solution I have found.</p>