title: Fastest way to process a large file?
body: <p>I have multiple 3 GB tab delimited files. There are 20 million rows in each file. All the rows have to be independently processed, no relation between any two rows. My question is, what will be faster?</p>
  <ol>
  <li><p>Reading line-by-line?</p>
  <pre><code>with open() as infile:
      for line in infile:
  </code></pre>
  </li>
  <li><p>Reading the file into memory in chunks and processing it, say 250 MB at a time?</p>
  </li>
  </ol>
  <p>The processing is not very complicated, I am just grabbing value in column1 to <code>List1</code>, column2 to <code>List2</code> etc. Might need to add some column values together.</p>
  <p>I am using python 2.7 on a linux box that has 30GB of memory. ASCII Text.</p>
  <p>Any way to speed things up in parallel? Right now I am using the former method and the process is very slow. Is using any <code>CSVReader</code> module going to help?
  I don't have to do it in python, any other language or database use ideas are welcome.</p>