title: Optimize python program to parse two large files at the same time
body: <p>I am trying to parse two large files with Python3 at the same time. As shown here:</p>
  
  <pre class="lang-py prettyprint-override"><code>dict = {}
  row = {}
  with open(file1, "r") as f1, open(file2, "r") as f2:
    zipped = zip(f1, f2)
    for line_f1, line_f2 in zipped:
      # parse the lines and save the line information in a dictionary 
      row = {"ID_1":line_f1[0], "ID_2":line_f2[0], ...}
  
      # This process takes roughly 0.0005s each time
      # it parses each pair of lines at once and returns an output
      # it doesn't depend on previous lines or lines after
      output = process(row) 
  
      # output is a string, add it to dict
      if output in dict:
         dict[output] += 1
      else:
         dict[output] = 1
  return dict
  </code></pre>
  
  <p>When I tested the above code with two smaller text files (30,000 lines each, file size = 13M) and it takes roughly 150s to finish the loop.</p>
  
  <p>When I tested with two large text files (9,000,000 lines each, file size = 3.8G) without the process step in the loop it takes roughly 670s. </p>
  
  <p>When I tested with the same two large text files with the process step. I timed that for every 10,000 items it will take roughly 60s. The time didn't grow when the number of iterations gets large.</p>
  
  <p>However, when I submit this job to a shared cluster it takes more than 36 hours for one pair of large files to finish processing. I am trying to figure out if there is any other way to process the files so it can be faster. Any suggestions would be appreciated. </p>
  
  <p>Thanks in advance!</p>