title: Most efficient way in Python to iterate over a large file (10GB+)
body: <p>I'm working on a Python script to go through two files - one containing a list of UUIDs, the other containing a large amount of log entries - each line containing one of the UUIDs from the other file. The purpose of the program is to create a list of the UUIDS from file1, then for each time that UUID is found in the log file, increment the associated value for each time a match is found.</p>
  
  <p>So long story short, count how many times each UUID appears in the log file.
  At the moment, I have a list which is populated with UUID as the key, and 'hits' as the value. Then another loop which iterates over each line of the log file, and checking if the UUID in the log matches a UUID in the UUID list. If it matches, it increments the value.</p>
  
  <pre><code>    for i, logLine in enumerate(logHandle):         #start matching UUID entries in log file to UUID from rulebase
          if logFunc.progress(lineCount, logSize):    #check progress
              print logFunc.progress(lineCount, logSize)  #print progress in 10% intervals
          for uid in uidHits:
              if logLine.count(uid) == 1:             #for each UUID, check the current line of the log for a match in the UUID list
                  uidHits[uid] += 1                   #if matched, increment the relevant value in the uidHits list
                  break                                #as we've already found the match, don't process the rest
          lineCount += 1               
  </code></pre>
  
  <p>It works as it should - but I'm sure there is a more efficient way of processing the file. I've been through a few guides and found that using 'count' is faster than using a compiled regex. I thought reading files in chunks rather than line by line would improve performance by reducing the amount of disk I/O time but the performance difference on a test file ~200MB was neglible. If anyone has any other methods I would be very grateful :)</p>