title: Match a list of IP addresses in a compressed log file
body: <p>I need to find a match to a list of IP Addresses(464k) from some (74k) of our compressed log files. When I decompress them manually and search them my script works, however, I don't have matches when the files are compressed. </p>
  
  <pre><code>from os import listdir
  from os.path import isfile, join
  import gzip
  
  realList=open('ipList', 'r')
  testlist=open('testList', 'r')
  offender=open('offenders', 'w')
  # DEV Log Dir Uncompressed #
  logDir = 'ulogs'
  
  # DEV Log Dir Compressed #
  #logDir = 'clogs'
  
  logFile=[ logFiles for logFiles in listdir(logDir) if isfile(join(logDir, logFiles))]
  count=0
  
  with testlist as host:
    ips = set(l.strip() for l in host)
    for lfile in logFile:
      print(lfile)
      # Uncompressed # Results: 870,328 Hits
      with open(logDir + '/' + lfile, 'r') as log:
      # Compressed # Results: 0 Hits
      #with gzip.open(logDir + '/' + lfile, 'r') as log:
        for line in log:
          #print(line)
          elements = line.split()
          if ips.intersection(elements):
            #print(elements[3])
            offender.write(elements[3] + '\n');
            count+=1
      print(count)
  </code></pre>