title: How to parse files larger than 100GB in Python?
body: <p>I have text files with about 100 Gb size with the below format (with duplicate records of line and ips and domains) :</p>
  
  <pre><code>domain|ip 
  yahoo.com|89.45.3.5
  bbc.com|45.67.33.2
  yahoo.com|89.45.3.5
  myname.com|45.67.33.2
  etc.
  </code></pre>
  
  <p>I am trying to parse them using the following python code but I still get Memory error. Does anybody know a more optimal way of parsing such files? (Time is an important element for me)</p>
  
  <pre><code>files = glob(path)
  for filename in files:
      print(filename)
      with open(filename) as f:
          for line in f:
              try:
                  domain = line.split('|')[0]
                  ip = line.split('|')[1].strip('\n')
                  if ip in d:
                      d[ip].add(domain)
                  else:
                      d[ip] = set([domain])
              except:
                  print (line)
                  pass
  
      print("this file is finished")
  
  for ip, domains in d.iteritems():
      for domain in domains:
          print("%s|%s" % (ip, domain), file=output)
  </code></pre>