title: Python MemoryError with Queue and threading
body: <p>I'm currently writing a script that reads reddit comments from a large file (5 gigs compressed, ~30 gigs of data being read). My script reads the comments, checks for some text, parses them, and sends them off to a Queue function (running in a seperate thread). No matter what I do, I always get a MemoryError on a specific iteration (number 8162735 if it matters in the slightest). And I can't seem to handle the error, Windows just keeps shutting down python when it hits. Here's my script: </p>
  
  <pre><code>import ujson
  from tqdm import tqdm
  import bz2
  import json
  import threading
  import spacy
  import Queue
  import time
  nlp = spacy.load('en')
  def iter_comments(loc):
      with bz2.BZ2File(loc) as file_:
          for i, line in (enumerate(file_)):
              yield ujson.loads(line)['body']
  objects = iter_comments('RC_2015-01.bz2')
  q = Queue.Queue()
  f = open("reddit_dump.bin", 'wb')
  def worker():
      while True:
          item = q.get()
          f.write(item)
          q.task_done()
  for i in range(0, 2):
      t = threading.Thread(target=worker)
      t.daemon = True
      t.start()
  def finish_parse(comment):
      global q
      try:
          comment_parse = nlp(unicode(comment))
          comment_bytes = comment_parse.to_bytes()
          q.put(comment_bytes)
      except MemoryError:
          print "MemoryError with comment {0}, waiting for Queue to empty".format(comment)
          time.sleep(2)
      except AssertionError:
          print "AssertionError with comment {0}, skipping".format(comment)
  for comment in tqdm(objects):
      comment = str(comment.encode('ascii', 'ignore'))
      if "&amp;gt;" in comment:
          c_parse_thread = threading.Thread(target=finish_parse, args=(comment,))
          c_parse_thread.start()          
  q.join()
  f.close()
  </code></pre>
  
  <p>Does anybody know what I'm doing wrong?</p>