title: How can I process xml asynchronously in python?
body: <p>I have a large XML data file (>160M) to process, and it seems like SAX/expat/pulldom parsing is the way to go. I'd like to have a thread that sifts through the nodes and pushes nodes to be processed onto a queue, and then other worker threads pull the next available node off the queue and process it.</p>
  
  <p>I have the following (it should have locks, I know - it will, later)</p>
  
  <pre><code>import sys, time
  import xml.parsers.expat
  import threading
  
  q = []
  
  def start_handler(name, attrs):
      q.append(name)
  
  def do_expat():
      p = xml.parsers.expat.ParserCreate()
      p.StartElementHandler = start_handler
      p.buffer_text = True
      print("opening {0}".format(sys.argv[1]))
      with open(sys.argv[1]) as f:
          print("file is open")
          p.ParseFile(f)
          print("parsing complete")
  
  
  t = threading.Thread(group=None, target=do_expat)
  t.start()
  
  while True:
      print(q)
      time.sleep(1)
  </code></pre>
  
  <p>The problem is that the body of the <code>while</code> block gets called only once, and then I can't even ctrl-C interrupt it. On smaller files, the output is as expected, but that seems to indicate that the handler only gets called when the document is fully parsed, which seems to defeat the purpose of a SAX parser.</p>
  
  <p>I'm sure it's my own ignorance, but I don't see where I'm making the mistake.</p>
  
  <p>PS: I also tried changing <code>start_handler</code> thus:</p>
  
  <pre><code>def start_handler(name, attrs):
      def app():
          q.append(name)
      u = threading.Thread(group=None, target=app)
      u.start()
  </code></pre>
  
  <p>No love, though.</p>