title: Twisted DeferredList only runs its callback half the time
body: <p>I'm trying to make a fairly simple web scraper using Twisted. I have it working, but whenever I try to scrape more than a few hundred sites, it will hang indefinitely for no discernible reason. Everything seems to work, except when it stops at the very end with a couple sites left to process.</p>
  
  <p>I used the tutorial here: <a href="http://technicae.cogitat.io/2008/06/async-batching-with-twisted-walkthrough.html" rel="nofollow">http://technicae.cogitat.io/2008/06/async-batching-with-twisted-walkthrough.html</a> as a blueprint.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>class Spider:
      """Twisted-based html retrieval system."""
  
      def __init__(self, queue, url_list):
          self.process_queue = queue
          self.start_urls = []
          for url in url_list:
              self.start_urls.append(url)
  
      def crawl(self):
          """Extracts information from each website in start_urls."""
          deferreds = []
          sem = defer.DeferredSemaphore(30)
          for url in self.start_urls:
              d = sem.run(self._crawl, url, self.process_queue)
              deferreds.append(d)
          dl = defer.DeferredList(deferreds, consumeErrors=1)
          dl.addCallback(self.finish, self.process_queue)
          dl.addCallback(self.shutdown)
          reactor.run()
  
      def _crawl(self, url, queue):
          d = getPage(url, timeout=10)
          d.addCallback(self.parse, url, queue)
          d.addErrback(self.parse_error, url, queue)
          return d
  
      def parse(self, result, url, queue):
          print 'Parsing:', url
          data = {'body': result, 'url': url}
          response = Response(data['url'], data['body'])
          queue.put(response)
          return data
  
      def parse_error(self, result, url, queue):
          print 'Errback from:', url
          data = {'body': 'error', 'url': url}
          response = Response(data['url'], data['body'])
          queue.put(response)
          return data
  
      def finish(self, results, queue):
          for (valid, data) in results:
              if valid:
                  print 'Success:', data['url']
              else:
                  print 'Failed:', data['url']
          finish_signal = Response('FINISHED', 'DONE')
          queue.put(finish_signal)
  
      def shutdown(self, ignore):
          reactor.stop()
  </code></pre>
  
  <p>I am running this section of code in a larger program, hence the Queue.</p>
  
  <p>Any suggestions for making the DeferredList always fire? Or ideas as to why it's only firing half the time, and failing without any exceptions the other half?</p>
  
  <p>It's very frustrating, especially since it works perfectly with small numbers of URL's (1-100) but fails when scaled up. I am new to Twisted, so I probably just messed up something with the errbacks, but I can't figure out what, or how to fix it...</p>
  
  <p>Also, before anyone answers with 'use Scrapy!' I can't use Scrapy for reasons I won't get into here. Assume this program is my last hope and must work.</p>
  
  <p>EDIT:</p>
  
  <p>Full standalone code so people can run it directly:</p>
  
  <pre><code>import sys
  from twisted.internet import defer, reactor
  from twisted.web.client import getPage
  
  class SeerSpider:
      """Twisted-based html retrieval system."""
  
      def __init__(self, queue, url_list):
          self.process_queue = queue
          self.start_urls = []
          for url in url_list:
              self.start_urls.append(url)
  
      def crawl(self):
          """Extracts information from each website in url_list."""
          deferreds = []
          sem = defer.DeferredSemaphore(30)
          for url in self.start_urls:
              d = sem.run(self._crawl, url, self.process_queue)
              deferreds.append(d)
          dl = defer.DeferredList(deferreds, consumeErrors=True)
          dl.addCallback(self.finish, self.process_queue)
          dl.addCallback(self.shutdown)
          reactor.run()
  
      def _crawl(self, url, queue):
          d = getPage(url, timeout=10)
          d.addCallback(self.parse, url, queue)
          d.addErrback(self.parse_error, url, queue)
          return d
  
      def parse(self, result, url, queue):
          data = {'body': result, 'url': url}
          response = Response(data['url'], data['body'])
          print response.url
          return data
  
      def parse_error(self, result, url, queue):
          data = {'body': 'error','url': url}
          response = Response(data['url'], data['body'])
          print response.url
          return data
  
      def finish(self, results, queue):
          finish_signal = Response('FINISHED', 'DONE')
          print finish_signal.url
  
      def shutdown(self, ignore):
          reactor.stop()
  
  class Response:
      def __init__(self, url, text):
          self.url = url
          self.body = text
  
  url_list = ['http://google.com/', 'http://example.com', 'http://facebook.com'] # this will work, make the list bigger to find the bug
  spider = SeerSpider(None, url_list)
  spider.crawl()
  </code></pre>