title: Python ThreadPoolExecutor wait for all futures to complete
body: <p>I am trying to write a module which needs to crawl some URLs concurrently/parallelly. since this would be a more expensive Network IO operation instead of CPU heavy. I am using ThreadPoolExecutor.</p>
  
  <p>Now in my code, multiple functions add tasks to the shared thread pool.</p>
  
  <blockquote>
    <p>my issue is Main thread gets suspended before all future objects are
    done processing in the callback functions.</p>
  </blockquote>
  
  <p>I am a beginner dealing with futures and ThreadPoolExecutor. Any help would be appreciated.  </p>
  
  <pre><code>import settings
  from concurrent.futures import ThreadPoolExecutor
  import concurrent.futures
  
  
  class Test(Base):
  
      WORKER_THREADS = settings.WORKER_THREADS
  
      def __init__(self, urls):
          super(Test, self).__init__()
          self.urls = urls
          self.worker_pool = ThreadPoolExecutor(max_workers=Test.WORKER_THREADS)
  
  
      def add_to_worker_queue(self, task, callback, **kwargs):
          self.logger.info("Adding task %s to worker pool.", task.func_name)
          self.worker_pool.submit(task, **kwargs).add_done_callback(callback)
          return
  
      def load_url(self, url):
          response = self.make_requests(urls=url) # make_requests is in Base class (it just makes a HTTP req)
          # response is a generator, so to get the data out of it need to iterate through it.
          for res in response:
              return res
  
      def handle_response(self, response):
          # do some stuff with response and add it again to the worker queue for further parallel processing
          self.add_to_worker_queue(some_task, callback_func, data=response)
          return
  
      def start(self):
          for url in self.urls:
              self.add_to_worker_queue(self.load_url, self.handle_response, url=[url])
          return
  
      def stop(self):
          self.worker_pool.shutdown(wait=True)
          return
  
  
  if __name__ == "__main__":
      start_urls = [ 'http://stackoverflow.com/'
                  , 'https://docs.python.org/3.3/library/concurrent.futures.html'
                    ]
      test = Test(urls=start_urls)
      test.start()
      test.stop()
  </code></pre>
  
  <p>PS I tried using executer with "with" statement, according to this example. <a href="https://docs.python.org/3.3/library/concurrent.futures.html#threadpoolexecutor-example" rel="nofollow">https://docs.python.org/3.3/library/concurrent.futures.html#threadpoolexecutor-example</a>
  but as I submit tasks to the pool one by one and above example wait for future objects to be completed which defeats my purpose.</p>