title: Too many threads in python threading - Recursive traversal
body: <p>I have a script to traverse an AWS S3 bucket to do some aggregation at the file level.</p>
  
  <pre class="lang-py prettyprint-override"><code>from threading import Semaphore, Thread
  class Spider:
      def __init__(self):
          self.sem = Semaphore(120)
          self.threads = list()
  
      def crawl(self, root_url):
          self.recursive_harvest_subroutine(root_url)
          for thread in self.threads:
              thread.join()
  
      def recursive_harvest_subroutine(self, url):
          children = get_direct_subdirs(url)
          self.sem.acquire()
          if len(children) == 0:
              queue_url_to_do_something_later(url)  # Done
          else:
              for child_url in children:
                  try:
                      thread = Thread(target=self.recursive_harvest_subroutine, args=(url,))
                      self.threads.append(thread)
                      thread.start()
          self.sem.release()
  </code></pre>
  
  <p>This used to run okay, until I encountered a bucket of several TB of data with hundreds of thousand sub-directories. The number of Thread objects in self.threads increases very fast and soon the server reported to me</p>
  
  <pre class="lang-py prettyprint-override"><code>RuntimeError: can't start new thread
  </code></pre>
  
  <p>There is some extra processing I have to do in the script so I can't just get all files from the bucket.</p>
  
  <p>Currently I'm putting a depth of at least 2 before the script can go parallelized but it's just a workaround. Any suggestion is appreciated.</p>