title: Rate Limit Downloads Amongst Multiple Processes
body: <p>I want to download and process a lot of files from website. The terms of service for the site restrict the number of files you're permitted to download per second. </p>
  
  <p>The time that it takes to process the files is actually the bottle neck, so I'd like to be able process multiple files in parallel. But I don't want the different processes to combine to violate the download limit. So I need something that limits the over request rate. I was thinking something like the following, but I'm not exactly an expert with the <code>multiprocessing</code> module.</p>
  
  <pre><code>import multiprocessing
  from multiprocessing.managers import BaseManager
  import time
  
  class DownloadLimiter(object):
  
      def __init__(self, time):
          self.time = time
          self.lock = multiprocessing.Lock()
  
      def get(self, url):
          self.lock.acquire()
          time.sleep(self.time)
          self.lock.release()
          return url
  
  
  class DownloadManager(BaseManager):
      pass
  
  DownloadManager.register('downloader', DownloadLimiter)
  
  
  class Worker(multiprocessing.Process):
  
      def __init__(self, downloader, queue, file_name):
          super().__init__()
          self.downloader = downloader
          self.file_name = file_name
          self.queue = queue
  
      def run(self):
          while not self.queue.empty():
              url = self.queue.get()
              content = self.downloader.get(url)
              with open(self.file_name, "a+") as fh:
                  fh.write(str(content) + "\n")
  </code></pre>
  
  <p>Then somewhere else running the downloads with</p>
  
  <pre><code>manager = DownloadManager()
  manager.start()
  downloader = manager.downloader(0.5)
  queue = multiprocessing.Queue()
  
  urls = range(50)
  for url in urls:
      queue.put(url)
  
  job1 = Worker(downloader, queue, r"foo.txt")
  job2 = Worker(downloader, queue, r"bar.txt")
  jobs = [job1, job2]
  
  for job in jobs:
      job.start()
  
  for job in jobs:
      job.join()
  </code></pre>
  
  <p>This seems to do the job on a small scale, but I'm a little wary about whether the locking is really being done correctly.  </p>
  
  <p>Also, if there's a better pattern for achieving the same goal, I'd love to hear it.</p>