title: Python: Serial, Threading, Multiprocessing. Which is the fastest way?
body: <p>so I have a code that needs to do HTTP requests (let's say 1000). I approached it in 3 ways so far with 50 HTTP requests. The results and codes are below.</p>
  <p>The fastest is the approach using Threads, issue is that I lose some data (from what I understood due to the GIL). My questions are the following:</p>
  <ol>
  <li><p>My understanding it that the correct approach in this case is to use Multiprocessing. Is there any way I can improve the speed of that approach? Matching the Threading time would be great.</p>
  </li>
  <li><p>I would guess that the higher the amount of links I have, the more time the Serial and Threading approach would take, while the Multiprocessing approach would increase much more slowly. Do you have any source that will allow me to get an estimate of the time it would take to run the code with n links?</p>
  </li>
  </ol>
  <p><strong>Serial - Time To Run around 10 seconds</strong></p>
  <pre><code>def get_data(link, **kwargs):
      data = requests.get(link)
      if &quot;queue&quot; in kwargs and isinstance(kwargs[&quot;queue&quot;], queue.Queue):
          kwargs[&quot;queue&quot;].put(data)
      else:
          return data
  
  links = [link_1, link_2, ..., link_n]
  matrix = []
  for link in links:
      matrix.append(get_data(link))
  </code></pre>
  <p><strong>Threads - Time To Run around 0.8 of a second</strong></p>
  <pre><code>def get_data_thread(links):
      q = queue.Queue()
      for link in links:
          data = threading.Thread(target = get_data, args = (link, ), kwargs = {&quot;queue&quot; : q})
          data.start()
      data.join()
      return q
  
  matrix = []
  q = get_data_thread(links)
  while not q.empty():
      matrix.append(q.get())
  </code></pre>
  <p><strong>Multiprocessing - Time To Run around 5 seconds</strong></p>
  <pre><code>def get_data_pool(links):
      p = mp.Pool()
      data = p.map(get_data, links)
      return data
  
  if __name__ == &quot;__main__&quot;:
      matrix = get_data_pool(links)
  </code></pre>