title: Python requests.get and threading with different servers
body: <p>I am working on a simple web scraper and rn trying to implement some multithreading. While my code works as intended with some servers(reducing time of execution vastly), my primary goal is to make it work with few specific ones. So when I try it with the ones in <strong>sites</strong> list, I get performance like I am still using sequential code. Any guesses what can cause this?</p>
  <pre><code>import requests, time
  from bs4 import BeautifulSoup
  from threading import Thread
  from random import choice
  
  # Enable to get some logging info
  #---------------------------------
  # import logging
  # import http.client
  # http.client.HTTPConnection.debuglevel = 1
  # logging.basicConfig()
  # logging.getLogger().setLevel(logging.DEBUG)
  # requests_log = logging.getLogger(&quot;requests.packages.urllib3&quot;)
  # requests_log.setLevel(logging.DEBUG)
  # requests_log.propagate = True
  
  sites = [
      &quot;https://pikabu.ru/community/blackhumour&quot;,
      &quot;https://www.pikabu.ru/tag/%D0%9C%D0%B5%D0%BC%D1%8B/hot&quot;
  ]
  
  class Pikabu_Downloader(Thread):
      def __init__(self, url, name, *args, **kwargs):
          super().__init__(*args, **kwargs)
          self.url = url
          self.name = name
          self.begin = time.time()
  
      def run(self):
          print(&quot;Beginning with thread number&quot;,self.name, &quot;,&quot;, round(time.time()-self.begin, 4), &quot; seconds has passed&quot;)
          html_data = self._get_html()
          print(&quot;After requests.get with thread number&quot;, self.name, &quot;,&quot;, round(time.time()-self.begin, 4), &quot; seconds has passed&quot;)
          if html_data is None:
              return
          self.soup = BeautifulSoup(html_data, &quot;html.parser&quot;)
          print(&quot;After making soup with thread number&quot;, self.name, &quot;,&quot;, round(time.time() - self.begin, 4), &quot; seconds has passed&quot;)
  
      def _get_html(self):
          try:
              user_agents = ('Mozilla/5.0 (Windows NT 10.0; Win64; x64)', 'AppleWebKit/537.36 (KHTML, like Gecko)', 'Chrome/74.0.3729.169', 'Safari/537.36')
              print(f&quot;Go {self.url}...&quot;)
              res = requests.get(self.url, headers={'User-Agent': choice(user_agents)}, stream = True)#, allow_redirects=False)
          except Exception as exc:
              print(exc)
          else:
              return res.text
  
  test = &quot;https://readingbooks.site/read/?name=1984&amp;&quot;
  
  def download():
      pikabu_urls = []
      for url in sites:
          pikabu = [url + &quot;?page=&quot; + str(x) for x in range(1, 10)]
          pikabu_urls = pikabu_urls + pikabu
  
      pikabu_dls = [Pikabu_Downloader(url=page, name=str(i)) for i, page in enumerate(pikabu_urls)]
      # Comment the string above and enable 2 underlying strings to get result from test server
      # tests = [test + &quot;page=&quot; + str(x) for x in range(1, pages)]
      # pikabu_dls = [Pikabu_Downloader(url=page, name=str(i)) for i, page in enumerate(tests)]
  
      for pikabu_dl in pikabu_dls:
          pikabu_dl.start()
  
      for pikabu_dl in pikabu_dls:
          pikabu_dl.join()
  
  download()
  </code></pre>
  <p>And the result is something like</p>
  <pre><code>...
  After requests.get with thread number 1 , 1.6904  seconds has passed
  After making soup with thread number 1 , 1.7554  seconds has passed
  After requests.get with thread number 2 , 2.9805  seconds has passed
  After making soup with thread number 2 , 3.0455  seconds has passed
  After requests.get with thread number 3 , 4.3225  seconds has passed
  After making soup with thread number 3 , 4.3895  seconds has passed
  ...
  </code></pre>
  <p>What can cause such latency between thread executions? I was hoping to get each thread to finish almost simultaneously and to get more...asynchronous output, like with server from <strong>test</strong>. If I set a timeout of 5 sec inside requests.get, most of the requests wont even work.</p>