title: Can't make use of different proxies to perform subsequesnt requests
body: <p>I've written a script in python using proxies to scrape the links of different posts traversing different pages from a webpage. <code>My goal here is to make two subsequesnt requests using different proxies from a list</code>. </p>
  
  <p><strong><em>The script takes random proxies from the list and sends request by <code>make_requests()</code> function and then again makes another request by picking another proxy from the list using the newly populated links by <code>make_ano_requests()</code> function.</em></strong></p>
  
  <p>Finally, <code>get_title()</code> function prints the result.</p>
  
  <p>However, if any proxy doesn't work then it gets kicked out from the list by either of the two functions <code>make_requests()</code> or <code>make_ano_requests()</code>.</p>
  
  <blockquote>
    <p>When I run the script, it seems to be working but somewhere within it's execution, the script gets stuck and never acoomplish the task. How can I accomplish the task?</p>
  </blockquote>
  
  <p>This is what I've written so far (<strong><em>proxyVault</em></strong> contains fake proxies here):</p>
  
  <pre><code>import random
  import requests
  from random import choice
  from bs4 import BeautifulSoup
  from urllib.parse import urljoin
  
  base_url = 'https://stackoverflow.com/questions/tagged/web-scraping'
  lead_urls = [f'https://stackoverflow.com/questions/tagged/web-scraping?sort='
              f'newest&amp;page={page}&amp;pagesize=50' for page in range(1, 5)]
  
  linkList = []
  
  proxyVault = ['103.110.37.244:36022', '180.254.218.229:8080', '110.74.197.207:50632', '1.20.101.95:49001', '200.10.193.90:8080', '173.164.26.117:3128', '103.228.118.66:43002', '178.128.231.201:3128', '1.2.169.54:55312', '181.52.85.249:31487', '97.64.135.4:8080', '190.96.214.123:53251', '52.144.107.142:31923', '45.5.224.145:52035', '89.218.22.178:8080', '192.241.143.186:80', '113.53.29.218:38310', '36.78.131.182:39243']
  
  def make_requests(url):
      proxy_url = choice(proxyVault)
      proxy = {'https': f'http://{proxy_url}'}
      try:
          res = requests.get(url, proxies=proxy)
          soup = BeautifulSoup(res.text, "lxml")
          linkList.extend([urljoin(base_url, item.get("href")) for item in soup.select(".summary .question-hyperlink")])
      except requests.exceptions.ProxyError:
          if proxy_url in proxyVault:
              proxyVault.remove(proxy_url)
              print(f'kicked out bad proxy by first func: {proxy_url}')
          return make_requests(url)
  
  def make_ano_requests(url):
      proxy_url = choice(proxyVault)
      proxy = {'https': f'http://{proxy_url}'}
      try:
          res = requests.get(url, proxies=proxy)
          get_title(res.text)
      except requests.exceptions.ProxyError:
          if proxy_url in proxyVault:
              proxyVault.remove(proxy_url)
              print(f'kicked out bad proxy by second func: {proxy_url}')
          return make_ano_requests(url)
  
  def get_title(response):
      soup = BeautifulSoup(response, "lxml")
      print(soup.select_one("h1[itemprop='name'] a").text)
  
  if __name__ == '__main__':
      for lead_url in lead_urls:
          make_requests(lead_url)
  
      for single_link in linkList:
          make_ano_requests(single_link)
  </code></pre>