title: Resetting TCP connection programmingly in Scrapy
body: <p>I'm currently crawling a website that bans IP if it requested too many pages in a short period of time. When that happens, it gives a <code>403 status code</code> in the response. If the IP address doesn't renew, the crawler will fail all the following requests.</p>
  
  <p>So I added an HTTP proxy, it is a hub hosting hundreds of IPs and assigns a random one per TCP connection.</p>
  
  <pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; import requests
  &gt;&gt;&gt; proxies = {"https": "https://user:pass@proxyservice.com"}
  &gt;&gt;&gt; s = requests.Session()
  &gt;&gt;&gt; 
  &gt;&gt;&gt; print("\n persisted connection:")
  &gt;&gt;&gt; for i in range(3):
  &gt;&gt;&gt;     print(s.get("https://ifconfig.co", proxies=proxies).text)
  &gt;&gt;&gt;
  &gt;&gt;&gt; print("\n new connection every request:")
  &gt;&gt;&gt; for i in range(3):
  &gt;&gt;&gt;     print(requests.get("https://ifconfig.co", proxies=proxies).text)
  
  persisted connection:
  123.123.123.123
  123.123.123.123
  123.123.123.123
  
  new connection every request:
  123.111.111.111
  123.222.222.222
  123.110.110.110
  </code></pre>
  
  <p>I working with Scrapy in my project, it uses persisted connection by default, which means it will use the same proxy IP per connection:</p>
  
  <pre class="lang-py prettyprint-override"><code>class TestSpider(scrapy.Spider):
      name = 'test'
      allowed_domains = ['ifconfig.co']
      custom_settings = {
          "CONCURRENT_REQUESTS": 2
      }
  
      def __init__(self):
          self.url = "https://ifconfig.co"
          self.headers = {"user-agent": "curl"}
          self.proxy = "https://user:pass@proxyservice.com"
  
      def start_requests(self):
          # We have 2 concurrent requests (persisted connections).
          yield Request(url=self.url, headers=self.headers, meta={"proxy": self.proxy}, dont_filter=True)
          yield Request(url=self.url, headers=self.headers, meta={"proxy": self.proxy}, dont_filter=True)
  
      def parse(self, response):
          self.logger.info(response.text)
          yield Request(url=self.url, headers=self.headers, meta={"proxy": self.proxy}, dont_filter=True)
  </code></pre>
  
  <p>Which gives us:</p>
  
  <pre><code>2019-06-27 12:13:21 [test] INFO: 181.xx.xx.197
  
  2019-06-27 12:13:21 [test] INFO: 38.xx.xx.199
  
  2019-06-27 12:13:21 [test] INFO: 181.xx.xx.197
  
  2019-06-27 12:13:22 [test] INFO: 38.xx.xx.199
  
  2019-06-27 12:13:22 [test] INFO: 181.xx.xx.197
  
  2019-06-27 12:13:22 [test] INFO: 38.xx.xx.199
  
  2019-06-27 12:13:22 [test] INFO: 181.xx.xx.197
  
  2019-06-27 12:13:23 [test] INFO: 181.xx.xx.197
  
  2019-06-27 12:13:23 [test] INFO: 38.xx.xx.199
  
  2019-06-27 12:13:23 [test] INFO: 181.xx.xx.197
  
  2019-06-27 12:13:24 [test] INFO: 38.xx.xx.199
  </code></pre>
  
  <p>How can I reset the TCP connection with the proxy server so that I can get a new IP address when the response gives a 403?</p>