title: raise AttributeError: Response content isn't text Scarpy proxy pool. How to solve?
body: <p>Raises <strong>AttributeError: Response content isn't text</strong>. I am using scrapy_proxy_pool and scrapy_user_agents. I am trying to find each and every link of target website.</p>
  <pre><code>import scrapy
  
  class LinksSpider(scrapy.Spider):
  
      name = 'links'
      allowed_domains = ['www.chotosite.com','chotosite.com']
  
      extracted_links = []
  
      def start_requests(self):
          start_urls = 'https://www.chotosite.com'
          yield scrapy.Request(url=start_urls, callback=self.extract_link)
  
      def extract_link(self, response):
          # eleminating images url from links
          str_response_content_type = str(response.headers.get('content-type'))
          if str_response_content_type == &quot;b'text/html; charset=UTF-8'&quot; :
              
              links = response.xpath(&quot;//a/@href&quot;).extract()
  
              for link in links:
                  if &quot;chotosite&quot; in link and link not in self.extracted_links:
                      self.extracted_links.append(link)
                      yield scrapy.Request(url=link, callback=self.extract_link)
  
                      yield {
                          &quot;links&quot;: link
                      }
  </code></pre>
  <p>Here is my settings.py file</p>
  <pre><code>BOT_NAME = 'chotosite'
  
  SPIDER_MODULES = ['chotosite.spiders']
  NEWSPIDER_MODULE = 'chotosite.spiders'
  
  # Obey robots.txt rules
  ROBOTSTXT_OBEY = False
  PROXY_POOL_ENABLED = True
  
  
  DOWNLOADER_MIDDLEWARES = {
      'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
      'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
      'scrapy_proxy_pool.middlewares.ProxyPoolMiddleware': 610,
      'scrapy_proxy_pool.middlewares.BanDetectionMiddleware': 620,
  }
  
  AUTOTHROTTLE_ENABLED = True
  
  </code></pre>
  <p>Here is the big console output in pastebin <a href="https://pastebin.com/tRbfvxdN" rel="nofollow noreferrer">https://pastebin.com/tRbfvxdN</a></p>
  <p>And here is the traceback:</p>
  <pre><code>Traceback (most recent call last):
    File &quot;/usr/lib/python3/dist-packages/twisted/internet/defer.py&quot;, line 1418, in _inlineCallbacks
      result = g.send(result)
    File &quot;/usr/lib/python3/dist-packages/scrapy/core/downloader/middleware.py&quot;, line 44, in process_request
      defer.returnValue((yield download_func(request=request, spider=spider)))
    File &quot;/usr/lib/python3/dist-packages/twisted/internet/defer.py&quot;, line 1362, in returnValue
      raise _DefGen_Return(val)
  twisted.internet.defer._DefGen_Return: &lt;200 https://www.chotosite.com&gt;
   
  During handling of the above exception, another exception occurred:
   
  Traceback (most recent call last):
    File &quot;/usr/lib/python3/dist-packages/twisted/internet/defer.py&quot;, line 1418, in _inlineCallbacks
      result = g.send(result)
    File &quot;/usr/lib/python3/dist-packages/scrapy/core/downloader/middleware.py&quot;, line 53, in process_response
      response = yield method(request=request, response=response, spider=spider)
    File &quot;/usr/local/lib/python3.8/dist-packages/scrapy_proxy_pool/middlewares.py&quot;, line 287, in process_response
      ban = is_ban(request, response)
    File &quot;/usr/local/lib/python3.8/dist-packages/scrapy_proxy_pool/policy.py&quot;, line 15, in response_is_ban
      if self.BANNED_PATTERN.search(response.text):
    File &quot;/usr/lib/python3/dist-packages/scrapy/http/response/__init__.py&quot;, line 93, in text
      raise AttributeError(&quot;Response content isn't text&quot;)
  AttributeError: Response content isn't text
  </code></pre>
  <p>And this is what installed on my system</p>
  <pre><code>Scrapy       : 1.7.3
  lxml         : 4.5.0.0
  libxml2      : 2.9.10
  cssselect    : 1.1.0
  parsel       : 1.5.2
  w3lib        : 1.21.0
  Twisted      : 18.9.0
  Python       : 3.8.5 (default, Jul 28 2020, 12:59:40) - [GCC 9.3.0]
  pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1f  31 Mar 2020)
  cryptography : 2.8
  Platform     : Linux-5.4.0-53-generic-x86_64-with-glibc2.29
  
  </code></pre>