title: Scrapy timeout after a while
body: <p>I work on crawling the text from <a href="https://www.dailynews.co.th" rel="nofollow noreferrer">https://www.dailynews.co.th</a>, and here is my question.</p>
  
  <p>My spider worked almost perfect at first, crawled about 4000 pages. </p>
  
  <pre><code>2018-09-28 20:05:00 [scrapy.extensions.logstats] INFO: Crawled 4161 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  </code></pre>
  
  <p>And then it started to raise tons of TimeoutErrors from almost all urls, like this.</p>
  
  <pre><code>2018-09-28 20:06:06 [scrapy.core.scraper] ERROR: Error downloading &lt;GET https://www.dailynews.co.th/tags/When%20Will%20You%20Marry&gt;
  Traceback (most recent call last):
  File "/usr/local/app/.local/share/virtualenvs/monolingual-6kEg5ui2/lib/python2.7/site-packages/Twisted-18.7.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1416, in _inlineCallbacks
  result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/app/.local/share/virtualenvs/monolingual-6kEg5ui2/lib/python2.7/site-packages/Twisted-18.7.0-py2.7-linux-x86_64.egg/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
  return g.throw(self.type, self.value, self.tb)
  File "/usr/local/app/.local/share/virtualenvs/monolingual-6kEg5ui2/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
  defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/usr/local/app/.local/share/virtualenvs/monolingual-6kEg5ui2/lib/python2.7/site-packages/Twisted-18.7.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 654, in _runCallbacks
  current.result = callback(current.result, *args, **kw)
  File "/usr/local/app/.local/share/virtualenvs/monolingual-6kEg5ui2/lib/python2.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 351, in _cb_timeout
  raise TimeoutError("Getting %s took longer than %s seconds." % (url, timeout))
  TimeoutError: User timeout caused connection failure: Getting https://www.dailynews.co.th/tags/When%20Will%20You%20Marry took longer than 5.0 seconds..
  </code></pre>
  
  <p>This was my second try and I decreased CONCURRENT_REQUESTS from 32 to 16, AUTOTHROTTLE_TARGET_CONCURRENCY from 32.0 to 4.0, and DOWNLOAD_TIMEOUT from 15 to 5. The problem was not solved but I got more pages than the first try (from 1000 to 4000).</p>
  
  <p>I also tried <code>scrapy shell</code> on the failed urls (while my spider was still running), I got a response with 200 which means the connection is fine itself.</p>
  
  <p>I wonder if I got banned or something else, can anybody give me a clue on this? Thanks a lot.</p>
  
  <p>FYI, here is my settings file.</p>
  
  <pre><code># Enable or disable downloader middlewares
  # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
  DOWNLOADER_MIDDLEWARES = {
  'protocol.middlewares.RotateUserAgentMiddleware': 110,
  'protocol.middlewares.MaximumAbsoluteDepthFilterMiddleware': 80,
  'protocol.middlewares.ProxyMiddleware': 543
  }
  
  # Enable and configure the AutoThrottle extension (disabled by default)
  # See https://doc.scrapy.org/en/latest/topics/autothrottle.html
  AUTOTHROTTLE_ENABLED = True
  # The initial download delay
  AUTOTHROTTLE_START_DELAY = 0.1
  # The maximum download delay to be set in case of high latencies
  AUTOTHROTTLE_MAX_DELAY = 10
  # The average number of requests Scrapy should be sending in parallel to
  # each remote server
  AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
  # Enable showing throttling stats for every response received:
  AUTOTHROTTLE_DEBUG = False
  
  DOWNLOAD_TIMEOUT = 5
  DEPTH_LIMIT = 100
  DEPTH_PRIORITY = 1
  SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'
  SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'
  </code></pre>
  
  <p>And here is my spider code.</p>
  
  <pre><code># -*- coding: utf-8 -*-
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.linkextractors import LinkExtractor
  from parse_config import parse_config
  
  
  class ProtocolSpider(CrawlSpider):
      name = 'protocol'
      start_urls = ['https://www.dailynews.co.th,']
      custom_settings = {
          'JOBDIR': 'crawl_job'
      }
  
      def __init__(self, **kwargs):
          super(ProtocolSpider, self).__init__(**kwargs)
          self.arg_dict = parse_config(kwargs)
          self.start_urls = self.arg_dict['start_urls']
          # print self.start_urls
          self.allowed_domains = self.arg_dict['allowed_domains']
          self.output_file = open(self.arg_dict['output_file'], 'ab')
          self.rules = (
              Rule(LinkExtractor(allow=self.arg_dict['allow_url'], deny=self.arg_dict['deny_url']),
                   callback="parse_all", follow=True),
          )
          self._compile_rules()
          self.use_web_proxy = self.arg_dict['use_web_proxy']
  
      def parse(self, response):
          self.parse_all(response)
          return super(ProtocolSpider, self).parse(response)
  
      def parse_all(self, response):
          self._record_url(response)
          self._extract_all_p(response)
          self._extract_all_div(response)
  
      def _record_url(self, response):
          self.output_file.write('url_marker: %s' % response.url + '\n')
  
      def _extract_all_p(self, response):
          if self.arg_dict['extract_all_p']:
              p_ls = response.xpath('//p/text()').extract()
              p_string = '\n'.join([p.strip().encode('utf8') for p in p_ls if p.strip()])
              self.output_file.write(p_string + '\n')
  
      def _extract_all_div(self, response):
          if self.arg_dict['extract_all_div']:
              div_ls = response.xpath('//div/text()').extract()
              div_string = '\n'.join([div.strip().encode('utf8') for div in div_ls if div.strip()])
              self.output_file.write(div_string + '\n')
  
      def close(self, spider, reason):
          self.output_file.close()
          return super(ProtocolSpider, self).close(spider, reason)
  </code></pre>