title: Scrapy twisted connection lost in non-clean fashion. No proxy. Already tried headers
body: <p>I am trying to crawl this site </p>
  
  <p><a href="https://www5.apply2jobs.com/jupitermed/ProfExt/index.cfm?fuseaction=mExternal.searchJobs" rel="nofollow noreferrer">https://www5.apply2jobs.com/jupitermed/ProfExt/index.cfm?fuseaction=mExternal.searchJobs</a></p>
  
  <p>with scrapy and keep getting twisted request/disconnection errors. I am not using a proxy and I tried both setting the user agent and actually setting all the headers based on <a href="https://stackoverflow.com/a/42471673/8528618">this answer</a> </p>
  
  <p>here is the code generating the request</p>
  
  <pre><code>def start_requests(self):
      url = 'https://www5.apply2jobs.com/jupitermed/ProfExt/index.cfm?fuseaction=mExternal.searchJobs'
  
      headers = {
          'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
          'Accept-Encoding': 'gzip, deflate, br',
          'Accept-Language': 'en-US,en;q=0.8',
          'Connection': 'keep-alive',
          'DNT': '1',
          'Host': 'www5.apply2jobs.com',
          'Referer': 'https://www5.apply2jobs.com/jupitermed/ProfExt/index.cfm?fuseaction=mExternal.showJob&amp;RID=2524&amp;CurrentPage=2',
          'Upgrade-Insecure-Requests': '1',
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36'
      }
  
      yield Request(url=url, headers=headers, callback=self.parse)
  </code></pre>
  
  <p>and this is my traceback:</p>
  
  <pre><code>2017-08-28 13:34:13 [scrapy.core.engine] INFO: Spider opened
  2017-08-28 13:34:13 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2017-08-28 13:34:13 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
  2017-08-28 13:34:13 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET https://www5.apply2jobs.com/robots.txt&gt; (failed 1 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
  2017-08-28 13:34:13 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET https://www5.apply2jobs.com/robots.txt&gt; (failed 2 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
  2017-08-28 13:34:13 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying &lt;GET https://www5.apply2jobs.com/robots.txt&gt; (failed 3 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
  2017-08-28 13:34:13 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading &lt;GET https://www5.apply2jobs.com/robots.txt&gt;: [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
  ResponseNeverReceived: [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
  2017-08-28 13:34:13 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET https://www5.apply2jobs.com/jupitermed/ProfExt/index.cfm?fuseaction=mExternal.searchJobs&gt; (failed 1 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
  2017-08-28 13:34:13 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET https://www5.apply2jobs.com/jupitermed/ProfExt/index.cfm?fuseaction=mExternal.searchJobs&gt; (failed 2 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
  2017-08-28 13:34:13 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying &lt;GET https://www5.apply2jobs.com/jupitermed/ProfExt/index.cfm?fuseaction=mExternal.searchJobs&gt; (failed 3 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
  2017-08-28 13:34:13 [scrapy.core.scraper] ERROR: Error downloading &lt;GET https://www5.apply2jobs.com/jupitermed/ProfExt/index.cfm?fuseaction=mExternal.searchJobs&gt;: [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
  2017-08-28 13:34:13 [scrapy.core.engine] INFO: Closing spider (finished)
  </code></pre>