title: Scrapy Shell: twisted.internet.error.ConnectionLost although USER_AGENT is set
body: <p>When I try to scrape a certain web site (with both, spider and shell), I get the following error:</p>
  
  <pre><code>twisted.web._newclient.ResponseNeverReceived: [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.&gt;]
  </code></pre>
  
  <p>I found out that this can happen, when no user agent is set.
  But after setting it manually, I still got the same error.</p>
  
  <p>You can see the whole output of scrapy shell here: <a href="http://pastebin.com/ZFJZ2UXe" rel="nofollow noreferrer">http://pastebin.com/ZFJZ2UXe</a></p>
  
  <h2>Notes:</h2>
  
  <p>I am not behind a proxy, and I can access other sites via scrapy shell without problems. I am also able to access the site with Chrome, so it is not a network or connection issue.</p>
  
  <p>Maybe someone can give me a hint how I could solve this problem?</p>