title: scrapy stopped abruptly with out any error messages
body: <p>Python Scrapy: I am trying to scrape big web site through their api. It has around 100 main urls and it will come with sub urls. My code parsed around a million in 4 hours and stopped with out any error message.  I am running in 11 threads in one process.</p>