title: Scrapy spider finishing early for no apparent reason
body: <p>I have a scrapy spider (code at this <a href="https://gist.github.com/da1b487b2d7790b6f954" rel="nofollow">gist</a>) which seems to run fine, apart from the fact that it suddenly stops for no apparent reason. When it stops, the last bit of the log file is:</p>
  
  <pre><code>2012-12-28 23:42:04+0000 [church] DEBUG: Crawled (200) &lt;GET http://www.achurchnearyou.com/cogges-st-mary/&gt; (referer: http://www.achurchnearyou.com/clifton-reynes-st-mary-the-virgin/)
  2012-12-28 23:42:04+0000 [church] DEBUG: Scraped from &lt;200 http://www.achurchnearyou.com/cogges-st-mary/&gt;
      {'archdeaconry': u'OXFORD',
       'archdeaconry_id': u'271',
       'benefice': u'Cogges and S Leigh',
       'benefice_id': u'27',
       'deanery': u'WITNEY',
       'deanery_id': u'27109',
       'legal_name': u'Cogges',
       'parish_id': u'270245'}
  2012-12-28 23:42:04+0000 [church] DEBUG: Redirecting (301) to &lt;GET http://www.achurchnearyou.com//&gt; from &lt;GET http://www.achurchnearyou.com/venue.php?V=0083&gt;
  2012-12-28 23:42:04+0000 [church] INFO: Closing spider (finished)
  </code></pre>
  
  <p>Is there any reason that a spider might decide it is finished straight after redirecting a URL? The interesting thing is that I have some custom DownloaderMiddleware which will catch a redirect like this and create a new request instead (basically some URLs that I'm trying will redirect to the homepage, and I want to ignore those and create a different URL instead).</p>