title: Scrapy Spider gets stuck in middle of crawling
body: <p>I'm new to scrapy and I'm trying to build a spider that will crawl a website and get all the phone numbers, emails, pdfs, etc. from it (and I want it to follow all the links from the main page so it searches the entire domain). </p>
  
  <p>This question had a similar issue but it wasn't resolved: <a href="https://stackoverflow.com/questions/18987719/why-scrapy-crawler-stops">Why scrapy crawler stops?</a></p>
  
  <p>Here is the code for my spider:</p>
  
  <pre><code>import scrapy
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.selector import Selector
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from mobilesuites.items import MobilesuitesItem
  import re
  
  class ExampleSpider(CrawlSpider):
      name = "hyatt"
      allowed_domains = ["hyatt.com"]
      start_urls = ( 
          'http://www.hyatt.com/',
      )   
  
      #follow only non-javascript links
      rules = ( 
              Rule(SgmlLinkExtractor(deny = ('.*\.jsp.*')), follow = True, callback = 'parse_item'),
              )   
  
      def parse_item(self, response):
          #self.log('The current url is %s' % response.url)
  
          selector = Selector(response)
          item = MobilesuitesItem()
          #get url
          item['url'] = response.url
  
          #get page title
          titles = selector.select("//title")
          for t in titles:
              item['title'] = t.select("./text()").extract()
  
          #get all phone numbers, emails, and pdf links
          text = response.body
          item['phone'] = '|'.join(re.findall('\d{3}[-\.\s]\d{3}[-\.\s]\d{4}|\(?\d{3}\)?[-\.\s]?\d{3}[-\.\s]\d{4}|\d{3}[-\.\s]\d{4}', text))
          item['email'] = '|'.join(re.findall("[^\s@]+@[^\s@]+\.[^\s@]+", text))
          item['pdfs'] = '|'.join(re.findall("[^\s\"&lt;]*\.pdf[^\s\"&gt;]*", text))
  
          #check to see if dining is mentioned on the page
          item['dining'] = bool(re.findall("\s[dD]ining\s|\s[mM]enu\s|\s[bB]everage\s", text))
          return item
  </code></pre>
  
  <p>Here's the last part of the crawl log before it hangs:</p>
  
  <pre><code>2014-07-21 18:18:57-0500 [hyatt] DEBUG: Scraped from &lt;200 http://www.place.hyatt.com/en/hyattplace/eat-and-drink/24-7-gallery-menu.html&gt;
      {'email': '',
       'phone': '',
       'title': [u'24/7 Gallery Menu'],
       'url': 'http://www.place.hyatt.com/en/hyattplace/eat-and-drink/24-7-gallery-menu.html'}
  2014-07-21 18:18:57-0500 [hyatt] DEBUG: Ignoring response &lt;404 http://hyatt.com/gallery/thrive/siteMap.html&gt;: HTTP status code is not handled or not allowed
  2014-07-21 18:18:57-0500 [hyatt] DEBUG: Crawled (200) &lt;GET http://www.hyatt.com/hyatt/pure/contact/&gt; (referer: http://www.hyatt.com/hyatt/pure/?icamp=HY_HyattPure_HPLS)
  2014-07-21 18:18:57-0500 [hyatt] DEBUG: Crawled (200) &lt;GET http://www.house.hyatt.com/en/hyatthouse/aboutus.html&gt; (referer: http://www.house.hyatt.com/en/hyatthouse.html)
  2014-07-21 18:18:57-0500 [hyatt] DEBUG: Crawled (200) &lt;GET http://www.place.hyatt.com/en/hyattplace/eat-and-drink/eat-and-drink.html&gt; (referer: http://www.place.hyatt.com/en/hyattplace/eat-and-drink.html)
  2014-07-21 18:18:57-0500 [hyatt] DEBUG: Crawled (200) &lt;GET http://www.park.hyatt.com/en/parkhyatt/newsandannouncements.html?icamp=park_hppsa_new_hotels&gt; (referer: http://www.park.hyatt.com/en/parkhyatt.html)
  2014-07-21 18:18:57-0500 [hyatt] DEBUG: Crawled (200) &lt;GET http://www.regency.hyatt.com/en/hyattregency/meetingsandevents.html&gt; (referer: http://www.regency.hyatt.com/en/hyattregency.html)
  2014-07-21 18:18:57-0500 [hyatt] DEBUG: Crawled (200) &lt;GET http://www.house.hyatt.com/en/hyatthouse/specialoffers.html&gt; (referer: http://www.house.hyatt.com/en/hyatthouse.html)
  2014-07-21 18:18:57-0500 [hyatt] DEBUG: Crawled (200) &lt;GET http://www.house.hyatt.com/en/hyatthouse/locations.html&gt; (referer: http://www.house.hyatt.com/en/hyatthouse.html)
  </code></pre>