title: Scrapy SgmlLinkExtractor
body: <p>I am trying to get a scrapy spider working, but there seems to be a problem with SgmlLinkExtractor.</p>
  
  <p>Here is the signature:</p>
  
  <blockquote>
    <p>SgmlLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), restrict_xpaths(), tags=('a', 'area'), attrs=('href'), canonicalize=True, unique=True, process_value=None)</p>
  </blockquote>
  
  <p>I am using the allow() option, here is my code:</p>
  
  <pre><code>start_urls = ['http://bigbangtrans.wordpress.com']
  rules = [Rule(SgmlLinkExtractor(allow=[r'series-\d{1}-episode-\d{2}.']), callback='parse_item')]
  </code></pre>
  
  <p>A sample url looks like <a href="http://bigbangtrans.wordpress.com/series-1-episode-11-the-pancake-batter-anomaly/" rel="nofollow">http://bigbangtrans.wordpress.com/series-1-episode-11-the-pancake-batter-anomaly/</a></p>
  
  <p>the output of <code>scrapy crawl tbbt</code> contains</p>
  
  <blockquote>
    <p>[tbbt] DEBUG: Crawled (200) http://bigbangtrans.wordpress.com/series-3-episode-17-the-precious-fragmentation/> (referer: <a href="http://bigbangtrans.wordpress.com" rel="nofollow">http://bigbangtrans.wordpress.com</a>)</p>
  </blockquote>
  
  <p>The parse_item callback, however, is not called and I can not figure out why.</p>
  
  <p>This is the whole spider code:</p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import HtmlXPathSelector
  
  class TbbtSpider(CrawlSpider):
      #print '\n TbbtSpider \n'
      name = 'tbbt'
      start_urls = ['http://bigbangtrans.wordpress.com'] # urls from which the spider will start crawling
      rules = [Rule(SgmlLinkExtractor(allow=[r'series-\d{1}-episode-\d{2}.']), callback='parse_item')]
  
  
      def parse_item(self, response):
          print '\n parse_blogpost \n'
          hxs = HtmlXPathSelector(response)
          item = TbbtItem()
          # Extract title
          item['title'] = hxs.select('//div[@id="post-5"]/div/p/span/text()').extract() # XPath selector for title
          return item
  </code></pre>