title: Creating editable CrawlSpider rules in Scrapy
body: <p>I've been trying to create a simple Scrapy CrawlSpider script that can be changed easily, but I'm having trouble figuring out how to get the link extractor Rules to work properly.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>class LernaSpider(CrawlSpider):
  """Our ad-hoc spider"""
  
  name = "lerna"
  
  def __init__(self, url, allow_follow='.*', deny_follow='', allow_extraction='.*', deny_extraction=''):
      parsed_url = urlparse(url)
      domain = str(parsed_url.netloc)
      self.allowed_domains = [domain]
      self.start_urls = [url]
      self.rules = (
          # Extract links
          # and follow links from them (since no callback means follow=True by default).
          Rule(SgmlLinkExtractor(allow=(allow_follow, ), deny=(deny_follow, ))),
  
          # Extract links and parse them with the spider's method parse_item
          Rule(SgmlLinkExtractor(allow=(allow_extraction, ), deny=(deny_extraction, )), callback='parse_item'),
      )
  
      super(LernaSpider, self).__init__()
  
  def parse_item(self, response):
  
      print 'Crawling... %s' % response.url
      # more stuff here
  </code></pre>
  
  <p>I have this code, but I can never get the allow/deny rules to work properly and I don't really get why. Is leaving the empty strings causing it to deny everything? I thought since it was a RE it would only do a blanket denial if I input '.*' or whatever.</p>
  
  <p>Any help would be appreciated.</p>