title: Setting rules with scrapy crawlspider
body: <p>I'm trying out the scrapy CrawlSpider subclass for the first time. I've created the following spider strongly based on the docs example at <a href="https://doc.scrapy.org/en/latest/topics/spiders.html#crawlspider-example" rel="nofollow">https://doc.scrapy.org/en/latest/topics/spiders.html#crawlspider-example</a>:</p>
  
  <pre><code>class Test_Spider(CrawlSpider):
  
      name = "test"
  
      allowed_domains = ['http://www.dragonflieswellness.com']
      start_urls = ['http://www.dragonflieswellness.com/wp-content/uploads/2015/09/']
  
      rules = (
          # Extract links matching 'category.php' (but not matching 'subsection.php')
          # and follow links from them (since no callback means follow=True by default).
          # Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),
  
          # Extract links matching 'item.php' and parse them with the spider's method parse_item
          Rule(LinkExtractor(allow='.jpg'), callback='parse_item'),
      )
  
      def parse_item(self, response):
          self.logger.info('Hi, this is an item page! %s', response.url)
          print(response.url)
  </code></pre>
  
  <p>I'm trying to get the spider to loop start at the prescibed directory and then extract all the '.jpg' links in the directory, but I see :</p>
  
  <pre><code>2016-09-29 13:07:35 [scrapy] INFO: Spider opened
  2016-09-29 13:07:35 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2016-09-29 13:07:35 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
  2016-09-29 13:07:36 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.dragonflieswellness.com/wp-content/uploads/2015/09/&gt; (referer: None)
  2016-09-29 13:07:36 [scrapy] INFO: Closing spider (finished)
  </code></pre>
  
  <p>How can I get this working?</p>