title: What is the purpose of setting follow to true in a LinkExtractor in a CrawlSpider?
body: <p>I saw on the documentation they have this sample code for a CrawlSpider:</p>
  
  <pre><code>import scrapy
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.linkextractors import LinkExtractor
  
  class MySpider(CrawlSpider):
      name = 'example.com'
      allowed_domains = ['example.com']
      start_urls = ['http://www.example.com']
  
      rules = (
          # Extract links matching 'category.php' (but not matching 'subsection.php')
          # and follow links from them (since no callback means follow=True by default).
          Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),
  
          # Extract links matching 'item.php' and parse them with the spider's method parse_item
          Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
      )
  
      def parse_item(self, response):
          self.logger.info('Hi, this is an item page! %s', response.url)
          item = scrapy.Item()
          item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
          item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()
          item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()
          return item
  </code></pre>
  
  <p>From my understanding these steps will happen:</p>
  
  <ol>
  <li>The Scrapy Spider (<code>MySpider</code>) above will get a response from the Scrapy Engine for the <code>'http://www.example.com'</code> link (which was in the <code>start_url</code> list). Then the <code>LinkExtractor</code> will extract all of the links from that response based on the two rules provided above.</li>
  <li>Now let's say the second <code>LinkExtractor</code> (with the callback) got 3 links (<code>'http://www.example.com/item1.php','http://www.example.com/item2.php','http://www.example.com/item3.php'</code>) and the first <code>LinkExtractor</code> without the callback got 1 link (<code>www.example.com/category1.php</code>).</li>
  </ol>
  
  <p>For the 3 links found above, the designated callback, <code>parse_item</code>, will be simply called. BUT, for that one link (<code>www.example.com/category1.php</code>) what will happen with that since there is no callback associated with it? Will the two <code>LinkExtractors</code> run on that one link again? Is that assumption correct?  </p>