title: Scrapy is not Crawling the next page url
body: <p>My spider is not crawling the page 2 but the XPath is returning the correct next page link which is an absolute link to next page. </p>
  
  <p>Here is my code </p>
  
  <pre><code>from scrapy import Spider
  from scrapy.http import Request, FormRequest
  
  
  
  class MintSpiderSpider(Spider):
  
      name = 'Mint_spider'
      allowed_domains = ['example.com']
      start_urls = ['http://www.example.com/']
  
      def parse(self, response):
          urls =  response.xpath('//div[@class = "post-inner post-hover"]/h2/a/@href').extract()
  
          for url in urls:
              yield Request(url, callback=self.parse_lyrics)
  
          next_page_url = response.xpath('//li[@class="next right"]/a/@href').extract_first()
          if next_page_url:
              yield scrapy.Request(next_page_url, callback=self.parse)
  
  
      def parse_foo(self, response):
          info = response.xpath('//*[@class="songinfo"]/p/text()').extract()
          name =  response.xpath('//*[@id="lyric"]/h2/text()').extract()
  
          yield{
              'name' : name,
              'info': info
          }
  </code></pre>