title: Scrapy: AttributeError: 'YourCrawler' object has no attribute 'parse_following_urls'
body: <p>I am writing a scrapy spider. I have been reading this question: <a href="https://stackoverflow.com/questions/27984064/scrapy-scraping-a-list-of-links">Scrapy: scraping a list of links</a>, and I can make it recognise the urls in a listpage, but I cant make it go inside the urls and save the data I want to see.</p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider
  from scrapy.selector import Selector
  from scrapy.http import Request
  
  class YourCrawler(CrawlSpider):
      name = "bookstore_2"
      start_urls = [
      'https://example.com/materias/?novedades=LC&amp;p',
      ]
  
      def parse(self, response):
          # go to the urls in the list
          s = Selector(response)
          page_list_urls = s.xpath('///*[@id="results"]/ul/li/div[1]/h4/a[2]/@href').extract()
          for url in page_list_urls:
              yield Request(url, callback=self.parse_following_urls, dont_filter=True)
  
              # For the urls in the list, go inside, and in div#main, take the div.ficha &gt; div.caracteristicas &gt; ul &gt; li
              def parse_following_urls(self, response):
                  #Parsing rules go here
                  for each_book in response.css('div#main'):
                      yield {
                      'book_isbn': each_book.css('div.ficha &gt; div.caracteristicas &gt; ul &gt; li').extract(),
                      }
                      # Return back and go to bext page in div#paginat ul li.next a::attr(href) and begin again
                      next_page = response.css('div#paginat ul li.next a::attr(href)').extract_first()
                      if next_page is not None:
                          next_page = response.urljoin(next_page)
                          yield scrapy.Request(next_page, callback=self.parse)
  </code></pre>
  
  <p>It gives an error: </p>
  
  <pre><code>AttributeError: 'YourCrawler' object has no attribute 'parse_following_urls'
  </code></pre>
  
  <p>And I don't understand why!</p>
  
  <p>EDIT -- </p>
  
  <p>As the response says, I had to close the method with the indentation like here:</p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider
  from scrapy.selector import Selector
  from scrapy.http import Request
  
  class YourCrawler(CrawlSpider):
      name = "bookstore_2"
      start_urls = [
      'https://example.com/materias/?novedades=LC&amp;p',
      ]
  
      def parse(self, response):
          # go to the urls in the list
          s = Selector(response)
          page_list_urls = s.xpath('///*[@id="results"]/ul/li/div[1]/h4/a[2]/@href').extract()
          for url in page_list_urls:
              yield Request(url, callback=self.parse_following_urls, dont_filter=True)
  
      # For the urls in the list, go inside, and in div#main, take the div.ficha &gt; div.caracteristicas &gt; ul &gt; li
      def parse_following_urls(self, response):
          #Parsing rules go here
          for each_book in response.css('div#main'):
              yield {
              'book_isbn': each_book.css('div.ficha &gt; div.caracteristicas &gt; ul &gt; li').extract(),
              }
              # Return back and go to bext page in div#paginat ul li.next a::attr(href) and begin again
              next_page = response.css('div#paginat ul li.next a::attr(href)').extract_first()
              if next_page is not None:
                  next_page = response.urljoin(next_page)
                  yield scrapy.Request(next_page, callback=self.parse)
  </code></pre>
  
  <p>But there is another problem, I think related to the urls, and now I am having this traceback:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/usr/local/lib/python2.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
      yield next(it)
    File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
      for x in result:
    File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in &lt;genexpr&gt;
      return (_set_referer(r) for r in result or ())
    File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in &lt;genexpr&gt;
      return (r for r in result or () if _filter(r))
    File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in &lt;genexpr&gt;
      return (r for r in result or () if _filter(r))
    File "/Users/nikita/scrapy/bookstore_2/bookstore_2/spiders/bookstore_2.py", line 16, in parse
      yield Request(url, callback=self.parse_following_urls, dont_filter=True)
    File "/usr/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
      self._set_url(url)
    File "/usr/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 58, in _set_url
      raise ValueError('Missing scheme in request url: %s' % self._url)
  ValueError: Missing scheme in request url: /book/?id=9780374281083
  </code></pre>
  
  <p>Maybe because I have to tell scrappy what is the base url? Should I add somewhere a urljoin?</p>
  
  <p>EDIT_2 ---</p>
  
  <p>Ok, the problem was with the urls. Adding </p>
  
  <pre><code>response.urljoin(
  </code></pre>
  
  <p>solved this issue.</p>