title: Scrapy & Selenium - Load next pages
body: <p>I am trying to scrape the following website: <a href="https://sabobic.blogabet.com" rel="nofollow noreferrer">https://sabobic.blogabet.com</a></p>
  
  <p>My crawler already crawls the content I need. But after clicking on the "See older"-Button I do not know how to start my method "crawltips" again.</p>
  
  <p>This is my current code:</p>
  
  <pre><code>class AlltipsSpider(Spider):
      name = 'alltips'
      allowed_domains = ['blogabet.com']
  
      def start_requests(self):
          self.driver = webdriver.Chrome('C:\webdrivers\chromedriver.exe')
          # Place all user urls here
          url = "https://sabobic.blogabet.com"
          self.driver.get(url)
          yield scrapy.http.Request (url, callback=self.crawltips)
  
  
      def crawltips(self, response):
          sel = Selector(text=self.driver.page_source)
          allposts = sel.xpath('//*[@class="block media _feedPick feed-pick"]')
  
          for post in allposts:
              username = post.xpath('.//div[@class="col-sm-7 col-lg-6 no-padding"]/a/@title').extract()
              publish_date = post.xpath('.//*[@class="bet-age text-muted"]/text()').extract()
  
              yield{'Username': username,
                  'Publish date': publish_date
                  }
  
          try:
              self.driver.find_element_by_id('last_item').click()
              sleep(5)
          except NoSuchElementException:
              self.logger.info('No more tipps')
          yield Request(url, callback=self.crawltips)
  </code></pre>
  
  <p>I think something is wrong with the yield function because I dont have a new url...</p>