title: How to crawl a website that has pagination using Scrapy?
body: <p>I am trying to crawl a website that has pagination. If i click on "next" button at the bottom of page, New items will be generated. My scrapy program is not able to fetch dynamic data. Is there way i can fetch this data?</p>
  
  <p>HTML of next button looks like below</p>
  
  <pre><code>&lt;div id="morePaginationID"&gt;
  
  &lt;a href="javascript:void(0);" onclick="lazyPagingNew('db')"&gt;&lt;/a&gt;
  </code></pre>
  
  <p>and My Spider is</p>
  
  <pre><code>class ExampleSpider(CrawlSpider):
  
      name = "example"
      domain_name = "example.com"
      allowed_domains = ["example.com"]
      start_urls = ["http://example.com/beauty/90?utm_source=viewallbea"]
      rules = ( Rule(SgmlLinkExtractor(allow=('.*',),restrict_xpaths=('//div[@id="morePaginationID"]',)), callback = "parse_zero" , follow= True), )
      def parse_zero(self,response):
          hxs = HtmlXPathSelector(response)
          paths = hxs.select('//div[@id="containerDiv"]/div[@id="loadFilterResults"]/ul[@id="categoryPageListing"]/li')
          m = len(paths)
          for i in range(m):
  
              item = ExampleItem()
  
              item["dealUrl"] = paths[i].select("figure/figcaption/a/@href").extract()[0]
  
              url = str(item["Url"])
              yield Request(url, callback=self.parselevelone, meta={"item":item})
          spider = ExampleSpider()
      def parselevelone(self, response):
          hxs = HtmlXPathSelector(response)
          item = response.meta["item"]
          item["Title2"] = hxs.select('//div[@class="fullDetail"]/div/figure/figcaption/h2/text()').extract()[0]
          items.append(item)
          return item
  </code></pre>