title: using scrapy to scrape asp.net website with javascript buttons and ajax requests
body: <p>I'd been trying to scrape some date from as asp.net website, the start page should be the following one:
  <a href="http://www.e3050.com/Items.aspx?cat=SON" rel="nofollow noreferrer">http://www.e3050.com/Items.aspx?cat=SON</a></p>
  
  <p>First, I want to display 50 item per page (from the select element)
  Second, I want to paginate through pages.</p>
  
  <p>I tried the following code for 50 items per page, but didn't work:</p>
  
  <pre><code>start_urls = ["http://www.e3050.com/Items.aspx?cat=SON"]    
  def parse(self, response):
          requests = []
          hxs = HtmlXPathSelector(response)
  
          # Check if there's more than 1 page         
          if len(hxs.select('//span[@id="ctl00_ctl00_ContentPlaceHolder1_ItemListPlaceHolder_lbl_PageSize"]/text()').extract()) &gt; 0:
              # Get last page number
              last_page = hxs.select('//span[@id="ctl00_ctl00_ContentPlaceHolder1_ItemListPlaceHolder_lbl_PageSize"]/text()').extract()[0]
              i = 1
  
              # preparing requests for each page
              while i &lt; (int(last_page) / 5)  + 1:
                  requests.append(Request("http://www.e3050.com/Items.aspx?cat=SON", callback=self.parse_product))
                  i +=1 
  
              # posting form date (50 items and next page button)
              requests.append(FormRequest.from_response(
                      response,
                      formdata={'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$pagesddl':'50',
                                '__EVENTTARGET':'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$pager1$ctl00$ctl01'},
                      callback=self.parse_product,
                      dont_click=True
                  )
              )
  
              for request in requests:
                  yield request
  </code></pre>