title: Troubles using scrapy with javascript __doPostBack method
body: <p>Trying to automatically grab the search results from a public search, but running into some trouble. The URL is of the form </p>
  
  <pre><code>http://www.website.com/search.aspx?keyword=#&amp;&amp;page=1&amp;sort=Sorting
  </code></pre>
  
  <p>As I click through the pages, after visiting this page, it changes slightly to </p>
  
  <pre><code>http://www.website.com/search.aspx?keyword=#&amp;&amp;sort=Sorting&amp;page=2
  </code></pre>
  
  <p>Problem being, if I then try to directly visit the second link without first visiting the first link, I am redirected to the first link. My current attempt at this is defining a long list of start_urls in scrapy.</p>
  
  <pre><code>class websiteSpider(BaseSpider):
      name = "website"
      allowed_domains = ["website.com"]
      baseUrl = "http://www.website.com/search.aspx?keyword=#&amp;&amp;sort=Sorting&amp;page="
      start_urls = [(baseUrl+str(i)) for i in range(1,1000)]
  </code></pre>
  
  <p>Currently this code simply ends up visiting the first page over and over again. I feel like this is probably straightforward, but I don't quite know how to get around this.</p>
  
  <p>UPDATE:
  Made some progress investigating this and found that the site updates each page by sending a POST request to the previous page using __doPostBack(arg1, arg2). My question now is how exactly do I mimic this POST request using scrapy. I know how to make a POST request, but not exactly how to pass it the arguments I want.</p>
  
  <p>SECOND UPDATE:
  I've been making a lot of progress! I think... I looked through examples and documentation and eventually slapped together this version of what I think should do the trick:</p>
  
  <pre><code>def start_requests(self):
      baseUrl = "http://www.website.com/search.aspx?keyword=#&amp;&amp;sort=Sorting&amp;page="
      target = 'ctl00$empcnt$ucResults$pagination'
      requests = []
      for i in range(1, 5):
          url = baseUrl + str(i)
          argument = str(i+1)
          data = {'__EVENTTARGET': target, '__EVENTARGUMENT': argument}
          currentPage = FormRequest(url, data)
          requests.append(currentPage)
      return requests
  </code></pre>
  
  <p>The idea is that this treats the POST request just like a form and updates accordingly. However, when I actually try to run this I get the following traceback(s) (Condensed for brevity):</p>
  
  <pre><code>2013-03-22 04:03:03-0400 [guru] ERROR: Unhandled error on engine.crawl()
  dfd.addCallbacks(request.callback or spider.parse, request.errback)
        File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 280, in addCallbacks
          assert callable(callback)
      exceptions.AssertionError: 
  
  2013-03-22 04:03:03-0400 [-] ERROR: Unhandled error in Deferred:
  2013-03-22 04:03:03-0400 [-] Unhandled Error
      Traceback (most recent call last):
      Failure: scrapy.exceptions.IgnoreRequest: Skipped (request already seen)
  </code></pre>
  
  <p>Changing question to be more directed at what this post has turned into.</p>
  
  <p>Thoughts?</p>
  
  <p>P.S. When the second errors happen scrapy is unable to cleany shutdown and I have to send a SIGINT twice to get things to actually wrap up.</p>