title: Scrapy: FormRequest doesn't auto-populate ASP.net hidden fields
body: <p>I'm using Scrapy to scrape some gold that's behind an authentication screen. The website uses <code>ASP.net</code> and ASP's got some stupid hidden fields littered all over the form (like <code>__VIEWSTATE</code>, <code>__EVENTTARGET</code>). </p>
  
  <p>When I call <code>FormRequest.from_response(response,...</code> I'm expecting that it reads these hidden fields automatically from the response and populates them in the <code>formdata</code> dictionary - which is what <a href="http://doc.scrapy.org/en/latest/topics/request-response.html#scrapy.http.FormRequest.from_response" rel="nofollow">Scrapy's FormRequest</a> documentation says it <em>should</em> do.</p>
  
  <p>But if that's the case, then why does the login process only work when I explicitly list these fields and populate them?</p>
  
  <pre><code>class ItsyBitsy(Spider):
      name = "itsybitsy"
      allowed_domains = ["website.com"]
      start_urls = ["http://website.com/cpanel/Default.aspx"]
  
      def parse(self, response):
          # Performs authentication to get past the login form
          sel = Selector(response)
          return [FormRequest.from_response(response,
          formdata={
          'tb_Username':'admin',
          'tb_Password':'password',
  
          # The following fields should be auto populated, right?
          # So why does removing 'em break the login (w/500 Server Error)
          '__VIEWSTATE':
                sel.xpath("//input[@name='__VIEWSTATE']/@value").extract(),
          '__EVENTVALIDATION':
                sel.xpath("//input[@name='__EVENTVALIDATION']/@value").extract(),
          '__EVENTTARGET': 'b_Login'
  
          },
          callback=self.after_login,
          clickdata={'id':'b_Login'},
          dont_click=True)]
  
      def after_login(self, response):
          # Mmm, scrumptious
          pass
  </code></pre>
  
  <h2>Edit: Adding form HTML</h2>
  
  <pre><code>&lt;form id="form1" action="Default.aspx" method="post" name="form1"&gt;
  &lt;div&gt;
  &lt;input type="hidden" value="" id="__EVENTTARGET" name="__EVENTTARGET"&gt;
  &lt;input type="hidden" value="" id="__EVENTARGUMENT" name="__EVENTARGUMENT"&gt;
  &lt;input type="hidden" value="/wEPDwULLTE2OTg2NjA1NTAPZBYCAgMPZBYGAgMPD2QWAh4Kb25rZXlwcmVzcwUlcmV0dXJuIGNsaWNrQnV0dG9uKGV2ZW50LCAnYl9Mb2dpbicpO2QCBQ8PZBYCHwAFJXJldHVybiBjbGlja0J1dHRvbihldmVudCwgJ2JfTG9naW4nKTtkAgcPD2QWAh4Hb25jbGljawUPcmV0dXJuIGxvZ2luKCk7ZGRKt/WTOQThVTxB9Y0QcIuRqylCIw==" id="__VIEWSTATE" name="__VIEWSTATE"&gt;
  &lt;/div&gt;
  
  &lt;script type="text/javascript"&gt;
  //&amp;lt;![CDATA[
  var theForm = document.forms['form1'];
  if (!theForm) {
  theForm = document.form1;
  }
  function __doPostBack(eventTarget, eventArgument) {
  if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
  theForm.__EVENTTARGET.value = eventTarget;
  theForm.__EVENTARGUMENT.value = eventArgument;
  theForm.submit();
  }
  }
  //]]&amp;gt;
  &lt;/script&gt;
  
  
  &lt;div&gt;
  
  &lt;input type="hidden" value="/wEWBAK0o8DDCQLxz5rcDwLF8dCIDALHyYWSA+rA4VJNaEpFIycMDHQPUOz393TI" id="__EVENTVALIDATION" name="__EVENTVALIDATION"&gt;
  &lt;input type="text" onkeypress="return clickButton(event, 'b_Login');" size="28" class="textfield-text" id="tb_Username" name="tb_Username"&gt;
  &lt;input type="password" onkeypress="return clickButton(event, 'b_Login');" size="28" class="textfield-text" id="tb_Password" name="tb_Password"&gt;
  &lt;a href="javascript:__doPostBack('b_Login','')" class="button-link" id="b_Login" onclick="return login();"&gt;Login&lt;/a&gt;
  &lt;/form&gt;
  </code></pre>