title: How to use Scrapy FormRequest.from_response() method to submit a form?
body: <p>I am trying to parse some info from <a href="http://www.zooroyal.de/muehldorfer-pferdeleckerli-bio-bronchial?c=6259" rel="nofollow">this</a> page and then reload the page with other options from the drop-down menu called "Packungsgröße". There are 3 options, so I am trying to scrape the same page for all the options.</p>
  
  <p>I found out that I can use the form of <code>class="upprice_config"</code> and <code>POST</code> it with the correct data. I do not know exactly which data I need to supply, so I supply all I have. Then I call the same function for parsing.</p>
  
  <p>Any way, it does not work the way I want. It does not give me any error and seems it executes the form submission, but I do not get the required page.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>... def parse_product(self, response):
  120     sc_menu = response.xpath('//form[@class="upprice_config"]/div/select')
  121         if sc_menu:
  122             sel_name = sc_menu[0].xpath('@name').extract()[0]
  124             for opt in sc_menu[0].xpath('option'):
  126                 if opt.xpath('@selected'):
  127                     selected = opt.xpath('text()').extract()[0]
  129                     product['options'] = selected.strip()
  130 
  131                     yield product
  132 
  133                 # extract value
  134                 value = opt.xpath('@value').extract()[0]
  136                 # submit form for the new option
  137                 self.submit_form(response, value, sel_name)
  138         else:
  139             product['options'] = ''
  140 
  141             yield product
  142 
  143 
  144 # submit form for the new option
  145 def submit_form(self, response, value, sel_name):
  146     formdata = {'method': 'POST',
  147                 'action': response.url,
  148                 'value': value}
  149     scrapy.http.FormRequest.from_response(response, 
  150                 formdata = formdata, 
  151                 clickdata = {'name': sel_name},
  152                 callback = self.parse_product)
  </code></pre>
  
  <p>Can somebody point me to what I am doing wrong here. If there is a better way to do it, please share with me.</p>