title: Using Nokogiri to scrape search results from POST form
body: <p>I would like to scrape search results from <a href="http://maxdelivery.com" rel="nofollow">http://maxdelivery.com</a>, but unfortunately, they are using POST instead of GET for their search form. I found this description of how to use Nokogiri and RestClient to fake a post form submission, but it's not returning any results for me: <a href="http://ruby.bastardsbook.com/chapters/web-crawling/" rel="nofollow">http://ruby.bastardsbook.com/chapters/web-crawling/</a></p>
  
  <p>I've worked with Nokogiri before, but not for the results of a POST form submission.</p>
  
  <p>Here's my code right now, only slightly modified from the example at the link above:</p>
  
  <pre><code>class MaxDeliverySearch
  
    REQUEST_URL = "http://www.maxdelivery.com/nkz/exec/Search/Display"
  
    def initialize(search_term)
      @term = search_term
    end
  
    def search
      if page = RestClient.post(REQUEST_URL, {
          'searchCategory'=&gt;'*',
          'searchString'=&gt;@term,
          'x'=&gt;'0',
          'y'=&gt;'0'
        })
        puts "Success finding search term: #{@term}"
  
        File.open("temp/Display-#{@term}.html", 'w'){|f| f.write page.body}
  
        npage = Nokogiri::HTML(page)
        rows = npage.css('table tr')
        puts "#{rows.length} rows"
  
        rows.each do |row|
          puts row.css('td').map{|td| td.text}.join(', ')
        end
      end
    end
  
  end
  </code></pre>
  
  <p>Now (ignoring the formatting stuff), I would expect <code>if page = RestClient.post(REQUEST_URL, {...}</code> to return some search results if passed a 'good' search term, but each time I just get the search results page back with no actual results, as if I had pasted the URL into the browser.</p>
  
  <p>Anyone have any idea what I'm missing? Or, just how to get back the results I'm looking for with another gem?</p>
  
  <p>With the class above, I would like to be able to do:</p>
  
  <pre><code>s = MaxDeliverySearch.new("ham")
  s.search #=&gt; big block of search results objects to traverse
  </code></pre>