title: How to send form data with a POST request?
body: <p>I want to view and scrape job listings on the <a href="https://www.akzonobel.com/nl/careers/vacatures/" rel="nofollow">https://www.akzonobel.com/nl/careers/vacatures/</a> website. The country has to be "The Netherlands" and the Job level is "Entry level".</p>
  
  <p>I'm using <a href="https://github.com/jnunemaker/httparty" rel="nofollow">httparty</a> to send a POST request but it keeps returning the initial 10 job listings. The correct response should be 3 job listings.</p>
  
  <p>This is the code I'm using:</p>
  
  <pre><code>require 'httparty'
  require 'nokogiri'
  
  @base_url = 'https://www.akzonobel.com'
  
  url = "#{@base_url}/careers/vacatures/"
  
  data = {
    'ctl00$contentLeft$ctl01$ddlCountryExt' =&gt; 'NLD',
    'ctl00$contentLeft$ctl01$ddlJobLevelExt' =&gt; 'ENTRY_LEVEL'
  }
  
  response = HTTParty.post("#{@base_url}/nl/careers/vacatures/", :body =&gt; data)
  
  html = Nokogiri::HTML(response)
  
  jobs = html.xpath('//h3//a')
  
  jobs.each do |job|
    puts job.text
  end
  
  puts jobs.size
  </code></pre>
  
  <p>Which returns:</p>
  
  <pre><code>Regional Demand Planner Nordeuropa (m,w)
  Forecast Analyst - TiO2 Spend Area
  PS Regional Manager APAC
  Production leader
  Engineering Administrator - Temporary
  Procurement Manager EMEA
  Business Analyst, Americas
  HR Business Partner Supply Chain and R&amp;D
  AS Regional Manager
  Business Information Manager
  10
  </code></pre>
  
  <p>How can I send the form data needed to the site to get the correct response?</p>
  
  <hr>
  
  <p>Update:</p>
  
  <p>I've tried the following:</p>
  
  <pre><code>require 'httparty'
  require 'nokogiri'
  
  @base_url = 'https://www.akzonobel.com'
  
  url = "#{@base_url}/careers/vacatures/"
  
  data = {
    'ctl00$contentLeft$ctl01$ddlCountryExt' =&gt; 'NLD',
    'ctl00$contentLeft$ctl01$ddlJobLevelExt' =&gt; 'ENTRY_LEVEL',
    'ctl00$contentLeft$ctl01$ddlContinentExt' =&gt; 1,
    'ctl00$contentLeft$ctl01$ddlRegionEx' =&gt; 4,
    'ctl00$contentLeft$ctl01$ddlJobFamilyEx' =&gt; 45,
    'ctl00$contentLeft$ctl01$ddlBusinessUnitExt' =&gt; 22,
    'ctl00$contentLeft$ctl01$ddlJobLevelExt' =&gt; 1,
    'ctl00$contentLeft$ctl01$ddlCountryExt' =&gt; 1,
  }
  
  response = HTTParty.post("#{@base_url}/nl/careers/vacatures/", :body =&gt; data)
  
  html = Nokogiri::HTML(response)
  
  jobs = html.xpath('//h3//a')
  
  jobs.each do |job|
    puts job.text
  end
  
  puts jobs.size
  </code></pre>
  
  <p>Unfortunately the result is exactly the same.</p>
  
  <hr>
  
  <p>Update 2:</p>
  
  <p>Here's the updated code:</p>
  
  <pre><code>require 'httparty'
  require 'nokogiri'
  
  @base_url = 'https://www.akzonobel.com'
  
  url = "#{@base_url}/careers/vacatures/"
  
  data = {
    'contentLeft_ctl01_ddlContinentExt' =&gt; 'C_EUROPE',
    'contentLeft_ctl01_ddlCountryExt' =&gt; 'NLD',
    'contentLeft_ctl01_ddlRegionExt' =&gt; 'Gelderland',
    'contentLeft_ctl01_ddlRegionExt' =&gt; 'Limburg',
    'contentLeft_ctl01_ddlRegionExt' =&gt; 'North Holland',
    'contentLeft_ctl01_ddlRegionExt' =&gt; 'South Holland',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'General Management',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Integrated Supply Chain',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Sales &amp; Marketing',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'RD&amp;I',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Support',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Other',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Lvl2_General Management',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Manufacturing',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'HSE',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Engineering',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Procurement',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Distribution &amp; Logistics',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Sales',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Marketing',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Lvl2_RD&amp;I',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Finance',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'IM',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'HR',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Legal, IP &amp; Compliance',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Facilities',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Lvl2_Other',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '80200000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '80300000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81900000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81100000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '82000000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81200000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '80700000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '80400000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '80500000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '80800000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '80900000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '82100000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '82200000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81010000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81020000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81030000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81040000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81300000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81410000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81420000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81430000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81600000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; '81700000',
    'contentLeft_ctl01_ddlJobFamilyExt' =&gt; 'Lvl3_Other',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '52000100',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '52000200',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '52000300',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '52000900',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000010',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000013',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000020',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000022',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000026',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000033',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000038',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000041',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000054',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000055',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000056',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000061',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000063',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000100',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000300',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000900',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '53000901',
    'contentLeft_ctl01_ddlBusinessUnitExt' =&gt; '51000000',
    'contentLeft_ctl01_ddlJobLevelExt' =&gt; 'ENTRY_LEVEL'
  }
  
  response = HTTParty.post("#{@base_url}/nl/careers/vacatures/", :body =&gt; data)
  
  html = Nokogiri::HTML(response)
  
  jobs = html.xpath('//h3//a')
  
  jobs.each do |job|
    puts job.text
  end
  
  puts jobs.size
  </code></pre>
  
  <p>Giving me the exact same result as before.</p>