title: Can't Scrape Dynamically Loaded HTML Table in an Aspx Website
body: <p>I am trying to scrape some data from the <a href="https://azbomprod.azmd.gov/GLSuiteWeb/Clients/AZBOM/public/WebVerificationSearch.aspx?q=azmd&amp;t=20220622082816" rel="nofollow noreferrer">Arizona Medical Board</a>. I search for Anesthesiology in the specialty dropdown list and I find that the table (with the links to the profiles I want to scrape) are dynamically loaded into the website. I notice when hitting the 'specialty search' button, a POST request is made to the server and the html table is actually returned from the server. I have tried simulating this post request to see if I get receive this html table and then try to parse it with bs4. Is this possible, and if so, am I even on the right track?</p>
  <p>I have tried to included the form data I found in the network tab of the developer tools but I am not sure if this is the right data, or if I am forgetting some data here or in the header.</p>
  <p>Please let me know if I need to clarify, I understand this may not be worded the best. Thank you!</p>
  <pre><code>import requests
  # import re
  import formdata
  
  session = requests.Session()
  
  url = &quot;https://azbomprod.azmd.gov/GLSuiteWeb/Clients/AZBOM/public/WebVerificationSearch.aspx?q=azmd&amp;t=20220622123512&quot;
  
  headers = {'User-Agent': 'My-Agent-Placeholder'}
  
  res = session.get(url, headers=headers)
  
  print(&quot;Response: {}&quot;.format(res))
  
  payload = {
    &quot;__VIEWSTATE&quot;: formdata.state,
    &quot;__VIEWSTATEGENERATOR&quot;: formdata.generator,
    &quot;__EVENTVALIDATION&quot; : formdata.validation,
    &quot;ctl00%24ContentPlaceHolder1%24Name&quot;: 'rbName1',
    &quot;ctl00%24ContentPlaceHolder1%24Name&quot;: &quot;rbName1&quot;,
    &quot;ctl00%24ContentPlaceHolder1%24txtLastName&quot; : '', 
    &quot;ctl00%24ContentPlaceHolder1%24txtFirstName&quot; : '',
    &quot;ctl00%24ContentPlaceHolder1%24License&quot;: &quot;rbLicense1&quot;,
    &quot;ctl00%24ContentPlaceHolder1%24txtLicNum&quot;: '',
    &quot;ctl00%24ContentPlaceHolder1%24Specialty&quot;: &quot;rbSpecialty1&quot;,
    &quot;ctl00%24ContentPlaceHolder1%24ddlSpecialty&quot;: '12155',
    &quot;ctl00%24ContentPlaceHolder1%24ddlCounty&quot;: '15910',
    &quot;ctl00%24ContentPlaceHolder1%24txtCity&quot;: '',
    &quot;__EVENTTARGET&quot;: &quot;ctl00%24ContentPlaceHolder1%24btnSpecial&quot;,
    &quot;__EVENTARGUMENT&quot;: ''  
  }
  
  # params = {&quot;q&quot;: &quot;azmd&quot;,
  # &quot;t&quot;: &quot;20220622123512&quot;}
  
  # #url = &quot;https://azbomprod.azmd.gov/GLSuiteWeb/Clients/AZBOM/Public/Results.aspx&quot;
  
  res = session.post(url, data=payload, headers=headers)
  print(&quot;Post response: {}&quot;.format(res))
  print(res.text)
  
  # res = requests.get('https://azbomprod.azmd.gov/GLSuiteWeb/Clients/AZBOM/Public/Results.aspx', headers=headers)
  </code></pre>