title: Unable to mimic POST request with headers - VBA
body: <p>I have spent about 15 days trying to make a scraper in VBA, I've been making decent progresses day after day but these last two days I got stuck in the very last step to get the data.</p>
  <p>This is a continuation of my previous <a href="https://stackoverflow.com/questions/62252608/necessary-href-not-shown-in-the-source-code-web-scaping-with-vba">post</a>, which gave me a good guide to start.</p>
  <p>Here's the process I want to simulate usign MSXML (not Internet Explorer)</p>
  <ol>
  <li>Open <a href="https://beacon.schneidercorp.com/" rel="nofollow noreferrer">https://beacon.schneidercorp.com/</a></li>
  <li>Select &quot;Iowa State&quot;</li>
  <li>Select &quot;Boone County, IA&quot;</li>
  <li>Click on the popup link &quot;Property Search&quot;</li>
  <li>In the top red ribbon, click on the &quot;Comp Search&quot; label</li>
  <li>At the bottom of the resulting page, in the &quot;Agricultural Comparables Search&quot; section check the &quot;Sale Date&quot; checkbox</li>
  <li>Select 5 months in the &quot;Sale Date&quot; combobox</li>
  <li>Click on the &quot;Search&quot; button at the bottom of the &quot;Agricultural Comparables Search&quot; section</li>
  <li>In the resulting page, look for &quot;Parcel ID&quot; identified as &quot;088327354400002&quot; and click on the link on the &quot;Recording&quot; column (value &quot;2020-0418&quot;)</li>
  </ol>
  <p>I could achieve the first 8 steps but I haven't been able to get URL of the results that should be get from that last link held in &quot;2020-0418&quot;</p>
  <p>As I did to get from the 8th to the 9th step, I noticed that inside the Development ToolKit's &quot;Network&quot; Tab, the website sent a POST request, as shown below.</p>
  <pre><code>**General**
      Request URL: https://beacon.schneidercorp.com/Application.aspx?AppID=84&amp;LayerID=795&amp;PageTypeID=3&amp;PageID=579&amp;Q=1926372975
      Request Method: POST
      Status Code: 302 
      Remote Address: 52.168.93.150:443
      Referrer Policy: no-referrer-when-downgrade
  **Response Headers**
      alt-svc: quic=&quot;:443&quot;; ma=2592000; v=&quot;44,43,39&quot;
      cache-control: private
      content-encoding: gzip
      content-length: 187
      content-type: text/html; charset=utf-8
      date: Sat, 27 Jun 2020 00:46:42 GMT
      location: /Application.aspx?AppID=84&amp;LayerID=795&amp;PageTypeID=3&amp;PageID=551&amp;Q=1603287013
      status: 302
      vary: Accept-Encoding
  **Request Headers**
      :authority: beacon.schneidercorp.com
      :method: POST
      :path: /Application.aspx?AppID=84&amp;LayerID=795&amp;PageTypeID=3&amp;PageID=579&amp;Q=1926372975
      :scheme: https
      accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-   exchange;v=b3;q=0.9
      accept-encoding: gzip, deflate, br
      accept-language: es-ES,es;q=0.9
      cache-control: max-age=0
      content-length: 395
      content-type: application/x-www-form-urlencoded
      cookie: _ga=GA1.2.1299682399.1590279064; MODULES508=; MODULESVISIBILE508=18469; MODULES1024=;   MODULESVISIBILE1024=29489%7C29501; MODULES501=; MODULESVISIBILE501=10310; _gid=GA1.2.449363625.1593013300;  ASP.NET_SessionId=4xwgdh2cqto0kugirkani4vp; _gat=1
      origin: https://beacon.schneidercorp.com
      referer: https://beacon.schneidercorp.com/Application.aspx?AppID=84&amp;LayerID=795&amp;PageTypeID=3&amp;PageID=579&amp;Q=1926372975
      sec-fetch-dest: document
      sec-fetch-mode: navigate
      sec-fetch-site: same-origin
      sec-fetch-user: ?1
      upgrade-insecure-requests: 1
      user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116   Safari/537.36
  Query String Parameters (source view)
      AppID=84&amp;LayerID=795&amp;PageTypeID=3&amp;PageID=579&amp;Q=1926372975
  **Form Data (source view)**
  __EVENTTARGET=ctlBodyPane%24ctl02%24ctl01%24gvwAgCompResults%24ctl05%24lnkRecording&amp;__EVENTARGUMENT=&amp;__VIEWSTATE=cbg8zdrx99ofbjcpw9%2FCE8J0v2SY5W86N%2Fbx%2FU0CsnNPy9D3bcg%2F5YstkCGTwd03lObnZbF9%2B5QuO1lP658HYgyXsOmpImGVjhn47teNdO788MngiEN9qzZbzrOv8jZAd93B8QXltxoPV5dLVu0%2BELpETwwTteNsmbKNEr1IpBz2aSxsN1spJUTKy42SUE37HkdUqVpsQlCPHPyIomJH4b6CoepL2uG9y45pMbUYFZxPG5ob&amp;__VIEWSTATEGENERATOR=569DB96F
  </code></pre>
  <p>Next, I show a sample of my code</p>
  <pre><code>Sub ScrapingTest()
     Dim XMLpagina As New MSXML2.ServerXMLHTTP60
     Dim htmlDoc As New MSHTML.htmlDocument
     Dim strURL As String, strBodyRequest As String
     Dim strETarget As String, strVState As String
     Dim strT1 As String, strT2 As String, strT3 As String
     Dim strPageID As String, strPageTypeID As String
     
     '====================
     'FOR VIEWING PURPOSES I ONLY SHOW A SHORT VERSION OF MY ORIGINAL CODE,
     'TRYING TO CUT AS MUCH AS POSSIBLE...
     '====================
     
     'OPENING Comp Search Website - STEP 6
     strURL = &quot;https://beacon.schneidercorp.com/Application.aspx?AppID=84&amp;LayerID=795&amp;PageTypeID=2&amp;PageID=578&quot;
     'SEND GET REQUEST
     XMLpagina.Open &quot;GET&quot;, strURL, False
     XMLpagina.send
     htmlDoc.body.innerHTML = XMLpagina.responseText
     
     Call generarCopiaHtml(XMLpagina)
     
     'GETTING THE VALUES TO BE SEND ON THE REQUESTBODY OF THE NEXT REQUEST
     'GET THE EVENTTARGET
     strETarget = &quot;ctlBodyPane$ctl02$ctl01$btnSearch&quot; 'I DON'T SCRAPE FOR THIS BECAUSE IT'S ALWAYS THE SAME
     
     'GET THE VIEWSTATE VALUE
     strT1 = &quot;&lt;input type='hidden' name='__VIEWSTATE' id='__VIEWSTATE' value='&quot;
     strT1 = Replace(strT1, &quot;'&quot;, &quot;&quot;&quot;&quot;)
     strT2 = &quot;' /&gt;&quot;
     strT2 = Replace(strT2, &quot;'&quot;, &quot;&quot;&quot;&quot;)
     strVState = extraeVerg(XMLpagina.responseText, strT1, strT2) 'THIS CUSTOM FUNCTION EXTRACTS A TEXT LAYING BETWEEEN strT1 AND strT2
     
     'SETS THE REQUESTBODY
     strBodyRequest = &quot;__EVENTTARGET=ctlBodyPane%24ctl02%24ctl01%24btnSearch&quot;
     strBodyRequest = strBodyRequest &amp; &quot;&amp;__EVENTARGUMENT=&quot;
     strBodyRequest = strBodyRequest &amp; &quot;&amp;__VIEWSTATE=&quot; &amp; strVState
     strBodyRequest = strBodyRequest &amp; &quot;&amp;__VIEWSTATEGENERATOR=569DB96F&quot;
     strBodyRequest = strBodyRequest &amp; &quot;&amp;ctlBodyPane%24ctl02%24ctl01%24chkUseSaleDate=on&quot;
     strBodyRequest = strBodyRequest &amp; &quot;&amp;ctlBodyPane%24ctl02%24ctl01%24cboSaleDate=5&quot; 'DEFINES HOW MANY MONTHS THE SEARCH WILL GO THROUGH
     strBodyRequest = strBodyRequest &amp; &quot;&amp;ctlBodyPane%24ctl02%24ctl01%24txtSaleDateHigh_VCS3Ag=&quot; &amp; Month(Now) &amp; &quot;%2F&quot; &amp; Day(Now) &amp; &quot;%2F&quot; &amp; Year(Now)
     strBodyRequest = strBodyRequest &amp; &quot;&amp;ctlBodyPane%24ctl02%24ctl01%24txtCSRPointsHigh=&quot;
  
     'OPENING Comp Search Website(SHOWING RESULTS)- STEP 9
     'SEND THE REQUEST
     XMLpagina.Open &quot;POST&quot;, strURL, False
     XMLpagina.setRequestHeader &quot;Content-type&quot;, &quot;application/x-www-form-urlencoded&quot;
     XMLpagina.setRequestHeader &quot;Content-Length&quot;, Len(strBodyRequest)
     XMLpagina.send strBodyRequest
     
     'GENERATE A LOCAL COPY OF THE RESPONSE
     Call generarCopiaHtml(XMLpagina)
     
     'BUILDING THE URL FOR THE NEXT REQUEST
     strT1 = &quot;{'Name':'Comp Results','PageId':&quot;
     strT1 = Replace(strT1, &quot;'&quot;, &quot;&quot;&quot;&quot;)
     strT2 = &quot;,'PageTypeId':&quot;
     strT2 = Replace(strT2, &quot;'&quot;, &quot;&quot;&quot;&quot;)
     strT3 = &quot;,'Icon&quot;
     strT3 = Replace(strT3, &quot;'&quot;, &quot;&quot;&quot;&quot;)
     strPageID = extraeVerg(XMLpagina.responseText, strT1, strT2)
     strPageTypeID = extraeVerg(XMLpagina.responseText, strT1 &amp; strPageID &amp; strT2, strT3)
     'THE strURL MUST BE EXACTLY LIKE &quot;https://beacon.schneidercorp.com/Application.aspx?AppID=84&amp;LayerID=795&amp;PageTypeID=3&amp;PageID=579&quot;
     strURL = &quot;https://beacon.schneidercorp.com/Application.aspx?AppID=84&amp;LayerID=795&amp;PageTypeID=&quot; &amp; strPageTypeID &amp; &quot;&amp;PageID=&quot; &amp; strPageID
     
     'GETTING THE VALUES TO BE SEND ON THE REQUESTBODY
     
     strT1 = &quot;&lt;input type='hidden' name='__VIEWSTATE' id='__VIEWSTATE' value='&quot;
     strT1 = Replace(strT1, &quot;'&quot;, &quot;&quot;&quot;&quot;)
     strT2 = &quot;' /&gt;&quot;
     strT2 = Replace(strT2, &quot;'&quot;, &quot;&quot;&quot;&quot;)
     strVState = extraeVerg(XMLpagina.responseText, strT1, strT2)
  
     'SETS THE REQUESTBODY
     strETarget = &quot;ctlBodyPane$ctl02$ctl01$gvwAgCompResults$ctl45$lnkRecording&quot; 'THIS VALUE MIMICS THE CLICK ON THE RECORD &quot;2020-0418&quot; RELATED TO THE PARCEL ID &quot;088327354400002&quot;
     strBodyRequest = &quot;__EVENTTARGET=&quot; &amp; Application.WorksheetFunction.EncodeURL(strETarget)
     strBodyRequest = strBodyRequest &amp; &quot;&amp;__EVENTARGUMENT=&quot;
     strBodyRequest = strBodyRequest &amp; &quot;&amp;__VIEWSTATE=&quot; &amp; strVState
     strBodyRequest = strBodyRequest &amp; &quot;&amp;__VIEWSTATEGENERATOR=569DB96F&quot;
     'SEND THE REQUEST
     XMLpagina.Open &quot;POST&quot;, strURL, False
     XMLpagina.setRequestHeader &quot;Content-type&quot;, &quot;application/x-www-form-urlencoded&quot;
     XMLpagina.setRequestHeader &quot;Content-Length&quot;, Len(strBodyRequest)
     XMLpagina.send strBodyRequest
     
     'GENERATE A LOCAL COPY OF THE RESPONSE
     Call generarCopiaHtml(XMLpagina)
     
     'ON THIS POINT I SHOULD BE GETTING INSIDE THE &quot;Results&quot; WEBSITE WITH AN URL LIKE THIS
     ' &quot;https://beacon.schneidercorp.com/Application.aspx?AppID=84&amp;LayerID=795&amp;PageTypeID=3&amp;PageID=551&quot; 
     'WHICH GIVES A LIST OF THE PARCELS INVOLVED IN THE SALE, BUT IT STILL SHOWS THE LAST PAGE RESULTS...
     'I CAN'T SEE WHAT AM I DOING WRONG...
     
  End Sub
  </code></pre>
  <p>My real goal is to repeat this process to get data from some specific sales of all Iowa State Counties, but when I do the first all other won't be a problem.</p>
  <p><strong>Can someone show me what am I doing to wrong to make this work?</strong></p>
  <p>PS1: I apologize for another question related to the same problem, that I made about ten days ago, which was wrong from top to bottom, I was so tired then that I wrote some crazy stuff.</p>
  <p>PS2: Out there seems to be a lot of information about this, but whether I'm not prepared enough to get the solution or my case is not too frequent.</p>