title: Web Scraping XHR Dynamic pages with rvest and R
body: <p>I am attempting to scrape a Dynamic website Morningstar.com via XHR requests. </p>
  
  <p>The exact site I am scraping is: <a href="http://performance.morningstar.com/funds/etf/total-returns.action?t=SPY&amp;region=USA&amp;culture=en_US" rel="nofollow noreferrer">http://performance.morningstar.com/funds/etf/total-returns.action?t=SPY&amp;region=USA&amp;culture=en_US</a> </p>
  
  <p>What I am trying to scrape is the Quarterly performance number (1-month).  The result should be 0.64 as of today. </p>
  
  <pre><code>try(res &lt;- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
                   query = list(
                     t="SPY",
                     region="usa",
                     culture="en-US"
                   )
    ))
  
    tryCatch(x &lt;- content(res) %&gt;%
               html_nodes(xpath = '//*[@id="tab-quar-end-content"]/table/tbody/tr[1]/td[1]') %&gt;% 
               html_text() %&gt;%
               trimws() %&gt;% 
               as.numeric()
             , error = function(e) x &lt;-NA)
  </code></pre>
  
  <p>However, the result is numeric(0)</p>
  
  <p>Any idea what I am doing wrong? </p>
  
  <p>Sody</p>
  
  <p><strong>Update:</strong> </p>
  
  <p>I was able to get the html data with the following code: </p>
  
  <pre><code> try(res &lt;- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
                   query = list(
  
                     t = "SPY",
                     region = "usa",
                     culture = "en-US",
                     ops = "clear",
                     s = "0P0000J533",
                     ndec = "2",
                     ep = "true",
                     align = "q",
                     annlz = "true",
                     comparisonRemove = "false"
  
                   )
    ))
  </code></pre>
  
  <p>But I am still having problems pointing to the data using either the CSS selector or the xpath with rvest. </p>
  
  <p>What do you guys use to find those data points? is SelectorGadget still the go to? </p>
  
  <p>Cheers, Aaron</p>