title: 'RCurl' [R] package getURL webpage error when scraping API
body: <p>I am trying to scrape data on pages from an API using the getURL function of the RCurl package in R. My problem is that I can't replicate the response that I get when I open the URL in Chrome when I make the request using R. Essentially, when I open the API page (url below) in Chrome it works fine but if I request it in using getURL in R (or using incognito mode in Chrome) I get a '500 Internal Server Error' response and not the pretty JSON that I'm looking for. </p>
  
  <p>URL/API in question:
  <a href="http://www.bluenile.com/api/public/loose-diamond/diamond-details/panel?country=USA&amp;currency=USD&amp;language=en-us&amp;productSet=BN&amp;sku=LD04077082" rel="nofollow noreferrer">http://www.bluenile.com/api/public/loose-diamond/diamond-details/panel?country=USA&amp;currency=USD&amp;language=en-us&amp;productSet=BN&amp;sku=LD04077082</a></p>
  
  <p>Here is my (failed) request in [R].</p>
  
  <pre><code>test2 &lt;- fromJSON(getURL("http://www.bluenile.com/api/public/loose-diamond/diamond-details/panel?country=USA&amp;currency=USD&amp;language=en-us&amp;productSet=BN&amp;sku=LD04077082", ssl.verifypeer = FALSE, useragent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36"))
  </code></pre>
  
  <p><strong>My Research so Far</strong>
  First I looked at this prior question on stack and added in my useragent to the request (did not solve problem but may still be necessary):
  <a href="https://stackoverflow.com/questions/24418691/viralheat-api-issues-with-geturl-command-in-rcurl-package">ViralHeat API issues with getURL() command in RCurl package</a></p>
  
  <p>Next I looked at this helpful post which guides my rationale:
  <a href="https://stackoverflow.com/questions/25173211/r-disparity-between-browser-and-get-geturl">R Disparity between browser and GET / getURL</a></p>
  
  <p><strong>My Ideas About the Solution</strong>
  This is not my area of expertise but my guess is that the request is lacking a cookie needed to complete the request (hence why it doesn't work in my browser in incognito mode). I compared the requests and responses from the successful request to the unsuccessful request:</p>
  
  <p><strong>Successful request:</strong>
  <img src="https://i.stack.imgur.com/SwWMB.png" alt="enter image description here"></p>
  
  <p><strong>Unsuccessful request:</strong></p>
  
  <p><img src="https://i.stack.imgur.com/STpbr.png" alt="enter image description here"></p>
  
  <p>Anyone have any ideas? Should I try using the package <code>RSelenium</code> package that was suggested by MrFlick in the 2nd post I made.</p>