title: R scraping entire html, not just page view
body: <p>Bit of a difficult question to put into one sentence, but I am trying to scrape some html from the following page</p>
  
  <pre><code>http://www.ncbi.nlm.nih.gov/snp/?term=(human[Organism])+AND+GLRA3[Gene Name]
  </code></pre>
  
  <p>where I can scrape what I need using R, but because the browser only displays the first 20 entries, only the corresponding html is available to me. This causes a problem, because I want to scrape <strong>all</strong> entries, not just the entries that are being served up by the page on the browser. Any way, here is my R code</p>
  
  <pre><code>library(XML)
  library(httr)
  
  #Go to Nectar Mutation and get SNP refs
  dbsnp.searchterm="(human[Organism])+AND+GLRA1[Gene Name]"
  dbsnp.url=paste0("http://www.ncbi.nlm.nih.gov/snp/?term=",dbsnp.searchterm)
  dbsnp.get=GET(dbsnp.url)
  dbsnp.content=content(dbsnp.get, as="text")
  links&lt;-xpathSApply(htmlParse(dbsnp.content), "//a[contains(@href, 'snp_ref')]",xmlGetAttr,"href")
  </code></pre>
  
  <p>and result</p>
  
  <pre><code>&gt; links
   [1] "/projects/SNP/snp_ref.cgi?rs=116474260"
   [2] "/projects/SNP/snp_ref.cgi?rs=121918408"
   [3] "/projects/SNP/snp_ref.cgi?rs=121918409"
   [4] "/projects/SNP/snp_ref.cgi?rs=121918410"
   [5] "/projects/SNP/snp_ref.cgi?rs=121918411"
   [6] "/projects/SNP/snp_ref.cgi?rs=121918412"
   [7] "/projects/SNP/snp_ref.cgi?rs=121918413"
   [8] "/projects/SNP/snp_ref.cgi?rs=121918414"
   [9] "/projects/SNP/snp_ref.cgi?rs=121918415"
  [10] "/projects/SNP/snp_ref.cgi?rs=121918416"
  [11] "/projects/SNP/snp_ref.cgi?rs=121918417"
  [12] "/projects/SNP/snp_ref.cgi?rs=121918418"
  [13] "/projects/SNP/snp_ref.cgi?rs=267600494"
  [14] "/projects/SNP/snp_ref.cgi?rs=267606848"
  [15] "/projects/SNP/snp_ref.cgi?rs=281864912"
  [16] "/projects/SNP/snp_ref.cgi?rs=281864913"
  [17] "/projects/SNP/snp_ref.cgi?rs=281864914"
  [18] "/projects/SNP/snp_ref.cgi?rs=281864915"
  [19] "/projects/SNP/snp_ref.cgi?rs=281864916"
  [20] "/projects/SNP/snp_ref.cgi?rs=281864917"
  </code></pre>
  
  <p>You will note that there are 4058 entries that I require.</p>