title: R memory issues while webscraping with rvest
body: <p>I am using <code>rvest</code> to webscrape in R, and I'm running into memory issues. I have a 28,625 by 2 data frame of strings called <code>urls</code> that contains the links to the pages I'm scraping. A row of the frame contains two related links. I want to generate a 28,625 by 4 data frame <code>Final</code> with information scraped from the links. One piece of information is from the second link in a row, and the other three are from the first link. The xpaths to the three pieces of information are stored as strings in the vector <code>xpaths</code>. I am doing this with the following code:</p>
  
  <pre><code>data &lt;- rep("", 4 * 28625)
  
  k &lt;- 1
  
  for (i in 1:28625) {
  
    name &lt;- html(urls[i, 2]) %&gt;%
      html_node(xpath = '//*[@id="seriesDiv"]/table') %&gt;%
      html_table(fill = T)
  
    data[k] &lt;- name[4, 3]
  
    data[k + 1:3] &lt;- html(urls[i, 1]) %&gt;% 
      html_nodes(xpath = xpaths) %&gt;%
      html_text()
  
    k &lt;- k + 4
  
  }
  
  dim(data) &lt;- c(4, 28625)
  Final &lt;- as.data.frame(t(data))
  </code></pre>
  
  <p>It works well enough, but when I open the task manager, I see that my memory usage has been monotonically increasing and is currently at 97% after about 340 iterations. I'd like to just start the program and come back in a day or two, but all of my RAM will be exhausted before the job is done. I've done a bit of research on how R allocates memory, and I've tried my best to preallocate memory and modify in place, to keep the code from making unnecessary copies of things, etc. </p>
  
  <p>Why is this so memory intensive? Is there anything I can do to resolve it?</p>