title: How can I time scraping news stories from a list of urls with R?
body: <p>I am trying to download the text of newspaper articles for textual analysis using R. I have a large list of urls to individual articles and want to use Rvest to extract each of these articles' text and title and convert it into a data frame.</p>
  
  <p>As an example, I have a subset of my dataset with articles from The Guardian:</p>
  
  <pre><code>&gt; items$link[1:8]
  
  [1] "https://www.theguardian.com/uk-news/2019/nov/16/concerns-raised-cladding-bolton-student-building-fire"                                   
  [2] "https://www.theguardian.com/uk-news/2019/nov/16/top-lawyer-calls-prince-andrew-bbc-interview-catastrophic-error"                         
  [3] "https://www.theguardian.com/politics/live/2019/nov/16/general-election-labour-meet-decide-manifesto-clause-v-live-news"                  
  [4] "https://www.theguardian.com/politics/2019/nov/16/priti-patel-block-rescue-british-isis-children"                                         
  [5] "https://www.theguardian.com/politics/2019/nov/16/police-assessing-claims-that-tories-offered-peerages-to-brexit-party"                   
  [6] "https://www.theguardian.com/world/2019/nov/16/paris-police-fire-teargas-on-anniversary-of-gilets-jaunes-protests"                        
  [7] "https://www.theguardian.com/us-news/2019/nov/16/trump-personally-kept-pressure-ukraine-impeachment-inquiry-witness-david-holmes-diplomat"
  [8] "https://www.theguardian.com/world/2019/nov/16/hong-kong-chinese-troops-deployed-to-help-clear-roadblocks"  
  </code></pre>
  
  <p>My code so far is:</p>
  
  <pre><code>## SETUP ##
  rm(list=ls())
  library(tidyverse)
  library(rvest)
  library(stringr)
  library(readtext)
  library(quanteda)
  library(beepr)
  
  setwd("uk")
  
  ## Functions ##
  parse_texts &lt;- function(nod){
    body &lt;- str_squish(as.character(nod) %&gt;% read_html() %&gt;%
                         html_nodes('.js-article__body &gt; p') %&gt;% #collects all text in article
                         html_text())
    one_body &lt;- paste(body, collapse = " ") # puts all of the text together
    data.frame(title = str_squish(nod %&gt;% read_html() %&gt;% 
                                    html_node('.content__headline') %&gt;% 
                                    html_text()),
               date_time = str_squish(nod %&gt;% read_html() %&gt;% 
                                        html_node('.content__dateline-wpd--modified') %&gt;% 
                                        html_text()),
               text = one_body,
               stringsAsFactors = FALSE)
  }
  
  #extract file text
  test_df &lt;- lapply(items$link[1:5], parse_texts) %&gt;% bind_rows()
  </code></pre>
  
  <p>This works, for the most part. My problem is that I have thousands of urls in my data. How can I automate a script that will slowly work through this list? </p>