title: R: How can I open a list of links to scrape the homepage of a news website?
body: <p>I'm trying to build a web scraper to <strong>scrape</strong> articles published on <a href="http://www.20min.ch" rel="nofollow noreferrer">www.20min.ch</a>, a news website, with R. Their api is openly accessible so I could create a dataframe containing titles, urls, descriptions, and timestamps with rvest. The next step would be to access every single link and create a list of article texts and combine it with my dataframe. However I don't know how to automatize the access to those articles. Ideally, I would like to read_html link 1, then copy the text with html node and then proceed to link 2...</p>
  <p>This is what I wrote so far:</p>
  <pre><code>site20min &lt;- read_xml(&quot;https://api.20min.ch/rss/view/1&quot;)
  
  site20min
  
  url_list &lt;- site20min %&gt;% html_nodes('link') %&gt;% html_text()
  
  df20min &lt;- data.frame(Title = character(),
                        Zeit = character(),
                        Lead = character(),
                        Text = character()
                        )
  
   for(i in 1:length(url_list)){
        myLink &lt;- url_list[i]
        site20min &lt;- read_html(myLink)
              
        titel20min &lt;- site20min %&gt;% html_nodes('h1 span') %&gt;% html_text()
        zeit20min &lt;- site20min %&gt;% html_nodes('#story_content .clearfix span') %&gt;% html_text()
        lead20min &lt;- site20min %&gt;% html_nodes('#story_content h3') %&gt;% html_text()
        text20min &lt;- site20min %&gt;% html_nodes('.story_text') %&gt;% html_text()  
             
        df20min_a &lt;- data.frame(Title = titel20min)
        df20min_b &lt;- data.frame(Zeit = zeit20min)
        df20min_c &lt;- data.frame(Lead = lead20min)
        df20min_d &lt;- data.frame(Text = text20min)
                }
  </code></pre>
  <p>What I need is R to open every single link and extract some information:</p>
  <pre><code>site20min_1 &lt;- read_html(&quot;https://www.20min.ch/schweiz/news/story/-Es-liegen-auch-Junge-auf-der-Intensivstation--14630453&quot;)
  
    titel20min_1 &lt;- site20min_1 %&gt;% html_nodes('h1 span') %&gt;% html_text()
    zeit20min_1 &lt;- site20min_1 %&gt;% html_nodes('#story_content .clearfix span') %&gt;% html_text()
    lead20min_1 &lt;- site20min_1 %&gt;% html_nodes('#story_content h3') %&gt;% html_text()
    text20min_1 &lt;- site20min_1 %&gt;% html_nodes('.story_text') %&gt;% html_text()
  </code></pre>
  <p>It should not be too much of a problem to rbind this to a dataframe. but at the moment some of my results turn out empty.</p>
  <p>thx for your help!</p>