title: Web-scraping dynamic Javascript page with RSelenium and rvest
body: <p>I am trying to create a dataframe of color IDs, description, and dates from <a href="https://www.pantone.com/pages/iphone/iphone_colorstrology.html" rel="nofollow noreferrer">this site</a>, which takes  day and month input through dropdown menus and returns, I think, a dynamic JS generated page. I'm new to coding and thought this would be a fun toy project. I'd like to use RSelenium to automate the dropdown selection, and rvest to scrape the generated content. The data frame structure I'm hoping for will look like:</p>
  
  <pre><code>description, date, meta
  "paragraph about birthday", Jun 01, "DAFFODIL PANTONE 17-1512 POWERFUL KNOWING EXPRESSIVE"
  </code></pre>
  
  <p>I'm attempting to first use a for loop to just iterate through each month of the year on a single day then work my way up to get every day for every month. </p>
  
  <p>I'm stuck on simply getting the loop to iterate through each month, and getting the content. I could use some conceptual help first on this part of the task and appreciate any insight!</p>
  
  <pre><code>library(RSelenium)
  library(rvest)
  library(tidyverse)
  library(xml2)
  
  ## first run: docker run -d -p 4445:4444 selenium/standalone-chrome
  ## open a new connection to Chrome
  remDr &lt;- RSelenium::remoteDriver(remoteServerAddr = "localhost",
                                   port = 4445L,
                                   browserName = "chrome")
  
  remDr$open()
  remDr$navigate("https://www.pantone.com/pages/iphone/iphone_colorstrology.html#___1__") #Entering our URL gets the browser to navigate to the page
  remDr$screenshot(display = TRUE) 
  
  #### create list of month/days
   month_day&lt;- read_html(remDr$getPageSource()[[1]])
   page_i &lt;- month_day %&gt;%
     html_nodes(".list") %&gt;%
     html_children() %&gt;% 
     html_text()
  
  months &lt;- page_i[1:12]
  months &lt;- (paste("'", months,"'", sep=''))
  days &lt;- page_i[13:43]
  days &lt;- as.numeric(days)
  
  
  ## create an object for month xpath elements
  for (m in months){
    elements &lt;- paste0("//option[contains(text(),",months,")]")
  }
  
  ## attempt at loop
  
  total &lt;- data.frame()
  
  for (e in elements){
  remDr$navigate("https://www.pantone.com/pages/iphone/iphone_colorstrology.html#___1__") 
        print(e)
        month &lt;- remDr$findElement(using = 'xpath', e)
        month$clickElement()
        day &lt;- remDr$findElement(using = 'xpath', "//select[@id='lstDay']//option[5]") ## arbitrarily picking the 5th of each month
        day$clickElement()
        submit &lt;- remDr$findElement(using = 'xpath', "/html[1]/body[1]/form[1]/div[1]/a[1]")
        submit$clickElement()
        html &lt;- read_html(remDr$getPageSource()[[1]])
        description &lt;- html %&gt;%  html_nodes(xpath = "//tr//tr[2]//td[1]") %&gt;% html_text() %&gt;% gsub("^\\s+|\\s+$", "", .)
        meta &lt;- html %&gt;% html_nodes(xpath = "//td[@id='tdBg']") %&gt;%  html_text() %&gt;% gsub("^\\s+|\\s+$", "", .) 
        date &lt;- html %&gt;% html_nodes(xpath = "//td[@id='bgHeaderDate']//div") %&gt;%  html_text() %&gt;% gsub("^\\s+|\\s+$", "", .)
        df &lt;- data.frame(cbind(description,meta,date))
        total &lt;- rbind(total, df)
  }
  
  </code></pre>
  
  <p>Not getting any errors but the results are unexpected each time. Either it would repeat on a single month/day combination like Jan05 * 12 times or jan05 * 3 times, Apr 05 *3 times, etc.</p>