title: Scraping data from a site where URL doesn't change on clicking 'Show More'
body: <p>I'm trying to scrape all the article links from a site and have I been successful in doing so. </p>
  
  <p>The site page has a <code>Show more</code> button for loading more articles. </p>
  
  <p>I'm using Selenium to click on this button which also works. </p>
  
  <p>The problem is that clicking on <code>Show more</code> doesn't change the URL of the page, therefore I'm being able to scrape only the initial links displayed by default.</p>
  
  <p>Here is the code snip:</p>
  
  <pre><code>def startWebDriver():
      global driver
      options = Options()
      options.add_argument("--disable-extensions")
      driver = webdriver.Chrome(executable_path = '/home/Downloads/chromedriver_linux64/chromedriver',options=options)
  
  startWebDriver()
  count = 0 
  s = set()
  
  driver.get('https://www.nytimes.com/search? endDate=20181231&amp;query=trump&amp;sort=best&amp;startDate=20180101')
  time.sleep(4)
  element = driver.find_element_by_xpath('//*[@id="site-content"]/div/div/div[2]/div[2]/div/button')
  
  while(count &lt; 10):
      element.click()
      time.sleep(4)
      count=count+1
  
  url = driver.current_url
  </code></pre>
  
  <p>I expect to get all article links displayed on the page after clicking on <code>Show More</code> 10 times</p>