title: Navigate with Selenium and scrape with BeautifulSoup in Python
body: <p>Ok so here is what I am trying to archieve:</p>
  
  <ol>
  <li>Call a URL with a list of dynamically filtered search results</li>
  <li>Click on the first search result (5/page)</li>
  <li><p>Scrape the headlines, paragraphs and images and store them as a json object in a a seperate file e.g.</p>
  
  <p>{<br>
   "Title": "Headline element of the individual entry",<br>
  "Content" : "Pargraphs and images in DOM order od the individual entry"
  <br>}</p></li>
  <li><p>Navigate back to the search results overview page and repeat steps 2 - 3</p></li>
  <li>After 5/5 results hav ebeen scraped go to te next page (click pagination link)</li>
  <li>Repeat steps 2 - 5 until no entry is left</li>
  </ol>
  
  <p>To visualize once more what is intedned:
  <a href="https://i.stack.imgur.com/QJPSA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QJPSA.png" alt="enter image description here"></a></p>
  
  <p>What I have so far is:</p>
  
  <pre><code>#import libraries
  from selenium import webdriver
  from bs4 import BeautfifulSoup
  
  #URL
  url = "https://URL.com"
  
  #Create a browser session
  driver = webdriver.Chrome("PATH TO chromedriver.exe")
  driver.implicitly_wait(30)
  driver.get(url)
  
  #click consent btn on destination URL ( overlays rest of the content )
  python_consentButton = driver.find_element_by_id('acceptAllCookies')
  python_consentButton.click() #click cookie consent btn
  
  #Seleium hands the page source to Beautiful Soup
  soup_results_overview = BeautifulSoup(driver.page_source, 'lxml')
  
  
  for link in soup_results_overview.findAll("a", class_="searchResults__detail"):
  
    #Selenium visits each Search Result Page
    searchResult = driver.find_element_by_class_name('searchResults__detail')
    searchResult.click() #click Search Result
  
    #Ask Selenium to go back to the search results overview page
    driver.back()
  
  #Tell Selenium to click paginate "next" link 
  #probably needs to be in a sourounding for loop?
  paginate = driver.find_element_by_class_name('pagination-link-next')
  paginate.click() #click paginate next
  
  driver.quit()
  </code></pre>
  
  <p><strong>Problem</strong><br>
  The list count resets every time Selenium navigates back to te search results overview page
  so it clicks the first entry 5 times, navigates to the next 5 items and stops</p>
  
  <p>This is probably a predestinated case for a recursive approach, not sure.</p>
  
  <p>Any advice on how to tackle this issue is appreciated.</p>