title: Filter out a paragraph before
body: <p>I'm trying to scrap linkedin Data Analysts job descriptions, so far I've been able to extract the Title and the Descriptions, the problem is i dont need all the description, i need only the skills / requirements / qualifications section; here is a sample job desc</p>
  <pre class="lang-html prettyprint-override"><code>We are looking for a passionate Data Analyst whos main responsibilities include conducting full lifecycle analysis to include
  
  requirements, activities, and design. Data analysts will develop analysis and reporting capabilities. They will also monitor
  
  performance and quality control plans to identify improvements.
  Act as a single point of contact for SBU/FU for their data needs
  Process data requests from SBU/FU that do not require building data products
  Perform and deliver analytical work on ad-hoc requests from SBU/CFU
  Identify data needs that require building a product and channel them to Data Governance
  Define detailed use cases for analytic and monetization initiatives
  Execute implementation of analytic and monetization initiatives
  Perform training delivery and handover of analytic and monetization initiatives
  Analyze and approve improvement requests for analytic and monetization
  
  Requirements
  BSc or MSc in STEM' or bus. admin
  2+ years of experience in data analysis
  Strong analytical skills
  Knowledge of advanced statistical and analytical tools and methodologies
  Knowledge of qualitative research methodologies
  Prior experience in travel industry and digital marketing is preferred
  Fluent English, fluent Arabic (preferred)
  &gt;&gt;&gt;
  </code></pre>
  <p>using selenium and bs4, how do i take only the last part which is <strong>Requirements</strong>, and what if it was in the middle how do i handle it?</p>
  <p>this my code, i tried putting the entire description in a list and filter it using pandas.replace but it proved to be cumbersome</p>
  <pre><code>lst_of_desc = []
  for i in range(1,26):
      driver.find_element(By.XPATH, f'//html/body/div[6]/div[3]/div[3]/div[2]/div/section[1]/div/div/ul/li[{i}]/div/div[1]').click()
      sleep(2)
      desc_to_bs4 = driver.find_element(By.CSS_SELECTOR,&quot;div[class*='html-content']&quot;).text
      desc = BeautifulSoup(desc_to_bs4)
      lst_of_desc.append(desc)
  
  print(lst_of_desc)
  
  pd.set_option('display.max_colwidth', None)
  df = pd.DataFrame(lst_of_desc)
  
  df = df.replace([List of Words to Filter By], regex=True)
  </code></pre>