title: Dynamically extract text from webpage using Python BeautifulSoup
body: <p>I'm trying to extract player position from many players' webpages (here's an example for <a href="https://www.sports-reference.com/cbb/players/malcolm-brogdon-1.html" rel="nofollow noreferrer">Malcolm Brogdon</a>). I'm able to extract Malcolm Brogdon's position using the following code:</p>
  <pre><code>player_id = 'malcolm-brogdon-1'
  
  # Import libraries
  from urllib.request import Request, urlopen
  from bs4 import BeautifulSoup as soup
  import pandas as pd
  import numpy as np
  
  url = &quot;https://www.sports-reference.com/cbb/players/{}.html&quot;.format(player_id)
  req = Request(url , headers={'User-Agent': 'Mozilla/5.0'})
  webpage = urlopen(req).read()
  page_soup = soup(webpage, &quot;html.parser&quot;)
  
  pos = page_soup.p.find(&quot;strong&quot;).next_sibling.strip()
  pos
  </code></pre>
  <p>However, I want to be able to do this in a more dynamic way (that is, to locate &quot;Position:&quot; and then find what comes after). There are other players for which the webpage is structured slightly differently, and my current code wouldn't return position (i.e. <a href="https://www.sports-reference.com/cbb/players/anthony-cat-barber-1.html" rel="nofollow noreferrer">Cat Barber</a>).</p>
  <p>I've tried doing something like <code>page_soup.find(&quot;strong&quot;, text=&quot;Position:&quot;)</code> but that doesn't seem to work.</p>
  <p><a href="https://i.stack.imgur.com/8XHiN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8XHiN.png" alt="Malcolm Brogdon's Sports-Reference webpage" /></a></p>