title: A general way to scrape link titles from any site in Python?
body: <p>Is there a "general" way to scrape link titles from any website in Python? For example, if I use the following code:</p>
  
  <pre><code>from urllib.request import url open
  from bs4 import BeautifulSoup
  
  site = "https://news.google.com"
  html = urlopen(site)
  soup = BeautifulSoup(html.read(), 'lxml');
  
  titles = soup.findAll('span', attrs = { 'class' : 'titletext' }) 
  for title in titles:
      print(title.contents)
  </code></pre>
  
  <p>I am able to extract nearly every headline title from news.google.com. However, if I use the same code at www.yahoo.com, I am unable to due to a different HTML formatting. </p>
  
  <p>Is there a more general way to do this so that it works for most sites?</p>