title: Extract main text from any webpage using BeautifulSoup and Python
body: <p>I'm writing some code (Python) to scrape text from web pages. My goal is to find a way to filter/delete the paragraphs on webpages that are not in the main article (e.g. advertisement, links to other articles, etc.).</p>
  <p>So far I've been using the <code>.find_all(&quot;p&quot;)</code> command to extract only paragraphs from the text, which although successful also scrapes a lot of rudimentary paragraphs which are not in the main/body of each article. This is my code now:</p>
  <pre><code>from urllib.request import Request, urlopen
  from bs4 import BeautifulSoup
  
  URLs = [
  &quot;https://www.elsoldetoluca.com.mx/local/proponen-sistemas-para-captar-agua-pluvial-en-el-edomex-6585661.html&quot;,
  &quot;https://www.elsoldetoluca.com.mx/local/agua-de-acuifero-del-valle-de-toluca-solo-debe-ser-para-uso-de-consumo-humano-especialista-4146232.html&quot;
          ]
  
  for url in URLs:
      req = Request(url, headers={&quot;User-Agent&quot;: 'Mozilla/5.0'})
      page = urlopen(req)
      paragraphs = []
      htmlParse = BeautifulSoup(page.read(), 'lxml')    
      for para in htmlParse.find_all(&quot;p&quot;):
          paragraph = para.get_text().replace(&quot;\n&quot;, &quot; &quot;)
          paragraphs = paragraphs + [paragraph]
           
      text  = str(&quot;\n\n&quot;.join(paragraphs)) 
  </code></pre>
  <p>So I'm looking for a clever way to filter out the paragraphs that are not in the main article. It is essential that this method can be applied on <em>any</em> webpage, since I'm using this code on ~100 random websites. Things I've been looking into already is to filter paragraphs containing certain words, however, I would rather not do this because this way a lot of information/paragraphs are left out. Also, I've been looking at leaving out HTML sections with certain names (such as <a href="https://matix.io/extract-text-from-webpage-using-beautifulsoup-and-python/" rel="nofollow noreferrer">https://matix.io/extract-text-from-webpage-using-beautifulsoup-and-python/</a>), but I find this is not very effective...</p>
  <p>Anyone any tips on how to do this elegantly? Thanks!</p>