title: Find or select elements from python to scrape with beautifulsoup
body: <p>I am not sure how to select below items inside the table class=&quot;table-info&quot;</p>
  <p>Using python and beautifulsoup, I want to extract the:</p>
  <ol>
  <li><p><strong>phone</strong></p>
  </li>
  <li><p><strong>email</strong></p>
  </li>
  <li><p><strong>website</strong></p>
  </li>
  <li><p><strong>main activity</strong> (li element text without the div)
  &quot;Computer consultancy activities&quot;.</p>
  <pre><code> &lt;table class=&quot;table-info&quot;&gt;
   &lt;tbody&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;Business name&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;Company XYZ&lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;Register code:&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;112233558&lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
  
  
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;Operating address:&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;&lt;a target=&quot;googlemaps&quot; href=&quot;https://www.google.com/maps/place/Some-location&quot;
                       class=&quot;link-location&quot;&gt;Some location strt. 233&lt;/a&gt;&lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;Legal address&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;
                   &lt;a class=&quot;link-location&quot; href=&quot;https://www.google.com/maps/place/Some-location&quot; target=&quot;_new&quot;&gt;Some
                       location
                   &lt;/a&gt;
               &lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;VAT No:&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;&lt;a href=&quot;javascript:void(0)&quot; onclick=&quot;return getVAT(this, '12345678')&quot;&gt;Get VAT
                       liability&lt;/a&gt;&lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;Age:&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;1 year&amp;nbsp;3 months&lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;Founded:&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;20/09/2019&lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;Capital:&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;2000 USD&lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td colspan=&quot;2&quot; class=&quot;sep&quot;&gt;&lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;Phone:&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;123456789&lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;E-mail:&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;&lt;a href=&quot;mailto:some@one.com&quot;&gt;some@one.com&lt;/a&gt;&lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td colspan=&quot;2&quot; class=&quot;sep&quot;&gt;&lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;Representatives:&lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot;&gt;
                   &lt;div class=&quot;box-message&quot;&gt;
                       &lt;p class=&quot;desc&quot;&gt;To access information, please&lt;/p&gt;
                       &lt;p&gt;
                           &lt;a href=&quot;#&quot; onclick=&quot;return loginClicked(this, '#');&quot;
                               class=&quot;btn btn-small btn-purple link-login&quot;&gt;Log in&lt;/a&gt;
                       &lt;/p&gt;
                   &lt;/div&gt;
               &lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td colspan=&quot;2&quot; class=&quot;sep&quot;&gt;&lt;/td&gt;
       &lt;/tr&gt;
       &lt;tr&gt;
           &lt;td class=&quot;col-1&quot;&gt;
               &lt;div class=&quot;col-1-text&quot;&gt;
                   Main activity:
                   &lt;span class=&quot;tip info&quot; title=&quot;&quot;
                       data-original-title=&quot;Activities are classified according to EMTAK 2008&quot;&gt;&lt;/span&gt;
               &lt;/div&gt;
           &lt;/td&gt;
           &lt;td class=&quot;col-2&quot;&gt;
               &lt;div class=&quot;col-2-text&quot; id=&quot;activity_top5ffe2eab23d13&quot;&gt;
                   &lt;ul&gt;
                       &lt;li&gt;
                           Computer consultancy activities
                           &lt;div class=&quot;main_activities_top_link_wrapper&quot;&gt;
                               &lt;a href=&quot;https://www.somesite.com/&quot; target=&quot;_blank&quot;
                                   onclick=&quot;ga('send', 'event', 'check', 'top_btn', 'Anonym');&quot;
                                   class=&quot;btn btn-simple btn-open-graph&quot;&gt;
                                   &lt;span&gt;Open TOP 20&lt;/span&gt; &lt;/a&gt;
                           &lt;/div&gt;
                       &lt;/li&gt;
                   &lt;/ul&gt;
  
               &lt;/div&gt;
           &lt;/td&gt;
       &lt;/tr&gt;
  
  
   &lt;/tbody&gt;
  </code></pre>
   
  </li>
  </ol>
  <p>Note: Above code is one query result / html example, but sometimes query result / company does not have email or website / vice versa. So, its important that code does not run into error if it does not find the html content what its looking for. I find its better to follow the class names or ids rather than counting how deep the table/div nesting goes (xpath).</p>
  <p>I have code which is not working great atm:</p>
  <pre><code>import csv
  import requests
  import datetime
  import time
   
  from requests import get
  from bs4 import BeautifulSoup
   
   
  with open('data.csv', encoding='utf8') as csvfile:
      reader = csv.reader(csvfile, delimiter=';')
      next(reader)
   
      count = 0
       
      for row in reader:
           
          timestamp = datetime.datetime.now().strftime(&quot;%Y-%m-%d %H:%M:%S&quot;)
   
          url = f'https://www.somedomain.com/result?country=en&amp;q={row[1]}'
           
          headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
          cookies = {'__test': '1bb6e881021f013463740eeb74840b18'}
          content = get(url, headers=headers,  cookies=cookies).content
          soup = BeautifulSoup(content, &quot;lxml&quot;)
   
          table_info = soup.select_one('.table-info')
   
          mail = table_info.select_one('.col-2 a[href^=mailto]')
          mail = mail.get('href')
          mail_clean = mail.split(':')[1]
   
          website = soup.find(text='Website:')
          website = table_info.select_one('.col-2 a[target^=_blank]')
          website = website.get('href') 
           
          collected_data = row[1], mail_clean, website, timestamp
   
          data_list = [[&quot;Regcode&quot;, &quot;Email&quot;, &quot;Website&quot;, &quot;Timestamp&quot;],collected_data]
          with open('extracted.csv', 'w', newline='') as file:
              writer = csv.writer(file, delimiter=';')
              writer.writerows(data_list)
   
          print(row[1], &quot;|&quot;, mail_clean,&quot;|&quot;, website,&quot;|&quot;, timestamp)
          #print(&quot;Waiting 3 seconds...&quot;)
          #time.sleep(3)
          count+=1
       
    
  </code></pre>