title: beautiful soup misses text when scraping html
body: <p>Here is a sample of my HTML. I need to parse all content for each record. This is one sample record.</p>
  <pre><code>&lt;div class=&quot;list-group-item card-contact&quot;&gt;
      &lt;div class=&quot;card-base&quot;&gt;
          &lt;div class=&quot;card-name hot&quot;&gt;&lt;a href=&quot;contact.php?leaduuid=e888888e&quot;&gt;ronald &lt;b&gt;loudia&lt;/b&gt;&lt;/a&gt;&lt;/div&gt;
          &lt;div class=&quot;card-last-seen&quot;&gt;&lt;i&gt;Inquired:&lt;/i&gt; 8 day ago, Value $205,000 Engaged &lt;/div&gt;
      &lt;/div&gt;
      &lt;a class=&quot;card-expand&quot; href=&quot;#&quot;&gt;&lt;/a&gt;
      &lt;div class=&quot;card-mobile&quot;&gt;
          &lt;ul class=&quot;card-activities&quot; data-score=&quot;27&quot;&gt;
              &lt;li class=&quot;card-update&quot;&gt;&lt;a class=&quot;btn&quot; href=&quot;wrap.php?leaduuid=e888888e&amp;amp;aid=0&quot;&gt;Update&lt;/a&gt;&lt;/li&gt;
              &lt;li class=&quot;card-notes&quot;&gt;&lt;a class=&quot;btn&quot; href=&quot;contact-notes.php?leaduuid=e888888e&quot;&gt;Notes&lt;/a&gt;&lt;/li&gt;
              &lt;li class=&quot;card-email&quot;&gt;
                  &lt;a class=&quot;btn&quot; href=&quot;mailto:lou@ad.fr&quot; title=&quot;leald@icloud.fr&quot;&gt;
                      &lt;div class=&quot;activity-label&quot;&gt;Email:&lt;/div&gt;
                      &lt;div class=&quot;activity-value&quot;&gt;lou@ad.fr&lt;/div&gt;
                  &lt;/a&gt;
              &lt;/li&gt;
              &lt;li class=&quot;card-sms&quot;&gt;&lt;a class=&quot;btn&quot; href=&quot;sms:(222) 125-4444&quot;&gt;SMS&lt;/a&gt;&lt;/li&gt;
              &lt;li class=&quot;card-phone&quot;&gt;
                  &lt;a class=&quot;btn&quot; href=&quot;tel:(222) 125-4444&quot;&gt;
                      &lt;div class=&quot;activity-label&quot;&gt;Phone:&lt;/div&gt;
                      &lt;div class=&quot;activity-value&quot;&gt;(222) 125-4444&lt;/div&gt;
                  &lt;/a&gt;
              &lt;/li&gt;
          &lt;/ul&gt;
      &lt;/div&gt;
  &lt;/div&gt;
  </code></pre>
  <p>I can't seem to get all the content between the tags.
  Here is my code</p>
  <pre><code>from bs4 import BeautifulSoup
  import re
  import mechanize
  import pandas as pd
  
  br = mechanize.Browser()
  
  br.open(url)
  mylist = []
  html_doc = br.response().read()
  soup = BeautifulSoup(html_doc, 'html.parser')
  mydivs = soup.find_all('div', ['card-name',
                                 'card-last-seen',
                                 'activity-value',
                                 'card-activities',
                                 'card-last-seen',
                                 'activity-value',
                                 'card-mobile'])
     
  st = ''
  for div in mydivs:
      if re.search('^\([0-9][0-9][0-9]\)', div.text):
          st += f'{div.text}\n'
      else:
          st += f'{div.text}, '
  mylist.append(st)
  #print(mylist)
  
  smallerlist = [l.split(', ') for l in ', '.join(mylist).split('\n')]
  smallerlist
  df = pd.DataFrame(smallerlist)
  </code></pre>
  <p>I only get some of the content, missing data-score and other content.</p>
  <p>Not sure how to get both div and ul content.
  Html has many records that I need to loop through and write to pandas DataFrame.</p>
  <p><strong>The expected output in dataframe:</strong></p>
  <pre><code>FirstName   LastName    LastLogin   Value   Score   Email   sms phone-activity-Value
  ronald  Lecloudia   8 day ago   $205,000 Engaged    27  lou@ad.fr   (222) 125-4444      (222) 125-4444
  </code></pre>