title: Selected LinkedIn profiles not loading completely in Selenium using Python
body: <p>I have written a code to fetch LinkedIn profile details but sometimes the entire HTML is not loaded for some user profiles.</p>
  
  <p>I have already used the classic wait mechanisms i.e.</p>
  
  <pre><code>driver.implicitly_wait(10)
  
  time.sleep(10)
  
  element_present = EC.presence_of_element_located((By.CLASS_NAME, '.pv-profile-section__card-item-v2.pv-profile-section.pv-position-entity.ember-view'))
  WebDriverWait(driver, 300).until(element_present)
  </code></pre>
  
  <p>but none of them seem to work.</p>
  
  <p>A snippet of my code:</p>
  
  <pre><code>firstName = urllib.parse.quote(userFirstName)
  lastName = urllib.parse.quote(userLastName)
  company = urllib.parse.quote(userCompany)
  
  driver.get('https://www.linkedin.com/search/results/people/?company='+company+'&amp;firstName='+firstName+'&amp;lastName='+lastName+'&amp;origin=FACETED_SEARCH')
  
  results = len(driver.find_elements_by_css_selector('.name.actor-name'))
  for i in range(1):
      print(i)
      driver.find_elements_by_css_selector('.name.actor-name')[i].click()
      time.sleep(10)
      print(driver.current_url)
  
      content = driver.execute_script("return document.getElementsByTagName('html')[0].innerHTML")
      driver.implicitly_wait(2)
      soup = BeautifulSoup(content, "html.parser")
      #print(soup)
  
      companyList = soup.findAll('section',{'class':'pv-profile-section__card-item-v2 pv-profile-section pv-position-entity ember-view'})
      print("Company list length: "+str(len(companyList)))
  </code></pre>
  
  <p>The code does give company list for many users but it simply fails in some cases. I checked those profiles on my browser and the elements in the code did exist.</p>
  
  <p>Any help/ past experience on this would be appreciated. I know trying to solve this problem will also take efforts so thanks in advance!</p>
  
  <p>P.S.: A part of HTML (the experience part I care about):</p>
  
  <pre><code>&lt;ul class="pv-profile-section__section-info section-info pv-profile-section__section-info--has-no-more"&gt;
  &lt;li class="pv-entity__position-group-pager pv-profile-section__list-item ember-view" id="ember394"&gt; &lt;section class="pv-profile-section__card-item-v2 pv-profile-section pv-position-entity ember-view" id="ember396"&gt; &lt;div class="display-flex justify-space-between full-width"&gt;
  &lt;a class="full-width ember-view" data-control-name="background_details_company" href="/search/results/index/?keywords=Aditya%20Birla%20Direct" id="ember397"&gt; &lt;div class="pv-entity__company-details"&gt;
  &lt;div class="pv-entity__logo company-logo"&gt;
  &lt;img alt="Aditya Birla Direct" class="pv-entity__logo-img pv-entity__logo-img EntityPhoto-square-5 lazy-image ember-view" id="ember399"/&gt;
  &lt;/div&gt;
  &lt;div class="pv-entity__company-summary-info"&gt;
  &lt;h3 class="t-16 t-black t-bold"&gt;
  &lt;span class="visually-hidden"&gt;Company Name&lt;/span&gt;
  &lt;span&gt;Aditya Birla Direct&lt;/span&gt;
  &lt;/h3&gt;
  &lt;h4 class="t-14 t-black t-normal"&gt;
  &lt;span class="visually-hidden"&gt;Total Duration&lt;/span&gt;
  &lt;span&gt;2 yrs 6 mos&lt;/span&gt;
  &lt;/h4&gt;
  &lt;/div&gt;
  &lt;/div&gt;
  &lt;/a&gt;
  &lt;!-- --&gt; &lt;/div&gt;
  &lt;ul class="pv-entity__position-group mt2 ember-view" id="ember400"&gt;&lt;li class="pv-entity__position-group-role-item sortable-item ember-view" id="ember402"&gt; &lt;div class="ember-view" id="ember403"&gt;&lt;div class="pv-entity__role-details"&gt;
  &lt;span class="pv-entity__timeline-node"&gt;&lt;/span&gt;
  &lt;div class="display-flex justify-space-between full-width"&gt;
  &lt;div class="pv-entity__role-container"&gt;
  &lt;div class="pv-entity__role-details-container pv-entity__role-details-container--timeline pv-entity__role-details-container--bottom-margin"&gt;
  &lt;div class="pv-entity__summary-info-v2 pv-entity__summary-info--background-section pv-entity__summary-info-margin-top"&gt;
  &lt;h3 class="t-14 t-black t-bold"&gt;
  &lt;span class="visually-hidden"&gt;Title&lt;/span&gt;
  &lt;span&gt;Product Designer&lt;/span&gt;
  &lt;/h3&gt;
  &lt;!-- --&gt; &lt;div class="display-flex"&gt;
  &lt;h4 class="pv-entity__date-range t-14 t-black--light t-normal"&gt;
  &lt;span class="visually-hidden"&gt;Dates Employed&lt;/span&gt;
  &lt;span&gt;Jun 2018 – Present&lt;/span&gt;
  &lt;/h4&gt;
  &lt;h4 class="t-14 t-black--light t-normal"&gt;
  &lt;span class="visually-hidden"&gt;Employment Duration&lt;/span&gt;
  &lt;span class="pv-entity__bullet-item-v2"&gt;1 yr 5 mos&lt;/span&gt;
  &lt;/h4&gt;
  &lt;/div&gt;
  &lt;h4 class="pv-entity__location t-14 t-black--light t-normal block"&gt;
  &lt;span class="visually-hidden"&gt;Location&lt;/span&gt;
  &lt;span&gt;Mumbai, Maharashtra, India&lt;/span&gt;
  &lt;/h4&gt;
  &lt;/div&gt;
  &lt;!-- --&gt; &lt;/div&gt;
  &lt;/div&gt;
  &lt;!-- --&gt; &lt;/div&gt;
  &lt;/div&gt;
  &lt;/div&gt;
  &lt;/li&gt;&lt;li class="pv-entity__position-group-role-item sortable-item ember-view" id="ember405"&gt; &lt;div class="ember-view" id="ember406"&gt;&lt;div class="pv-entity__role-details"&gt;
  &lt;span class="pv-entity__timeline-node"&gt;&lt;/span&gt;
  &lt;div class="display-flex justify-space-between full-width"&gt;
  &lt;div class="pv-entity__role-container"&gt;
  &lt;div class="pv-entity__role-details-container"&gt;
  &lt;div class="pv-entity__summary-info-v2 pv-entity__summary-info--background-section pv-entity__summary-info-margin-top"&gt;
  &lt;h3 class="t-14 t-black t-bold"&gt;
  &lt;span class="visually-hidden"&gt;Title&lt;/span&gt;
  &lt;span&gt;UI/UX Designer&lt;/span&gt;
  &lt;/h3&gt;
  &lt;!-- --&gt; &lt;div class="display-flex"&gt;
  &lt;h4 class="pv-entity__date-range t-14 t-black--light t-normal"&gt;
  &lt;span class="visually-hidden"&gt;Dates Employed&lt;/span&gt;
  &lt;span&gt;May 2017 – Present&lt;/span&gt;
  &lt;/h4&gt;
  &lt;h4 class="t-14 t-black--light t-normal"&gt;
  &lt;span class="visually-hidden"&gt;Employment Duration&lt;/span&gt;
  &lt;span class="pv-entity__bullet-item-v2"&gt;2 yrs 6 mos&lt;/span&gt;
  &lt;/h4&gt;
  &lt;/div&gt;
  &lt;h4 class="pv-entity__location t-14 t-black--light t-normal block"&gt;
  &lt;span class="visually-hidden"&gt;Location&lt;/span&gt;
  &lt;span&gt;Mumbai, Maharashtra, India&lt;/span&gt;
  &lt;/h4&gt;
  &lt;/div&gt;
  &lt;!-- --&gt; &lt;/div&gt;
  &lt;/div&gt;
  &lt;!-- --&gt; &lt;/div&gt;
  &lt;/div&gt;
  &lt;/div&gt;
  &lt;/li&gt;
  &lt;/ul&gt;
  &lt;!-- --&gt;&lt;/section&gt;
  &lt;/li&gt;&lt;li class="pv-entity__position-group-pager pv-profile-section__list-item ember-view" id="ember408"&gt; &lt;section class="pv-profile-section__card-item-v2 pv-profile-section pv-position-entity ember-view" id="1192970710"&gt; &lt;div class="display-flex justify-space-between full-width"&gt;
  &lt;div class="display-flex flex-column full-width"&gt;
  &lt;a class="full-width ember-view" data-control-name="background_details_company" href="/search/results/index/?keywords=improove%20technology%20pvt%20ltd" id="ember411"&gt; &lt;div class="pv-entity__logo company-logo"&gt;
  &lt;img alt="improove technology pvt ltd" class="pv-entity__logo-img pv-entity__logo-img EntityPhoto-square-5 lazy-image ghost-company ember-view" id="ember413"/&gt;
  &lt;/div&gt;
  &lt;div class="pv-entity__summary-info pv-entity__summary-info--background-section"&gt;
  &lt;h3 class="t-16 t-black t-bold"&gt;UI/UX Designer&lt;/h3&gt;
  &lt;p class="visually-hidden"&gt;Company Name&lt;/p&gt;
  &lt;p class="pv-entity__secondary-title t-14 t-black t-normal"&gt;improove technology pvt ltd&lt;/p&gt;
  &lt;!-- --&gt;
  &lt;div class="display-flex"&gt;
  &lt;h4 class="pv-entity__date-range t-14 t-black--light t-normal"&gt;
  &lt;span class="visually-hidden"&gt;Dates Employed&lt;/span&gt;
  &lt;span&gt;May 2015 – May 2017&lt;/span&gt;
  &lt;/h4&gt;
  &lt;h4 class="t-14 t-black--light t-normal"&gt;
  &lt;span class="visually-hidden"&gt;Employment Duration&lt;/span&gt;
  &lt;span class="pv-entity__bullet-item-v2"&gt;2 yrs 1 mo&lt;/span&gt;
  &lt;/h4&gt;
  &lt;/div&gt;
  &lt;h4 class="pv-entity__location t-14 t-black--light t-normal block"&gt;
  &lt;span class="visually-hidden"&gt;Location&lt;/span&gt;
  &lt;span&gt;Delhi&lt;/span&gt;
  &lt;/h4&gt;
  &lt;/div&gt;
  &lt;/a&gt;
  &lt;!-- --&gt; &lt;/div&gt;
  &lt;!-- --&gt; &lt;/div&gt;
  &lt;/section&gt;
  &lt;/li&gt;
  </code></pre>
  
  <p>I basically need the company name, role and dates employed.</p>