title: Sign in and scrape search results on LinkedIn using Python
body: <p>Sorry for this stupid question but I am so new to programming and have a deadline to meet for my thesis. I greatly appreciate your patience and help!</p>
  
  <p>My goal is to get data from LinkedIn. One possible way I just knew (from youtube videos) is to search on google then scrape links for further scraping. </p>
  
  <pre><code>import mechanize
  import re
  from bs4 import BeautifulSoup
  
  br = mechanize.Browser()
  br.addheaders = [('User-agent','Mozilla/5.0')]
  br.set_handle_robots(False)
  html = br.open('http://google.com/search?q=site:linkedin.com%2Fjob%2F+big+data')
  html = html.read().lower()
  
  #let handle the unicode character
  html = unicode(html,errors='ignore')
  
  #raw result
  #print html
  
  soup = BeautifulSoup(html)
  #print soup
  </code></pre>
  
  <p>In the soup I see this portion that contains the link that I wanna scrape. </p>
  
  <pre><code>&lt;div data-async-context="query:site%3Alinkedin.com%2Fjob%2F%20big%20data" id="ires"&gt;
  ...    
  &lt;a href="/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=5&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0CGIQFjAE&amp;amp;url=https%3A%2F%2Fwww.linkedin.com%2Fjob%2Fgoogle%2Fbig-data-jobs%2F&amp;amp;ei=U9TsU8mECZOkyQSLxoCYAg&amp;amp;usg=AFQjCNGlr9VqOU8WSGS_8eJpVqWpKM3rIQ&amp;amp;sig2=lFrE7FgtzqS4dfSD-k_cKA&amp;amp;bvm=bv.72938740,d.aWw" onmousedown="return rwt(this,'','','','5','AFQjCNGlr9VqOU8WSGS_8eJpVqWpKM3rIQ','lFrE7FgtzqS4dfSD-k_cKA','0CGIQFjAE','','',event)" wrc_done="true" data-href="https://www.linkedin.com/job/google/big-data-jobs/"&gt;&lt;em&gt;Big Data&lt;/em&gt; Jobs at Google | LinkedIn&lt;/a&gt;
  ...
  &lt;/div&gt;
  </code></pre>
  
  <p>The link that I want is "<a href="https://www.linkedin.com/job/google/big-data-jobs/" rel="nofollow">https://www.linkedin.com/job/google/big-data-jobs/</a>" </p>
  
  <p>However, I don't know why my following code does not work. Can someone advice?</p>
  
  <pre><code>for google_body in soup.findAll('div',attrs={'data-async-context':"query:site%3Alinkedin.com%2Fjob%2F%20big%20data"}):
      for google_links in google_body.findAll('a'):
          print google_links.get('data-href')
  </code></pre>