title: how to scrape deeply embeded links with python beautifulSoup
body: <p>I'm trying to build a spider/web crawler for academic purposes to grab text from academic publications and append related links to a URL stack. I'm trying to crawl 1 website called 'PubMed'. I can't seem to grab the links I need though. Here is my code with an example page, this page should be representative of others in their database:</p>
  
  <pre><code> website = 'http://www.ncbi.nlm.nih.gov/pubmed/?term=mtap+prmt'
   from bs4 import BeautifulSoup
   import requests
   r = requests.get(website)
   soup = BeautifulSoup(r.content)
  </code></pre>
  
  <p>I have broken the html tree down into several variables just for readability so that it can all fit on 1 screen width.</p>
  
  <pre><code> key_text = soup.find('div', {'class':'grid'}).find('div',{'class':'col twelve_col nomargin shadow'}).find('form',{'id':'EntrezForm'})
   side_column = key_text.find('div', {'xmlns:xi':'http://www.w3.org/2001/XInclude'}).find('div', {'class':'supplemental col three_col last'})
   side_links = side_column.find('div').findAll('div')[1].find('div', {'id':'disc_col'}).findAll('div')[1]
  
   for link in side_links:
        print link
  </code></pre>
  
  <p>if you look at the html source code using chrome inspect element there should be several other nested divs with links within 'side_links'. However the above code produces the following error:</p>
  
  <pre><code> Traceback (most recent call last):
   File "C:/Users/ballbag/Copy/web_scraping/google_search.py", line 22, in &lt;module&gt;
   side_links = side_column.find('div').findAll('div')[1].find('div',      {'id':'disc_col'}).findAll('div')[1]
   IndexError: list index out of range
  </code></pre>
  
  <p>if you go to the url there is a column on the right called 'related links' containing the urls that I wish to scrape. But I can't seem to get to them. There is a statement saying  under the div i am trying to get into and I suspect this has something to do with it. Can anyone help grab these links? I'd really appreciate any pointers</p>