title: Web Scraper (Python 3.6) crashes when encountering javascript in string
body: <p>I coded a (probably inefficient) web scraper with BeautifulSoup for this <a href="http://www.poliscirumors.com" rel="nofollow noreferrer">website</a>. While it works, the function to get the posts crashes when encountering a post that contains javascript as the loop that goes through the post content (<code>for item in i.find_all("p")[1:]:</code>) stops and a later request for meta data (<code>i.select('span')[0].get_text()</code>) of the post is unable to find the particular element. An example is the last post <a href="http://www.poliscirumors.com/topic/tenure-denial-blog/page/23" rel="nofollow noreferrer">here</a>. While I could code an exception, I would prefer to understand the problem and solve it directly. What am I doing wrong?</p>
  
  <pre><code>from urllib.request import urlopen
  import requests as rs
  from bs4 import BeautifulSoup as BS
  import re
  from itertools import chain
  posts = []
  def post_data(postlist, weblink, rmin, rmax):
      page = rs.get(weblink)
      soup = BS(page.content, 'lxml')
      for d in range(rmin, rmax):
          for i in soup.find_all("div", id="position-"+str(d)):
              text = []
              for item in i.find_all("p")[1:]:
                  text.append(item.get_text().replace("\n" , "/" ).replace("," , "$" ))
              text = "".join(text)
              text.replace("\n", "/").replace("," , "$" )
              postlist.append((weblink, str(d), i.find("strong").get_text() , text , i.select('span')[0].get_text(), i.select('span')[1].get_text(), i.span["id"][1:], list(i.find("div", class_="poststuff"))[0]))
      postlist=list((chain.from_iterable(postlist)))
  post_data(posts, "http://www.poliscirumors.com/topic/tenure-denial-blog/page/23", 460, 461)
  </code></pre>
  
  <p>The error is the following:</p>
  
  <pre><code>File "p3.py", line 20, in post_data
  postlist.append((weblink, str(d), i.find("strong").get_text() , text , i.select('span')[0].get_text(), i.select('span')[1].get_text(), i.span["id"][1:], list(i.find("div", class_="poststuff"))[0]))
  IndexError: list index out of range
  </code></pre>