title: BeautifulSoup - access content on page that is loaded dynamically
body: <p>I am writing a program that goes through this site: <a href="http://www.whatuni.com" rel="nofollow">http://www.whatuni.com</a>, <br>specifically: <a href="http://www.whatuni.com/degree-courses/search?subject=software-engineering&amp;pageno=" rel="nofollow">http://www.whatuni.com/degree-courses/search?subject=software-engineering&amp;pageno=</a><br>
  Pages 1 to 6.<br>
  It searches through every page and looks for every university that is Russell Group and does software engineering then looks through all modules. It is meant to reject any module with a hardware component and add the rest to a list. So far it is searching through the modules pages but there is a problem: The topics in the module in the website are not displayed by default, it is loaded by JavaScript after a button is pressed. This is a page that shows the situation:
  <a href="http://www.whatuni.com/degrees/artificial-intelligence-and-software-engineering-beng-hons/university-of-edinburgh/cd/54977454/5508/" rel="nofollow">http://www.whatuni.com/degrees/artificial-intelligence-and-software-engineering-beng-hons/university-of-edinburgh/cd/54977454/5508/</a><br>
  I want to know how I can access these modules so I can loop through them, it is the last complicated step in this program. Would I have to simulate user input or would that not work because it does not parse through the browser? Or do I have to look in a script? I am using python 3.4, and beautiful soup for this program.</p>
  
  <pre><code>from general import *
  
  class ModuleCrawler:
  
  base_url = ""
  page_url = ""
  bad_words = ("computer systems" or "computer architecture" or "hardware" or "electronics"
               or "information technology")
  
  def __init__(self, base_url, page_url):
      self.base_url = base_url
      self.page_url = page_url
      self.crawl_topics(self.get_description(get_html(page_url)))
  
  def crawl_topics(self, description):
      if self.bad_words in description.lower():
          print("crap uni")
      else:
          #print(description.lower())
  
      append_to_file("PossibleUnis.txt", self.page_url)
  
  def get_description(self, html_string):
      return ', '.join(str(s) for s in (get_by_id(html_string, "#fullDesc")))
  </code></pre>
  
  <p>That is the module crawler, the beautiful soup code is in another file. I made it get the description from the page instead but that sometimes leaves out those words and my text file has many modules that have those topics.</p>