title: Scrape the HTML from a webpage after rendering the JSON/Javascript without Selenium while supporting POSTing
body: <p>I'm trying to use the following code, which works fine but it doesn't let the JavaScript run first, which means I don't get the desired HTML from the webpage.</p>
  
  <p>I've had a look at <code>DryScrape</code> but as far as I can see it doesn't support posting like you can see in the <code>auto_login()</code> function, the same goes for <code>PyQt4</code>.</p>
  
  <p>The website in question has 4 bits of JSON "lists"? that form the page on load/render; if I view the source it doesn't show nicely and I can't easily find stuff in it, however if I "Inspect Element" on the page the HTML looks perfect and then I can easily look through it with <code>BeautifulSoup</code>.</p>
  
  <p>I know I could use <code>Selenium</code>, but this isn't what I want to do mainly because of wanting it to run in the background, I may be able to use <code>PhantomJS</code> or <code>PyVirtualDisplay</code> to do this but this would only be as a last resort.</p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  HEADERS = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'en-US,en;q=0.8', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36'}
  SESSION = requests.session()
  RESPONSE = requests.Response
  
  email = "mail@mail.com"
  pass = "password123"
  
  def auto_login():
      global RESPONSE
      url = "https://website.com/log.php"
      payload = {
          "log":email,
          "pwd":pass,
      "finish":"https://website.com/listed/public/gen1/",
      }
      RESPONSE = SESSION.post(url, data=payload, headers=HEADERS, verify=False)
  
  def process_html():
      PROCESSED_HTML = BeautifulSoup(RESPONSE.content, 'html.parser')
      return PROCESSED_HTML
  
  def main():
      auto_login()
      PROCESSED_HTML = get_html()
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>How can I easily render a webpage's JavaScript using my script (modified), preferably with <code>PyQt4</code> (<code>DryScrape</code> won't install correctly on my Windows 10, Python 2.7 for some reason) without using <code>Selenium</code>. </p>
  
  <p>Any ideas would be appreciated.</p>