title: Scraping a website with python 3 that requires login
body: <p>Just a question regarding some scraping authentication.  Using <code>BeautifulSoup</code>:</p>
  
  <pre><code>#importing the requests lib  
  import requests
  from bs4 import BeautifulSoup
  
  #specifying the page
  page = requests.get("http://localhost:8080/login?from=%2F")
  #parsing through the api
  soup = BeautifulSoup(page.content, 'html.parser')
  print(soup.prettify())
  </code></pre>
  
  <p>From here the output, I think would be important:</p>
  
  <pre><code> &lt;table&gt;
     &lt;tr&gt;
      &lt;td&gt;
       User:
      &lt;/td&gt;
      &lt;td&gt;
       &lt;input autocapitalize="off" autocorrect="off" id="j_username" name="j_username" type="text"/&gt;
      &lt;/td&gt;
     &lt;/tr&gt;
     &lt;tr&gt;
      &lt;td&gt;
       Password:
      &lt;/td&gt;
      &lt;td&gt;
       &lt;input name="j_password" type="password"/&gt;
      &lt;/td&gt;
     &lt;/tr&gt;
     &lt;tr&gt;
      &lt;td align="right"&gt;
       &lt;input id="remember_me" name="remember_me" type="checkbox"/&gt;
      &lt;/td&gt;
      &lt;td&gt;
       &lt;label for="remember_me"&gt;
        Remember me on this computer
       &lt;/label&gt;
      &lt;/td&gt;
     &lt;/tr&gt;
    &lt;/table&gt;
  </code></pre>
  
  <p>This scrapes the website fine, but it requires a login. Here I am using the <code>mechanicalsoup</code> library:</p>
  
  <pre><code>import mechanicalsoup
  
  browser = mechanicalsoup.StatefulBrowser()
  browser.open("http://localhost:8080/login?from=%2F")
  browser.get_url()
  browser.get_current_page()
  browser.get_current_page().find_all('form')
  browser["j_username"] = "admin"
  browser ["j_password"] = "password"
  browser.launch_browser()
  </code></pre>
  
  <p>However it still won't let me login. </p>
  
  <p>Has anyone used a scraping tool for python 3 that lets them scrape a site that has authentication?</p>