title: Scraping a site that requires login username and password on two separate pages
body: <p>I'm trying to scrape information from my companies Intranet so that I can display information on our office wall board via dashing dashboard. I'm trying to work with the provided information from:<a href="http://scraperblog.blogspot.com/2012/12/scraping-site-that-requires-login-in.html" rel="nofollow">This Site</a>.The problem that I'm having other than being a noob is that in order to gain access to the information I want to scrape, I need to login to our Intranet providing my username on one page then submitting to another so that I can provide my password. Once I'm logged in, I can then link and scrape my data.</p>
  
  <p>Here is some source code from my login username page:</p>
  
  <pre><code>&lt;form action='loginauthpwd.asp?PassedURL=' method='post' style='margin: 0px;'&gt;&lt;table border='0' cellspacing='1' width='999' height='350'&gt;&lt;tr&gt;&lt;td width='100'&gt;&amp;nbsp;&lt;/td&gt;&lt;td valign='center' width='100'&gt;&lt;table style='width: 350px; background-color: #EEEEEE; border: 1px solid gray;'&gt;&lt;tr&gt;&lt;td class='fontBlack' style='padding: 10px; vertical-align: top;'&gt;&lt;span style='font-weight: bold;'&gt;Username:&lt;/span&gt;&lt;br&gt;&lt;input type='text' class='normal' autocomplete='off' id='LoginUser' name='LoginUser' style='border: 1px solid gray; height: 16px; font-family: arial; font-size: 11; width: 180px;' maxlength='30'&gt;&lt;input class='normal_button' type='button' value='Go' style='border: 1px solid gray; font-weight: bold; width: 80px; margin-left: 10px;' onclick="var username=document.getElementById('LoginUser').value; if (username.length &gt; 2) { submit(); } else { alert('Enter your Username.'); }"&gt;&lt;/form&gt;
  </code></pre>
  
  <p>Here is some source from my login password page:</p>
  
  <pre><code>&lt;form action='loginauthprocess.asp?UserName=******&amp;Page=&amp;PassedURL=' target='_top' method='post' onsubmit='checkMyBrowser();' style='margin: 0px;'&gt;&lt;table border='0' cellspacing='1' width='999' height='350'&gt;&lt;tr&gt;&lt;td width='100'&gt;&amp;nbsp;&lt;/td&gt;&lt;td valign='center' width='100'&gt;&lt;table style='width: 350px; background-color: #EEEEEE; border: 1px solid gray;'&gt;&lt;tr&gt;&lt;td class='fontBlack' style='padding: 10px; vertical-align: top;'&gt;&lt;span style='font-weight: bold;'&gt;Password:&lt;/span&gt;&lt;br&gt;&lt;input class='normal' type='password' autocomplete='off' id='LoginPassword' name='LoginPassword' style='border: 1px solid gray; height: 16px; font-family: arial; font-size: 11; width: 180px;' maxlength='30'&gt;&lt;input class='normal_button' type='submit' value='Log In' style='border: 1px solid gray; font-weight: bold; width: 80px; margin-left: 10px;' onclick="var password=document.getElementById('LoginPassword').value; if (password.length &gt; 2) { submit(); } else { alert('Enter your Password.'); }"&gt;&lt;/form&gt;
  </code></pre>
  
  <p>Using said resource's example this is what I think should work but doesn't seem to be:</p>
  
  <pre><code>require 'mechanize'
  @agent = Mechanize.new
  @agent.verify_mode = OpenSSL::SSL::VERIFY_NONE
  
  ##Login Page:
  page = @agent.get 'http://www.website_here.com/intranet/login.asp'
  
  ##Username Page:
  form = page.forms[0]
  form['USER NAME HERE'] = LoginUser
  ##Submit User:
  page = form.submit
  
  ##Password Page:
  form = page.forms[0]
  form['USER PASSWORD HERE'] = LoginPassword
  ##Submit Password:
  page = form.submit
  </code></pre>
  
  <p>When I test my code I get the following output:</p>
  
  <p><strong>test.rb:10:in `': uninitialized constant LoginUser (NameError)</strong></p>
  
  <p>Can anyone point out what I'm doing wrong?</p>
  
  <p>Thanks</p>
  
  <p><strong>EDIT 3/27/15:</strong></p>
  
  <p>Using @seoyoochan resource I tried to form my code like this:</p>
  
  <pre><code>require 'rubygems'
  require 'mechanize'
  login_page  = agent.get "http://www.website_here.com/intranet/loginauthusr.asp?Page="
  login_form = login_page.form_with(action: '/sessions') 
  user_field = login_form.field_with(name: "session[user]") 
  user.value = 'My User Name'
  
  login_form.submit
  </code></pre>
  
  <p>When I try to run my code I'm now getting this output:</p>
  
  <p><strong>test.rb:4:in <code>&lt;main&gt;': undefined local variable or method</code>agent' for main:Object (NameError)</strong></p>
  
  <p>I'm needing an example on how to assign the right names/classes that my provided form will work with.</p>
  
  <p><strong>EDIT 4/4/15:</strong></p>
  
  <p>Okay, Now using @tylermauthe example I'm trying to test the following code:</p>
  
  <pre><code>require 'mechanize'
  require 'io/console'
  
  agent = Mechanize.new
  page = agent.get('http://www.website_here.com/intranet/loginauthusr.asp?Page=')
  
  form = page.forms.find{|form| form.action.include?("loginauthpwd.asp?PassedURL=")}
  
  puts "Login:"
  form.login = gets.chomp
  page = agent.submit(form)
  pp page
  </code></pre>
  
  <p>Now my thoughts are that this code should allow me to enter and submit my username bringing me to my next page that would ask for my password. BUT, when I try to run it and enter my username, I get the following output:</p>
  
  <p><strong>/var/lib/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize/form.rb:217:in <code>method_missing': undefined method</code>loginUser=' for # (NoMethodError)
      from scraper.rb:10:in `'</strong></p>
  
  <p>What am I missing or have entered wrong? Please refer to my first edit to see how my form is coded. Also to be clear I did not code the forms this way. I'm only trying to learn how to code and scrape data needed to display on my Dashing Dashboard project.</p>