title: Scrape password protected asp page
body: <p>I would like to develop automatic scraper for asp password protected web page. I have a login/password for this page.</p>
  <p>First of all, a look in to Firebug log during authorization via firefox. What I have found:</p>
  <ol>
  <li>When I open login page, I get cookie with &quot;__RequestVerificationToken&quot;. i.e <code>http://mysite</code></li>
  <li>When I press Login button FF makes POST query to <code>http://mysite/Account/Login</code> with parameters UserName, Password and __RequestVerificationToken, also it uses cookie saved on step 1</li>
  <li>In case of successful authorisation I get another cookie .ASPXAUTH and goes to <code>http://mysite/Account/Index</code> (page which I want to scrape)</li>
  </ol>
  <p>My code</p>
  <pre><code>//1. Get __RequestVerificationToken cookie
  
      $urlLogin = &quot;http://mysite&quot;;
      $cookieFile = &quot;cookie.txt&quot;;
      $regs=array();
      
      $ch = curl_init();
      
      curl_setopt($ch, CURLOPT_URL, $urlLogin);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
      curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
      curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
      curl_setopt($ch, CURLOPT_STDERR,$f = fopen(&quot;answer.txt&quot;, &quot;w+&quot;));
      curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1; rv:18.0) Gecko/20100101 Firefox/18.0' );
      curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile); 
      
      $data=curl_exec($ch);
  
  //2. Parse token value for the post request
  
  $hash=file_get_contents(&quot;answer.txt&quot;);
  preg_match_all('/=(.*); p/i',$hash, $regs);
  
  //3. Make a post request
  
      $postData = '__RequestVerificationToken='.$regs[1][0].'&amp;UserName=someLogin'.'&amp;Password=somePassword';
      $urlSecuredPage = &quot;http://mysite/Account/Login&quot;;
      curl_setopt($ch, CURLOPT_URL, $urlSecuredPage); 
      curl_setopt($ch, CURLOPT_POST, TRUE);
      curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
      curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile); 
      curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile); 
  
      $data = curl_exec($ch);
      curl_close($ch);
  </code></pre>
  <p>At step 3 my cookie saved on step 1 is rewriting with new value of __RequestVerificationToken. I don`t understand why it happens. As a result I can not authorize due to wrong __RequestVerificationToken and get HTTP 500 error.</p>
  <p>Where I`m wrong?</p>