title: Fiddler Website web request Mimicking for screen scraping
body: <p>I'm trying to log into a website to download data through my account. This is the raw Fiddler Request for the POST login form.</p>
  
  <pre><code>POST login/login.jsp HTTP/1.1
  Host: server.com
  Connection: keep-alive
  Content-Length: 73
  Cache-Control: max-age=0
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  Origin: https://server.com
  User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36
  Content-Type: application/x-www-form-urlencoded
  Referer: https://server.com/login/login.jsp
  Accept-Encoding: gzip,deflate,sdch
  Accept-Language: en-US,en;q=0.8
  Cookie: __utma=109610308.114257620.1370889472.1373479499.1371761934.3; __utmc=109613338; __utmz=109610308.1373249472.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _bcvm_vid_424161365915852877=4393500994580715020; _bcvm_vrid_424161365915852877=4393492275825713189; WT_FPC=id=199.234.233.42-2645888112.30303753:lv=1371356395815:ss=1371758333825; JSESSIONID=RGJGy4yQ2WCXRPbnhxCTKGb2rZh39b67d8g8PktTQLqfsBQTlTlYLTD!1154156211; BIGipServeresuite_prod_pool=295635768.2713643.0000
  </code></pre>
  
  <p>It then responds with:</p>
  
  <pre><code>HTTP/1.1 302 Moved Temporarily
  Date: Fri, 21 Jun 2013 12:39:46 GMT
  Location: https://server.com/login/redirect.jsp?APPLICATION=0
  Content-Type: text/html
  Set-Cookie: SECURITY_SESSION_ID=383826514*198399234219875960; domain=.server.com; path=/
  Connection: Close
  Set-Cookie: BIGipServeresuite_prod_pool=294168768.27163.0000; expires=Fri, 21-Jun-2013 13:09:47 GMT; path=/
  Content-Length: 3669
  </code></pre>
  
  <p>That SECURITY_SESSION_ID is what's needed to do anything on the site.</p>
  
  <p>To mimic it I coded wrote this:</p>
  
  <pre><code>   //GET the Login page - I preform a quick get to pick up the first two important cookies
  
            HttpWebRequest GETLoginRequest = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
            GETLoginRequest.Method = "GET";
            GETLoginRequest.Accept = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*";
            GETLoginRequest.AllowAutoRedirect = false;
            GETLoginRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)";
            GETLoginRequest.CookieContainer = cookieJar;  
  
            HttpWebResponse GETLoginResponse = (HttpWebResponse)GETLoginRequest.GetResponse(); //Gets the JSession and BIGipServer cookies
            Console.Write(" \n 3rd count after GETLoginResponse : " + cookieJar.Count + "\n");
  
   //POST Login
  
            HttpWebRequest POSTLoginRequest = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
            POSTLoginRequest.Method = "POST";
            WebHeaderCollection myWebHeaderCollection = POSTLoginRequest.Headers;
            POSTLoginRequest.AllowAutoRedirect = true;
            byte[] bytes = Encoding.ASCII.GetBytes(formParams);
  
         ///Cache
            POSTLoginRequest.Headers.Add(HttpRequestHeader.CacheControl, "max-age=0"); 
  
          //Client
            POSTLoginRequest.Accept = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*";
            POSTLoginRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate,sdch"); 
            myWebHeaderCollection.Add("Accept-Language:en-US");
            POSTLoginRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)";
          //Cookies/Login
            POSTLoginRequest.CookieContainer = cookieJar; //The cookie jar contains mainly added
          //Entity
            POSTLoginRequest.ContentLength = bytes.Length;
            POSTLoginRequest.ContentType = "Content-Type: application/x-www-form-urlencoded";
          //Miscellanous
            POSTLoginRequest.Headers.Add("Origin: https://server.com");
            POSTLoginRequest.Referer = "https://server.com/login/login.jsp";
          //Transport
  
                //Fix I found to allow Connection: Keep-Alive
            var sp = POSTLoginRequest.ServicePoint;
            var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
            prop.SetValue(sp, (byte)0, null);
  
            ServicePointManager.Expect100Continue = false;
  
            POSTLoginRequest.Host = "server.com";
  
            using (Stream os = POSTLoginRequest.GetRequestStream())
            {
                os.Write(bytes, 0, bytes.Length);
            }
            HttpWebResponse POSTLoginResponse = (HttpWebResponse)POSTLoginRequest.GetResponse();
            Console.Write(" \n 4th count after POSTLoginResponse : " + cookieJar.Count + "\n");
  </code></pre>
  
  <p>at the end of the day my request in fiddler looks like this:</p>
  
  <pre><code> POST /login/login.jsp HTTP/1.1
  Cache-Control: max-age=0
  Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
  Accept-Encoding: gzip,deflate,sdch
  Accept-Language: en-US
  User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)
  Content-Type: Content-Type: application/x-www-form-urlencoded
  Origin: https://server.com
  Referer: https://server.com/login/login.jsp
  Host: esuite.pjm.com
  Cookie: __utma=1096103034.235016339.1371048460.1371048460.1371048460.1; __utmz=109610428.1371048460.1.1.utmcsr=bing|utmccn=(organic)|utmcmd=organic|utmctr=pjm; _bcvm_vrid_424161365915852877=4393493862784729423; WT_FPC=id=199.234.233.42-3603288592.30304123:lv=1371044861062:ss=1371044859892; JSESSIONID=RGbYQd7JnPdNkTvtGCzQ9NLyFgfBnnyLFzbvKPg2Y0gLnhL2hp8F!-1770592471; BIGipServeresuite_prod_pool=327723200.27163.0000
  Content-Length: 73
  Connection: Keep-Alive
  </code></pre>
  
  <p>which is pretty much the same and yet I get this as a response:</p>
  
  <pre><code>HTTP/1.1 200 OK
  Date: Fri, 21 Jun 2013 13:58:17 GMT
  Content-Length: 3356
  Content-Type: text/html
  Set-Cookie: BIGipServeresuite_prod_pool=327723200.27163.0000; expires=Fri, 21-Jun-2013 14:28:17 GMT; path=/
  </code></pre>
  
  <p>I think one thing has to do with that in the browser the reply is a 302 Moved Temporarily but I really don't know. The two important cookies seem to be JSession and BIGIp because
  those are set by the site. The other cookies I manually added in and they seem to be google analytic cookies and I don't think they matter all to much. Anyway the headers are nearly
  identicaly but it's still not responding with the SECURITY_SESSION_ID that I'm looking for. Does anyone have any idea on what I'm doing wrong? </p>