title: HttpWebRequest throws exception in code but not browser
body: <p>I'm currently writing an API that gets data from a Point of Sale System's web interface. So far, I haven't had any problems logging in and generating reports to get data from until this situation.</p>
  
  <p>In general, I can use the follow method to return an HttpWebRequest object that does the trick for most request to the web server.</p>
  
  <pre><code>private HttpWebRequest DefaultRequestObject(string path)
  {
      var request = (HttpWebRequest)WebRequest.Create(_baseUrl + path);
      request.Method = "GET";
      request.Host = _baseUrl.Substring(8); // Cut off the https://
      request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0";
      request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
      request.Headers.Add("Accept-Language", "en-US,en;q=0.5");
      request.Headers.Add("Accept-Encoding", "gzip, deflate");
  
      request.SendChunked = false;
      request.AllowAutoRedirect = true;
  
      request.ProtocolVersion = HttpVersion.Version11;
  
      var sp = request.ServicePoint;
      sp.Expect100Continue = false;
      var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
      prop.SetValue(sp, (byte)0, null);
  
      request.CookieContainer = _cookieJar;
  
      if (!String.IsNullOrEmpty(_cookieString))
          request.Headers.Add(HttpRequestHeader.Cookie, _cookieString);
  
      return request;
  }
  </code></pre>
  
  <p>When I send a GET request, I use the following method:</p>
  
  <pre><code>public MgrngResponse GetContent(string serverPath, Dictionary&lt;string, string&gt; args)
  {
      if (!serverPath.StartsWith("/"))
          serverPath = "/~pos/mgrng/" + serverPath;
  
      var requestString = serverPath + "?" + HttpDataFormat(args);
      var request = DefaultRequestObject(requestString);
  
      try
      {
          var response = (HttpWebResponse)request.GetResponse();
          var mgrngResponse = new MgrngResponse(response);
  
          if (!String.IsNullOrEmpty(mgrngResponse.HttpResponse.GetResponseHeader("Set-Cookie")))
              SaveMgrngResponseCookies(mgrngResponse);
  
          _sessionExpiration = DateTime.Now.AddMinutes(15);
          UpdateStatus(mgrngResponse.Content);
  
          return mgrngResponse;
      }
      catch (WebException webException)
      {
          using (WebResponse response = webException.Response)
          {
              var httpResponse = (HttpWebResponse)response;
              Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
              using (Stream data = response.GetResponseStream())
              using (var reader = new StreamReader(data))
              {
                  string text = reader.ReadToEnd();
                  Console.WriteLine(text);
              }
          }
          var eventArgs = new SessionUpdateEventArgs(SessionStatus.ConnectionError, "Unable to GET data");
          RaiseStatusChangeEvent(eventArgs);
          return null;
      }
  }
  </code></pre>
  
  <p>This works well for all of the pages I've attempted so far, but now I'm running into a new problem where when I try to get the response for a particular page, the method throw a WebException with a 500 Internal Server Error.</p>
  
  <p>I used Fiddler to match a browser's request exactly with mine (with the exception of the order of the headers and the cookie values obviously) but I'm still getting the Internal Server Error.</p>
  
  <p>Below are the Raw Requests from Fiddler. The first one is from Firefox and the second one is from my program.</p>
  
  <hr>
  
  <pre><code>GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&amp;csv_delimeter=csv_delimeter_comma&amp;format=text&amp;format1=csv&amp;format1=txt&amp;format1=pdf&amp;format1=html HTTP/1.1
  Host: location.posprovider.com
  User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  Accept-Language: en-US,en;q=0.5
  Accept-Encoding: gzip, deflate
  Cookie: USER=af182fda473354eb3199522726ca61c9d5516c95f165cffd63b9522726ca61c9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9fc516c950a6607ae63b9522726ca61c9; PHPSESSID=9d7f54f9a1769a3e0572745fe0db3d97
  Connection: keep-alive
  </code></pre>
  
  <hr>
  
  <pre><code>GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&amp;csv_delimeter=csv_delimeter_comma&amp;format=text&amp;format1=csv&amp;format1=txt&amp;format1=pdf&amp;format1=html HTTP/1.1
  User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  Accept-Language: en-US,en;q=0.5
  Accept-Encoding: gzip, deflate
  Host: location.posprovider.com
  Cookie: PHPSESSID=16ad21c9d69fe85b3d185ab284f8620b; USER=30dca66f355e0ba89b6eda3c3e822ea24a95e5209e0f90bec94eda3c3e822ea243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e826395e520780b58edc94eda3c3e822ea2
  Connection: Keep-Alive
  </code></pre>
  
  <hr>
  
  <p>I even tried logging into the web interface and then copy and pasting my generated request string into the browser and I got the desired data.</p>
  
  <p>Any ideas?</p>