title: Scraping a website with Html Agility Pack. Response of GET not as expected
body: <p>Using System.Net.HttpRequest I would like to imitate a users search on the following search engine in my code.</p>
  
  <p><a href="http://www.scirus.com" rel="nofollow noreferrer">http://www.scirus.com</a></p>
  
  <p>An example of the search URL is as follows:</p>
  
  <p><a href="http://www.scirus.com/srsapp/search?q=core+facilities&amp;t=all&amp;sort=0&amp;g=s" rel="nofollow noreferrer">http://www.scirus.com/srsapp/search?q=core+facilities&amp;t=all&amp;sort=0&amp;g=s</a></p>
  
  <p>I have the following code to perform the HTTP GET. Note I'm using the HtmlAgilityPack.</p>
  
  <pre><code>protected override HtmlDocument MakeRequestHtml(string requestUrl)
  {
      try
      {
          HttpWebRequest request = WebRequest.Create(requestUrl) as HttpWebRequest;
          request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
          HttpWebResponse response = request.GetResponse() as HttpWebResponse;
  
          HtmlDocument htmlDoc = new HtmlDocument();
          htmlDoc.Load(response.GetResponseStream());
          return (htmlDoc);
  
      }
      catch (Exception e)
      {
          Console.WriteLine(e.Message);
          Console.Read();
          return null;
      }
  }
  </code></pre>
  
  <p>Where "requestUrl" is the example search URL shown above.</p>
  
  <p>The contents of htmlDoc.DocumentNode.InnerHtml contains no search results and looks nothing like the search results page you would get if you copy pasted the example search URL shown above into your browser.</p>
  
  <p>I'm guessing the reason for this is because you must first have a session in order to be able to perform requests. Can anybody advise if there is a feasible way to replicate the behavior of the user agent? Or perhaps there is a better way of achieving the goal of "scraping" the search results that I don't know about? Suggestions please.</p>
  
  <p>Robots.txt contents:</p>
  
  <pre><code># / robots.txt file for http://www.scirus.com
  
  User-agent: NetMechanic
  Disallow: /srsapp/sciruslink
  
  User-agent: *
  Disallow: /srsapp/sciruslink
  Disallow: /srsapp/search
  Disallow: /srsapp/search_simple
  Disallow: /search_simple
  # for dev and accept server uncomment below line at Build time to disallow robots completely
  ##Disallow: /
  </code></pre>
  
  <p>Content of htmlDoc.DocumentNode.InnerHtml</p>
  
  <p><img src="https://i.stack.imgur.com/YJ82N.jpg" alt="Response"></p>