title: Screaming Frog and other online checker returning 200 status code but while debugging from c# returning 500
body: <p>When i scan the URL through screaming frog as well as other online checker it returns status code 200 (OK) but when i do the same from C# code for same URL , it always return 500 (Internal Server Error) without any detail error, but while browsing the URL is running smoonthly.</p>
  <pre class="lang-cs prettyprint-override"><code>try
  {
      HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
      using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
      {
          if (response.StatusCode == HttpStatusCode.NotFound)
          {
              EventLogProvider.LogInformation(&quot;Test&quot;, &quot;Not found&quot;, &quot;Not found&quot;);
          }
      }
  }
  catch (WebException ex)
  {
      var httpResponse = (HttpWebResponse)ex.Response;
  }
  </code></pre>
  <p>I would like to test if the status code is 404 or not but everytime i gets 500</p>