title: Really weird HTTP client using TcpClient in C#
body: <p>I am implementing a simple HTTP Client that just connects to a web server and gets its default homepage. Here it is and it works nice:</p>
  
  <pre><code>using System;
  using System.Net.Sockets;
  
  namespace ConsoleApplication1
  {
      class Program
      {
          static void Main(string[] args)
          {
              TcpClient tc = new TcpClient();
              tc.Connect("www.google.com", 80);
  
              using (NetworkStream ns = tc.GetStream())
              {
                  System.IO.StreamWriter sw = new System.IO.StreamWriter(ns);
                  System.IO.StreamReader sr = new System.IO.StreamReader(ns);
  
                  string req = "";
                  req += "GET / HTTP/1.0\r\n";
                  req += "Host: www.google.com\r\n";
                  req += "\r\n";
  
                  sw.Write(req);
                  sw.Flush();
  
                  Console.WriteLine("[reading...]");
                  Console.WriteLine(sr.ReadToEnd());
              }
              tc.Close();
              Console.WriteLine("[done!]");
              Console.ReadKey();
          }
      }
  }
  </code></pre>
  
  <hr>
  
  <p>When I delete the below line from above code, the program blocks on <strong>sr.ReadToEnd</strong>. </p>
  
  <pre><code>req += "Host: www.google.com\r\n";
  </code></pre>
  
  <p>I even replaced <strong>sr.ReadToEnd</strong> with <strong>sr.Read</strong>, but it cannot read anything. I used Wireshark to see what's happen: </p>
  
  <p><a href="http://www.imagechicken.com/uploads/1252514718052893500.jpg" rel="nofollow noreferrer">Screenshot of captured packets using Wireshark http://www.imagechicken.com/uploads/1252514718052893500.jpg</a></p>
  
  <p>As you see, after my GET request Google doesn't respond and the request is retransmitted again and again. It seems we HAVE TO specify the <strong>Host</strong> part in HTTP request. The weird part is WE DON'T. I used <strong>telnet</strong> to send this request and got the respond from Google. I also captured the request sent by telnet and it was exactly same as my request.</p>
  
  <p>I tried many other websites (e.g. Yahoo, Microsoft) but the result is same.</p>
  
  <p>So, does the delay in telnet cause the web-server act differently (because in telnet we actually <em>type</em> characters instead of sending them together in 1 packet).</p>
  
  <hr>
  
  <p>Another weird problem is when I change <strong>HTTP/1.0</strong> to <strong>HTTP/1.1</strong>, the program always blocks on <strong>sr.ReadToEnd</strong> line. I guess that's because the web server don't close the connection.</p>
  
  <p>One solution is using <strong>Read</strong> (or <strong>ReadLine</strong>) and <strong>ns.DataAvailable</strong> to read the response. But I cannot be sure that I have read all of the response. How I can read the response and be sure there is no more bytes left in the response of a HTTP/1.1 request? </p>
  
  <hr>
  
  <p><strong>Note:</strong>
  As W3 says,</p>
  
  <blockquote>
    <p><a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html" rel="nofollow noreferrer">the <strong>Host</strong> request-header field MUST accompany all <strong>HTTP/1.1</strong>
    requests</a></p>
  </blockquote>
  
  <p>(and I did it for my HTTP/1.1 requests). But I haven't seen such thing for <strong>HTTP/1.0</strong>. Also sending a request without <strong>Host</strong> header using telnet works without any problem.</p>
  
  <hr>
  
  <p><strong>Update:</strong></p>
  
  <p><strong>Push</strong> flag has been set to 1 in the TCP segment. I also have tried <strong>netsh winsock reset</strong> to reset my TCP/IP stack. There is no firewalls nor anti-viruses on the testing computer. The packet are actually sent because Wireshark installed on another computer can capture it.</p>
  
  <p>I also have tried some other requests. For Instance,</p>
  
  <pre><code>string req = "";
  req += "GET / HTTP/1.0\r\n";
  req += "s df slkjfd sdf/ s/fd \\sdf/\\\\dsfdsf \r\n";
  req += "qwretyuiopasdfghjkl\r\n";
  req += "Host: www.google.com\r\n";
  req += "\r\n";
  </code></pre>
  
  <p>In all kind of requests, if I omit the <strong>Host:</strong> part, the web-server doesn't respond and if with a <strong>Host:</strong> part, even an invalid request (just like the above request) will be responded (by a 400: HTTP Bad Request).</p>
  
  <p><a href="https://stackoverflow.com/users/126769/nos">nos</a> says the <strong>Host:</strong> part is not required on his machine, and this makes the situation more weird.</p>