title: Python - getting source code with socket
body: <p>I wanna send http get request and receive source code from webpage, this has to be done through sockets. I set buffer size to 4096, but my script download only small part from the page</p>
  
  <pre><code>import socket
  sock = socket.socket ( socket.AF_INET, socket.SOCK_STREAM )
  sock.connect ( ( "edition.cnn.com", 80 ) )
  
  host = socket.gethostbyname("edition.cnn.com")
  sock.sendall('GET http://edition.cnn.com/index.html HTTP/1.1\r\n'\
      + 'User-Agent: agent123\r\n'\
      + 'Host: '+host+'\r\n'\
      + '\r\n')
  
  print sock.recv(4096)
  sock.close()
  </code></pre>
  
  <p>After I run this code data I get are</p>
  
  <pre><code>HTTP/1.1 200 OK
  
  Server: nginx
  
  Date: Wed, 01 Jan 2014 18:31:25 GMT
  
  Content-Type: text/html
  
  Transfer-Encoding: chunked
  
  Connection: keep-alive
  
  Set-Cookie: CG=GR:44:Réthimnon; path=/
  
  Last-Modified: Wed, 01 Jan 2014 18:31:22 GMT
  
  Vary: Accept-Encoding
  
  Cache-Control: max-age=60, private
  
  Expires: Wed, 01 Jan 2014 18:32:25 GMT
  
  
  
  ac2a
  
  
  &lt;!DOCTYPE HTML&gt;
  &lt;html lang="en-US"&gt;
  &lt;head&gt;
  &lt;title&gt;CNN.com International - Breaking, World, Business, Sports, Entertainment and Video News&lt;/title&gt;
  &lt;meta http-equiv="X-UA-Compatible" content="IE=edge"/&gt;
  &lt;meta http-equiv="content-type" content="text/html;charset=utf-8"/&gt;
  &lt;meta http-equiv="last-modified" content="2014-01-01T18:28:34Z"/&gt;
  &lt;meta http-equiv="refresh" content="1800;url=http://edition.cnn.com/?refresh=1"/&gt;
  &lt;meta name="robots" content="index,follow"/&gt;
  &lt;meta name="googlebot" content="noarchive"/&gt;
  &lt;meta name="description" content="CNN.com International delivers breaking news from across the globe and information on the latest top stories, business, sports and entertainment headlines. Follow the news as it happens through: special reports, videos, audio, photo galleries plus interactive maps and timelines."/&gt;
  &lt;meta name="keywords" content="CNN, CNN news, CNN International, CNN International news, CNN Edition, Edition news, news, news online, breaking news, U.S. news, world news, global news, weather, business, CNN Money, sports, politics, law, technology, entertainment, education,
  </code></pre>
  
  <p>Which isn't even first 13 rows from source code... view-source:<a href="http://edition.cnn.com/index.html" rel="nofollow">http://edition.cnn.com/index.html</a></p>
  
  <hr>
  
  <p>And another problem, when I try address google.com like a host </p>
  
  <pre><code>import socket
  sock = socket.socket ( socket.AF_INET, socket.SOCK_STREAM )
  sock.connect ( ( "google.com", 80 ) )
  
  host = socket.gethostbyname("google.com")
  sock.sendall('GET http://google.com/index.html HTTP/1.1\r\n'\
      + 'User-Agent: agent123\r\n'\
      + 'Host: '+host+'\r\n'\
      + '\r\n')
  print sock.recv(4096)
  sock.close()
  </code></pre>
  
  <p>I get this response</p>
  
  <pre><code>HTTP/1.1 301 Moved Permanently
  
  Location: http://www.google.com/index.html
  
  Content-Type: text/html; charset=UTF-8
  
  Date: Wed, 01 Jan 2014 18:38:57 GMT
  
  Expires: Fri, 31 Jan 2014 18:38:57 GMT
  
  Cache-Control: public, max-age=2592000
  
  Server: gws
  
  Content-Length: 229
  
  X-XSS-Protection: 1; mode=block
  
  X-Frame-Options: SAMEORIGIN
  
  Alternate-Protocol: 80:quic
  
  
  
  &lt;HTML&gt;&lt;HEAD&gt;&lt;meta http-equiv="content-type" content="text/html;charset=utf-8"&gt;
  &lt;TITLE&gt;301 Moved&lt;/TITLE&gt;&lt;/HEAD&gt;&lt;BODY&gt;
  &lt;H1&gt;301 Moved&lt;/H1&gt;
  The document has moved
  &lt;A HREF="http://www.google.com/index.html"&gt;here&lt;/A&gt;.
  
  &lt;/BODY&gt;&lt;/HTML&gt;
  </code></pre>
  
  <p>which says that page is moved to the same address like i wanted download...</p>