title: TCP Receive window
body: <p>I am trying to understand how the receiver window affect the throughput over a high latency connection.</p>
  
  <p>I have a simple client-server pair of apps on two machines, far apart, with the connection between the two of 250mSec latency RTT. I ran this test with both Windows (XP, 7), and Linux (Ubuntu 10.x), with the same results, so for simplicity let's assume the case of:
  Client receiving data: WinXP Pro
  Server sending data: Win7 Pro
  Again, latency is 250mSec RTT.</p>
  
  <p>I run my TCP test without changing the receiver buffer size on the client (default is 8Kb), and I see on the wire (using Wireshark):</p>
  
  <ul>
  <li>the client send ACKS to the server and the TCP packets contains RWIN=65k</li>
  <li>server send data and report RWIN=65k</li>
  </ul>
  
  <p>Looking at the trace I see a bursts of 3-4 packets (with a payload of 1460 bytes), immediately followed by the ACK sent from the client machine to the server, then nothing for approx 250mSec then a new burst of packets from the server to the client.</p>
  
  <p>So, in conclusion it appears that the server doesn't send new data even before it fills up the receiver's window.</p>
  
  <p>To do more tests, I also ran the same test this time changing the receiver's buffer size on the client machine (on Windows, changing the receiver's buffer size ends up affecting the RWIN advertised by the machine). I would expect to see large burst of packets before blocking for ACK... and at least a higher throughput.</p>
  
  <p>In this case I set recv buffer size to 100,000,000. The packets from the client to the server have now a RWIN=99,999,744 (well, that's nice), but unfortunately the pattern of the data sent FROM the server to the client is still the same: a short burst followed by a long wait.
  To confirm also what I see on the wire, I also measure the time to send a chunk of data from the server to the client. I don't see ANY changes in using a large RWIN or using the default. </p>
  
  <p>Can anybody help me understanding why changing the RWIN doesn't really affect the throughput?</p>
  
  <p>Few notes:
   - server send data as fast as possible using write() of chunks of 8Kb
   - as I said before, I see similar effects using Linux as well. changing the receiver buffer size affects the RWIN used by a node, but the throughput remains the same.
   - I analyze the trace after several hundred packets, to give enough time to the TCP slow start mechanism to enlarge the CWIN size.</p>
  
  <hr>
  
  <p>As suggested, I'm adding a small snapshot of a wire trace here</p>
  
  <pre><code>No.     Time        Source                Destination           Protocol Length Info
       21 2.005080    CCC.CCC.CCC.CCC       sss.sss.sss.sss       TCP      60     57353 &gt; 21500 [ACK] Seq=1 Ack=11681 Win=99999744 Len=0
       22 2.005109    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=19305 Ack=1 Win=65536 Len=1460
       23 2.005116    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=20765 Ack=1 Win=65536 Len=1460
       24 2.005121    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=22225 Ack=1 Win=65536 Len=1460
       25 2.005128    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      946    21500 &gt; 57353 [PSH, ACK] Seq=23685 Ack=1 Win=65536 Len=892
       26 2.005154    CCC.CCC.CCC.CCC       sss.sss.sss.sss       TCP      60     57353 &gt; 21500 [ACK] Seq=1 Ack=14601 Win=99999744 Len=0
       27 2.007106    CCC.CCC.CCC.CCC       sss.sss.sss.sss       TCP      60     57353 &gt; 21500 [ACK] Seq=1 Ack=16385 Win=99999744 Len=0
       28 2.007398    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=24577 Ack=1 Win=65536 Len=1460
       29 2.007401    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=26037 Ack=1 Win=65536 Len=1460
       30 2.007403    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=27497 Ack=1 Win=65536 Len=1460
       31 2.007404    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=28957 Ack=1 Win=65536 Len=1460
       32 2.007406    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=30417 Ack=1 Win=65536 Len=1460
       33 2.007408    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      946    21500 &gt; 57353 [PSH, ACK] Seq=31877 Ack=1 Win=65536 Len=892
       34 2.007883    CCC.CCC.CCC.CCC       sss.sss.sss.sss       TCP      60     57353 &gt; 21500 [ACK] Seq=1 Ack=19305 Win=99999744 Len=0
       35 2.257143    CCC.CCC.CCC.CCC       sss.sss.sss.sss       TCP      60     57353 &gt; 21500 [ACK] Seq=1 Ack=22225 Win=99999744 Len=0
       36 2.257160    CCC.CCC.CCC.CCC       sss.sss.sss.sss       TCP      60     57353 &gt; 21500 [ACK] Seq=1 Ack=24577 Win=99999744 Len=0
       37 2.257358    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=32769 Ack=1 Win=65536 Len=1460
       38 2.257362    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=34229 Ack=1 Win=65536 Len=1460
       39 2.257364    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=35689 Ack=1 Win=65536 Len=1460
       40 2.257365    sss.sss.sss.sss       CCC.CCC.CCC.CCC       TCP      1514   21500 &gt; 57353 [ACK] Seq=37149 Ack=1 Win=65536 Len=1460
  </code></pre>
  
  <p>As you see, the server stop sending data at packet #33.</p>
  
  <p>Client send ACK at packet #34 of an old packet (seq=19305, sent on packet #20, not shown here).
  With an RWIN of 100Mb, I would expect the server NOT to block for a while.</p>
  
  <p>After 20-30 packets, the congestion window on the server side should be large enough to send more packets than I see...
  I assume the congestion window eventually is going to grow up to the RWIN... but still, even after hundred of packets, the pattern is the same: data data then block for 250mSec...</p>