title: Weird TCP issue with Amazon S3
body: <blockquote>
    <p>This question exists because it has
    historical significance, but it is not
    considered a good, on-topic question
    for this site, so please do not use it
    as evidence that you can ask similar
    questions here.</p>
    
    <p>More info: <a href="https://stackoverflow.com/faq">https://stackoverflow.com/faq</a></p>
  </blockquote>
  
  <p>I am having a weird TCP issue that makes no sense at all.  I am tearing my hair out.</p>
  
  <p>When downloading a file from <a href="http://en.wikipedia.org/wiki/Amazon_S3" rel="nofollow noreferrer">Amazon S3</a> (and only Amazon S3- every other site works fine) some percentage of the time the connections dies.   This only happens from our servers in Seattle with a web server behind a <a href="http://en.wikipedia.org/wiki/Network_address_translation" rel="nofollow noreferrer">NAT</a> router.  Connecting directly from the router works fine. From our servers here in Victoria everything works fine, and we can't replicate this at all.</p>
  
  <ul>
  <li>Web server in Seattle (tried 10 different servers) ->  <a href="http://www.cisco.com/en/US/docs/ios/12_2t/12_2t13/feature/guide/ftsnat.html#wp1027129" rel="nofollow noreferrer">SNAT</a> router (2 different routers; old kernels and new kernels) -> Amazon S3 =  doesn't work ~10% of the time</li>
  <li>Web server in Seattle  ->  SNAT router -> any other website =  works</li>
  <li>Router box -> Amazon S3 =  works</li>
  <li>Web server in Victoria -> SNAT router in Victoria -> Amazon S3 = works</li>
  </ul>
  
  <p>Things I've tried:</p>
  
  <ul>
  <li>Disabling window scaling</li>
  <li>Lowering the window size</li>
  <li>upgrading the router to the newest kernel</li>
  <li>Disabling the firewall entirely</li>
  </ul>
  
  <p>In the failure case Amazon (or something along the way) sends us a 868 length packet instead of the expected 1500 byte packet.  The server upon seeing the packets replies with a RST packet (error has occurred) and the connection is torn down.</p>
  
  <p>Any help would be greatly appreciated!</p>
  
  <p>Below are two TCP dumps- </p>
  
  <pre><code>------------ Begin Error Case TCP Dump -------------
  tcpdump -r /tmp/s3doesntwork-died 
  reading from file /tmp/s3doesntwork-died, link-type EN10MB (Ethernet)
  19:58:42.542189 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: S 193799772:193799772(0) win 5840 &lt;mss 1460,sackOK,timestamp 760821159 0,nop,wscale 5&gt;
  19:58:42.544115 IP 189-81.amazon.com.www &gt; locum.sparklit.com.39491: S 3148664267:3148664267(0) ack 193799773 win 8190 &lt;mss 1460,nop,wscale 6,nop,nop,sackOK&gt;
  19:58:42.545176 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: . ack 1 win 183
  19:58:42.545184 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: P 1:212(211) ack 1 win 183
  19:58:42.548113 IP 189-81.amazon.com.www &gt; locum.sparklit.com.39491: . ack 212 win 916
  19:58:42.558108 IP 189-81.amazon.com.www &gt; locum.sparklit.com.39491: . 1:1461(1460) ack 212 win 916
  19:58:42.558117 IP 189-81.amazon.com.www &gt; locum.sparklit.com.39491: . 1461:2921(1460) ack 212 win 916
  19:58:42.558123 IP 189-81.amazon.com.www &gt; locum.sparklit.com.39491: . 2921:4381(1460) ack 212 win 916
  19:58:42.558128 IP 189-81.amazon.com.www &gt; locum.sparklit.com.39491: . 4381:5841(1460) ack 212 win 916
  19:58:42.559108 IP 189-81.amazon.com.www &gt; locum.sparklit.com.39491: . 5841:7301(1460) ack 212 win 916
  19:58:42.559118 IP 189-81.amazon.com.www &gt; locum.sparklit.com.39491: . 7301:8129(828) ack 212 win 916
  19:58:42.559138 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: R 193799984:193799984(0) win 0
  19:58:42.559169 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: . ack 1461 win 274
  19:58:42.559176 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: . ack 2921 win 365
  19:58:42.559180 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: . ack 4381 win 457
  19:58:42.559188 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: . ack 5841 win 548
  19:58:42.560167 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: . ack 7301 win 639
  19:58:45.308618 IP locum.sparklit.com.39491 &gt; 189-81.amazon.com.www: F 212:212(0) ack 7301 win 639
  19:58:45.310512 IP 189-81.amazon.com.www &gt; locum.sparklit.com.39491: R 3148671568:3148671568(0) win 8201
  </code></pre>
  
  <p>Update:  Traceroute to Amazon</p>
  
  <pre><code>1  64.34.33.195 (64.34.33.195)  0.795 ms  0.775 ms  0.762 ms
  2  six01-sea4.amazon.com (206.81.80.147)  0.746 ms  0.732 ms  0.715 ms
  3  72.21.222.183 (72.21.222.183)  2.657 ms  2.651 ms  2.638 ms
  4  72.21.222.179 (72.21.222.179)  2.618 ms  2.604 ms  2.591 ms
  5  * * *
  6  * * *
  7  * * *
  </code></pre>
  
  <p>Update:  The offending incoming packet that causes the RST to be sent</p>
  
  <pre><code>19:58:42.559118 IP (tos 0x0, ttl 57, id 18905, offset 0, flags [DF], proto TCP (6), length 868) 189-81.amazon.com.www &gt;     locum.sparklit.com.39491: ., cksum 0x9f6e (correct), 7301:8129(828) ack 212 win 916
  0x0000:  4500 0364 49d9 4000 3906 05d7 cfab bd51  E..dI.@.9......Q
      0x0010:  4022 21c5 0050 9a43 bbac ea50 0b8d 2730  @"!..P.C...P..'0
      0x0020:  5010 0394 9f6e 0000 7c75 a901 cd57 0718  P....n..|u...W..
      0x0030:  a786 e954 4160 3734 f563 5029 e7ad 48a7  ...TA`74.cP)..H.
      0x0040:  34c0 b11b 75a2 a341 c1e6 8aab a03c 31ee  4...u..A.....&lt;1.
      0x0050:  1496 c9ef df22 aadc b87e 8431 fc2a dcd6  ....."...~.1.*..
      0x0060:  e72d 8cf7 aa92 5b12 4923 3f51 50cd 5195  .-....[.I#?QP.Q.
      0x0070:  7910 6ce4 0fc0 d63c f115 276b b7e7 5bf7  y.l....&lt;..'k..[.
      0x0080:  508e d8fa d655 c5b3 1638 3cd6 6cd1 198c  P....U...8&lt;.l...
      0x0090:  1c7f 1b7e 59a4 4370 9c87 523d 0ae2 adb4  ...~Y.Cp..R=....
      0x00a0:  4d8e 7ad5 7954 c6ac 79e3 9e05 4148 6c97  M.z.yT..y...AHl.
      0x00b0:  b711 f262 47fa 363f 9d52 bcd4 a58e 177f  ...bG.6?.R......
      0x00c0:  b33b 1033 b530 7351 d8eb 29a9 dbcf 2c6b  .;.3.0sQ..)...,k
      0x00d0:  b161 99e0 3e67 192b 8c9f a735 89ad f886  .a..&gt;g.+...5....
      0x00e0:  4d3a ff00 462e b0ad 8dec 8f04 9bdb 9121  M:..F..........!
      0x00f0:  4263 0ac4 81c9 18e7 ae6a 1a65 d8db d3ee  Bc.......j.e....
      0x0100:  3722 b608 cf24 1182 2ba1 b39b 728a f29b  7"...$..+...r...
      0x0110:  df1e db68 9af5 b69a e51e 5923 2ed0 29fd  ...h......Y#..).
      0x0120:  f0e4 6303 f8bb f4ae ef44 d5e3 bc67 8f70  ..c......D...g.p
      0x0130:  5b88 8299 a30d 931e e1c0 38fc 6bca a917  [.........8.k...
      0x0140:  7b9d 8a2d 2bb3 5ef7 9e0d 625e c59c d6dc  {..-+.^...b^....
      0x0150:  e778 cd67 5c26 ecd2 a64b 6737 345b 643d  .x.g\&amp;...Kg74[d=
      0x0160:  6a3d b57e f22f 98d5 455e b5da b546 0c60  j=.~./..E^...F.`
      0x0170:  5c51 5281 8a69 4e69 888c af15 0b0a b240  \QR..iNi.......@
      0x0180:  c540 c334 0101 1935 04b5 6996 a091 7ad0  .@.4...5..i...z.
      0x0190:  2676 ba0d 8bbc 30cc 3385 c7f2 a835 ab19  &amp;v....0.3....5..
      0x01a0:  2e0b 003a d75b e17b 453a 03b6 070a a7f4  ...:.[.{E:......
      0x01b0:  a6cf 671e 7270 6b59 ae5e 5f43 28eb 73cd  ..g.rpkY.^_C(.s.
      0x01c0:  53c2 8f34 9c8e 3e95 d268 fe0f 4864 4661  S..4..&gt;..h..HdFa
      0x01d0:  9008 2702 b75e 782d 81e0 0fc2 b3ae bc51  ..'..^x-.......Q
      0x01e0:  0c00 e080 477a 9bb6 3b1f 08fc 7efd 943e  ....Gz..;...~..&gt;
      0x01f0:  26eb ff00 163c 4bad 695e 1b97 51d2 2f6e  &amp;....&lt;K.i^..Q./n
      0x0200:  4cb6 f35b dca3 7c98 1d46 ec8f a62b c675  L..[..|..F...+.u
      0x0210:  4fd9 dfc7 1a4b b2df e812 da63 fe7e 0b2f  O....K.....c.~./
      0x0220:  eb8a fd38 bff1 d055 386f c735 cf5f f8fa  ...8...U8o.5._..
      0x0230:  7705 44ac 54f6 2723 f2ae d862 2a25 6b1c  w.D.T.'#...b*%k.
      0x0240:  f2a5 06ee 7e6b a7c1 bd79 f2a2 2b1d c3b1  ....~k...y..+...
      0x0250:  ba5c d759 e03f 849a e787 3564 d6af 2d6d  .\.Y.?....5d..-m
      0x0260:  db4e b7fd d48f 0ca1 8866 1c0e dcd7 db37  .N.......f.....7
      0x0270:  b2d9 eb2e 4dd6 9767 704f 5692 d949 fcf1  ....M..gpOV..I..
      0x0280:  53da 7c3c d0fc 45a3 ea3a 4cf6 3f67 b379  S.|&lt;..E..:L.?g.y
      0x0290:  a197 cbb5 7311 0dcf 20d6 8f11 2d99 2a94  ....s.......-.*.
      0x02a0:  7a1e 69e1 9fda 77c5 7e0d d3ad 74f8 eeac  z.i...w.~...t...
      0x02b0:  e6b3 b641 1c71 df5b 6085 1db7 a907 f3cd  ...A.q.[`.......
      0x02c0:  7a3e 81fb 6b4c 42ff 0069 6831 4ebd dec6  z&gt;..kLB..ih1N...
      0x02d0:  e837 fe3a c335 3d9f ec75 e01d 576b 5c47  .7.:.5=..u..Wk\G
      0x02e0:  ab12 c3a7 f683 8fe9 5a29 fb07 7c31 3f32  ........Z)..|1?2
      0x02f0:  dbea 81fd 4ea7 2e7f 98ae 37ec 99aa 53e8  ....N.....7...S.
      0x0300:  cb5a 97ed cfe0 ad1f 4d6b 99f4 bd56 79c1  .Z......Mk...Vy.
      0x0310:  ff00 8f48 d151 c8ef 8627 15f4 1681 acc5  ...H.Q...'......
      0x0320:  e21d 0b4e d56d e378 adef add2 e238 e5c6  ...N.m.x.....8..
      0x0330:  f556 1900 e38c d781 d97e c3bf 0ce0 71f6  .V.......~....q.
      0x0340:  9b1d 42ea 31fc 0fa9 cf8f fd0a be80 d2b4  ..B.1...........
      0x0350:  f834 8d3a d2c2 d13c ab4b 5896 1850 9270  .4.:...&lt;.KX..P.p
      0x0360:  8a30 073e                                .0.&gt;
  </code></pre>