title: Dial tcp addr:port: i/o timeout, EOF, and Connection reset by peer
body: <p>Hi I have a simple server setup that intermittently errors out with the following errors. </p>
  
  <blockquote>
    <ul>
    <li>dial tcp address:port: i/o timeout</li>
    <li>EOF</li>
    <li>connection reset by peer</li>
    </ul>
  </blockquote>
  
  <p>Client code: </p>
  
  <pre><code>request, err := http.NewRequest("POST", "http://someaddress/abc?key=somekey", bytes.NewReader(value))
  if err != nil {
      return
  }
  request.Header.Set("Content-Type", "application/json")
  request.Header.Set("Content-Length", strconv.Itoa(len(value)))
  request.Header.Set("Connection", "Keep-Alive")
  
  response, err := client.Do(request)
  if response != nil {
      io.Copy(ioutil.Discard, response.Body)
      response.Body.Close()
  }
  </code></pre>
  
  <p>Server code:</p>
  
  <pre><code>func HandleMe(response http.ResponseWriter, request *http.Request) {
      query := request.URL.Query()
  
      key := query.Get("key")
      if key == "" {
          response.Write(nil)
          return
      }
  
      body, err := ioutil.ReadAll(request.Body)
      if err != nil {
          response.Write(nil)
          return
      }
  
      index(key, string(body))
      response.Write(nil)
      return
  }
  </code></pre>
  
  <p>The index function basically takes the request and adds to a buffered channel and returns immediately. The channel has capacity of 100000 entries for burst loads. Even during the peak times with burst load, the channel does not have more than 500 entries. On the other side of the channel are 20 workers which do additional processing for the data before adding the processed data to database.</p>
  
  <p>Now the weird issue is that I get the 3 errors outlined above occasionally. The server processes gets around 30k requests per second at peak times. The three errors outlined above happen 2-3 times per minute, and in vast majority of the cases it's dial tcp address:port: i/o timeout issue. Does anyone happen to know what I'm doing wrong here? </p>
  
  <p>Edit: 
  Some stats on that server</p>
  
  <blockquote>
    <p>ss -s Total: 648 (kernel 0) TCP:   547 (estab 528, closed 1, orphaned
    4, synrecv 0, timewait 0/0), ports 0</p>
    
    <p>Transport Total     IP        IPv6
    *   0         -         - RAW   0         0         0 UDP   12        9         3 TCP   546       31        515 INET    558       40<br>
    518 FRAG    0         0         0</p>
    
    <p>netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n
          1 established)
          1 Foreign
          4 LAST_ACK
         14 LISTEN
         24 TIME_WAIT
        234 SYN_RECV</p>
  </blockquote>