title: nginx 502 errors consistently with no application errors
body: <p>I ran into a strange problem suddenly a few days back. </p>
  
  <p>My application has been running for quite sometime with no issues. Suddenly I started seeing a good number of 502 errors consistently. We operate at about 50000 requests per minute with an average server response time of 12 ms.</p>
  
  <p>There are no application panics (errors) and nginx config also allows upto 10000 worker connections.</p>
  
  <p>Other configuration..</p>
  
  <pre><code>sendfile        on;
  keepalive_timeout  600;
  server_tokens off;
  client_max_body_size 20m;
  </code></pre>
  
  <p>Can anyone help me with the direction I should be looking at solving the problem? I get one of these below errors and mostly the first one (sendfile()) below.</p>
  
  <pre><code>2019/08/16 15:01:42 [error] 30#0: *60729 sendfile() failed (32: Broken pipe) while sending request to upstream, client: &lt;IP&gt;, server: &lt;hostname&gt;, request: "POST &lt;endpoint&gt; HTTP/1.1", upstream: "http://127.0.0.1:8080/&lt;endpoint&gt;", host: "&lt;hostname&gt;"
  
  2019/08/16 15:01:45 [error] 30#0: *60821 readv() failed (104: Connection reset by peer) while reading upstream, client: &lt;IP&gt;, server: &lt;hostname&gt;, request: "POST &lt;endpoint&gt; HTTP/1.1", upstream: "http://127.0.0.1:8080/&lt;endpoint&gt;", host: "&lt;hostname&gt;"
  
  2019/08/16 14:55:27 [error] 20#0: *42152 upstream timed out (110: Connection timed out) while reading response header from upstream, client: &lt;IP&gt;, server: &lt;hostname&gt;, request: "POST &lt;endpoint&gt; HTTP/1.1", upstream: "http://127.0.0.1:8080/&lt;endpoint&gt;", host: "&lt;hostname&gt;"
  </code></pre>
  
  <p>We are on golang and gin framework, if it helps in any debugging.</p>