<p>We have an <a href="https://cloud.google.com/load-balancing/docs/https/" rel="noreferrer">HTTP(s) Load Balancer</a> created by a kubernetes ingress, which points to a backend formed by set of pods running nginx and Ruby on Rails.</p>
<p>Taking a look to the load balancer logs we have detected an increasing number of requests with a response code of <code>0</code> and <a href="https://cloud.google.com/load-balancing/docs/https/https-logging-monitoring#what_is_logged" rel="noreferrer"><code>statusDetails</code></a> = <code>client_disconnected_before_any_response</code>.</p>
<p>We're trying to understand why this his happening, but we haven't found anything relevant. There is nothing in the nginx access or error logs.</p>
<p>This is happening for multiple kind of requests, from GET to POST.</p>
<p>We also suspect that sometimes despite of the request being logged with that error, the requests is actually passed to the backend. For instance we're seeing PG::UniqueViolation errors, due to idential sign up requests being sent twice to the backend in our sign up endpoint.</p>
<p>Any kind of help would be appreciated. Thanks!</p>
<hr />
<h2> UPDATE 1</h2>
<p>As requested <a href="https://gist.github.com/javiercr/ae9bdbea6bbcf29e0d39e24fdb483533" rel="noreferrer">here is the yaml</a> file for the ingress resource:</p>
<hr />
<h3> UPDATE 2</h3>
<p>I've created a log-based Stackdriver metric, to count the number of requests that present this behavior. Here is the chart:</p>
<p><a href="https://i.stack.imgur.com/a5cD1.png" rel="noreferrer"><img src="https://i.stack.imgur.com/a5cD1.png" alt="chart" /></a></p>
<p>The big peaks approximately match the timestamp for these kubernetes events:</p>
<p><a href="https://i.stack.imgur.com/A8FQj.png" rel="noreferrer"><img src="https://i.stack.imgur.com/A8FQj.png" alt="events" /></a></p>
<p>Full error: <code>Readiness probe failed: Get http://10.48.1.28:80/health_check: net/http: request canceled (Client.Timeout exceeded while awaiting headers)&quot;</code></p>
<p>So it seems sometimes the readiness probe for the pods behind the backend fails, but not always.</p>
<p>Here is the definition of the readinessProbe</p>
<pre><code>readinessProbe:
  failureThreshold: 3
  httpGet:
    httpHeaders:
    - name: X-Forwarded-Proto
      value: https
    - name: Host
      value: [redacted]
    path: /health_check
    port: 80
    scheme: HTTP
  initialDelaySeconds: 1
  periodSeconds: 30
  successThreshold: 1
  timeoutSeconds: 5
</code></pre>

<p>A response code of 0 and statusDetails = client_disconnected_before_any_response means the client closed the connection before the Load Balancer being able to provide a response as per this <a href="https://cloud.google.com/load-balancing/docs/https/https-logging-monitoring#statusdetail_http_failure_messages" rel="nofollow noreferrer">GCP documentation</a>.</p>

<p>Investigating why it did not respond in time, one of the reasons could be the difference between the <a href="https://cloud.google.com/load-balancing/docs/https/#timeouts_and_retries" rel="nofollow noreferrer">keepalive timeouts</a> from nginx and the GCP Load Balancer, even if this will most-like provide a backend_connection_closed_before_data_sent_to_client caused by a <a href="https://blog.percy.io/tuning-nginx-behind-google-cloud-platform-http-s-load-balancer-305982ddb340" rel="nofollow noreferrer">502 Bad Gateway race condition</a>.</p>

<p>To make sure the backend responds to the request and to see if how long it takes, you can repeat this process for a couple of times (since you still get some valid responses):</p>

<p>curl response time</p>

<p>$ curl -w "@curl.txt" -o /dev/null -s IP_HERE</p>

<p>curl.txt content(create and save this file first):</p>

<pre><code>   time_namelookup:  %{time_namelookup}\n
      time_connect:  %{time_connect}\n
   time_appconnect:  %{time_appconnect}\n
  time_pretransfer:  %{time_pretransfer}\n
     time_redirect:  %{time_redirect}\n
time_starttransfer:  %{time_starttransfer}\n
                ----------\n
        time_total:  %{time_total}\n
</code></pre>

<p>If this is the case, please review the sign up endpoint code for any type of loop like the PG::UniqueViolation errors that you mentioned.</p>
