<p>I've got a Next.js app which has 2 simple <code>readiness</code> and <code>liveness</code> endpoints with the following implementation:</p>
<pre><code>return res.status(200).send('OK');
</code></pre>
<p>I've created the endpoints as per <a href="https://nextjs.org/docs/api-routes/introduction" rel="noreferrer">the api routes docs</a>. Also, I've got a <code>/stats</code> basePath as per <a href="https://nextjs.org/docs/api-reference/next.config.js/basepath" rel="noreferrer">the docs here</a>. So, the probes endpoints are at <code>/stats/api/readiness</code> and <code>/stats/api/liveness</code>.</p>
<p>When I build and run the app in a Docker container locally - the probe endpoints are accessible and returning 200 OK.</p>
<p>When I deploy the app to my k8s cluster, though, the probes fail. There's plenty of <code>initialDelaySeconds</code> time, so that's not the cause.</p>
<p>I connect to the <code>service</code> of the pod thru <code>port-forward</code> and when the pod has just started, before it fails, I can hit the endpoint and it returns 200 OK. And a bit after it starts failing as usual.</p>
<p>I also tried accessing the failing pod thru a healthy pod:</p>
<pre><code>k exec -t [healthy pod name] -- curl -l 10.133.2.35:8080/stats/api/readiness
</code></pre>
<p>And the same situation - in the beginning, while the pod hasn't failed yet, I get 200 OK on the curl command. And a bit after, it start failing.</p>
<p>The error on the probes that I get is:</p>
<pre><code>Readiness probe failed: Get http://10.133.2.35:8080/stats/api/readiness: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
</code></pre>
<p>Funny experiment - I tried putting a random, non-existent endpoint for the probes, and I get the same error. Which leads me to the thought that the probes fail because it cannot access the proper endpoints?</p>
<p>But then again, the endpoints are accessible for a period of time before the probes start failing. So, I have literally no idea why this is happening.</p>
<p>Here is my k8s deployment config for the probes:</p>
<pre><code>      livenessProbe:
        httpGet:
          path: /stats/api/liveness
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 10
        timeoutSeconds: 3
        periodSeconds: 3
        successThreshold: 1
        failureThreshold: 5
      readinessProbe:
        httpGet:
          path: /stats/api/readiness
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 10
        timeoutSeconds: 3
        periodSeconds: 3
        successThreshold: 1
        failureThreshold: 3
</code></pre>
<p><strong>Update</strong></p>
<p>used <code>curl -v</code> as requested from comments. The result is:</p>
<pre><code>*   Trying 10.133.0.12:8080...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.133.0.12 (10.133.0.12) port 8080 (#0)
&gt; GET /stats/api/healthz HTTP/1.1
&gt; Host: 10.133.0.12:8080
&gt; User-Agent: curl/7.76.1
&gt; Accept: */*
&gt;
* Mark bundle as not supporting multiuse
&lt; HTTP/1.1 200 OK
&lt; ETag: &quot;2-nOO9QiTIwXgNtWtBJezz8kv3SLc&quot;
&lt; Content-Length: 2
&lt; Date: Wed, 16 Jun 2021 18:42:23 GMT
&lt; Connection: keep-alive
&lt; Keep-Alive: timeout=5
&lt;
{ [2 bytes data]
100     2  100     2    0     0    666      0 --:--:-- --:--:-- --:--:--   666
* Connection #0 to host 10.133.0.12 left intact
OK%
</code></pre>
<p>Then, ofcourse, once it starts failing, the result is:</p>
<pre><code>*   Trying 10.133.0.12:8080...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* connect to 10.133.0.12 port 8080 failed: Connection refused
* Failed to connect to 10.133.0.12 port 8080: Connection refused
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (7) Failed to connect to 10.133.0.12 port 8080: Connection refused
command terminated with exit code 7
</code></pre>

<p>Error tells you: <code>Client.Timeout exceeded while awaiting headers</code>. Meaning the TCP connection is established (not refused, nor timing out).</p>
<p>Your liveness/readiness probe timeout is too low. Your application doesn't have enough time to respond.</p>
<p>Could be due to CPU or memory allocations being smaller than when using your laptop, due to higher concurrency, maybe a LimitRange that sets some defaults when you did not.</p>
<p>Check with:</p>
<pre><code>time kubectl exec -t [healthy pod name] -- curl -l 127.0.0.1:8080/stats/api/readiness
</code></pre>
<p>If you can't allocate more CPU, double that time, round it up, and fix your probes:</p>
<pre><code>  livenessProbe:
    ...
    timeoutSeconds: 10

  readinessProbe:
    ...
    timeoutSeconds: 10
</code></pre>
<p>Alternatively, though probably less in the spirit, you could replace those httpGet checks with tcpSocket ones. They would be faster, though may miss actual issues.</p>
