<p>I am facing a very weird problem.</p>
<p>I have a deployment with nginx container(six pods) work on GKE, I use Liveness/Readiness Probe on nginx container, all the pods are working fine.</p>
<p>One day, all pods were dead because of readiness/liveness failed. And I use <code>kubectl describe pod</code> that pop up ...</p>
<pre><code>Liveness probe failed: Get http://10.28.6.19:80/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get http://10.28.6.19:80/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
</code></pre>
<p>I try to delete pods to let six pods reborn, the pod is very normal.</p>
<p>Some posts said: this is pods hit the limit of resource, I swear my pod resource usage is very low. And I don't set limit for my pods.</p>
<p>I have encounter this problem third, until now I still don't know the root cuse and any solution of this kind of problem.</p>
<p>Deployment Spec as below, Nginx will direct repose 200 status code according to <code>/healthz</code>, so nginx config have to add a location</p>
<pre><code>readinessProbe:
  httpGet:
    path: /healthz
    port: 80
  initialDelaySeconds: 10
  timeoutSeconds: 2
livenessProbe:
  httpGet:
    path: /healthz
    port: 80
  initialDelaySeconds: 10
  timeoutSeconds: 2
</code></pre>

<p>When your app starts hitting the resource limit, the Kubernetes starts throttling your container. Resulting in probe failure. You should always set the resource request and limit for avoiding such events:</p>
<pre><code>  containers:
  - name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        memory: &quot;64Mi&quot;
        cpu: &quot;250m&quot;
      limits:
        memory: &quot;128Mi&quot;
        cpu: &quot;500m&quot;
</code></pre>
