<p>We are running flask microservices on GKE. The main application that accepts all traffic and divides it in other services is restarting.</p>

<p>POD's readiness &amp; liveness start going timeout at random intervals. however, we are running 5 pods of particular service and it's stateless application. One thing I have noticed memory also keeps increasing with time.</p>

<p>is it due to the docker <code>python-slim</code> image at a certain level no able to handle the application and for continuous memory increasing in pod is it like OS <code>python-slim</code> not releasing memory? </p>

<p><strong>Note</strong> : This behavior only on production not on staging(running single application pod). </p>

<p>what cloud be the reason behind this, please help. Thanks </p>

<p><strong>Update</strong> 
liveness &amp; readiness probe config</p>

<pre><code>readinessProbe:
            httpGet:
              path: /k8/readiness
              port: 9595
            initialDelaySeconds: 25
            periodSeconds: 8
            timeoutSeconds: 10
            successThreshold: 1
            failureThreshold: 30
        livenessProbe:
            httpGet:
              path: /k8/liveness
              port: 9595
            initialDelaySeconds: 30
            periodSeconds: 8
            timeoutSeconds: 10
            successThreshold: 1
            failureThreshold: 30
</code></pre>

Harsh Manvar
<p>While it's a bit difficult to provide an answer without looking at the manifests, events, or other traces from your cluster, I've often seen this case happens when people misunderstand/misconfigure readiness and liveness probes and/or not scaling correctly.</p>

<p>For example, it is possible that you have a probe:</p>

<pre><code>readinessProbe:
  httpGet:
    path: /healthz
    port: 443
  failureThreshold: 1
  periodSeconds: 10
</code></pre>

<p>This means, check every 10 seconds, if http <code>GET</code> to <code>/healthz:443</code> is OK, if it fails once, stop sending traffic (since it's a readiness probe only).</p>

<p>If you don't set <code>timeoutSeconds</code>, by default the value is 1 second.</p>

<p>What can often happen under load, is that the <code>/healthz:443</code> endpoint takes longer and longer to respond if additional pods are not added and the latency keeps increasing.</p>

<p>Eventually, once it hovers around the 1 second, a single timeout would cause readiness to fail - this is a best case scenario.</p>

<p>If your liveness probe is configured this way, you would have pod restarts which is way worse.</p>

<p>This <a href="https://srcco.de/posts/kubernetes-liveness-probes-are-dangerous.html" rel="nofollow noreferrer">article</a> does a great job of explaining why it's not always wise to use liveness probes (unless you have a very specific check for it).</p>

<p>In the case of load causing this, you might actually be OK with a 1 second timeout (default value), but you might want to use something like HPA to add additional pods if your latency is increasing.</p>
