<p>I have an application deployed to kubernetes.
Here is techstack:
<em>Java 11, Spring Boot 2.3.x or 2.5.x, using hikari 3.x or 4.x</em></p>
<p>Using spring actuator to do healthcheck. Here is <code>liveness</code> and <code>readiness</code> configuration within application.yaml:</p>
<pre><code>  endpoint:
    health:
      group:
        liveness:
          include: '*'
          exclude:
            - db
            - readinessState
        readiness:
          include: '*'
</code></pre>
<p>what it does if DB is down -</p>
<ol>
<li>Makes sure <code>liveness</code> doesn't get impacted - meaning, application
container should keep on running even if there is DB outage.</li>
<li><code>readinesss</code> will be impacted making sure no traffic is allowed to hit the container.</li>
</ol>
<p><code>liveness</code> and <code>readiness</code> configuration in container spec:</p>
<pre><code>livenessProbe:
  httpGet:
      path: actuator/health/liveness
      port: 8443
      scheme: HTTPS
  initialDelaySeconds: 30
  periodSeconds: 30
  timeoutSeconds: 5
readinessProbe:
  httpGet:
      path: actuator/health/readiness
      port: 8443
      scheme: HTTPS
  initialDelaySeconds: 30
  periodSeconds: 30
  timeoutSeconds: 20
</code></pre>
<p>My application is started and running fine for few hours.</p>
<p><strong>What I did:</strong></p>
<p>I brought down DB.</p>
<p><strong>Issue Noticed:</strong></p>
<p>When DB is down, after 90+ seconds I see 3 more pods getting spinned up. When a pod is described I see Status and condition like below:</p>
<pre><code>Status:       Running
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
</code></pre>
<p>when I list all running pods:</p>
<pre><code>NAME                                                  READY   STATUS    RESTARTS   AGE
application-a-dev-deployment-success-5d86b4bcf4-7lsqx    0/1     Running   0          6h48m
application-a-dev-deployment-success-5d86b4bcf4-cmwd7    0/1     Running   0          49m
application-a-dev-deployment-success-5d86b4bcf4-flf7r    0/1     Running   0          48m
application-a-dev-deployment-success-5d86b4bcf4-m5nk7    0/1     Running   0          6h48m
application-a-dev-deployment-success-5d86b4bcf4-tx4rl    0/1     Running   0          49m
</code></pre>
<p><strong>My Analogy/Finding:</strong></p>
<p>Per <code>ReadinessProbe</code> configuration: <code>periodSeconds</code> is set to 30 seconds and <code>failurethreshold</code> is defaulted to 3 per k8s documentation.</p>
<p>Per application.yaml <code>readiness</code> includes db check, meaning after every 30 seconds <code>readiness</code> check failed. When it fails 3 times, <code>failurethreshold</code> is met and it spins up new pods.</p>
<p>Restart policy is default to Always.</p>
<p><strong>Questions:</strong></p>
<ol>
<li>Why it spinned new pods?</li>
<li>Why it spinned specifically only 3 pods and not 1 or 2 or 4 or any number?</li>
<li>Does this has to do anything with <code>restartpolicy</code>?</li>
</ol>

Shivraj
<ol>
<li>As you answered to yourself, it spinned new pods after 3 times tries according to <code>failureThreshold</code>. You can change your <code>restartPolicy</code> to <code>OnFailure</code>, it will allow you to restart the job only if it fails or <code>Never</code> if you don't want have the cluster to be restarted. The difference between the statuses you can find <a href="https://stackoverflow.com/a/40534364/16860542">here</a>. Note this:</li>
</ol>
<blockquote>
<p>The restartPolicy <strong>applies to all containers</strong> in the Pod. restartPolicy only refers to r<strong>estarts of the containers</strong> by the kubelet <strong>on the same node</strong>. After containers in a Pod exit, the kubelet restarts them with an exponential back-off delay (10s, 20s, 40s, …), that is capped at five minutes. Once a container has executed for 10 minutes without any problems, the kubelet resets the restart backoff timer for that container.</p>
</blockquote>
<ol start="2">
<li><p>Share your full <code>Deployment</code> file, I suppose that you've set <code>replicas</code> number to <code>3</code>.</p>
</li>
<li><p>Answered in the answer for the 1st question.</p>
</li>
</ol>
<p>Also note this, if this works for you:</p>
<blockquote>
<p>Startup probes are useful for Pods that have containers that take a long time to come into service. Rather than set a long liveness interval, you can configure a separate configuration for probing the container as it starts up, allowing a time longer than the liveness interval would allow.</p>
<p>If your container usually starts in more than initialDelaySeconds + failureThreshold × periodSeconds, you should specify a startup probe that checks the same endpoint as the liveness probe. The default for periodSeconds is 10s. You should then set its failureThreshold high enough to allow the container to start, without changing the default values of the liveness probe. This helps to protect against deadlocks.</p>
</blockquote>
