<p>I have deployed Hashicorp's Vault in my Kubernetes clusters on AWS using the Helm chart for it.   </p>

<p>The number of replicas in the deployment is specified as <code>3</code>.  </p>

<p>Out of these 3 pods, <code>1</code> was ready(<code>1/1</code>) while the other two replica pods were not ready(<code>0/1</code>).
I killed the ready pod and while it was expected that Kubernetes will deploy a new pod to replace it, it deployed two new pods. </p>

<p>Now I have two ready pods and two not ready pods. On deleting one of these pods, now Kubernetes recreates only one pod. So I have <code>4</code> instead of <code>3</code> pods for my vault deployment.
What could be the reason behind this and how can we prevent this?</p>

Uddhav Bhosle
<p>Your deployment is not working because HA (high availability) is not available when using the s3 storage backend. You’ll need Hashicorp’s Consul or AWS’s DynamoDB, or a different backend provider for that. Change the number of replicas to 1 if you're sticking with the s3 backend provider. </p>

<p>As far as why your seeing 4 pods instead of 3, you need to provide more details. Paste the output of <code>kubectl get pods -l app=vault</code> as well as <code>kubectl describe deploy -l app=vault</code> and I will update this answer.</p>

<p>I can only offer speculation for what it's worth. With Deployment objects there's a <code>maxSurge</code> property that allows rolling updates to scale up beyond the desired number of replicas. It defaults to 25%, rounded up, which in your case would be an additional 1 pod.</p>

<blockquote>
  <p><strong>Max Surge</strong></p>
  
  <p><code>.spec.strategy.rollingUpdate.maxSurge</code> is an optional field that
  specifies the maximum number of Pods that can be created over the
  desired number of Pods. The value can be an absolute number (for
  example, 5) or a percentage of desired Pods (for example, 10%). The
  value cannot be 0 if MaxUnavailable is 0. The absolute number is
  calculated from the percentage by rounding up. The default value is
  25%.</p>
  
  <p>For example, when this value is set to 30%, the new ReplicaSet can be
  scaled up immediately when the rolling update starts, such that the
  total number of old and new Pods does not exceed 130% of desired Pods.
  Once old Pods have been killed, the new ReplicaSet can be scaled up
  further, ensuring that the total number of Pods running at any time
  during the update is at most 130% of desired Pods.</p>
</blockquote>

<p>It's possible that deleting the one <code>Running (1/1)</code> pod, along with the NotReady state of the other pods, put your Deployment into a state of "rolling update" or something along those lines, which allowed your deployment to scale up to its <code>maxSurge</code> setting. </p>
