<p>We use <code>kubectl set image</code> to rollout a new version <code>2.0.0</code> of an existing application. We then use <code>kubectl rollout status</code> to wait for the new pod to become ready so that we can run some basic tests.</p>
<p>The problem is, <code>kubectl rollout status</code> returns (implying the new v2 pod is ready) but when we use <code>kubectl exec</code> we ALWAYS land in the old v1 pod.</p>
<pre class="lang-bash prettyprint-override"><code>$ date
Mon 13 Feb 2023 02:33:50 PM CET
$ k set image deploy/myapp myapp=myapp:2.0.0 &amp;&amp; k rollout status deploy/myapp
deployment.apps/myapp image updated
Waiting for deployment &quot;myapp&quot; rollout to finish: 1 old replicas are pending termination...
Waiting for deployment &quot;myapp&quot; rollout to finish: 1 old replicas are pending termination...
deployment &quot;myapp&quot; successfully rolled out
</code></pre>
<p>Here, we assume the new version is running. Let's check:</p>
<pre class="lang-bash prettyprint-override"><code>$ k exec deploy/myapp -- show_version
1.0.0
</code></pre>
<p>Nope, it's still the old version.<br />
Check the deplyoment:</p>
<pre class="lang-bash prettyprint-override"><code>$ k get deploy/myapp
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
myapp   1/1     1            1           273d
</code></pre>
<p>Looks ready (K9S shows 1 pod &quot;Terminating&quot; and 1 pod ready).<br />
Check again:</p>
<pre class="lang-bash prettyprint-override"><code>$date
Mon 13 Feb 2023 02:34:00 PM CET
$ k exec deploy/myapp -- show_version
1.0.0
</code></pre>
<p>Nope, check the pods:</p>
<pre class="lang-bash prettyprint-override"><code>kubectl get pod | grep myapp-
myapp-79454d746f-zw5kg         1/1     Running       0              31s
myapp-6c484f86d4-2zsk5         1/1     Terminating   0              3m5s
</code></pre>
<p>So our pod is running, we just can't exec into it - it always &quot;picks&quot; the terminating pod:</p>
<pre class="lang-bash prettyprint-override"><code>$date
Mon 13 Feb 2023 02:34:10 PM CET
$ k exec deploy/myapp -- show_version
1.0.0
</code></pre>
<p>Wait 20-30s:</p>
<pre class="lang-bash prettyprint-override"><code>$ date
Mon 13 Feb 2023 02:34:25 PM CET
$ k exec deploy/myapp -- show_version
2.0.0
</code></pre>
<p>Finally we have <code>exec</code> on the correct pod.</p>
<p>Why/how can we wait for the old pod to terminate?
OR
How can we ensure we exec into the correct pod for testing?</p>

Marc
<h2>Update</h2>
<blockquote>
<p>Even better would be to get the new_pod id and exec directly into that.</p>
</blockquote>
<p>Also possible, yes. Try this:</p>
<pre><code>k rollout status deploy/myapp &gt;/dev/null &amp;&amp; \
  k get po -l app=myapp | grep Running | awk '{print $1}' | xargs -I{}  kubectl exec {} -- show_version
</code></pre>
<blockquote>
<p>I would love to know what controls that 30s time.</p>
</blockquote>
<p>This can be configured using the <a href="https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace" rel="nofollow noreferrer">terminationGracePeriodSeconds</a> field in the pod's spec. The value defaults to, you guessed it right, 30s. If you're not concerned about data loss (due to the immediate shutdown), it can be set to 0. After that you can directly exec into the new pod:</p>
<pre class="lang-yaml prettyprint-override"><code>    spec:
      terminationGracePeriodSeconds: 0
</code></pre>
<pre><code>k rollout status deploy/myapp &gt;/dev/null &amp;&amp; k exec deploy/myapp -- show_version
</code></pre>
<hr />
<p>While being &quot;Terminated&quot; the old pod is still in <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase" rel="nofollow noreferrer">phase <em>Running</em></a>, and the <code>kubectl exec deploy/myapp</code> seems to use the <a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec" rel="nofollow noreferrer">first <em>Running</em> pod of the deployment </a>.</p>
<p>I would suggest:</p>
<ol>
<li>Retrieve and store the name of the currently running pod in a temp variable prior to deployment (assuming the pod has the label <code>app=myapp</code>)</li>
</ol>
<pre><code>$ old_pod=$(kubectl get pods -l app=myapp -o jsonpath='{.items[0].metadata.name}')
</code></pre>
<ol start="2">
<li>Deploy</li>
</ol>
<pre><code>$ k apply -f Deployment.yaml
</code></pre>
<ol start="3">
<li>Wait <a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-status-em-" rel="nofollow noreferrer">until the rollout is done</a></li>
</ol>
<pre><code>$ k rollout status deploy/myapp
</code></pre>
<ol start="4">
<li>Wait <a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#wait" rel="nofollow noreferrer">until the <code>old_pod</code> is deleted</a></li>
</ol>
<pre><code>$ k wait --for=delete pod/$old_pod --timeout -1s
</code></pre>
<ol start="5">
<li>Check the new pod</li>
</ol>
<pre><code>$ k exec deploy/myapp -- show_version
</code></pre>
