<p>Let's say I've a deployment. For some reason it's not responding after sometime. Is there any way to tell Kubernetes to rollback to previous version automatically on failure?</p>

<p>You mentioned that:</p>
<blockquote>
<p>I've a deployment. For some reason it's not responding after sometime.</p>
</blockquote>
<p>In this case, you can use <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/" rel="noreferrer">liveness and readiness</a> probes:</p>
<blockquote>
<p>The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.</p>
</blockquote>
<blockquote>
<p>The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.</p>
</blockquote>
<p>The above probes may prevent you from deploying a corrupted version, however liveness and readiness probes aren't able to rollback your Deployment to the previous version. There was a similar <a href="https://github.com/kubernetes/kubernetes/issues/23211" rel="noreferrer">issue</a> on Github, but I am not sure there will be any progress on this matter in the near future.</p>
<p>If you really want to automate the rollback process, below I will describe a solution that you may find helpful.</p>
<hr />
<p>This solution requires running <code>kubectl</code> commands from within the Pod.
In short, you can use a script to continuously monitor your Deployments, and when errors occur you can run <code>kubectl rollout undo deployment DEPLOYMENT_NAME</code>.</p>
<p>First, you need to decide how to find failed Deployments. As an example, I'll check Deployments that perform the update for more than 10s with the following command:<br />
<strong>NOTE:</strong> You can use a different command depending on your need.</p>
<pre><code>kubectl rollout status deployment ${deployment} --timeout=10s
</code></pre>
<p>To constantly monitor all Deployments in the <code>default</code> Namespace, we can create a Bash script:</p>
<pre><code>#!/bin/bash

while true; do
    sleep 60
    deployments=$(kubectl get deployments --no-headers -o custom-columns=&quot;:metadata.name&quot; | grep -v &quot;deployment-checker&quot;)
    echo &quot;====== $(date) ======&quot;
    for deployment in ${deployments}; do
        if ! kubectl rollout status deployment ${deployment} --timeout=10s 1&gt;/dev/null 2&gt;&amp;1; then
            echo &quot;Error: ${deployment} - rolling back!&quot;
            kubectl rollout undo deployment ${deployment}
        else
            echo &quot;Ok: ${deployment}&quot;
        fi
    done
done
</code></pre>
<p>We want to run this script from inside the Pod, so I converted it to <code>ConfigMap</code> which will allow us to mount this script in a volume (see: <a href="https://kubernetes.io/docs/concepts/configuration/configmap/#using-configmaps-as-files-from-a-pod" rel="noreferrer">Using ConfigMaps as files from a Pod</a>):</p>
<pre><code>$ cat check-script-configmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: check-script
data:
  checkScript.sh: |
    #!/bin/bash

    while true; do
        sleep 60
        deployments=$(kubectl get deployments --no-headers -o custom-columns=&quot;:metadata.name&quot; | grep -v &quot;deployment-checker&quot;)
        echo &quot;====== $(date) ======&quot;
        for deployment in ${deployments}; do
            if ! kubectl rollout status deployment ${deployment} --timeout=10s 1&gt;/dev/null 2&gt;&amp;1; then
                echo &quot;Error: ${deployment} - rolling back!&quot;
                kubectl rollout undo deployment ${deployment}
            else
                echo &quot;Ok: ${deployment}&quot;
            fi
        done
    done        

$ kubectl apply -f check-script-configmap.yml
configmap/check-script created
</code></pre>
<p>I've created a separate <code>deployment-checker</code> ServiceAccount with the <code>edit</code> Role assigned and our Pod will run under this ServiceAccount:<br />
<strong>NOTE:</strong> I've created a Deployment instead of a single Pod.</p>
<pre><code>$ cat all-in-one.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployment-checker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: deployment-checker-binding
subjects:
  - kind: ServiceAccount
    name: deployment-checker
    namespace: default
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: deployment-checker
  name: deployment-checker
spec:
  selector:
    matchLabels:
      app: deployment-checker
  template:
    metadata:
      labels:
        app: deployment-checker
    spec:
      serviceAccountName: deployment-checker
      volumes:
        - name: check-script
          configMap:
            name: check-script
      containers:
      - image: bitnami/kubectl
        name: test
        command: [&quot;bash&quot;, &quot;/mnt/checkScript.sh&quot;]
        volumeMounts:
        - name: check-script
          mountPath: /mnt
      
      
</code></pre>
<p>After applying the above manifest, the <code>deployment-checker</code> Deployment was created and started monitoring Deployment resources in the <code>default</code> Namespace:</p>
<pre><code>$ kubectl apply -f all-in-one.yaml
serviceaccount/deployment-checker created
clusterrolebinding.rbac.authorization.k8s.io/deployment-checker-binding created
deployment.apps/deployment-checker created

$ kubectl get deploy,pod | grep &quot;deployment-checker&quot;
deployment.apps/deployment-checker   1/1     1            
pod/deployment-checker-69c8896676-pqg9h   1/1     Running   
</code></pre>
<p>Finally, we can check how it works. I've created three Deployments (<code>app-1</code>, <code>app-2</code>, <code>app-3</code>):</p>
<pre><code>$ kubectl create deploy app-1 --image=nginx
deployment.apps/app-1 created

$ kubectl create deploy app-2 --image=nginx
deployment.apps/app-2 created

$ kubectl create deploy app-3 --image=nginx
deployment.apps/app-3 created
</code></pre>
<p>Then I changed the image for the <code>app-1</code> to the incorrect one (<code>nnnginx</code>):</p>
<pre><code>$ kubectl set image deployment/app-1 nginx=nnnginx
deployment.apps/app-1 image updated
</code></pre>
<p>In the <code>deployment-checker</code> logs we can see that the <code>app-1</code> has been rolled back to the previous version:</p>
<pre><code>$ kubectl logs -f  deployment-checker-69c8896676-pqg9h
...
====== Thu Oct  7 09:20:15 UTC 2021 ======
Ok: app-1
Ok: app-2
Ok: app-3
====== Thu Oct  7 09:21:16 UTC 2021 ======
Error: app-1 - rolling back!
deployment.apps/app-1 rolled back
Ok: app-2
Ok: app-3
</code></pre>
