<p>I don't know what to test or how to test to make sure that my application doesn't go into an unrecoverable state when deployments are being upgraded (new versions are rolled out). I understand that Kubernetes deployment provides rolling upgrade, which means the old pod will not be killed until the new pod is ready. But I would still assume that there will be gRPC requests lost during the transition. Is there anyway that I could test it?</p>

qw_ask_questions
<p>To make your deployment image upgrade full proof and with <code>0</code> downtime, you need two things in your deployment file  <code>readiness probe</code> and <code>rollingUpdate</code> Strategy.</p>

<p><code>readiness probe</code> is a check that Kubernetes does in order to make sure that your pod is ready to send traffic to it. Until it is not ready, Kubernetes will not use your pod. Easy! In our case, it looks like this:</p>

<pre><code>readinessProbe:
  httpGet:
    path: /
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 5
  successThreshold: 1
</code></pre>

<p>We are basically telling Kubernetes to send an http get request to the path / every five seconds and if it is successful, mark the pod ready and start sending traffic to it.</p>

<p>Another thing you should know is <code>RollingUpdate strategy</code>, it looks like this:</p>

<pre><code>strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1
</code></pre>

<p>It basically tells Kubernetes that there should be zero unavailable pods while deploying (<code>maxUnavailable: 0</code>) and there should be one new pod at a time (<code>maxSurge: 1</code>).</p>

<p>So your deployment yaml should look like:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp-container
        image: prafull/myapp:1
        imagePullPolicy: Always
        ports:
          - containerPort: 80
            protocol: TCP
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1    
</code></pre>

<p>If you now upgrade your image using <code>kubectl apply -f deployment.yaml</code> there will be no downtime for your requests.</p>
