<p>I've followed <a href="http://rahmonov.me/posts/zero-downtime-deployment-with-kubernetes/" rel="nofollow noreferrer">http://rahmonov.me/posts/zero-downtime-deployment-with-kubernetes/</a> blog, created two docker images with index.html returning 'Version 1 of an app' and 'Version 2 of an app'. What I want to achieve is zero downtime release. I'm using</p>

<pre><code>kubectl apply -f mydeployment.yaml
</code></pre>

<p>with <code>image: mynamespace/nodowntime-test:v1</code> inside.</p>

<p>to deploy v1 version to k8s and then run:</p>

<pre><code>while True
    do
            printf "\n---------------------------------------------\n"
            curl "http://myhosthere"
            sleep 1s
    done
</code></pre>

<p>So far everything works. After short time curl returns 'Version 1 of an app'.
Then I'm applying same k8s deployment file with <code>image: mynamespace/nodowntime-test:v2</code>. And well, it works, but there is one ( always one ) <strong>Gateway Timeout</strong> response between v1 and v2. So its not really no downtime release ; ) It is much better than without RollingUpdate but not perfect.</p>

<p>I'm using <code>RollingUpdate</code> strategy and <code>readinessProbe:</code></p>

<pre><code>---                              
apiVersion: apps/v1              
kind: Deployment                 
metadata:                        
  name: nodowntime-deployment    
spec:                            
  replicas: 1                    
  strategy:                      
    type: RollingUpdate          
    rollingUpdate:               
      maxUnavailable: 0          
      maxSurge: 1                
  selector:                      
    matchLabels:                 
      app: nodowntime-test       
  template:                      
    metadata:                    
      labels:                    
        app: nodowntime-test     
    spec:                        
      containers:                
      ...                        
        readinessProbe:          
          httpGet:               
            path: /              
            port: 80             
          initialDelaySeconds: 5 
          periodSeconds: 5       
          successThreshold: 5 
</code></pre>

<p>Can I do it better? Is it some issue with syncing all of that with ingress controller? I know I can tweak it by using <code>minReadySeconds</code> so old and new pod overlaps for some time but is it the only solution?</p>

<p>I've recreated the mentioned experiment and changed the number of requests to something close to 30 per second by starting the three simultaneous processes of the following:</p>

<pre><code>While True
    do
        curl -s https://&lt;NodeIP&gt;:&lt;NodePort&gt;/ -m 0.1 --connect-timeout 0.1 | grep Version || echo "fail"
done
</code></pre>

<p>After editing deployment and changing image version several times, there was no packet loss at all during the transition process.
I even caught a short moment of serving requests by both images at the same time.</p>

<pre><code>  Version 1 of my awesome app! Money is pouring in!
  Version 1 of my awesome app! Money is pouring in!
  Version 1 of my awesome app! Money is pouring in!
  Version 2 of my awesome app! More Money is pouring in!
  Version 1 of my awesome app! Money is pouring in!
  Version 1 of my awesome app! Money is pouring in!
  Version 2 of my awesome app! More Money is pouring in!
  Version 2 of my awesome app! More Money is pouring in!
  Version 2 of my awesome app! More Money is pouring in!
</code></pre>

<p>Therefore, if you send the request to service directly, it will work as expected. </p>

<p><a href="https://www.bountysource.com/issues/44399804-gateway-timeout-when-rolling-update-a-scaled-service-in-docker-swarm-mode" rel="nofollow noreferrer">“Gateway Timeout”</a> error is a reply from Traefik proxy. It opens TCP connection to backend through a set of iptables rules.<br>
When you do the RollingUpdates, iptables rules have changed but Traefic doesn't know that, so the connection is still considered as open from Traefik point of view. And after the first unsuccessful attempt to go through nonexistent iptables rule Traefik reports "Gateway Timeout" and closes tcp connection. On the next try, it opens a new connection to the backend through the new iptables rule, and everything goes well again.</p>

<p>It could be fixed by <a href="https://docs.traefik.io/configuration/commons/#retry-configuration" rel="nofollow noreferrer">enabling retries</a> in Traefik.</p>

<pre><code># Enable retry sending request if network error
[retry]

# Number of attempts
#
# Optional
# Default: (number servers in backend) -1
#
# attempts = 3
</code></pre>

<p><strong>Update:</strong></p>

<p>finally we worked around it without using 'retry' feature of traefik which could potentially need idempotent processing on all services ( which is good to have anyway but we could not afford forcing all projects to do that ). What you need is kubernetes RollingUpdate strategy + ReadinessProbe configured and graceful shutdown implemented in your app.</p>
