<p>We have some payment endpoints in our Phoenix app that have a timeout of 153 seconds. These wrap around CardConnect Bolt API endpoints which require a very long timeout.</p>
<p>Having payment endpoints interrupted can lead to double charge and we'd like to avoid that if possible.</p>
<p>I think there is two parts to this.</p>
<ol>
<li>Stop GKE from sending new requests to pods running old images once I have one pod with the new image deployed</li>
<li>Keep the old pod alive until 153 seconds has passed to let all these payment endpoints finish there work.</li>
</ol>
<p>#1 is probably a Kubernetes configuration, and #2 might be Kubernetes and phoenix configuration.</p>
<p>The only thing that I've seen to partially address this is the <code>terminationGracePeriodSeconds</code> for the pod <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#podspec-v1-core" rel="nofollow noreferrer">https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#podspec-v1-core</a>.</p>
<p>This seems like a common problem, but searching documentation hasn't yielded much for me.</p>

CallMeNorm
<p>Kubernetes has Services. Services have selectors that are used to match pods. The matching record is stored as an <code>Endpoint</code>. What you're describe is possible. Let's see if we can get the requirement together:</p>
<ol>
<li>You need to ensure K8s brings up the new pod before shutting this one down.</li>
<li>You need to ensure that K8s has probes to check when the new one can start serving connections (then it will be added to the <code>Endpoint</code> object and remove the old one). Removing it from the Endpoint does not break open TCP connections. It just stops new ones from going to the old pod.</li>
<li>You need to allow the running pod enough time to finish in-flight request BEFORE starting to shutdown.</li>
<li>You need to allow Kubernetes to kill the pod if it exceeds the in-flight time plus a generous shutdown time.</li>
</ol>
<p>Now onto the YAML</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: web-app
  namespace: default
spec:
  ports:
    - name: web
      port: 8080
      protocol: TCP
      targetPort: web
  selector:
    app: web-app
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web-app
  name: web-app
  namespace: default
spec:
  revisionHistoryLimit: 1
  replicas: 1
  selector:
    matchLabels:
      app: web-app
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - image: nginx:latest
          imagePullPolicy: IfNotPresent
          name: web-app
          lifecycle:
            # This &quot;sleep&quot; preStop hook delays the
            # Pod shutdown 
            preStop:
              exec:
                command:
                  - /bin/sleep
                  - &quot;154&quot; # The max timeout for in flight requests + plus 1 second
          readinessProbe:
            initialDelaySeconds: 5
            periodSeconds: 3
            failureThreshold: 3
            successThreshold: 1
            httpGet:
              port: web
              path: /
          livenessProbe:
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 1
            successThreshold: 1
            httpGet:
              port: web
              path: /
          ports:
            - containerPort: 80
              name: web
              protocol: TCP
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 155  # sleep period plus the shutdown time

</code></pre>
<p>Try this out and see if it works for you.</p>
