<p>I have a stateful application deployed in Kubernetes cluster. Now the challenge is how do I scale down the cluster in a graceful way so that each pod while terminating (during scale down) completes it’s pending tasks and then gracefully shuts-down. The scenario is similar to what is explained below but in my case the pods terminating will have few inflight tasks to be processed.</p>
<p><a href="https://medium.com/@marko.luksa/graceful-scaledown-of-stateful-apps-in-kubernetes-2205fc556ba9" rel="nofollow noreferrer">https://medium.com/@marko.luksa/graceful-scaledown-of-stateful-apps-in-kubernetes-2205fc556ba9</a> 1</p>
<p>Do we have an official feature support for this from kubernetes api.</p>
<pre><code>Kubernetes version: v1.11.0

Host OS: linux/amd64

CRI version: Docker 1.13.1
</code></pre>
<p><strong>UPDATE :</strong></p>
<p>Possible Solution - While performing a statefulset scale-down the preStop hook for the terminating pod(s) will send a message notification to a queue with the meta-data details of the resp. task(s) to be completed. Afterwards use a K8 Job to complete the tasks. Please do comment if the same is a recommended approach from K8 perspective.</p>
<p>Thanks In Advance!</p>
<p>Regards,
Balu</p>

Balu R
<p>Your pod will be scaled down only after the in-progress job is completed. You may additionally configure the lifecycle in the deployment manifest with <code>prestop</code> attribute which will gracefully stop your application. This is one of the best practices to follow. Please refer <a href="https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/#define-poststart-and-prestop-handlers" rel="nofollow noreferrer">this</a> for detailed explanation and syntax.</p>
<h2>Updated Answer</h2>
<p>This is the yaml I tried to deploy on my local and tried generating the load to raise the cpu utilization and trigger the hpa.</p>
<p>Deployment.yaml</p>
<pre class="lang-yaml prettyprint-override"><code>kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: whoami
  labels:
    app: whoami
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami
          resources:
            requests:
              cpu: 30m
            limits:
              cpu: 40m
          ports:
            - name: web
              containerPort: 80
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - echo &quot;Starting Sleep&quot;; date; sleep 600; echo &quot;Pod will be terminated now&quot;
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: whoami
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: whoami
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 40
#    - type: Resource
#      resource:
#        name: memory
#        targetAverageUtilization: 10
---
apiVersion: v1
kind: Service
metadata:
  name: whoami-service
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    app: whoami

</code></pre>
<p>Once the pod is deployed, execute the below command which will generate the load.</p>
<pre class="lang-sh prettyprint-override"><code>kubectl run -i --tty load-generator --image=busybox /bin/sh

while true; do wget -q -O- http://whoami-service.default.svc.cluster.local; done
</code></pre>
<p>Once the replicas are created, I stopped the load and the pods are terminated after 600 seconds. This scenario worked for me. I believe this would be the similar case for statefulset as well. Hope this helps.</p>
