<p>I was running my kafka-connect on two ec2 machines. So irrespective of number of tasks, these two machines would always stay up running tasks. Hence under used machines.
Recently I migrated kafka-connect on kubernetes. I achieved good cpu/memory efficiency. </p>

<p><strong>But the problem arises when downscaling of kubernetes happens</strong>. <strong>Downscaling of pods does not happen gracefully</strong>. </p>

<p><strong>Eg.</strong> Suppose there are 2 pods p1 and p2.
p1 is running 3 tasks t1,t2,t3
p2 is running 2 tasks t4,t5
(here task t5 is task for source connector that brings data from postgres to kafka)</p>

<p>When any pod vanishes during downscaling, tasks running on it are rebalanced on other pods.
Suppose pod p2 vanishes.</p>

<p>After task rebalancing new state of cluster is:-
P1 is running 5 tasks t1,t2,t3,t4_new,t5_new</p>

<p>But logs for my source connector says that some other task(presumably task running on older pod t5) is still running and accessing postgres db data.</p>

<p>How can i make sure whenever pod downscales, it happens gracefully in the sense that all task running on pod are stopped.</p>

<p>It could be that the default <code>grace period</code> is not enough for you a
aplication to finish its tasks after recevied <code>SIGTERM</code> singal. </p>

<p><code>SIGTERM</code> signal is sent the the main process in the container and once the signal is recevied container should start a graceful shutdown of the running application and exit. </p>

<p>There is a very good explanation/flow described in <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods" rel="nofollow noreferrer">kubernetes official documentation</a> about <code>Termination of Pods</code>. </p>

<p>You could try to extend the <code>terminationGracePeriodSeconds</code> in your deployment to see if that helps (The default is 30): </p>

<pre><code>spec:
    replicas: 
    template:
        spec:
            containers:
              - name: test
                image: ...
            terminationGracePeriodSeconds: 60
</code></pre>

<p>The other way is to use <code>preStop</code> hook. <code>preStop</code> hook is executed immediately before a container is terminated. 
How it works is when container needs to be terminated, Kubelet will run the pre-stop hook and only then send <code>SIGTERM</code> to the process. This can be used to initate a graceful shutdown of the container. </p>

<p>It can be also used to perform some other operations before shutdown without having to implement those in the app itself. </p>

<p>This is an simple example how it works (It is a <code>HTTP GET</code> request that will be sent to `http:///shutdown): </p>

<pre><code>lifecycle:
   preStop:
      httpGet:
         port: 80
         path: shutdown
</code></pre>

<p>Here is also link to <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/" rel="nofollow noreferrer">kubernetes documentation</a> about <code>Container hooks</code>. Let me know if this was heplful. </p>
