<p>I built a service that utilizes docker pods to process data.  The time it takes varies from as little as 15 minutes to as much as 1 hour.</p>
<p>My applications captures SIGTERM to ensure a graceful shutdown takes place when demand drops while Pods and Nodes are decommissioned.</p>
<p>In each docker image I placed code to report back if it shutdown because it completed the work and if a SIGTERM event took place and thus completed its processing and terminated.</p>
<p>My system is deployed in AWS using EKS.  I use EKS to manage node deployment when demand goes up and spindown nodes when demand drops.  I use KEDA to manage POD deployment which is what helps trigger whether additional nodes are needed or not.  In KEDA I have the cooldownPeriod defined for 2 hours the maximum I expect a pod to take even though the max it would ever take is 1 hour.</p>
<p>In AWS EKS, I have defined the terminationGracePeriodSeconds for 2 hours as well.</p>
<p>I isolated the issue during Node scale down that when nodes are being terminated, the terminationGracePeriodSeconds is not being honored and my Pods are being shutdown within ~30 minutes.  Because the Pods are abruptly removed I am unable to look at their logs to see what happened.</p>
<p>I tried to simulate this issue by issuing a kubernetes node drain and kept my pod running</p>
<pre><code>kubectl drain &lt;MY NODE&gt;
</code></pre>
<p>I saw the SIGTERM come through, and I also noticed that the pod was only terminated after 2 hours and not before.</p>
<p>So for a brief minute I thought maybe I did not configure the terminationGracePeriod properly, so I checked:</p>
<pre><code>kubectl get deployment test-mypod -o yaml|grep terminationGracePeriodSeconds
  terminationGracePeriodSeconds: 7200
</code></pre>
<p>I even redeployed the config but that made no difference.</p>
<p>However, I was able to reproduce the issue by modifying the desiredSize of the Node group.  I can reproduce it programmatically in Python by doing this:</p>
<pre><code>        resp = self.eks_client.update_nodegroup_config(clusterName=EKS_CLUSTER_NAME,
                                                       nodegroupName=EKS_NODE_GROUP_NAME,
                                                       scalingConfig={'desiredSize': configured_desired_size})
</code></pre>
<p>or by simply going to AWS console and modifying the desiredSize there.</p>
<p>I see EKS choosing a node and if it happens that there is a pod processing data that will take about an hour, the pod is sometimes prematurely terminated.</p>
<p>I have logged on to that node that is being scaled down and found no evidence of the prematurely terminated Pod in the logs.</p>
<p>I was able to capture this information once</p>
<pre><code>kubectl get events | grep test-mypod-b8dfc4665-zp87t
54m         Normal    Pulling    pod/test-mypod-b8dfc4665-zp87t         Pulling image ...
54m         Normal    Pulled     pod/test-mypod-b8dfc4665-zp87t         Successfully pulled image ...
54m         Normal    Created    pod/test-mypod-b8dfc4665-zp87t         Created container mypod
54m         Normal    Started    pod/test-mypod-b8dfc4665-zp87t         Started container mypod
23m         Normal    ScaleDown  pod/test-mypod-b8dfc4665-zp87t         deleting pod for node scale down
23m         Normal    Killing    pod/test-mypod-b8dfc4665-zp87t         Stopping container mypod
13m         Warning   FailedKillPod   pod/test-po-b8dfc4665-zp87t       error killing pod: failed to &quot;KillContainer&quot; for &quot;mypod&quot; with KillContainerError: &quot;rpc error: code = Unknown desc = operation timeout: context deadline exceeded&quot;
</code></pre>
<p>I once saw a pod removed for no reason as such when scaledown was disabled but it decided to remove my pod:</p>
<pre><code>kubectl get events | grep test-mypod-b8dfc4665-vxqhv
45m         Normal    Pulling    pod/test-mypod-b8dfc4665-vxqhv Pulling image ...
45m         Normal    Pulled     pod/test-mypod-b8dfc4665-vxqhv Successfully pulled image ...
45m         Normal    Created    pod/test-mypod-b8dfc4665-vxqhv Created container mypod
45m         Normal    Started    pod/test-mypod-b8dfc4665-vxqhv Started container mypod
40m         Normal    Killing    pod/test-mypod-b8dfc4665-vxqhv Stopping container mypod
</code></pre>
<p>This is the kuberenets version I have</p>
<pre><code>Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;18&quot;, GitVersion:&quot;v1.18.0&quot; GitCommit:&quot;9e991415386e4cf155a24b1da15becaa390438d8&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2020-03-25T14:58:59Z&quot;, GoVersion:&quot;go1.13.8&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;18+&quot;, GitVersion:&quot;v1.18.20-eks-8c49e2&quot;, GitCommit:&quot;8c49e2efc3cfbb7788a58025e679787daed22018&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-10-17T05:13:46Z&quot;, GoVersion:&quot;go1.13.15&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<p>To minimize this issue, I deployed a Pod Disruption Budget during peak hours to block scale down and in the evening during low demand I remove the PDB which initiates the scaledown. However, that is not the right solution and even during low peak there are still pods that get stopped prematurely.</p>

<p>We faced the same issue with AWS EKS and cluster-autoscaler - nodes were unexpectedly shut down, no preventive actions were working, and even the <strong>node</strong> annotation <code>cluster-autoscaler.kubernetes.io/scale-down-disabled=true</code> did not make any difference.</p>
<p>After two days of troubleshooting, we found the reason - it was because we use Multiple Availability Zone in ASG configuration, which has an automatic &quot;AZRebalance&quot; process. The AZRebalance tries to ensure that the number of nodes is approximately the same between all availability zones. Therefore, sometimes when the scale-up event occurs, it tries to rebalance nodes by killing one node and creating another in a different time zone. The message in the events log looks like this:</p>
<p><a href="https://i.stack.imgur.com/EKvJS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EKvJS.png" alt="enter image description here" /></a></p>
<p>Cluster-autoscaler does not control this process, so there are two systems (cluster-autoscaler and AWS ASG) that manage the number of nodes simultaneously, which leads to unexpected behavior.</p>
<p>As a workaround, we suspended the &quot;AZRebalance&quot; process in the ASG.
<a href="https://i.stack.imgur.com/h4h5a.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/h4h5a.png" alt="enter image description here" /></a></p>
<p>Another solution would be to use ASG for each availability zone separately and use <code>--balance-similar-node-groups</code> <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler" rel="nofollow noreferrer">feature</a> in the cluster-autoscaler.</p>
<p><a href="https://faun.pub/aws-eks-and-cluster-autoscaler-gotchas-97efa48c01c4" rel="nofollow noreferrer">Here's</a> the article about that  and <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler" rel="nofollow noreferrer">here's</a> the cluster-autoscaler documentation.</p>
