<p>I wonder if someone can help me.</p>
<p>Kubernetes (K8s 1.21 platform eks.4) is Terminating running pods without error or reason. The only thing I can see in the events is:</p>
<pre><code>7m47s       Normal    Killing                   pod/test-job-6c9fn-qbzkb                          Stopping container test-job
</code></pre>
<p>Because I've set up an anti-affinity rule, only one pod can run in one node. So every time a pod gets killed, autoscaler brings up another node.</p>
<p>These are the cluster-autoscaler logs</p>
<pre><code>I0208 19:10:42.336476       1 cluster.go:148] Fast evaluation: ip-10-4-127-38.us-west-2.compute.internal for removal
I0208 19:10:42.336484       1 cluster.go:169] Fast evaluation: node ip-10-4-127-38.us-west-2.compute.internal cannot be removed: pod annotated as not safe to evict present: test-job-6c9fn-qbzkb
I0208 19:10:42.336493       1 scale_down.go:612] 1 nodes found to be unremovable in simulation, will re-check them at 2022-02-08 19:15:42.335305238 +0000 UTC m=+20363.008486077

I0208 19:15:04.360683       1 klogx.go:86] Pod default/test-job-6c9fn-8wx2q is unschedulable
I0208 19:15:04.360719       1 scale_up.go:376] Upcoming 0 nodes
I0208 19:15:04.360861       1 scale_up.go:300] Pod test-job-6c9fn-8wx2q can't be scheduled on eks-ec2-8xlarge-84bf6ad9-ca4a-4293-a3e8-95bef28db16d, predicate checking error: node(s) didn't match Pod's node affinity/selector; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity/selector; debugInfo=
I0208 19:15:04.360901       1 scale_up.go:449] No pod can fit to eks-ec2-8xlarge-84bf6ad9-ca4a-4293-a3e8-95bef28db16d
I0208 19:15:04.361035       1 scale_up.go:300] Pod test-job-6c9fn-8wx2q can't be scheduled on eks-ec2-inf1-90bf6ad9-caf7-74e8-c930-b80f785bc743, predicate checking error: node(s) didn't match Pod's node affinity/selector; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity/selector; debugInfo=
I0208 19:15:04.361062       1 scale_up.go:449] No pod can fit to eks-ec2-inf1-90bf6ad9-caf7-74e8-c930-b80f785bc743
I0208 19:15:04.361162       1 scale_up.go:300] Pod test-job-6c9fn-8wx2q can't be scheduled on eks-ec2-large-62bf6ad9-ccd4-6e03-5c78-c3366d387d50, predicate checking error: node(s) didn't match Pod's node affinity/selector; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity/selector; debugInfo=
I0208 19:15:04.361194       1 scale_up.go:449] No pod can fit to eks-ec2-large-62bf6ad9-ccd4-6e03-5c78-c3366d387d50
I0208 19:15:04.361512       1 scale_up.go:412] Skipping node group eks-eks-on-demand-10bf6ad9-c978-9b35-c7fc-cdb9977b27cb - max size reached
I0208 19:15:04.361675       1 scale_up.go:300] Pod test-job-6c9fn-8wx2q can't be scheduled on eks-ec2-test-58bf6d43-13e8-9acc-5173-b8c5054a56da, predicate checking error: node(s) didn't match Pod's node affinity/selector; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity/selector; debugInfo=
I0208 19:15:04.361711       1 scale_up.go:449] No pod can fit to eks-ec2-test-58bf6d43-13e8-9acc-5173-b8c5054a56da
I0208 19:15:04.361723       1 waste.go:57] Expanding Node Group eks-ec2-xlarge-84bf6ad9-cb6d-7e24-7eb5-a00c369fd82f would waste 75.00% CPU, 86.92% Memory, 80.96% Blended
I0208 19:15:04.361747       1 scale_up.go:468] Best option to resize: eks-ec2-xlarge-84bf6ad9-cb6d-7e24-7eb5-a00c369fd82f
I0208 19:15:04.361762       1 scale_up.go:472] Estimated 1 nodes needed in eks-ec2-xlarge-84bf6ad9-cb6d-7e24-7eb5-a00c369fd82f
I0208 19:15:04.361780       1 scale_up.go:586] Final scale-up plan: [{eks-ec2-xlarge-84bf6ad9-cb6d-7e24-7eb5-a00c369fd82f 0-&gt;1 (max: 2)}]
I0208 19:15:04.361801       1 scale_up.go:675] Scale-up: setting group eks-ec2-xlarge-84bf6ad9-cb6d-7e24-7eb5-a00c369fd82f size to 1
I0208 19:15:04.361826       1 auto_scaling_groups.go:219] Setting asg eks-ec2-xlarge-84bf6ad9-cb6d-7e24-7eb5-a00c369fd82f size to 1
I0208 19:15:04.362154       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;kube-system&quot;, Name:&quot;cluster-autoscaler-status&quot;, UID:&quot;81b80048-920c-4bf1-b2c0-ad5d067d74f4&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;359476&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: setting group eks-ec2-xlarge-84bf6ad9-cb6d-7e24-7eb5-a00c369fd82f size to 1
I0208 19:15:04.374021       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;kube-system&quot;, Name:&quot;cluster-autoscaler-status&quot;, UID:&quot;81b80048-920c-4bf1-b2c0-ad5d067d74f4&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;359476&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: setting group eks-ec2-xlarge-84bf6ad9-cb6d-7e24-7eb5-a00c369fd82f size to 1
I0208 19:15:04.541658       1 eventing_scale_up_processor.go:47] Skipping event processing for unschedulable pods since there is a ScaleUp attempt this loop
I0208 19:15:04.541859       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:&quot;Pod&quot;, Namespace:&quot;default&quot;, Name:&quot;test-job-6c9fn-8wx2q&quot;, UID:&quot;67beba1d-4f52-4860-91af-89e5852e4cad&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;359507&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'TriggeredScaleUp' pod triggered scale-up: [{eks-ec2-xlarge-84bf6ad9-cb6d-7e24-7eb5-a00c369fd82f 0-&gt;1 (max: 2)}]
</code></pre>
<p>I'm running an EKS cluster with cluster-autoscaler and keda's aws-sqs trigger. I've set up an autoscaling node group with SPOT instances.</p>
<p>For testing purposes I've defined an ScaledJob consisting on a container with a simple python script, looping through time.sleep. The pod should run for 30 mins. But it never gets so far. In general it ends after 15 mins.</p>
<pre><code>{
            &quot;apiVersion&quot;: &quot;keda.sh/v1alpha1&quot;,
            &quot;kind&quot;: &quot;ScaledJob&quot;,
            &quot;metadata&quot;: {
                &quot;name&quot;: id,
                &quot;labels&quot;: {&quot;myjobidentifier&quot;: id},
                &quot;annotations&quot;: {
                    &quot;cluster-autoscaler.kubernetes.io/safe-to-evict&quot;: &quot;false&quot;
                },
            },
            &quot;spec&quot;: {
                &quot;jobTargetRef&quot;: {
                    &quot;parallelism&quot;: 1,
                    &quot;completions&quot;: 1,
                    &quot;backoffLimit&quot;: 0,
                    &quot;template&quot;: {
                        &quot;metadata&quot;: {
                            &quot;labels&quot;: {&quot;job-type&quot;: id},
                            &quot;annotations&quot;: {
                                &quot;cluster-autoscaler.kubernetes.io/safe-to-evict&quot;: &quot;false&quot;
                            },
                        },
                        &quot;spec&quot;: {
                            &quot;affinity&quot;: {
                                &quot;nodeAffinity&quot;: {
                                    &quot;requiredDuringSchedulingIgnoredDuringExecution&quot;: {
                                        &quot;nodeSelectorTerms&quot;: [
                                            {
                                                &quot;matchExpressions&quot;: [
                                                    {
                                                        &quot;key&quot;: &quot;eks.amazonaws.com/nodegroup&quot;,
                                                        &quot;operator&quot;: &quot;In&quot;,
                                                        &quot;values&quot;: group_size,
                                                    }
                                                ]
                                            }
                                        ]
                                    }
                                },
                                &quot;podAntiAffinity&quot;: {
                                    &quot;requiredDuringSchedulingIgnoredDuringExecution&quot;: [
                                        {
                                            &quot;labelSelector&quot;: {
                                                &quot;matchExpressions&quot;: [
                                                    {
                                                        &quot;key&quot;: &quot;job-type&quot;,
                                                        &quot;operator&quot;: &quot;In&quot;,
                                                        &quot;values&quot;: [id],
                                                    }
                                                ]
                                            },
                                            &quot;topologyKey&quot;: &quot;kubernetes.io/hostname&quot;,
                                        }
                                    ]
                                },
                            },
                            &quot;serviceAccountName&quot;: service_account.service_account_name,
                            &quot;containers&quot;: [
                                {
                                    &quot;name&quot;: id,
                                    &quot;image&quot;: image.image_uri,
                                    &quot;imagePullPolicy&quot;: &quot;IfNotPresent&quot;,
                                    &quot;env&quot;: envs,
                                    &quot;resources&quot;: {
                                        &quot;requests&quot;: requests,
                                    },
                                    &quot;volumeMounts&quot;: [
                                        {
                                            &quot;mountPath&quot;: &quot;/tmp&quot;,
                                            &quot;name&quot;: &quot;tmp-volume&quot;,
                                        }
                                    ],
                                }
                            ],
                            &quot;volumes&quot;: [
                                {&quot;name&quot;: &quot;tmp-volume&quot;, &quot;emptyDir&quot;: {}}
                            ],
                            &quot;restartPolicy&quot;: &quot;Never&quot;,
                        },
                    },
                },
                &quot;pollingInterval&quot;: 30,
                &quot;successfulJobsHistoryLimit&quot;: 10,
                &quot;failedJobsHistoryLimit&quot;: 100,
                &quot;maxReplicaCount&quot;: 30,
                &quot;rolloutStrategy&quot;: &quot;default&quot;,
                &quot;scalingStrategy&quot;: {&quot;strategy&quot;: &quot;default&quot;},
                &quot;triggers&quot;: [
                    {
                        &quot;type&quot;: &quot;aws-sqs-queue&quot;,
                        &quot;metadata&quot;: {
                            &quot;queueURL&quot;: queue.queue_url,
                            &quot;queueLength&quot;: &quot;1&quot;,
                            &quot;awsRegion&quot;: region,
                            &quot;identityOwner&quot;: &quot;operator&quot;,
                        },
                    }
                ],
            },
        }
</code></pre>
<p>I know this is not a problem of resources (dummy code and large instances), nor a problem of eviction (it's clear from the logs that the pod is safe from eviction), but I really don't know how to troubleshoot this anymore.</p>
<p>thanks a lot!!</p>
<p>EDIT:</p>
<p>Same behavior with On-Demand and SPOT instances.</p>
<p>EDIT 2:</p>
<p>I added the aws node termination handler, it seems that now I'm seeing other events:</p>
<pre><code>ip-10-4-126-234.us-west-2.compute.internal.16d223107de38c5f
NodeNotSchedulable
Node ip-10-4-126-234.us-west-2.compute.internal status is now: NodeNotSchedulable

test-job-p85f2-txflr.16d2230ea91217a9
FailedScheduling
0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) were unschedulable.
</code></pre>
<p>If I check the scaling group activity:</p>
<pre><code>Instance i-03d27a1cf341405e1 was taken out of service in response to a user request, shrinking the capacity from 1 to 0.
</code></pre>

Raúl
<p>I also have a similar problem because of HPA's scale-in.</p>
<p>When you don't write the <strong>minReplicaCount</strong> value, the value is set to 0 as default. Then, the pod is terminated because of the HPA's scale-in.</p>
<p>I recommend you should set the <strong>minReplicaCount</strong> value that you want (e.g. 1).</p>
