<p>I have created a k8s cluster with kops (1.21.4) on AWS and as per the <a href="https://kops.sigs.k8s.io/addons/#cluster-autoscaler" rel="nofollow noreferrer">docs on autoscaler</a>. I have done the required changes to my cluster but when the cluster starts, the cluster-autoscaler pod is unable to schedule on any node. When I describe the pod, I see the following:</p>
<pre><code>Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  4m31s (x92 over 98m)  default-scheduler  0/4 nodes are available: 1 Too many pods, 3 node(s) didn't match Pod's node affinity/selector.
</code></pre>
<p>Looking at the deployment for cluster I see the following <code>podAntiAffinity</code>:</p>
<pre><code>      affinity:                                                                 
        podAntiAffinity:                                                        
          preferredDuringSchedulingIgnoredDuringExecution:                      
          - podAffinityTerm:                                                    
              labelSelector:                                                    
                matchExpressions:                                               
                - key: app                                                      
                  operator: In                                                  
                  values:                                                       
                  - cluster-autoscaler                                          
              topologyKey: topology.kubernetes.io/zone                          
            weight: 100                                                         
          requiredDuringSchedulingIgnoredDuringExecution:                       
          - labelSelector:                                                      
              matchExpressions:                                                 
              - key: app                                                        
                operator: In                                                    
                values:                                                         
                - cluster-autoscaler                                            
            topologyKey: kubernetes.com/hostname
</code></pre>
<p>From this I understand that it want to prevent running pod on same node which already has cluster-autoscaler running. But that doesn't seem to justify the error seen in the pod status.</p>
<p>Edit: The pod for autoscaler has the following <code>nodeSelectors</code> and <code>tolerations</code>:</p>
<pre><code>Node-Selectors:              node-role.kubernetes.io/master=
Tolerations:                 node-role.kubernetes.io/master op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
</code></pre>
<p>So clearly, it should be able to schedule on master node too.</p>
<p>I am not sure what else do I need to do to make the pod up and running.</p>

Divick
<p>Posting the answer out of comments.</p>
<hr />
<p>There are <code>podAffinity</code> rules in place so first thing to check is if any errors in scheduling are presented. Which is the case:</p>
<pre><code>0/4 nodes are available: 1 Too many pods, 3 node(s) didn't match Pod's node affinity/selector.
</code></pre>
<p>Since there are 1 control plane (on which pod is supposed to be scheduled) and 3 worked nodes, that leads to the error <code>1 Too many pods</code> related to the control plane.</p>
<hr />
<p>Since cluster is running in AWS, there's a known limitation about amount of <code>network interfaces</code> and <code>private IP addresses</code> per machine type - <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI" rel="nofollow noreferrer">IP addresses per network interface per instance type</a>.</p>
<p><code>t3.small</code> was used which has 3 interfaces and 4 IPs per interface = 12 in total which was not enough.</p>
<p>Scaling up to <code>t3.medium</code> resolved the issue.</p>
<hr />
<p>Credits to <a href="https://stackoverflow.com/a/64972286/15537201">Jonas's answer</a> about the root cause.</p>
