<p>I have a kops cluster with a max of 75 nodes and added  with <a href="https://github.com/kubernetes/kops/tree/master/addons/cluster-autoscaler" rel="nofollow noreferrer">cluster autoscaler</a>. It uses <a href="https://github.com/kubernetes/kops/blob/master/docs/networking.md#kops-default-networking" rel="nofollow noreferrer">kubenet</a> networking. 
Things have currently stopped working - ie scale down is no longer happening.</p>

<p><strong>The cluster is running on max capacity ie 75 nodes even with almost no load. Not sure where to start to troubleshoot the problem.</strong></p>

<p>See the following errors in the cluster autoscaler pod</p>

<pre><code>    I0222 01:45:14.327164       1 static_autoscaler.go:97] Starting main loop
W0222 01:45:14.770818       1 static_autoscaler.go:150] Cluster is not ready for autoscaling
I0222 01:45:15.043126       1 leaderelection.go:199] successfully renewed lease kube-system/cluster-autoscaler
I0222 01:45:17.121507       1 leaderelection.go:199] successfully renewed lease kube-system/cluster-autoscaler
I0222 01:45:19.126665       1 leaderelection.go:199] successfully renewed lease kube-system/cluster-autoscaler
I0222 01:45:21.327581       1 leaderelection.go:199] successfully renewed lease kube-system/cluster-autoscaler
I0222 01:45:23.331802       1 leaderelection.go:199] successfully renewed lease kube-system/cluster-autoscaler
I0222 01:45:24.775124       1 static_autoscaler.go:97] Starting main loop
W0222 01:45:25.085442       1 static_autoscaler.go:150] Cluster is not ready for autoscaling
</code></pre>

<p>Autoscaling was working fine.</p>

<p>Update, also see the following errors when running <code>kops validate cluster</code></p>

<pre><code>    VALIDATION ERRORS
    KIND    NAME                MESSAGE
    Node    ip-172-20-32-173.ec2.internal   node "ip-172-20-32-173.ec2.internal" is not ready
 ...

I0221 22:16:02.688911    2403 node_conditions.go:60] node "ip-172-20-51-238.ec2.internal" not ready: &amp;NodeCondition{Type:NetworkUnavailable,Status:True,LastHeartbeatTime:2019-02-21 22:15:56 -0500 EST,LastTransitionTime:2019-02-21 22:15:56 -0500 EST,Reason:NoRouteCreated,Message:RouteController failed to create a route,}
</code></pre>

smk
<p>I found out the problem was that my Cluster had gone into an Unhealthy state because of <a href="https://github.com/kubernetes/kops/blob/master/docs/networking.md#kops-default-networking" rel="nofollow noreferrer">this limitation</a> in AWS VPC routing tables.My cluster had scaled to 75 nodes and then had become unhealthy and was not able to scale down.</p>

<p>From the link, </p>

<blockquote>
  <p>One important limitation when using kubenet networking is that an AWS routing table cannot have more than 50 entries, which sets a limit of 50 nodes per cluster. </p>
</blockquote>
