<p>I have a regional cluster set up in <strong>google kubernetes engine (GKE)</strong>. The node group is a single <strong>vm in each region (3 total)</strong>. I have a deployment with <strong>3 replicas minimum</strong> controlled by a HPA.
The <strong>nodegroup is configured to be autoscaling</strong> (cluster autoscaling aka CA).
The problem scenario:</p>

<p>Update deployment image. Kubernetes automatically creates new pods and the CA identifies that a new node is needed. I now have 4.
The old pods get removed when all new pods have started, which means I have the exact same CPU request as the minute before. But the after the 10 min maximum downscale time I still have 4 nodes.</p>

<p>The CPU requests for the nodes is now:</p>

<pre><code>CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ------------  ----------  ---------------  -------------
  358m (38%)    138m (14%)  516896Ki (19%)   609056Ki (22%)
--
  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ------------  ----------  ---------------  -------------
  800m (85%)    0 (0%)      200Mi (7%)       300Mi (11%)
--
  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ------------  ----------  ---------------  -------------
  510m (54%)    100m (10%)  410Mi (15%)      770Mi (29%)
--
  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ------------  ----------  ---------------  -------------
  823m (87%)    158m (16%)  484Mi (18%)      894Mi (33%)
</code></pre>

<p>The 38% node is running:</p>

<pre><code>Namespace                  Name                                                            CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                                            ------------  ----------  ---------------  -------------
  kube-system                event-exporter-v0.1.9-5c8fb98cdb-8v48h                          0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                fluentd-gcp-v2.0.17-q29t2                                       100m (10%)    0 (0%)      200Mi (7%)       300Mi (11%)
  kube-system                heapster-v1.5.2-585f569d7f-886xx                                138m (14%)    138m (14%)  301856Ki (11%)   301856Ki (11%)
  kube-system                kube-dns-autoscaler-69c5cbdcdd-rk7sd                            20m (2%)      0 (0%)      10Mi (0%)        0 (0%)
  kube-system                kube-proxy-gke-production-cluster-default-pool-0fd62aac-7kls    100m (10%)    0 (0%)      0 (0%)           0 (0%)
</code></pre>

<p>I suspect it wont downscale because heapster or kube-dns-autoscaler.
But the 85% pod contains:</p>

<pre><code>Namespace                  Name                                                            CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                                            ------------  ----------  ---------------  -------------
  kube-system                fluentd-gcp-v2.0.17-s25bk                                       100m (10%)    0 (0%)      200Mi (7%)       300Mi (11%)
  kube-system                kube-proxy-gke-production-cluster-default-pool-7ffeacff-mh6p    100m (10%)    0 (0%)      0 (0%)           0 (0%)
  my-deploy                  my-deploy-54fc6b67cf-7nklb                                      300m (31%)    0 (0%)      0 (0%)           0 (0%)
  my-deploy                  my-deploy-54fc6b67cf-zl7mr                                      300m (31%)    0 (0%)      0 (0%)           0 (0%)
</code></pre>

<p>The fluentd and kube-proxy pods are present on every node, so I assume they are not needed without the node. Which means that my deployment could be relocated to the other nodes since it only has a request of 300m (31% since only 94% of node CPU is allocatable).</p>

<p>So I figured that Ill check the logs. But if I run <code>kubectl get pods --all-namespaces</code> there are no pod visible on GKE for the CA. And if I use the command <code>kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml</code> it only tells me if it is about to scale, not why or why not.
Another option is to look at <code>/var/log/cluster-autoscaler.log</code> in the master node. I SSH:ed in the all 4 nodes and only found a <code>gcp-cluster-autoscaler.log.pos</code> file that says: <code>/var/log/cluster-autoscaler.log 0000000000000000        0000000000000000</code> meaning the file should be right there but is empty.
Last option according to the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md" rel="noreferrer">FAQ</a>, is to check the events for the pods, but as far as i can tell they are empty.</p>

<p>Anyone know why it wont downscale or atleast where to find the logs?</p>

<p>Answering myself for visibility.</p>

<p>The problem is that the CA never considers moving anything unless all the requirements mentioned in the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md" rel="noreferrer">FAQ</a> are met at the same time.
So lets say I have 100 nodes with 51% CPU requests. It still wont consider downscaling.</p>

<p>One solution is to increase the value at which CA checks, now 50%. But unfortunately that is not supported by GKE, see answer from google support @GalloCedrone:</p>

<blockquote>
  <p>Moreover I know that this value might sound too low and someone could be interested to keep as well a 85% or 90% to avoid your scenario.
  Currently there is a feature request open to give the user the possibility to modify the flag "--scale-down-utilization-threshold", but it is not implemented yet.</p>
</blockquote>

<p>The workaround I found is to decrease the CPU request (100m instead of 300m) of the pods and have the Horizontal Pod Autoscaler (HPA) create more on demand. This is fine for me but if your application is not suitable for many small instances you are out of luck. Perhaps a cron job that cordons a node if the total utilization is low?</p>
