<p>When I resize a replication controller using kubectl, if the cluster does not have enough resource, there will have one or more pods always in pending.</p>

<p>Is there has any tool will auto resize GKE cluster when the resource is running out?</p>

<p>I had a similar requirement (for the Go build system): wanted to know when scheduled vs. available CPU or memory was > 1, and scale out nodes when that was true (or, more accurately, when it was ~.8). There's not a built-in metric, but as you suggest you can do it with a custom metric.</p>

<p>This was all done in Go, but it will give you the basic idea:</p>

<ol>
<li><a href="https://github.com/golang/build/blob/c8b96fa9faa2d24f902a6b0b3f36979d5d0a2a4f/cmd/coordinator/kube.go#L158-L202" rel="nofollow">Create the metrics (memory and CPU, in my case</a></li>
<li><a href="https://github.com/golang/build/blob/c8b96fa9faa2d24f902a6b0b3f36979d5d0a2a4f/cmd/coordinator/kube.go#L227-L292" rel="nofollow">Put values to the metrics</a></li>
</ol>

<p>The key takeaway IMO is that you have to iterate over each <em>pod</em> in the cluster to determine how much capacity is consumed, then iterate over each <em>node</em> in the cluster to determine how much capacity is available. It's then just a matter of pointing your autoscaler to the custom metric(s). </p>

<p>Big big big thing worth noting: I ultimately determined that scaling on the built-in CPU utilization metric was just as good as (if not better than, but more on that in a bit) than the custom metric. Each pod we scheduled pegged the CPU, so when pods were maxed out so was CPU. The build-in CPU utilization metric is probably better because you don't have the latency that comes with periodically putting custom metrics.</p>
