<p>Original problem. I would like to have a Kubernetes cluster with at least 2 nodes with zero GPU consumption. If a job is coming and takes one node, then autoscaler should create another spare node.</p>
<p>I found out that I can rely on <code>DCGM_FI_DEV_GPU_UTIL</code> metrics. If <code>DCGM_FI_DEV_GPU_UTIL == 0</code> then the node is in &quot;idle&quot; mode. In PromQL I can just write <code>count(DCGM_FI_DEV_GPU_UTIL == 0)</code> and get the number of &quot;idle&quot; nodes.</p>
<p>However, I do not understand how to write metricsQuery in Prometheus Adapter config. All examples that I found are about</p>
<pre><code>(sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;)
</code></pre>
<p>However, I need something like <code>count(&lt;&lt;.Series&gt;&gt; == 0)</code>, but this does not work. Any idea how I can get this metrics for HPA  which indicates the number of nodes with no GPU consumption?</p>

Trarbish
<p>Probably your jobs are running in Kubernetes Pod. You may have a configuration where only one custom Pod with job can run on a single Node. The first step is to configure your metrics for the Prometheus adapter and it's described quite nicely <a href="https://blog.wyrihaximus.net/2021/01/scaling-php-fpm-based-on-utilization-demand-on-kubernetes/" rel="nofollow noreferrer">here</a>. This step will ensure that the Pod is added.</p>
<p>In the second step you need to configure a cluster autoscaler that will add another Node when needed. Cluster autoscaler is dependent on your Kubernetes solution provider (AWS, Azure, GCP...) and should be in their documentation. I personally use <a href="https://www.kubecost.com/kubernetes-autoscaling/kubernetes-cluster-autoscaler/" rel="nofollow noreferrer">Cluster autoscaler</a>, <a href="https://karpenter.sh/" rel="nofollow noreferrer">Karpenter</a>.</p>
