<h2>Problem</h2>
<p>I am trying to implement a Horizontal Pod Autoscaler (HPA) on my AKS cluster. However, I'm unable to retrieve the GPU metrics (auto-generated by Azure) that my HPA requires to scale.</p>
<h2>Example</h2>
<p>As a reference, see <a href="https://learn.microsoft.com/en-us/azure/aks/tutorial-kubernetes-scale#autoscale-pods" rel="nofollow noreferrer">this example</a> where the HPA scales based on <code>targetCPUUtilizationPercentage: 50</code>. That is, the HPA will deploy more/less pods to achieve a target of an average CPU utilization across all pods. Ideally, I want to achieve the same with the GPU.</p>
<h2>Setup</h2>
<p>I have deployed an AKS cluster with Azure Monitor enabled and my node size set to <code>Standard_NC6_Promo</code> - Azure's VM option that comes equipped with Nvidia's Tesla K80 GPU. However, in order to utilize the GPU, you must first install the appropriate plugin into your cluster, as explained <a href="https://learn.microsoft.com/en-us/azure/aks/gpu-cluster" rel="nofollow noreferrer">here</a>. Once you install this plugin a number of GPU metrics are automatically collected by Azure and logged to a table named &quot;InsightsMetrics&quot; (<a href="https://learn.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-gpu-monitoring" rel="nofollow noreferrer">see</a>). From what I can read, the metric <code>containerGpuDutyCycle</code> will be the most beneficial for monitoring GPU utilization.</p>
<h2>Current Situation</h2>
<p>I can successfully see the insight metrics gathered by installed plugin, where one of the metrics is <code>containerGpuDutyCycle</code>.</p>
<p><a href="https://i.stack.imgur.com/UONW7.png" rel="nofollow noreferrer">InsightsMetrics table inside of Logs tab of Kubernetes Service on Azure Portal</a></p>
<p>Now how to expose/provide this metric to my HPA?</p>
<h2>Possible Solutions</h2>
<p>What I've noticed is that if you navigate to the <strong>Metrics</strong> tab of your AKS cluster, you cannot retrieve these GPU metrics. I assume this is because these GPU &quot;metrics&quot; are technically logs and not &quot;official&quot; metrics. However, azure does support something called <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/platform/app-insights-metrics" rel="nofollow noreferrer">log-based metrics</a>, where the results of log queries can be treated as an &quot;official&quot; metric, but nowhere do I see how to create my own custom log-based metric.</p>
<p>Furthermore, <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-metrics-apis" rel="nofollow noreferrer">Kubernetes supports custom and external metrics</a> through their Metrics API, where metrics can be retrieved from external sources (such as Azure's Application Insights). Azure has an implementation of the Metrics API called <a href="https://github.com/Azure/azure-k8s-metrics-adapter" rel="nofollow noreferrer">Azure Kubernetes Metrics Adapter</a>. Perhaps I need to expose the <code>containerGpuDutyCycle</code> metric as an external metric using this? If so, how do I reference/expose the metric as external/custom?</p>
<h2>Alternative Solutions</h2>
<p>My main concern is exposing the GPU metrics for my HPA. I'm using Azure's Kubernetes Metrics Adapter for now as I assumed it would better integrate into my AKS cluster (same eco-system). However, it's in alpha stage (not production ready). If anyone can solve my problem using an alternative metric adapter (e.g. <a href="https://prometheus.io/" rel="nofollow noreferrer">Prometheus</a>), that would still be very helpful.</p>
<p>Many thanks for any light you can shed on this issue.</p>

<p>I managed to do this recently (just this week). I'll outline my solution and all the gotchas, in case that helps.</p>
<p>Starting with an AKS cluster, I installed the following components in order to harvest the GPU metrics:</p>
<ol>
<li>nvidia-device-plugin - to make GPU metrics collectable</li>
<li>dcgm-exporter - a daemonset to reveal GPU metrics on each node</li>
<li>kube-prometheus-stack - to harvest the GPU metrics and store them</li>
<li>prometheus-adapter - to make harvested, stored metrics available to the k8s metrics server</li>
</ol>
<p>The AKS cluster comes with a metrics server built in, so you don't need to worry about that. It is also possible to provision the cluster with the nvidia-device-plugin already applied, but currently not possible via terraform (<a href="https://stackoverflow.com/questions/66117018/is-it-possible-to-use-aks-custom-headers-with-the-azurerm-kubernetes-cluster-res">Is it possible to use aks custom headers with the azurerm_kubernetes_cluster resource?</a>), which is how I was deploying my cluster.</p>
<p>To install all this stuff I used a script much like the following:</p>
<pre><code>helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add gpu-helm-charts https://nvidia.github.io/gpu-monitoring-tools/helm-charts

helm repo update

echo &quot;Installing the NVIDIA device plugin...&quot;
helm install nvdp/nvidia-device-plugin \
--generate-name \
--set migStrategy=mixed \
--version=0.9.0

echo &quot;Installing the Prometheus/Grafana stack...&quot;
helm install prometheus-community/kube-prometheus-stack \
--create-namespace --namespace prometheus \
--generate-name \
--values ./kube-prometheus-stack.values

prometheus_service=$(kubectl get svc -nprometheus -lapp=kube-prometheus-stack-prometheus -ojsonpath='{range .items[*]}{.metadata.name}{&quot;\n&quot;}{end}')
helm install prometheus-adapter prometheus-community/prometheus-adapter \
--namespace prometheus \
--set rbac.create=true,prometheus.url=http://${prometheus_service}.prometheus.svc.cluster.local,prometheus.port=9090

helm install gpu-helm-charts/dcgm-exporter \
--generate-name
</code></pre>
<p>Actually, I'm lying about the <code>dcgm-exporter</code>. I was experiencing a problem (my first &quot;gotcha&quot;) where the <code>dcgm-exporter</code> was not responding to liveness requests in time, and was consistently entering a <code>CrashLoopBackoff</code> status (<a href="https://github.com/NVIDIA/gpu-monitoring-tools/issues/120" rel="nofollow noreferrer">https://github.com/NVIDIA/gpu-monitoring-tools/issues/120</a>). To get around this, I created my own <code>dcgm-exporter</code> k8s config (by taking details from here and modifying them slightly: <a href="https://github.com/NVIDIA/gpu-monitoring-tools" rel="nofollow noreferrer">https://github.com/NVIDIA/gpu-monitoring-tools</a>) and applied it.
In doing this I experienced my second &quot;gotcha&quot;, which was that in the latest <code>dcgm-exporter</code> images they have removed some GPU metrics, such as <code>DCGM_FI_DEV_GPU_UTIL</code>, largely because these metrics are resource intensive to collect (see <a href="https://github.com/NVIDIA/gpu-monitoring-tools/issues/143" rel="nofollow noreferrer">https://github.com/NVIDIA/gpu-monitoring-tools/issues/143</a>). If you want to re-enable them make sure you run the <code>dcgm-exporter</code> with the arguments set as: <code>[&quot;-f&quot;, &quot;/etc/dcgm-exporter/dcp-metrics-included.csv&quot;]</code> OR you can create your own image and supply your own metrics list, which is what I did by using this Dockerfile:</p>
<pre><code>FROM nvcr.io/nvidia/k8s/dcgm-exporter:2.1.4-2.3.1-ubuntu18.04

RUN sed -i -e '/^# DCGM_FI_DEV_GPU_UTIL.*/s/^#\ //' /etc/dcgm-exporter/default-counters.csv

ENTRYPOINT [&quot;/usr/local/dcgm/dcgm-exporter-entrypoint.sh&quot;]
</code></pre>
<p>Another thing you can see from the above script is that I also used my own Prometheus helm chart values file. I followed the instructions from nvidia's site (<a href="https://docs.nvidia.com/datacenter/cloud-native/kubernetes/dcgme2e.html" rel="nofollow noreferrer">https://docs.nvidia.com/datacenter/cloud-native/kubernetes/dcgme2e.html</a>), but found my third &quot;gotcha&quot; in the <code>additionalScrapeConfig</code>.</p>
<p>What I learned was that, in the final deployment, the HPA has to be in the same namespace as the service it's scaling (identified by <code>targetRef</code>), otherwise it can't find it to scale it, as you probably already know.</p>
<p><strong>But just as importantly</strong> the <code>dcgm-metrics</code> <code>Service</code> <em>also has to be in the same namespace</em>, otherwise the HPA can't find the metrics it needs to scale by.
So, I changed the <code>additionalScrapeConfig</code> to target the relevant namespace. I'm sure there's a way to use the <code>additionalScrapeConfig.relabel_configs</code> section to enable you to keep <code>dcgm-exporter</code> in a different namespace and still have the HPA find the metrics, but I haven't had time to learn that voodoo yet.</p>
<p>Once I had all of that, I could check that the DCGM metrics were being made available to the kube metrics server:</p>
<pre><code>$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq -r . | grep DCGM_FI_DEV_GPU_UTIL
</code></pre>
<p>In the resulting list you <em>really</em> want to see a <code>services</code> entry, like so:</p>
<pre><code>&quot;name&quot;: &quot;jobs.batch/DCGM_FI_DEV_GPU_UTIL&quot;,
&quot;name&quot;: &quot;namespaces/DCGM_FI_DEV_GPU_UTIL&quot;,
&quot;name&quot;: &quot;services/DCGM_FI_DEV_GPU_UTIL&quot;,
&quot;name&quot;: &quot;pods/DCGM_FI_DEV_GPU_UTIL&quot;,
</code></pre>
<p>If you don't it probably means that the dcgm-exporter deployment you used is missing the <code>ServiceAccount</code> component, and also the HPA still won't work.</p>
<p>Finally, I wrote my HPA something like this:</p>
<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
  namespace: my-namespace
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: X
  maxReplicas: Y
...
  metrics:
  - type: Object
    object:
      metricName: DCGM_FI_DEV_GPU_UTIL
      targetValue: 80
      target:
        kind: Service
        name: dcgm-exporter
</code></pre>
<p>and it all worked.</p>
<p>I hope this helps! I spent so long trying different methods shown by people on consultancy company blogs, medium posts etc before discovering that people who write these pieces have already made assumptions about your deployment which affect details that you really need to know about (eg: the namespacing issue).</p>
