<p>I am trying to get Kafka topic lag into Prometheus and finally to the APIServer in order to utilize an external metrics HPA for my application.</p>
<p>I am getting the error <strong>no metrics returned from external metrics API</strong></p>
<pre><code>70m         Warning   FailedGetExternalMetric        horizontalpodautoscaler/kafkademo-hpa   unable to get external metric default/kafka_lag_metric_sm0ke/&amp;LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
66m         Warning   FailedComputeMetricsReplicas   horizontalpodautoscaler/kafkademo-hpa   invalid metrics (1 invalid out of 1), first error is: failed to get external metric kafka_lag_metric_sm0ke: unable to get external metric default/kafka_lag_metric_sm0ke/&amp;LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API

</code></pre>
<p>This happens <strong>even though</strong> I can see the following output when querying the external API:</p>
<pre><code>kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq
{
  &quot;kind&quot;: &quot;APIResourceList&quot;,
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;groupVersion&quot;: &quot;external.metrics.k8s.io/v1beta1&quot;,
  &quot;resources&quot;: [
    {
      &quot;name&quot;: &quot;kafka_lag_metric_sm0ke&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: true,
      &quot;kind&quot;: &quot;ExternalMetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    }
  ]
}

</code></pre>
<p>Here's the set-up:</p>
<ul>
<li>Kafka: v2.7.0</li>
<li>Prometheus: v2.26.0</li>
<li>Prometheus Adapter: v0.8.3</li>
</ul>
<p><strong>Prometheus Adapter Values</strong></p>
<pre><code>rules:
  external:
  - seriesQuery: 'kafka_consumergroup_group_lag{topic=&quot;prices&quot;}'
    resources:
      template: &lt;&lt;.Resource&gt;&gt;
    name:
      as: &quot;kafka_lag_metric_sm0ke&quot;
    metricsQuery: 'avg by (topic) (round(avg_over_time(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])))'

</code></pre>
<p><strong>HPA</strong></p>
<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: kafkademo-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kafkademo
  minReplicas: 3
  maxReplicas: 12
  metrics:
  - type: External
    external:
      metricName: kafka_lag_metric_sm0ke
      metricSelector:
        matchLabels:
          topic: prices
      targetValue: 5

</code></pre>
<p><strong>HPA information</strong></p>
<pre><code>kubectl describe hpa kafkademo-hpa 
Name:                                       kafkademo-hpa
Namespace:                                  default
Labels:                                     &lt;none&gt;
Annotations:                                &lt;none&gt;
CreationTimestamp:                          Sat, 17 Apr 2021 20:01:29 +0300
Reference:                                  Deployment/kafkademo
Metrics:                                    ( current / target )
  &quot;kafka_lag_metric_sm0ke&quot; (target value):  &lt;unknown&gt; / 5
Min replicas:                               3
Max replicas:                               12
Deployment pods:                            3 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric default/kafka_lag_metric_sm0ke/&amp;LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
Events:
  Type     Reason                        Age                     From                       Message
  ----     ------                        ----                    ----                       -------
  Warning  FailedComputeMetricsReplicas  70m (x335 over 155m)    horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get external metric kafka_lag_metric_sm0ke: unable to get external metric default/kafka_lag_metric_sm0ke/&amp;LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
  Warning  FailedGetExternalMetric       2m30s (x366 over 155m)  horizontal-pod-autoscaler  unable to get external metric default/kafka_lag_metric_sm0ke/&amp;LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
</code></pre>
<p><strong>-- Edit 1</strong></p>
<p>When i query the default namespace i get this:</p>
<pre><code>kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/kafka_lag_metric_sm0ke |jq
{
  &quot;kind&quot;: &quot;ExternalMetricValueList&quot;,
  &quot;apiVersion&quot;: &quot;external.metrics.k8s.io/v1beta1&quot;,
  &quot;metadata&quot;: {},
  &quot;items&quot;: []
}

</code></pre>
<p>I can see that the &quot;items&quot; field is empty. What does this mean?</p>
<p>What i don't seem to comprehend is the chain of events that happen behind the scenes.</p>
<p>AFAIK this is what happens. <em><strong>Is this correct?</strong></em></p>
<ul>
<li>prometheus-adapter queries Prometheus, executes the seriesQuery, computes the metricsQuery and creates &quot;kafka_lag_metric_sm0ke&quot;</li>
<li>It registers an endpoint with the api server for external metrics.</li>
<li>The API Server will periodically update its stats based on that endpoint.</li>
<li>The HPA checks &quot;kafka_lag_metric_sm0ke&quot; from the API server and performs the scaling according to the supplied values.</li>
</ul>
<p>I also don't seem to understand the significance of namespaces in all this. I can see that the stat is namespaced. Does that mean that there will be 1 stat per namespace? How does that make sense?</p>

<p>In a long tradition of answering my own questions after I ask them, here's what's wrong with the above configuration.</p>
<p>The error lies in the prometheus-adapter yaml:</p>
<pre><code>rules:
  external:
    - seriesQuery: 'kafka_consumergroup_group_lag{topic=&quot;prices&quot;}'
      resources:
        template: &lt;&lt;.Resource&gt;&gt;
      name:
        as: &quot;kafka_lag_metric_sm0ke&quot;
      metricsQuery: 'avg by (topic) (round(avg_over_time(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])))'
</code></pre>
<p>I removed <code>&lt;&lt;.LabelMatchers&gt;&gt;</code> and now it works:</p>
<pre><code>kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/kafka_lag_metric_sm0ke |jq
{
  &quot;kind&quot;: &quot;ExternalMetricValueList&quot;,
  &quot;apiVersion&quot;: &quot;external.metrics.k8s.io/v1beta1&quot;,
  &quot;metadata&quot;: {},
  &quot;items&quot;: [
    {
      &quot;metricName&quot;: &quot;kafka_lag_metric_sm0ke&quot;,
  &quot;metricLabels&quot;: {
        &quot;topic&quot;: &quot;prices&quot;
      },
      &quot;timestamp&quot;: &quot;2021-04-21T16:55:18Z&quot;,
      &quot;value&quot;: &quot;0&quot;
    }
  ]
}
</code></pre>
<p>I am still unsure as to why it works. I know that <code>&lt;&lt;.LabelMatchers&gt;&gt;</code> in this case will be substituted with something that doesn't produce a valid query, but I don't know what it is.</p>
