<p>I was following this walk through (partially, am using EKS.
<a href="https://itnext.io/horizontal-pod-autoscale-with-custom-metrics-8cb13e9d475" rel="noreferrer">https://itnext.io/horizontal-pod-autoscale-with-custom-metrics-8cb13e9d475</a></p>

<p>I manage to get scaled up one deployment with this http_requests_total metric.</p>

<p>Now, I am trying to add a new metric. I have prometheus server, and it already scrapes cloudwatch and I have <code>aws_sqs_approximate_age_of_oldest_message_maximum</code> value there for many of my queues.</p>

<p>In the similar manner to the mentioned tutorial, I am adding the definition of a metric:</p>

<pre><code>  - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
    resources:
      overrides:
       kubernetes_namespace: {resource: "namespace"}
       kubernetes_pod_name: {resource: "pod"}
    name:
      matches: "^(.*)_total"
      as: "${1}_per_second"
    metricsQuery: 'sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[2m])) by (&lt;&lt;.GroupBy&gt;&gt;)'
</code></pre>

<p>vs</p>

<pre><code>  - seriesQuery: 'aws_sqs_approximate_age_of_oldest_message_maximum{queue_name!=""}'
    resources:
      overrides:
       kubernetes_namespace: {resource: "namespace"}
       kubernetes_pod_name: {resource: "pod"}
    metricsQuery: '&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}'
</code></pre>

<p>Or some version of the bottom one. But, I can never see it in:
<code>kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq</code></p>

<p>No matter what I try.</p>

<p>Any ideas how to move forward?
Thanks!</p>

<p>If you don't see the metric in <code>/apis/custom.metrics.k8s.io/v1beta1</code> it means that the Prometheus Adapter couldn't discover it.</p>

<p>The Prometheus Adapter discovers metrics by using the value of your <code>seriesQuery</code> field for an <a href="https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers" rel="noreferrer"><code>/api/v1/series</code></a> request to Prometheus (done periodically with a frequency defined by the <em>relist interval</em>).</p>

<p>Things to try:</p>

<ul>
<li><p>What do you get if you make the following request to Prometheus?</p>

<pre><code>http://&lt;prometheus-ip&gt;:9090/api/v1/series? match[]=aws_sqs_approximate_age_of_oldest_message_maximum{queue_name!=""}&amp;start=&lt;current-timestamp-sec&gt;
</code></pre></li>
<li><p>What do you get if you drop the following in the query text box of the Prometheus UI and press <em>Execute</em>?</p>

<pre><code>aws_sqs_approximate_age_of_oldest_message_maximum{queue_name!=""}
</code></pre></li>
</ul>

<p>If you get no data back in either case, then you just don't have any time series in Prometheus that match your <code>seriesQuery</code> specification.</p>
