<p>I installed the <a href="https://github.com/prometheus-operator/kube-prometheus" rel="nofollow noreferrer">kube-prometheus-0.9.0</a>, and want to deploy a sample application on which to test the Prometheus metrics autoscaling, with the following resource manifest file: (hpa-prome-demo.yaml)</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: hpa-prom-demo
spec:
  selector:
    matchLabels:
      app: nginx-server
  template:
    metadata:
      labels:
        app: nginx-server
    spec:
      containers:
      - name: nginx-demo
        image: cnych/nginx-vts:v1.0
        resources:
          limits:
            cpu: 50m
          requests:
            cpu: 50m
        ports:
        - containerPort: 80
          name: http
---
apiVersion: v1
kind: Service
metadata:
  name: hpa-prom-demo
  annotations:
    prometheus.io/scrape: &quot;true&quot;
    prometheus.io/port: &quot;80&quot;
    prometheus.io/path: &quot;/status/format/prometheus&quot;
spec:
  ports:
  - port: 80
    targetPort: 80
    name: http
  selector:
    app: nginx-server
  type: NodePort
</code></pre>
<p>For testing purposes, used a NodePort Service and luckly I can get the http repsonse after applying the deployment. Then I installed
Prometheus Adapter via Helm Chart by creating a new <code>hpa-prome-adapter-values.yaml</code> file to override the default Values values, as follows.</p>
<pre class="lang-yaml prettyprint-override"><code>rules:
  default: false
  custom:
  - seriesQuery: 'nginx_vts_server_requests_total'
    resources:
      overrides:
        kubernetes_namespace:
          resource: namespace
        kubernetes_pod_name:
          resource: pod
    name:
      matches: &quot;^(.*)_total&quot;
      as: &quot;${1}_per_second&quot;
    metricsQuery: (sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;))

prometheus:
  url: http://prometheus-k8s.monitoring.svc
  port: 9090

</code></pre>
<p>Added a rules rule and specify the address of Prometheus. Install Prometheus-Adapter with the following command.</p>
<pre class="lang-sh prettyprint-override"><code>$ helm install prometheus-adapter prometheus-community/prometheus-adapter -n monitoring -f hpa-prome-adapter-values.yaml
NAME: prometheus-adapter
LAST DEPLOYED: Fri Jan 28 09:16:06 2022
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
prometheus-adapter has been deployed.
In a few minutes you should be able to list metrics using the following command(s):

  kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

</code></pre>
<p>Finally the adatper was installed successfully, and can get the http response, as follows.</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl get po -nmonitoring |grep adapter
prometheus-adapter-665dc5f76c-k2lnl    1/1     Running   0          133m

$ kubectl get --raw=&quot;/apis/custom.metrics.k8s.io/v1beta1&quot; | jq
{
  &quot;kind&quot;: &quot;APIResourceList&quot;,
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;groupVersion&quot;: &quot;custom.metrics.k8s.io/v1beta1&quot;,
  &quot;resources&quot;: [
    {
      &quot;name&quot;: &quot;namespaces/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: false,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    }
  ]
}


</code></pre>
<p>But it was supposed to be like this,</p>
<pre class="lang-json prettyprint-override"><code>$  kubectl get --raw=&quot;/apis/custom.metrics.k8s.io/v1beta1&quot; | jq
{
  &quot;kind&quot;: &quot;APIResourceList&quot;,
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;groupVersion&quot;: &quot;custom.metrics.k8s.io/v1beta1&quot;,
  &quot;resources&quot;: [
    {
      &quot;name&quot;: &quot;namespaces/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: false,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    },
    {
      &quot;name&quot;: &quot;pods/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: true,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    }
  ]
}
</code></pre>
<p>Why I can't get the metrics <code>pods/nginx_vts_server_requests_per_second</code>? as a result, below query was also failed.</p>
<pre class="lang-sh prettyprint-override"><code> kubectl get --raw &quot;/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/nginx_vts_server_requests_per_second&quot; | jq .
Error from server (NotFound): the server could not find the metric nginx_vts_server_requests_per_second for pods
</code></pre>
<p>Anybody cloud please help? many thanks.</p>

Marco Mei
<p><strong>ENV</strong>:</p>
<ol>
<li>helm install all Prometheus charts from  <code>prometheus-community   https://prometheus-community.github.io/helm-chart</code></li>
<li>k8s cluster enabled by docker for mac</li>
</ol>
<p><strong>Solution</strong>:<br />
I met the same problem, from Prometheus UI, i found it had <code>namespace</code> label and no <code>pod</code> label in metrics as below.</p>
<pre><code>nginx_vts_server_requests_total{code=&quot;1xx&quot;, host=&quot;*&quot;, instance=&quot;10.1.0.19:80&quot;, job=&quot;kubernetes-service-endpoints&quot;, namespace=&quot;default&quot;, node=&quot;docker-desktop&quot;, service=&quot;hpa-prom-demo&quot;}
</code></pre>
<p>I thought Prometheus may <strong>NOT</strong> use <code>pod</code> as a label, so i checked Prometheus config and found:</p>
<pre><code>121       - action: replace
122         source_labels:
123         - __meta_kubernetes_pod_node_name
124         target_label: node

</code></pre>
<p>then searched
<a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/" rel="nofollow noreferrer">https://prometheus.io/docs/prometheus/latest/configuration/configuration/</a> and do the similar thing as below under every <code>__meta_kubernetes_pod_node_name</code> i searched（ie. 2 places)</p>
<pre><code>125       - action: replace
126         source_labels:
127         - __meta_kubernetes_pod_name
128         target_label: pod
</code></pre>
<p>after a while, the configmap reloaded, UI and API could find <code>pod</code> label</p>
<pre><code>$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq                                    
{
  &quot;kind&quot;: &quot;APIResourceList&quot;,
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;groupVersion&quot;: &quot;custom.metrics.k8s.io/v1beta1&quot;,
  &quot;resources&quot;: [
    {
      &quot;name&quot;: &quot;pods/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: true,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    },
    {
      &quot;name&quot;: &quot;namespaces/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: false,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    }
  ]
}
</code></pre>
