<p>I installed the <a href="https://github.com/prometheus-operator/kube-prometheus" rel="nofollow noreferrer">kube-prometheus-0.9.0</a>, and want to deploy a sample application on which to test the Prometheus metrics autoscaling, with the following resource manifest file: (hpa-prome-demo.yaml)</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: hpa-prom-demo
spec:
  selector:
    matchLabels:
      app: nginx-server
  template:
    metadata:
      labels:
        app: nginx-server
    spec:
      containers:
      - name: nginx-demo
        image: cnych/nginx-vts:v1.0
        resources:
          limits:
            cpu: 50m
          requests:
            cpu: 50m
        ports:
        - containerPort: 80
          name: http
---
apiVersion: v1
kind: Service
metadata:
  name: hpa-prom-demo
  annotations:
    prometheus.io/scrape: &quot;true&quot;
    prometheus.io/port: &quot;80&quot;
    prometheus.io/path: &quot;/status/format/prometheus&quot;
spec:
  ports:
  - port: 80
    targetPort: 80
    name: http
  selector:
    app: nginx-server
  type: NodePort
</code></pre>
<p>For testing purposes, used a NodePort Service and luckly I can get the http repsonse after applying the deployment. Then I installed
Prometheus Adapter via Helm Chart by creating a new <code>hpa-prome-adapter-values.yaml</code> file to override the default Values values, as follows.</p>
<pre class="lang-yaml prettyprint-override"><code>rules:
  default: false
  custom:
  - seriesQuery: 'nginx_vts_server_requests_total'
    resources:
      overrides:
        kubernetes_namespace:
          resource: namespace
        kubernetes_pod_name:
          resource: pod
    name:
      matches: &quot;^(.*)_total&quot;
      as: &quot;${1}_per_second&quot;
    metricsQuery: (sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;))

prometheus:
  url: http://prometheus-k8s.monitoring.svc
  port: 9090

</code></pre>
<p>Added a rules rule and specify the address of Prometheus. Install Prometheus-Adapter with the following command.</p>
<pre class="lang-sh prettyprint-override"><code>$ helm install prometheus-adapter prometheus-community/prometheus-adapter -n monitoring -f hpa-prome-adapter-values.yaml
NAME: prometheus-adapter
LAST DEPLOYED: Fri Jan 28 09:16:06 2022
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
prometheus-adapter has been deployed.
In a few minutes you should be able to list metrics using the following command(s):

  kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

</code></pre>
<p>Finally the adatper was installed successfully, and can get the http response, as follows.</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl get po -nmonitoring |grep adapter
prometheus-adapter-665dc5f76c-k2lnl    1/1     Running   0          133m

$ kubectl get --raw=&quot;/apis/custom.metrics.k8s.io/v1beta1&quot; | jq
{
  &quot;kind&quot;: &quot;APIResourceList&quot;,
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;groupVersion&quot;: &quot;custom.metrics.k8s.io/v1beta1&quot;,
  &quot;resources&quot;: [
    {
      &quot;name&quot;: &quot;namespaces/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: false,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    }
  ]
}


</code></pre>
<p>But it was supposed to be like this,</p>
<pre class="lang-json prettyprint-override"><code>$  kubectl get --raw=&quot;/apis/custom.metrics.k8s.io/v1beta1&quot; | jq
{
  &quot;kind&quot;: &quot;APIResourceList&quot;,
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;groupVersion&quot;: &quot;custom.metrics.k8s.io/v1beta1&quot;,
  &quot;resources&quot;: [
    {
      &quot;name&quot;: &quot;namespaces/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: false,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    },
    {
      &quot;name&quot;: &quot;pods/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: true,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    }
  ]
}
</code></pre>
<p>Why I can't get the metrics <code>pods/nginx_vts_server_requests_per_second</code>? as a result, below query was also failed.</p>
<pre class="lang-sh prettyprint-override"><code> kubectl get --raw &quot;/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/nginx_vts_server_requests_per_second&quot; | jq .
Error from server (NotFound): the server could not find the metric nginx_vts_server_requests_per_second for pods
</code></pre>
<p>Anybody cloud please help? many thanks.</p>

<p>It is worth knowing that using the <a href="https://github.com/prometheus-operator/kube-prometheus" rel="nofollow noreferrer">kube-prometheus</a> repository, you can also install components such as <strong>Prometheus Adapter for Kubernetes Metrics APIs</strong>, so there is no need to install it separately with Helm.</p>
<p>I will use your <code>hpa-prome-demo.yaml</code> manifest file to demonstrate how to monitor <code>nginx_vts_server_requests_total</code> metrics.</p>
<hr />
<p>First of all, we need to install Prometheus and Prometheus Adapter with appropriate configuration as described step by step below.</p>
<p>Copy the <a href="https://github.com/prometheus-operator/kube-prometheus" rel="nofollow noreferrer">kube-prometheus</a> repository and refer to the <a href="https://github.com/prometheus-operator/kube-prometheus#kubernetes-compatibility-matrix" rel="nofollow noreferrer">Kubernetes compatibility matrix</a> in order to choose a compatible branch:</p>
<pre><code>$ git clone https://github.com/prometheus-operator/kube-prometheus.git 
$ cd kube-prometheus
$ git checkout release-0.9
</code></pre>
<p>Install the <code>jb</code>, <code>jsonnet</code> and <code>gojsontoyaml</code> tools:</p>
<pre><code>$ go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
$ go install github.com/google/go-jsonnet/cmd/jsonnet@latest
$ go install github.com/brancz/gojsontoyaml@latest 
</code></pre>
<p>Uncomment the <code>(import 'kube-prometheus/addons/custom-metrics.libsonnet') +</code> line from the <code>example.jsonnet</code> file:</p>
<pre><code>$ cat example.jsonnet
local kp =
  (import 'kube-prometheus/main.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
  // (import 'kube-prometheus/addons/managed-cluster.libsonnet') +
  // (import 'kube-prometheus/addons/node-ports.libsonnet') +
  // (import 'kube-prometheus/addons/static-etcd.libsonnet') +
  (import 'kube-prometheus/addons/custom-metrics.libsonnet') +          &lt;--- This line
  // (import 'kube-prometheus/addons/external-metrics.libsonnet') +
...
</code></pre>
<p>Add the following rule to the <code>./jsonnet/kube-prometheus/addons/custom-metrics.libsonnet</code> file in the <code>rules+</code> section:</p>
<pre><code>      {
        seriesQuery: &quot;nginx_vts_server_requests_total&quot;,
        resources: {
          overrides: {
            namespace: { resource: 'namespace' },
            pod: { resource: 'pod' },
          },
        },
        name: { &quot;matches&quot;: &quot;^(.*)_total&quot;, &quot;as&quot;: &quot;${1}_per_second&quot; },
        metricsQuery: &quot;(sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;))&quot;,
      },
</code></pre>
<p>After this update, the <code>./jsonnet/kube-prometheus/addons/custom-metrics.libsonnet</code> file should look like this:<br />
<strong>NOTE:</strong> This is not the entire file, just an important part of it.</p>
<pre><code>$ cat custom-metrics.libsonnet
// Custom metrics API allows the HPA v2 to scale based on arbirary metrics.
// For more details on usage visit https://github.com/DirectXMan12/k8s-prometheus-adapter#quick-links

{
  values+:: {
    prometheusAdapter+: {
      namespace: $.values.common.namespace,
      // Rules for custom-metrics
      config+:: {
        rules+: [
          {
            seriesQuery: &quot;nginx_vts_server_requests_total&quot;,
            resources: {
              overrides: {
                namespace: { resource: 'namespace' },
                pod: { resource: 'pod' },
              },
            },
            name: { &quot;matches&quot;: &quot;^(.*)_total&quot;, &quot;as&quot;: &quot;${1}_per_second&quot; },
            metricsQuery: &quot;(sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;))&quot;,
          },
...
</code></pre>
<p>Use the jsonnet-bundler update functionality to update the <code>kube-prometheus</code> dependency:</p>
<pre><code>$ jb update
</code></pre>
<p>Compile the manifests:</p>
<pre><code>$ ./build.sh example.jsonnet
</code></pre>
<p>Now simply use <code>kubectl</code> to install Prometheus and other components as per your configuration:</p>
<pre><code>$ kubectl apply --server-side -f manifests/setup
$ kubectl apply -f manifests/
</code></pre>
<p>After configuring Prometheus, we can deploy a sample <code>hpa-prom-demo</code> Deployment:<br />
<strong>NOTE:</strong> I've deleted the annotations because I'm going to use a <a href="https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md#related-resources" rel="nofollow noreferrer">ServiceMonitor</a> to describe the set of targets to be monitored by Prometheus.</p>
<pre><code>$ cat hpa-prome-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hpa-prom-demo
spec:
  selector:
    matchLabels:
      app: nginx-server
  template:
    metadata:
      labels:
        app: nginx-server
    spec:
      containers:
      - name: nginx-demo
        image: cnych/nginx-vts:v1.0
        resources:
          limits:
            cpu: 50m
          requests:
            cpu: 50m
        ports:
        - containerPort: 80
          name: http
---
apiVersion: v1
kind: Service
metadata:
  name: hpa-prom-demo
  labels:
    app: nginx-server
spec:
  ports:
  - port: 80
    targetPort: 80
    name: http
  selector:
    app: nginx-server
  type: LoadBalancer
</code></pre>
<p>Next, create a <code>ServiceMonitor</code> that describes how to monitor our NGINX:</p>
<pre><code>$ cat servicemonitor.yaml
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  name: hpa-prom-demo
  labels:
    app: nginx-server
spec:
  selector:
    matchLabels:
      app: nginx-server
  endpoints:
  - interval: 15s
    path: &quot;/status/format/prometheus&quot;
    port: http
</code></pre>
<p>After waiting some time, let's check the <code>hpa-prom-demo</code> logs to make sure that it is scrapped correctly:</p>
<pre><code>$ kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
hpa-prom-demo-bbb6c65bb-49jsh   1/1     Running   0          35m

$ kubectl logs -f hpa-prom-demo-bbb6c65bb-49jsh
...
10.4.0.9 - - [04/Feb/2022:09:29:17 +0000] &quot;GET /status/format/prometheus HTTP/1.1&quot; 200 3771 &quot;-&quot; &quot;Prometheus/2.29.1&quot; &quot;-&quot;
10.4.0.9 - - [04/Feb/2022:09:29:32 +0000] &quot;GET /status/format/prometheus HTTP/1.1&quot; 200 3771 &quot;-&quot; &quot;Prometheus/2.29.1&quot; &quot;-&quot;
10.4.0.9 - - [04/Feb/2022:09:29:47 +0000] &quot;GET /status/format/prometheus HTTP/1.1&quot; 200 3773 &quot;-&quot; &quot;Prometheus/2.29.1&quot; &quot;-&quot;
10.4.0.9 - - [04/Feb/2022:09:30:02 +0000] &quot;GET /status/format/prometheus HTTP/1.1&quot; 200 3773 &quot;-&quot; &quot;Prometheus/2.29.1&quot; &quot;-&quot;
10.4.0.9 - - [04/Feb/2022:09:30:17 +0000] &quot;GET /status/format/prometheus HTTP/1.1&quot; 200 3773 &quot;-&quot; &quot;Prometheus/2.29.1&quot; &quot;-&quot;
10.4.2.12 - - [04/Feb/2022:09:30:23 +0000] &quot;GET /status/format/prometheus HTTP/1.1&quot; 200 3773 &quot;-&quot; &quot;Prometheus/2.29.1&quot; &quot;-&quot;
...
</code></pre>
<p>Finally, we can check if our metrics work as expected:</p>
<pre><code>$ kubectl get --raw &quot;/apis/custom.metrics.k8s.io/v1beta1/&quot; | jq . | grep -A 7 &quot;nginx_vts_server_requests_per_second&quot;
      &quot;name&quot;: &quot;pods/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: true,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    },
--
      &quot;name&quot;: &quot;namespaces/nginx_vts_server_requests_per_second&quot;,
      &quot;singularName&quot;: &quot;&quot;,
      &quot;namespaced&quot;: false,
      &quot;kind&quot;: &quot;MetricValueList&quot;,
      &quot;verbs&quot;: [
        &quot;get&quot;
      ]
    },

$ kubectl get --raw &quot;/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/nginx_vts_server_requests_per_second&quot; | jq .
{
  &quot;kind&quot;: &quot;MetricValueList&quot;,
  &quot;apiVersion&quot;: &quot;custom.metrics.k8s.io/v1beta1&quot;,
  &quot;metadata&quot;: {
    &quot;selfLink&quot;: &quot;/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/nginx_vts_server_requests_per_second&quot;
  },
  &quot;items&quot;: [
    {
      &quot;describedObject&quot;: {
        &quot;kind&quot;: &quot;Pod&quot;,
        &quot;namespace&quot;: &quot;default&quot;,
        &quot;name&quot;: &quot;hpa-prom-demo-bbb6c65bb-49jsh&quot;,
        &quot;apiVersion&quot;: &quot;/v1&quot;
      },
      &quot;metricName&quot;: &quot;nginx_vts_server_requests_per_second&quot;,
      &quot;timestamp&quot;: &quot;2022-02-04T09:32:59Z&quot;,
      &quot;value&quot;: &quot;533m&quot;,
      &quot;selector&quot;: null
    }
  ]
}
</code></pre>
