<p>I installed Prometheus on my Kubernetes cluster with Helm, using the community chart <a href="https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack" rel="nofollow noreferrer">kube-prometheus-stack</a> - and I get some beautiful dashboards in the bundled Grafana instance. I now wanted the recommender from the Vertical Pod Autoscaler to use Prometheus as a data source for historic metrics, <a href="https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/FAQ.md#how-can-i-use-prometheus-as-a-history-provider-for-the-vpa-recommender" rel="nofollow noreferrer">as described here</a>. Meaning, I had to make a change to the Prometheus scraper settings for cAdvisor, and <a href="https://stackoverflow.com/a/65421764/310937">this answer</a> pointed me in the right direction, as after making that change I can now see the correct <code>job</code> tag on metrics from cAdvisor.</p>
<p>Unfortunately, now some of the charts in the Grafana dashboards are broken. It looks like it no longer picks up the CPU metrics - and instead just displays &quot;No data&quot; for the CPU-related charts.</p>
<p>So, I assume I have to tweak the charts to be able to pick up the metrics correctly again, but I don't see any obvious places to do this in Grafana?</p>
<p>Not sure if it is relevant for the question, but I am running my Kubernetes cluster on Azure Kubernetes Service (AKS).</p>
<p>This is the full <code>values.yaml</code> I supply to the Helm chart when installing Prometheus:</p>
<pre class="lang-yaml prettyprint-override"><code>kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
kubeProxy:
  enabled: false
kubelet:
  serviceMonitor:
    # Diables the normal cAdvisor scraping, as we add it with the job name &quot;kubernetes-cadvisor&quot; under additionalScrapeConfigs
    # The reason for doing this is to enable the VPA to use the metrics for the recommender
    # https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/FAQ.md#how-can-i-use-prometheus-as-a-history-provider-for-the-vpa-recommender
    cAdvisor: false
prometheus:
  prometheusSpec:
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          # the azurefile storage class is created automatically on AKS
          storageClassName: azurefile
          accessModes: [&quot;ReadWriteMany&quot;]
          resources:
            requests:
              storage: 50Gi
    additionalScrapeConfigs:
      - job_name: 'kubernetes-cadvisor'
        scheme: https
        metrics_path: /metrics/cadvisor
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
</code></pre>
<p>Kubernetes version: 1.21.2</p>
<p>kube-prometheus-stack version: 18.1.1</p>
<p>helm version: version.BuildInfo{Version:&quot;v3.6.3&quot;, GitCommit:&quot;d506314abfb5d21419df8c7e7e68012379db2354&quot;, GitTreeState:&quot;dirty&quot;, GoVersion:&quot;go1.16.5&quot;}</p>

<p>Unfortunately, I don't have access to Azure AKS, so I've reproduced this issue on my GKE cluster. Below I'll provide some explanations that may help to resolve your problem.</p>
<p>First you can try to execute this <code>node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate</code> rule to see if it returns any result:</p>
<p><a href="https://i.stack.imgur.com/UZ4uk.png" rel="noreferrer"><img src="https://i.stack.imgur.com/UZ4uk.png" alt="enter image description here" /></a></p>
<p>If it doesn't return any records, please read the following paragraphs.</p>
<h2>Creating a scrape configuration for cAdvisor</h2>
<p>Rather than creating a completely new scrape configuration for cadvisor, I would suggest using one that is generated by default when <code>kubelet.serviceMonitor.cAdvisor: true</code>, but with a few modifications such as changing the label to <code>job=kubernetes-cadvisor</code>.</p>
<p>In my example, the 'kubernetes-cadvisor' scrape configuration looks like this:</p>
<p><strong>NOTE:</strong> I added this config under the <code>additionalScrapeConfigs</code> in the <code>values.yaml</code> file (the rest of the <code>values.yaml</code> file may be like yours).</p>
<pre><code>- job_name: 'kubernetes-cadvisor'
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics/cadvisor
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace
  - source_labels: [__metrics_path__]
    separator: ;
    regex: (.*)
    target_label: metrics_path
    replacement: $1
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: &quot;0&quot;
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: &quot;&quot;
    follow_redirects: true
    namespaces:
      names:
      - kube-system
</code></pre>
<h3>Modifying Prometheus Rules</h3>
<p>By default, Prometheus rules fetching data from cAdvisor use <code>job=&quot;kubelet&quot;</code> in their PromQL expressions:</p>
<p><a href="https://i.stack.imgur.com/oNFnF.png" rel="noreferrer"><img src="https://i.stack.imgur.com/oNFnF.png" alt="enter image description here" /></a></p>
<p>After changing <code>job=kubelet</code> to <code>job=kubernetes-cadvisor</code>, we also need to modify this label in the Prometheus rules:<br />
<strong>NOTE:</strong> We just need to modify the rules that have <code>metrics_path=&quot;/metrics/cadvisor</code> (these are rules that retrieve data from cAdvisor).</p>
<pre><code>$ kubectl get prometheusrules prom-1-kube-prometheus-sta-k8s.rules -o yaml
...
  - name: k8s.rules
    rules:
    - expr: |-
        sum by (cluster, namespace, pod, container) (
          irate(container_cpu_usage_seconds_total{job=&quot;kubernetes-cadvisor&quot;, metrics_path=&quot;/metrics/cadvisor&quot;, image!=&quot;&quot;}[5m])
        ) * on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (
          1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=&quot;&quot;})
        )
      record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
...
here we have a few more rules to modify...
</code></pre>
<p>After modifying Prometheus rules and waiting some time, we can see if it works as expected. We can try to execute <code>node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate</code> as in the beginning.</p>
<p>Additionally, let's check out our Grafana to make sure it has started displaying our dashboards correctly:
<a href="https://i.stack.imgur.com/Z7LRc.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Z7LRc.png" alt="enter image description here" /></a></p>
