<p>For the prometheus deployment's ClusterRole I have</p>
<pre><code># ClusterRole for the deployment
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [&quot;&quot;]
  resources:
  - nodes
  - nodes/proxy
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
- nonResourceURLs: [&quot;/metrics&quot;]
  verbs: [&quot;get&quot;]
</code></pre>
<p>With the ServiceAccount and the ClusterRoleBinding already put to place too.</p>
<p>And the following are the settings for the jobs inside <code>prometheus.yml</code> that are getting 403 error</p>
<pre><code>- job_name: 'kubernetes-cadvisor'

      scheme: https

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

- job_name: 'kubernetes-nodes'

      scheme: https

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
</code></pre>
<p>I don't get the reason why I keep getting 403 error even though the <code>ServiceAccount</code> and the <code>ClusterRole</code> has been binded together.</p>

MatsuzakaSteven
<p>Make sure that the <code>/var/run/secrets/kubernetes.io/serviceaccount/token</code> file contains the correct token. To do so, you can enter into Prometheus pod with:</p>
<p><code>kubectl exec -it -n &lt;namespace&gt; &lt;Prometheus_pod_name&gt; -- bash</code></p>
<p>and cat the token file. Then exit the pod and execute:</p>
<p><code>echo $(kubectl get secret -n &lt;namespace&gt; &lt;prometheus_serviceaccount_secret&gt; -o jsonpath='{.data.token}') | base64 --decode</code></p>
<p>If the tokens match, you can try querying the Kubernetes API server with Postman or Insomnia to see if the rules you put in your <code>ClusterRole</code> are correct. I suggest you to query both <code>/proxy/metrics/cadvisor</code> and <code>/proxy/metrics</code> URLs</p>
