<p>I am unable to identify what the exact issue with the permissions with my setup as shown below. I've looked into all the similar QAs but still unable to solve the issue. The aim is to deploy Prometheus and let it <strong>scrape</strong> <code>/metrics</code> endpoints that my other applications in the cluster expose fine.</p>
<pre class="lang-sh prettyprint-override"><code>Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \&quot;system:serviceaccount:default:default\&quot; cannot list resource \&quot;endpoints\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:default:default\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \&quot;system:serviceaccount:default:default\&quot; cannot list resource \&quot;services\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
...
...
</code></pre>
<p>The command below returns <code>no</code> to all services, nodes, pods etc.</p>
<pre class="lang-sh prettyprint-override"><code>kubectl auth can-i get services --as=system:serviceaccount:default:default -n default
</code></pre>
<p><strong>Minikube</strong></p>
<pre class="lang-sh prettyprint-override"><code>$ minikube start --vm-driver=virtualbox --extra-config=apiserver.Authorization.Mode=RBAC

😄  minikube v1.14.2 on Darwin 11.2
✨  Using the virtualbox driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing virtualbox VM for &quot;minikube&quot; ...
🐳  Preparing Kubernetes v1.19.2 on Docker 19.03.12 ...
    ▪ apiserver.Authorization.Mode=RBAC
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass, dashboard
🏄  Done! kubectl is now configured to use &quot;minikube&quot; by default
</code></pre>
<p><strong>Roles</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole

metadata:
  name: monitoring-cluster-role

rules:
  - apiGroups: [&quot;&quot;]
    resources: [&quot;nodes&quot;, &quot;services&quot;, &quot;pods&quot;, &quot;endpoints&quot;]
    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
  - apiGroups: [&quot;&quot;]
    resources: [&quot;configmaps&quot;]
    verbs: [&quot;get&quot;]
  - apiGroups: [&quot;extensions&quot;]
    resources: [&quot;deployments&quot;]
    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
</code></pre>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ServiceAccount

metadata:
  name: monitoring-service-account
  namespace: default
</code></pre>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding

metadata:
  name: monitoring-cluster-role-binding

roleRef:
  kind: ClusterRole
  name: monitoring-cluster-role
  apiGroup: rbac.authorization.k8s.io

subjects:
  - kind: ServiceAccount
    name: monitoring-service-account
    namespace: default
</code></pre>
<p><strong>Prometheus</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ConfigMap
 
metadata:
  name: prometheus-config-map
  namespace: default
 
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'kubernetes-service-endpoints'
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name 
</code></pre>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
 
metadata:
  name: prometheus-deployment
  namespace: default
  labels:
    app: prometheus
 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:latest
          ports:
            - name: http
              protocol: TCP
              containerPort: 9090
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus/
            - name: storage
              mountPath: /prometheus/
      volumes:
        - name: config
          configMap:
            name: prometheus-config-map
        - name: storage
          emptyDir: {}

</code></pre>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
 
metadata:
  name: prometheus-service
  namespace: default
 
spec:
  type: NodePort
  selector:
    app: prometheus
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9090
</code></pre>

<blockquote>
<p>User &quot;system:serviceaccount:default:default&quot; cannot list resource &quot;endpoints&quot; in API group &quot;&quot; at the cluster scope&quot;</p>
</blockquote>
<blockquote>
<p>User &quot;system:serviceaccount:default:default&quot; cannot list resource &quot;pods&quot; in API group &quot;&quot; at the cluster scope&quot;</p>
</blockquote>
<blockquote>
<p>User &quot;system:serviceaccount:default:default&quot; cannot list resource &quot;services&quot; in API group &quot;&quot; at the cluster scope&quot;</p>
</blockquote>
<p>Something running with ServiceAccount <code>default</code> in namespace <code>default</code> is doing things it does not have permissions for.</p>
<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: monitoring-service-account
</code></pre>
<p>Here you create a specific ServiceAccount. You also give it some Cluster-wide permissions.</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: default
</code></pre>
<p>You run Prometheus in namespace <code>default</code> but do not specify a specific ServiceAccount, so it will run with ServiceAccount <code>default</code>.</p>
<p>I think your problem is that you are supposed to set the ServiceAccount that you create in the Deployment-manifest for Prometheus.</p>
