<p>Not sure what I am missing. Please, find below all the config scripts I have used</p>
<pre><code>2022-07-21T07:26:56.903Z    info    service/collector.go:220    Starting otelcol... {&quot;service&quot;: &quot;my-prom-instance&quot;, &quot;Version&quot;: &quot;0.54.0&quot;, &quot;NumCPU&quot;: 4}
2022-07-21T07:26:56.903Z    info    service/collector.go:128    Everything is ready. Begin running and processing data. {&quot;service&quot;: &quot;my-prom-instance&quot;}
2022-07-21T07:26:56.902Z    debug   discovery/manager.go:309    Discoverer channel closed   {&quot;service&quot;: &quot;my-prom-instance&quot;, &quot;kind&quot;: &quot;receiver&quot;, &quot;name&quot;: &quot;prometheus&quot;, &quot;pipeline&quot;: &quot;metrics&quot;, &quot;provider&quot;: &quot;static/0&quot;}
W0721 07:26:56.964183       1 reflector.go:324] k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1.Pod: pods is forbidden: User &quot;system:serviceaccount:monitoring:otel-collector-collector&quot; cannot list resource &quot;pods&quot; in API group &quot;&quot; at the cluster scope
E0721 07:26:56.964871       1 reflector.go:138] k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User &quot;system:serviceaccount:monitoring:otel-collector-collector&quot; cannot list resource &quot;pods&quot; in API group &quot;&quot; at the cluster scope
W0721 07:26:58.435237       1 reflector.go:324] k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1.Pod: pods is forbidden: User &quot;system:serviceaccount:monitoring:otel-collector-collector&quot; cannot list resource &quot;pods&quot; in API group &quot;&quot; at the cluster scope
E0721 07:26:58.435924       1 reflector.go:138]
</code></pre>
<p>clusterRole.yaml</p>
<pre><code>apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
  namespace: monitoring
rules:
- apiGroups: [&quot;&quot;]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
- nonResourceURLs: [&quot;/metrics&quot;]
  verbs: [&quot;get&quot;]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: default
  namespace: monitoring

</code></pre>
<p>config-map.yaml</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.rules: |-
    groups:
    - name: devopscube demo alert
      rules:
      - alert: High Pod Memory
        expr: sum(container_memory_usage_bytes) &gt; 1
        for: 1m
        labels:
          severity: slack
        annotations:
          summary: High Memory Usage
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    rule_files:
      - /etc/prometheus/prometheus.rules
    alerting:
      alertmanagers:
      - scheme: http
        static_configs:
        - targets:
          - &quot;alertmanager.monitoring.svc:9093&quot;

    scrape_configs:
      - job_name: 'node-exporter'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_endpoints_name]
          regex: 'node-exporter'
          action: keep
      
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
        - role: endpoints
        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      - job_name: 'kubernetes-nodes'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics     
      
      - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
      
      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

      - job_name: 'kubernetes-cadvisor'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      
      - job_name: 'kubernetes-service-endpoints'

        kubernetes_sd_configs:
        - role: endpoints

        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name

</code></pre>
<p>prometheus-deployment.yaml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
  labels:
    app: prometheus-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          args:
            - &quot;--config.file=/etc/prometheus/prometheus.yml&quot;
            - &quot;--storage.tsdb.path=/prometheus/&quot;
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
  
        - name: prometheus-storage-volume
          emptyDir: {}

</code></pre>
<p>otel-deployment.yaml</p>
<pre><code>apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'kube-state-metrics'
              scrape_interval: 5s
              scrape_timeout: 1s
              static_configs:
                - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
            - job_name: k8s
              kubernetes_sd_configs:
              - role: pod
              relabel_configs:
              - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                regex: &quot;true&quot;
                action: keep
              metric_relabel_configs:
              - source_labels: [__name__]
                regex: &quot;(request_duration_seconds.*|response_duration_seconds.*)&quot;
                action: keep
    processors:
      batch:
    exporters:
      logging:
    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [logging]
      telemetry:
        logs:
          level: debug
          initial_fields:
            service: my-prom-instance
</code></pre>
<p>otel-service.yaml</p>
<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector-sa
  namespace: monitoring
</code></pre>

Ritesh Sinha
<p>the service account is defined with name otel-collector-sa, and your ClusterRoleBinding link to service account default</p>
