<p>i'm trying to get prometheus monitoring of my ingress nginx deployment working. i can view the default metrics in the prometheus ui, but nothing related to nginx.</p>
<ul>
<li>i've deployed prometheus-operator via cluster-monitoring (while updating the serviceMonitorSelector and serviceMonitorNamespaceSelector values): <a href="https://github.com/carlosedp/cluster-monitoring" rel="nofollow noreferrer">https://github.com/carlosedp/cluster-monitoring</a></li>
<li>i've deployed ingress-nginx via the official helm chart: <a href="https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx" rel="nofollow noreferrer">https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx</a></li>
</ul>
<p><code>kubectl get prometheus -A</code></p>
<pre><code>...
      runAsUser: 1000
    serviceAccountName: prometheus-k8s
    serviceMonitorNamespaceSelector:
      matchLabels:
        release: prometheus-operator
    serviceMonitorSelector:
      matchLabels:
        release: prometheus-operator
    version: v2.19.1
...
</code></pre>
<p><code>kubectl get ns nginx</code></p>
<pre><code>apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: &quot;2021-02-08T02:37:35Z&quot;
  labels:
    release: prometheus-operator
  managedFields:
  - apiVersion: v1
...
</code></pre>
<p><code>kubectl get servicemonitor nginx-ingress-nginx-controller -n monitoring -o yaml</code></p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: nginx
  creationTimestamp: &quot;2021-02-14T19:09:01Z&quot;
  generation: 5
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
    release: prometheus-operator
  managedFields:
...
spec:
  endpoints:
  - interval: 30s
    port: metrics
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      release: prometheus-operator
</code></pre>
<p><code>kubectl get svc nginx-ingress-nginx-controller-metrics -n nginx -o yaml</code></p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: nginx
    prometheus.io/port: &quot;10254&quot;
    prometheus.io/scrape: &quot;true&quot;
  creationTimestamp: &quot;2021-02-11T03:58:38Z&quot;
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
    release: prometheus-operator
...
spec:
  clusterIP: 10.43.90.125
  clusterIPs:
  - 10.43.90.125
  ports:
  - name: metrics
    port: 9913
    protocol: TCP
    targetPort: metrics
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: ClusterIP
</code></pre>
<p>UPDATE 2/17:
full output of <code>kubectl get prometheus -n monitoring -o yaml</code></p>
<pre><code>➜  ~ kubectl get -n monitoring prometheus -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: Prometheus
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {&quot;apiVersion&quot;:&quot;monitoring.coreos.com/v1&quot;,&quot;kind&quot;:&quot;Prometheus&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;labels&quot;:{&quot;prometheus&quot;:&quot;k8s&quot;},&quot;name&quot;:&quot;k8s&quot;,&quot;namespace&quot;:&quot;monitoring&quot;},&quot;spec&quot;:{&quot;affinity&quot;:{&quot;podAntiAffinity&quot;:{&quot;preferredDuringSchedulingIgnoredDuringExecution&quot;:[{&quot;podAffinityTerm&quot;:{&quot;labelSelector&quot;:{&quot;matchExpressions&quot;:[{&quot;key&quot;:&quot;prometheus&quot;,&quot;operator&quot;:&quot;In&quot;,&quot;values&quot;:[&quot;k8s&quot;]}]},&quot;namespaces&quot;:[&quot;monitoring&quot;],&quot;topologyKey&quot;:&quot;kubernetes.io/hostname&quot;},&quot;weight&quot;:100}]}},&quot;alerting&quot;:{&quot;alertmanagers&quot;:[{&quot;name&quot;:&quot;alertmanager-main&quot;,&quot;namespace&quot;:&quot;monitoring&quot;,&quot;port&quot;:&quot;web&quot;}]},&quot;externalUrl&quot;:&quot;http://prometheus.192.168.1.15.nip.io&quot;,&quot;image&quot;:&quot;prom/prometheus:v2.19.1&quot;,&quot;nodeSelector&quot;:{&quot;kubernetes.io/os&quot;:&quot;linux&quot;},&quot;podMonitorNamespaceSelector&quot;:{},&quot;podMonitorSelector&quot;:{},&quot;replicas&quot;:1,&quot;resources&quot;:{&quot;requests&quot;:{&quot;memory&quot;:&quot;400Mi&quot;}},&quot;retention&quot;:&quot;15d&quot;,&quot;ruleSelector&quot;:{&quot;matchLabels&quot;:{&quot;prometheus&quot;:&quot;k8s&quot;,&quot;role&quot;:&quot;alert-rules&quot;}},&quot;scrapeInterval&quot;:&quot;30s&quot;,&quot;scrapeTimeout&quot;:&quot;30s&quot;,&quot;securityContext&quot;:{&quot;fsGroup&quot;:2000,&quot;runAsNonRoot&quot;:true,&quot;runAsUser&quot;:1000},&quot;serviceAccountName&quot;:&quot;prometheus-k8s&quot;,&quot;serviceMonitorNamespaceSelector&quot;:{},&quot;serviceMonitorSelector&quot;:{&quot;matchLabels&quot;:{&quot;release&quot;:&quot;prometheus-operator&quot;}},&quot;version&quot;:&quot;v2.19.1&quot;}}
    creationTimestamp: &quot;2021-02-13T04:13:49Z&quot;
    generation: 4
    labels:
      prometheus: k8s
    managedFields:
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:prometheus: {}
        f:spec:
          .: {}
          f:affinity:
            .: {}
            f:podAntiAffinity:
              .: {}
              f:preferredDuringSchedulingIgnoredDuringExecution: {}
          f:alerting:
            .: {}
            f:alertmanagers: {}
          f:externalUrl: {}
          f:image: {}
          f:nodeSelector:
            .: {}
            f:kubernetes.io/os: {}
          f:podMonitorNamespaceSelector: {}
          f:podMonitorSelector: {}
          f:replicas: {}
          f:resources:
            .: {}
            f:requests:
              .: {}
              f:memory: {}
          f:retention: {}
          f:ruleSelector:
            .: {}
            f:matchLabels:
              .: {}
              f:prometheus: {}
              f:role: {}
          f:scrapeInterval: {}
          f:scrapeTimeout: {}
          f:securityContext:
            .: {}
            f:fsGroup: {}
            f:runAsNonRoot: {}
            f:runAsUser: {}
          f:serviceAccountName: {}
          f:serviceMonitorNamespaceSelector: {}
          f:serviceMonitorSelector:
            .: {}
            f:matchLabels:
              .: {}
              f:release: {}
          f:version: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: &quot;2021-02-15T15:07:05Z&quot;
    name: k8s
    namespace: monitoring
    resourceVersion: &quot;1321004&quot;
    uid: a9829d00-2765-416b-b068-2c1e57623832
  spec:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: prometheus
                operator: In
                values:
                - k8s
            namespaces:
            - monitoring
            topologyKey: kubernetes.io/hostname
          weight: 100
    alerting:
      alertmanagers:
      - name: alertmanager-main
        namespace: monitoring
        port: web
    externalUrl: http://prometheus.192.168.1.15.nip.io
    image: prom/prometheus:v2.19.1
    nodeSelector:
      kubernetes.io/os: linux
    podMonitorNamespaceSelector: {}
    podMonitorSelector: {}
    replicas: 1
    resources:
      requests:
        memory: 400Mi
    retention: 15d
    ruleSelector:
      matchLabels:
        prometheus: k8s
        role: alert-rules
    scrapeInterval: 30s
    scrapeTimeout: 30s
    securityContext:
      fsGroup: 2000
      runAsNonRoot: true
      runAsUser: 1000
    serviceAccountName: prometheus-k8s
    serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector:
      matchLabels:
        release: prometheus-operator
    version: v2.19.1
kind: List
metadata:
  resourceVersion: &quot;&quot;
  selfLink: &quot;&quot;
</code></pre>
<p>I've also experimented with getting monitoring working with other deployments, to no avail (so I'm obviously doing something wrong), but here are my prometheus logs:</p>
<pre><code>➜  ~ kubectl logs prometheus-k8s-0  -n monitoring -c prometheus --tail 25
level=error ts=2021-02-17T15:29:18.588Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;endpoints\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:29:25.978Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;mqtt\&quot;&quot;
level=error ts=2021-02-17T15:29:43.562Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:29:46.748Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;services\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:29:56.841Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;pihole\&quot;&quot;
level=error ts=2021-02-17T15:30:05.943Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;endpoints\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:30:14.735Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;mqtt\&quot;&quot;
level=error ts=2021-02-17T15:30:18.489Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;services\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:30:28.171Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;pihole\&quot;&quot;
level=error ts=2021-02-17T15:30:30.412Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:30:57.878Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;endpoints\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:30:59.128Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;mqtt\&quot;&quot;
level=error ts=2021-02-17T15:31:15.689Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;services\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:31:21.136Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:31:27.498Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;pihole\&quot;&quot;
level=error ts=2021-02-17T15:31:36.739Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;mqtt\&quot;&quot;
level=error ts=2021-02-17T15:31:54.742Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;endpoints\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:32:03.270Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;services\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:32:13.738Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:32:20.258Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;mqtt\&quot;&quot;
level=error ts=2021-02-17T15:32:24.315Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;pihole\&quot;&quot;
level=error ts=2021-02-17T15:32:44.099Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;endpoints\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:32:58.299Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;services\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:33:03.925Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; at the cluster scope&quot;
level=error ts=2021-02-17T15:33:05.050Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg=&quot;/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \&quot;system:serviceaccount:monitoring:prometheus-k8s\&quot; cannot list resource \&quot;pods\&quot; in API group \&quot;\&quot; in the namespace \&quot;mqtt\&quot;&quot;
</code></pre>

<p>After a good deal more research, the issue I faced involved the <code>ClusterRole</code> definition. The prometheus logs suggested a permissions issue with a number of the namespaces I was trying to monitor, so I ended up having to add permissions to my <code>ClusterRole</code> to include resources like <code>pods</code>, <code>services</code>, and <code>endpoints</code>, and to include verbs like <code>list</code> and <code>watch</code>:</p>
<pre><code> apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: prometheus-k8s
  rules:
  - apiGroups:
    - &quot;&quot;
    resources:
    - nodes/metrics
    - nodes
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
  - nonResourceURLs:
    - /metrics
    verbs:
    - get
</code></pre>
