<p>I want to monitor my Kafka deployed using Strimzi operator. I deployed Prometheus operator alongside a few basic monitoring tools using community chart <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="nofollow noreferrer">https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack</a>, then followed Strimzi documentation <a href="https://strimzi.io/docs/operators/latest/deploying.html#assembly-metrics-str" rel="nofollow noreferrer">https://strimzi.io/docs/operators/latest/deploying.html#assembly-metrics-str</a> to configure kafka exporter and other metrics to being exposed. Strimzi examples have pod monitors <code>cluster-operator-metrics</code>, <code>entity-operator-metrics</code> and <code>kafka-resources-metrics</code>. Everything discovered and scrapped successfully except one pod monitor <code>cluster-operator-metrics</code> which is even now showed up in Prometheus instance.</p>
<p><a href="https://i.stack.imgur.com/JjRYh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JjRYh.png" alt="enter image description here" /></a></p>
<p>I have following architecture:</p>
<ul>
<li>My strimzi operator deployed into <code>strimzi-system</code> namespace and watches for namespace <code>A</code></li>
<li>Kafka brokers, exporters, entity operator deployed into namespace <code>A</code></li>
<li>Kube prometheus stack chart (with prometheus operator) deployed into <code>monitoring</code> namespace</li>
<li>I have deployed second prometheus instance into namespace <code>A</code> to configure different retention and collect metrics in another place. This prometheus instance successfully discovered targets for <code>entity-operator-metrics</code> and <code>kafka-resources-metrics</code></li>
<li><code>entity-operator-metrics</code> and <code>kafka-resources-metrics</code> pod monitors deployed to namespace A</li>
<li><code>cluster-operator-metrics</code> deployed to <code>strimzi-system</code> namespace</li>
</ul>
<p>I want to scrape metrics for <code>cluster-operator-metrics</code> deployed in <code>strimzi-system</code> namespace by prometheus deployed in <code>monitoring</code> namespace.</p>
<p>Here is manifests which I'm using:</p>
<p><strong>Prometheus</strong> instance deployed to <code>monitoring</code> namespace</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: &quot;2023-02-06T10:29:49Z&quot;
  generation: 7
  labels:
    app: kube-prometheus-stack-prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 45.0.0
    chart: kube-prometheus-stack-45.0.0
    heritage: Helm
    release: prometheus
  name: prometheus-kube-prometheus-prometheus
  namespace: monitoring
  resourceVersion: &quot;8888421&quot;
  uid: 4fbc11f6-5547-45de-ad77-40fbd37d5e1d
spec:
  alerting:
    alertmanagers:
    - apiVersion: v2
      name: prometheus-kube-prometheus-alertmanager
      namespace: monitoring
      pathPrefix: /
      port: http-web
  enableAdminAPI: false
  evaluationInterval: 30s
  externalUrl: http://prometheus-kube-prometheus-prometheus.monitoring:9090
  hostNetwork: false
  image: quay.io/prometheus/prometheus:v2.42.0
  listenLocal: false
  logFormat: logfmt
  logLevel: info
  paused: false
  podMonitorNamespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: In
      values:
      - monitoring
      - strimzi-system
  podMonitorSelector: {}
  portName: http-web
  probeNamespaceSelector: {}
  probeSelector:
    matchLabels:
      release: prometheus
  replicas: 1
  retention: 10d
  routePrefix: /
  ruleNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      release: prometheus
  scrapeInterval: 30s
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-kube-prometheus-prometheus
  serviceMonitorNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: monitoring
  serviceMonitorSelector: {}
  shards: 1
  version: v2.42.0
  walCompression: true
</code></pre>
<p><strong>Prometheus</strong> instance deployed to namespace <code>A</code></p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  creationTimestamp: &quot;2023-02-06T11:23:09Z&quot;
  generation: 7
  name: kafka-prometheus
  namespace: A
  resourceVersion: &quot;8888395&quot;
  uid: 43ab949d-4b1f-4ad2-843c-5a22e7ecc1a1
spec:
  additionalScrapeConfigs:
    key: prometheus-additional.yaml
    name: additional-scrape-configs
  enableAdminAPI: false
  evaluationInterval: 30s
  podMonitorSelector:
    matchLabels:
      app: strimzi
  replicas: 1
  resources:
    requests:
      memory: 400Mi
  retention: 30d
  scrapeInterval: 30s
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: kafka-prometheus-server
  serviceMonitorSelector: {}
  storage:
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 10Gi
        storageClassName: standard
</code></pre>
<p><strong>PodMonitor</strong> for <code>entity-operator-metrics</code> in namespace A</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  creationTimestamp: &quot;2023-02-06T11:56:15Z&quot;
  generation: 1
  labels:
    app: strimzi
  name: kafka-entity-operator-metrics
  namespace: A
  resourceVersion: &quot;4290222&quot;
  uid: f1d14d38-e1ad-404f-94ba-7ff294923608
spec:
  namespaceSelector:
    matchNames:
    - A
  podMetricsEndpoints:
  - path: /metrics
    port: healthcheck
  selector:
    matchLabels:
      app.kubernetes.io/name: entity-operator
</code></pre>
<p><strong>PodMonitor</strong> for <code>kafka-resource-metrics</code> in namespace A</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  creationTimestamp: &quot;2023-02-06T11:56:15Z&quot;
  generation: 1
  labels:
    app: strimzi
  name: kafka-resource-metrics
  namespace: A
  resourceVersion: &quot;4290228&quot;
  uid: 48a4e919-51ab-433d-9dbe-8f0352d7341d
spec:
  namespaceSelector:
    matchNames:
    - A
  podMetricsEndpoints:
  - path: /metrics
    port: tcp-prometheus
    relabelings:
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(strimzi_io_.+)
      replacement: $1
      separator: ;
    - action: replace
      regex: (.*)
      replacement: $1
      separator: ;
      sourceLabels:
      - __meta_kubernetes_namespace
      targetLabel: namespace
    - action: replace
      regex: (.*)
      replacement: $1
      separator: ;
      sourceLabels:
      - __meta_kubernetes_pod_name
      targetLabel: kubernetes_pod_name
    - action: replace
      regex: (.*)
      replacement: $1
      separator: ;
      sourceLabels:
      - __meta_kubernetes_pod_node_name
      targetLabel: node_name
    - action: replace
      regex: (.*)
      replacement: $1
      separator: ;
      sourceLabels:
      - __meta_kubernetes_pod_host_ip
      targetLabel: node_ip
  selector:
    matchExpressions:
    - key: strimzi.io/kind
      operator: In
      values:
      - Kafka
      - KafkaConnect
</code></pre>
<p><strong>PodMonitor</strong> for <code>cluster-operator-metrics</code> in namespace <code>strimzi-system</code></p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  creationTimestamp: &quot;2023-02-12T14:30:33Z&quot;
  generation: 1
  labels:
    app: strimzi
  name: kafka-cluster-operator-metrics
  namespace: strimzi-system
  resourceVersion: &quot;8255294&quot;
  uid: 0986ba6a-dc83-4000-91b4-efc6f16462e4
spec:
  namespaceSelector:
    matchNames:
    - strimzi-system
  podMetricsEndpoints:
  - path: /metrics
    port: healthcheck
  selector:
    matchLabels:
      strimzi.io/kind: cluster-operator
</code></pre>
<p>What I have tried:</p>
<ul>
<li>I have checked, that strimzi cluster operator has target labels <code>strimzi.io/kind: cluster-operator</code> and here is a part of pod manifest of strimzi operator</li>
</ul>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: &quot;2023-01-31T16:22:36Z&quot;
  generateName: strimzi-cluster-operator-77f9c84bc-
  labels:
    name: strimzi-cluster-operator
    pod-template-hash: 77f9c84bc
    strimzi.io/kind: cluster-operator
  name: strimzi-cluster-operator-77f9c84bc-h9895
  namespace: strimzi-system
</code></pre>
<ul>
<li>Also, I have checked that Prometheus service account deployed in <code>monitoring</code> has cluster role with permission to access metrics endpoint:</li>
</ul>
<pre><code>apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: &quot;2023-02-06T10:29:46Z&quot;
  labels:
    app: kube-prometheus-stack-prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 45.0.0
    chart: kube-prometheus-stack-45.0.0
    heritage: Helm
    release: prometheus
  name: prometheus-kube-prometheus-prometheus
  resourceVersion: &quot;8223589&quot;
  uid: f13d2445-b185-4b75-aa92-ba97b7393da0
rules:
- apiGroups:
  - &quot;&quot;
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  - /metrics/cadvisor
  verbs:
  - get
</code></pre>
<ul>
<li>I verified that strimzi operator pod returns metrics on endpoint <code>GET /metrics</code></li>
<li>Tried to deploy PodMonitor into <code>monitoring</code> and <code>A</code> namespace</li>
<li>Tried to disable any restrictions to discover pod monitors in Prometheus in namespace <code>monitoring</code>. It discovered 2 other pod monitors which is worked before, but still didn't find strimzi cluster operator pod monitor. Followed by this answer <a href="https://stackoverflow.com/questions/60706343/prometheus-operator-enable-monitoring-for-everything-in-all-namespaces">prometheus operator - enable monitoring for everything in all namespaces</a></li>
</ul>

Kostya Zgara
<p>This is an old one, but here goes in case anyone finds it useful. I had some headache with a similar problem. In prometheus-stack's helm values, there's this.</p>
<pre><code>## PodMonitors to be selected for target discovery.
## If {}, select all PodMonitors
##
podMonitorSelector: {}
</code></pre>
<p><em>But</em> then there's also this:</p>
<pre><code>## If true, a nil or {} value for prometheus.prometheusSpec.podMonitorSelector will cause the
## prometheus resource to be created with selectors based on values in the helm deployment,
## which will also match the podmonitors created
##
podMonitorSelectorNilUsesHelmValues: true
</code></pre>
<p>Which is true by default. The PodMonitors will be scraped if that is set to false, or kube-prometheus-stack's labels are added to the PodMonitor.</p>
