<p>I am using Prometheus to monitor my Kubernetes cluster. I have set up Prometheus in a separate namespace. I have multiple namespaces and multiple pods are running. Each pod container exposes a custom metrics at this end point, <code>:80/data/metrics</code> . I am getting the Pods CPU, memory metrics etc, but how to configure Prometheus to pull data from <code>:80/data/metrics</code> in each available pod ? I have used this tutorial to set up Prometheus, <a href="https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/" rel="noreferrer">Link</a></p>

<p>You have to add this three annotation to your pods:</p>

<pre><code>prometheus.io/scrape: 'true'
prometheus.io/path: '/data/metrics'
prometheus.io/port: '80'
</code></pre>

<p>How it will work?</p>

<p>Look at the <code>kubernetes-pods</code> job of <code>config-map.yaml</code> you are using to configure prometheus,</p>

<pre><code>- job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
</code></pre>

<p>Check this three relabel configuration</p>

<pre><code>- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
</code></pre>

<p>Here, <code>__metrics_path__</code> and <code>port</code> and whether to scrap metrics from this pod are being read from pod annotations.</p>

<p>For, more details on how to configure Prometheus see <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration" rel="noreferrer">here</a>. </p>
