<p>I am deploying prometheus using stable/prometheus-operator chart. It is installed in <code>monitoring</code> namespace. In the <code>default</code> namespace I have a pod running named <code>my-pod</code> with three replicas. This pod spits out metrics on port 9009 (I have verified this by doing k port-forward and validating the metrics show up in localhost:9009). I would like prometheus-operator to scrape these metrics. So I added the configuration below to <code>values.yaml</code></p>

<pre><code>prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
    - job_name: 'my-pod-job'
      scrape_interval: 15s
      kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
          - default
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_name]
        action: keep   
        regex: 'my-pod'  
</code></pre>

<p>I then install prometheus using the command below:</p>

<pre><code>helm upgrade --install prometheus stable/prometheus-operator \
--set kubeEtcd.enabled=false \
--set kubeControllerManager.enabled=false \
--set kubeScheduler.enabled=false \
--set prometheusOperator.createCustomResource=true \
--set grafana.smtp.existingSecret=smtp-secret \
--set kubelet.serviceMonitor.https=true \
--set kubelet.enabled=true \
-f values.yaml --namespace monitoring
</code></pre>

<p>However, when I go to <code>/service-discover</code> I see </p>

<pre><code>my-pod-job (0/40 active targets)
</code></pre>

<p><strong>Question</strong></p>

<p>How can I configure prometheus such that it scrapes metrics from pods running in default namespace and spitting out metrics on port 9009?</p>

Anthony
<p>I'll try to give a <strong>fully working answer</strong> with explanation.</p>
<h3>Short answer</h3>
<p>When working with prometheus <strong>operator</strong> adding the following annotations is not enough:</p>
<pre><code>prometheus.io/path: &lt;path_to_scrape&gt;
prometheus.io/port: &quot;&lt;port&gt;&quot;
prometheus.io/scrape: &quot;true&quot;
</code></pre>
<p>You also need to add the following section in the helm chart under <code>prometheusSpec.additionalScrapeConfigs</code> :</p>
<pre><code>prometheusSpec:
  additionalScrapeConfigs:
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
</code></pre>
<p>Notice the <code>source_labels</code> - we're telling Prometheus to scrap pods with the <code>prometheus.io/scrape</code> annotation.</p>
<h3>But why the auto-discovery feature is different in the Prometheus Operator?</h3>
<p>If you worked before with Prometheus directly, you probably wondering why it is not scraping the annotations directly.</p>
<p>According to the Prometheus upstream team the annotation approach is very limited compare to the <code>*Monitor</code> (<code>ServiceMonitor</code> and <code>PodMonitor</code>).
Read more in <a href="https://github.com/prometheus-operator/prometheus-operator/issues/1547#issuecomment-401092041" rel="nofollow noreferrer">here</a>.</p>
<h3>I've added the relevant additionalScrapeConfigs and it is not working</h3>
<p>Notice that you can setup <code>additionalScrapeConfigs</code> as an <a href="https://github.com/rapid7/csp-ops-helm/blob/development/3rd-party/kube-prometheus-stack/values.yaml#L2661" rel="nofollow noreferrer">array</a> or as an <a href="https://github.com/rapid7/csp-ops-helm/blob/development/3rd-party/kube-prometheus-stack/values.yaml#L2692" rel="nofollow noreferrer">inline block</a>. In both cases, indentation issues might happen, so go to the Prometheus CRD (under the <code>monitoring.coreos.com/v1</code> api) and view the status, you might see some errors there.</p>
<h3>Where can I view the scrapped pods?</h3>
<p>In the Prometheus UI - go to Service Discovery:</p>
<p><a href="https://i.stack.imgur.com/NjKp7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NjKp7.png" alt="enter image description here" /></a></p>
<p>Where you will see all the jobs you've added in (<em>kubernetes-pods</em> and <em>kubernetes-service-endpoints</em> in my case):</p>
<p><a href="https://i.stack.imgur.com/Nq37j.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Nq37j.png" alt="enter image description here" /></a></p>
