<p>I am able to scrape Prometheus metrics from a Kubernetes service using this Prometheus job configuration:</p>

<pre><code>- job_name: 'prometheus-potapi'
  static_configs:
  - targets: ['potapi-service.potapi:1234']
</code></pre>

<p>It uses Kubernetes DNS and it gives me the metrics from any of my three pods I use for my service.</p>

<p>I would like to see the result for each pod. </p>

<p>I am able to see the data I want using this configuration:</p>

<pre><code>- job_name: 'prometheus-potapi-pod'
  static_configs:
  - targets: ['10.1.0.126:1234']
</code></pre>

<p>I have searched and experimented using the service discovery mechanism available in Prometheus. Unfortunately, I don't understand how it should be setup. The <a href="https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml" rel="nofollow noreferrer">service discovery reference</a> isn't really helpful if you don't know how it works.</p>

<p>I am looking for an example where the job using the IP number is replaced with some service discovery mechanism. Specifying the IP was enough for me to see that the data I'm looking for is exposed.</p>

<p>The pods I want to scrape metrics from all live in the same namespace, <code>potapi</code>. </p>

<p>The metrics are always exposed through the same port, <code>1234</code>.</p>

<p>Finally, the are all named like this:</p>

<pre><code>potapi-deployment-754d96f855-lkh4x
potapi-deployment-754d96f855-pslgg
potapi-deployment-754d96f855-z2zj2 
</code></pre>

<p>When I do </p>

<pre><code>kubectl describe pod potapi-deployment-754d96f855-pslgg -n potapi
</code></pre>

<p>I get this description:</p>

<pre><code>Name:           potapi-deployment-754d96f855-pslgg
Namespace:      potapi
Node:           docker-for-desktop/192.168.65.3
Start Time:     Tue, 07 Aug 2018 14:18:55 +0200
Labels:         app=potapi
  pod-template-hash=3108529411
Annotations:    &lt;none&gt;
Status:         Running
IP:             10.1.0.127
Controlled By:  ReplicaSet/potapi-deployment-754d96f855
Containers:
  potapi:
    Container ID:   docker://72a0bafbda9b82ddfc580d79488a8e3c480d76a6d17c43d7f7d7ab18458c56ee
    Image:          potapi-service
    Image ID:       docker://sha256:d64e94c2dda43c40f641008c122e6664845d73cab109768efa0c3619cb0836bb
    Ports:          4567/TCP, 4568/TCP, 1234/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 07 Aug 2018 14:18:57 +0200
    Ready:          True
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4fttn (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  default-token-4fttn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-4fttn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
  node.kubernetes.io/unreachable:NoExecute for 300s
Events:          &lt;none&gt;
</code></pre>

<p>How would you rewrite the job definition given these prerequisites?</p>

<p>Here they use <a href="https://github.com/prometheus/prometheus/blob/v2.3.2/documentation/examples/prometheus-kubernetes.yml#L253-L256" rel="nofollow noreferrer"><code>example.io/scrape=true</code></a> (and similar annotations for specifying the <a href="https://github.com/prometheus/prometheus/blob/v2.3.2/documentation/examples/prometheus-kubernetes.yml#L266-L269" rel="nofollow noreferrer">scrape port</a> and the <a href="https://github.com/prometheus/prometheus/blob/v2.3.2/documentation/examples/prometheus-kubernetes.yml#L258-L260" rel="nofollow noreferrer">scrape path</a> if it's not <code>/metrics</code>), which is how one achieves the "autodiscovery" part.</p>

<p>If you apply that annotation -- and the relevant config snippets in the Prom config -- to a <code>Service</code>, then Prom will scrape the port and path on the <code>Service</code>, meaning you will have stats for the <code>Service</code> itself, and not the individual Endpoints behind it. Similarly, if you label the <code>Pod</code>s, you will gather metrics for the <code>Pod</code>s but they would need to be rolled up to have a cross-<code>Pod</code> view of the state of affairs. There are multiple different resource types that can be autodiscovered, including <a href="https://prometheus.io/docs/prometheus/2.3/configuration/configuration/#node" rel="nofollow noreferrer">node</a> and <a href="https://prometheus.io/docs/prometheus/2.3/configuration/configuration/#ingress" rel="nofollow noreferrer">ingress</a>, also. They all behave similarly.</p>

<p>Unless you have grave CPU or storage concerns for your Prom instance, I absolutely wouldn't enumerate the scrape targets in the config like that: I would use the scrape annotations, meaning you can change who is scraped, what port, etc. without having to reconfigure Prom each time.</p>

<p>Be aware that if you want to use their example as-is, and you want to apply those annotations from within the kubernetes resource YAML, ensure that you quote the <code>: 'true'</code> value, otherwise YAML will promote that to be a boolean literal, and kubernetes annotations can only be string values.</p>

<p>Applying the annotations from the command line will work just fine:</p>

<pre><code>kubectl annotate pod -l app=potapi example.io/scrape=true
</code></pre>

<p>(BTW, they use <code>example.io/</code> in their example, but there is nothing special about that string except it namespaces the <code>scrape</code> part to keep it from colliding with something else named <code>scrape</code>. So feel free to use your organization's namespace if you wish to avoid having something weird named <code>example.io/</code> in your cluster)</p>
