<p>I am trying to get some custom application metrics captured in golang using the prometheus client library to show up in Prometheus.</p>

<p>I have the following working:</p>

<ul>
<li><p>I have a go application which is exposing metrics on localhost:8080/metrics as described in this article:</p>

<p><a href="https://godoc.org/github.com/prometheus/client_golang/prometheus" rel="noreferrer">https://godoc.org/github.com/prometheus/client_golang/prometheus</a></p></li>
<li><p>I have a kubernates minikube running which has Prometheus, Grafana and AlertManager running using the operator from this article:</p>

<p><a href="https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus" rel="noreferrer">https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus</a></p></li>
<li><p>I created a docker image for my go app, when I run it and go to localhost:8080/metrics I can see the prometheus metrics showing up in a browser.</p></li>
<li><p>I use the following pod.yaml to deploy my docker image to a pod in k8s</p></li>
</ul>

<blockquote>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
  labels:
    zone: prod
    version: v1
  annotations:
   prometheus.io/scrape: 'true'
   prometheus.io/port: '8080'

spec:
   containers:
    - name: my-container
      image: name/my-app:latest
      imagePullPolicy: IfNotPresent
      ports:
      - containerPort: 8080
</code></pre>
</blockquote>

<ul>
<li>If I connect to my pod using:</li>
</ul>

<blockquote>
  <p>kubectl exec -it my-app-pod -- /bin/bash</p>
</blockquote>

<p>then do wget on "localhost:8080/metrics", I can see my metrics</p>

<p>So far so good, here is where I am hitting a wall. I could have multiple pods running this same image. I want to expose all the images to prometheus as targets. How do I configure my pods so that they show up in prometheus so I can report on my custom metrics?</p>

<p>Thanks for any help offered!</p>

<p>The kubernetes_sd_config directive can be used to discover all pods with a given tag. Your Prometheus.yml config file should have something like so:
<code>
- job_name: 'some-app'
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      regex: python-app
      action: keep
</code></p>

<p>The source label [__meta_kubernetes_pod_label_app] is basically using the Kubernetes api to look at pods that have a label of 'app' and whose value is captured by the regex expression, given on the line below (in this case, matching 'python-app').</p>

<p>Once you've done this Prometheus will automatically discover the pods you want and start scraping the metrics from your app.</p>

<p>Hope that helps. You can follow blog post <a href="http://burhan.io/flask-application-monitoring-with-prometheus/" rel="nofollow noreferrer">here</a> for more detail.</p>

<p><strong>Note:</strong> it is worth mentioning that at the time of writing, kubernetes_sd_config is still in beta. Thus breaking changes to configuration may occur in future releases.</p>
