<p>We have a Prometheus running in our cluster and we are able to use grafana to watch our cluster / pods metrics, now I want to add some custom metrics , is there a way to do it ? if so how should I connect the code to Prometheus , I mean if i write golang program using Prometheus API , and deploy it as docker to k8s , now does the program know to connect with Prometheus ?
e.g. this program is exposing data to the /metrics endpoint but what else should I do to make prom to be able to read this data ? </p>

<p><a href="https://gist.github.com/sysdig-blog/3640f39a7bb1172f986d0e2080c64a75#file-prometheus-metrics-golang-go" rel="nofollow noreferrer">https://gist.github.com/sysdig-blog/3640f39a7bb1172f986d0e2080c64a75#file-prometheus-metrics-golang-go</a></p>

<p>You probably want Prometheus to discover the targets to scrape from automatically, if so you should configure <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config" rel="nofollow noreferrer">K8s service discovery</a> in Prometheus to discover pods, nodes and services from your cluster (maybe you already did something like this since you are already monitoring k8s metrics).</p>

<p>To get your go application monitored you can for example add annotations to your pods or to your services to enable scraping from this targets and define where the metrics are available (path, port). However this depends on your scrape configuration and relabeling. Good example can be found <a href="https://linuxacademy.com/blog/kubernetes/running-prometheus-on-kubernetes/" rel="nofollow noreferrer">here</a></p>

<p>If you are using <a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md" rel="nofollow noreferrer">Prometheus Operator</a> you need to define a servicemonitor resource instead.</p>
