<p>I'm using the helm prometheus-operator chart: <a href="https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml" rel="nofollow noreferrer">https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml</a> and I expected it to get my custom metrics from my golang api as I did previously by "hardcoding" the name of the service and the port in the values.yml file:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code> scrape_configs:
    - job_name: 'custom-api'
      static_configs:
        - targets: ['custom-api-service.backend.svc.cluster.local:8000']</code></pre>
</div>
</div>
</p>

<p>However, as I have more microservices I know that it can also be done dynamically using the <code>_meta</code> tags. Example: 
<code>__meta_kubernetes_service_name</code></p>

<p>However, I haven't figure it out what should I modify from the values.yaml file to make it work.</p>

<p>Grafana is getting my cpu and memory usage from the <code>custom-api</code> but <code>custom-api</code> is not appearing in the targets tab from the prometheus dashboard which is weird...</p>

<p>These are my services:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: custom-api-service
  namespace: backend
  labels:
    service: custom-api-service
spec:
  type: NodePort
  ports:
    - port: 8000
      targetPort: 8000
      nodePort: 30080
      protocol: TCP
      name: custom-api
  selector:
    component: goapi

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: servicemonitor
  namespace: backend
  labels:
    service: servicemonitor
spec:
  selector:
    matchLabels:
      service: custom-api-service
  endpoints:
  - port: custom-api</code></pre>
</div>
</div>
</p>

alex
<p>You will have to create a <a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md#servicemonitor" rel="nofollow noreferrer">Service monitor CRD</a> to scrape your metrics. </p>

<p>Let's say you have a k8s service (here: <code>example-app</code>) which is used to communicate with your microservices. Make sure that your microservice exposes Prometheus metrics at a certain port and the k8s service also includes that port (here: <code>prom</code>).</p>

<pre class="lang-yaml prettyprint-override"><code>kind: Service
apiVersion: v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
  - name: prom
    port: 8080
  - name: other-port
    port: xxxx
</code></pre>

<p>This Service object is discovered by a ServiceMonitor, which selects in the same way. You need to make sure that the <code>matchLabels</code> of serviceMonitor object matchs the <code>metadata.labels</code> of the service.</p>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: prom
</code></pre>

<p>Once you have created the serviceMonitor object, the operator controller will do the rest for you (ie. update the Prometheus configuration). You can also provide custom configuration via serviceMonitor object.</p>

<p>For more details visit <a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md#related-resources" rel="nofollow noreferrer">Getting started with Prometheus operator</a>.
<a href="https://i.stack.imgur.com/ksdLA.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ksdLA.jpg" alt="service monitor"></a></p>
