<p>I'm using the helm prometheus-operator chart: <a href="https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml" rel="nofollow noreferrer">https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml</a> and I expected it to get my custom metrics from my golang api as I did previously by "hardcoding" the name of the service and the port in the values.yml file:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code> scrape_configs:
    - job_name: 'custom-api'
      static_configs:
        - targets: ['custom-api-service.backend.svc.cluster.local:8000']</code></pre>
</div>
</div>
</p>

<p>However, as I have more microservices I know that it can also be done dynamically using the <code>_meta</code> tags. Example: 
<code>__meta_kubernetes_service_name</code></p>

<p>However, I haven't figure it out what should I modify from the values.yaml file to make it work.</p>

<p>Grafana is getting my cpu and memory usage from the <code>custom-api</code> but <code>custom-api</code> is not appearing in the targets tab from the prometheus dashboard which is weird...</p>

<p>These are my services:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: custom-api-service
  namespace: backend
  labels:
    service: custom-api-service
spec:
  type: NodePort
  ports:
    - port: 8000
      targetPort: 8000
      nodePort: 30080
      protocol: TCP
      name: custom-api
  selector:
    component: goapi

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: servicemonitor
  namespace: backend
  labels:
    service: servicemonitor
spec:
  selector:
    matchLabels:
      service: custom-api-service
  endpoints:
  - port: custom-api</code></pre>
</div>
</div>
</p>

alex
<p>The <code>Prometheus</code> resource includes a field called <code>serviceMonitorSelector</code>, which defines a selection of <code>ServiceMonitors</code> to be used. By default and before the version v0.19.0, <code>ServiceMonitors</code> must be installed in the same namespace as the Prometheus instance. With the Prometheus Operator v0.19.0 and above, <code>ServiceMonitors</code> can be selected outside the Prometheus namespace via the <code>serviceMonitorNamespaceSelector</code> field of the <code>Prometheus</code> resource</p>

<p>In the monitoring namespace create a <code>Prometheus</code> object which selects the <code>ServiceMonitor</code> by label <code>service: servicemonitor</code></p>

<pre><code>apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: &lt;service-account-name&gt;
  serviceMonitorSelector:
    matchLabels:
      service: servicemonitor
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false
</code></pre>

<p>The <code>serviceAccountName</code> you can find out in <code>monitoring</code> namespace as <code>helmreleasename-prometheus-operator-prometheus</code></p>
