<p>Recently, prometheus-operator has been promoted to stable helm chart (<a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="noreferrer">https://github.com/helm/charts/tree/master/stable/prometheus-operator</a>). </p>

<p>I'd like to understand how to add a custom application to monitoring by prometheus-operator in a k8s cluster. An example for say gitlab runner which by default provides metrics on 9252 would be appreciated (<a href="https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server" rel="noreferrer">https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server</a>).</p>

<p>I have a rudimentary yaml that obviously doesn't work but also not provides any feedback on <em>what</em> isn't working:</p>

<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gitlab-monitor
  # Change this to the namespace the Prometheus instance is running in
  namespace: default
  labels:
    app: gitlab-runner-gitlab-runner
    release: prometheus
spec:
  selector:
    matchLabels:
      app: gitlab-runner-gitlab-runner
  namespaceSelector:
    # matchNames:
    # - default
    any: true
  endpoints:
  - port: http-metrics
    interval: 15s
</code></pre>

<p>This is the prometheus configuration:</p>

<pre><code>&gt; kubectl get prometheus -o yaml

...
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
  matchLabels:
    release: prometheus
...
</code></pre>

<p>So the selectors should match. By "not working" I mean that the endpoints do not appear in the prometheus UI.</p>

<p>Thanks to Peter who showed me that it idea in principle wasn't entirely incorrect I've found the missing link. As a <code>servicemonitor</code> does monitor services (haha), I missed the part of creating a service which isn't part of the gitlab helm chart. Finally this yaml did the trick for me and the metrics appear in Prometheus:</p>

<pre><code># Service targeting gitlab instances
apiVersion: v1
kind: Service
metadata:
  name: gitlab-metrics
  labels:
    app: gitlab-runner-gitlab-runner
spec:
  ports:
  - name: metrics # expose metrics port
    port: 9252 # defined in gitlab chart
    targetPort: metrics
    protocol: TCP
  selector:
    app: gitlab-runner-gitlab-runner # target gitlab pods
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gitlab-metrics-servicemonitor
  # Change this to the namespace the Prometheus instance is running in
  # namespace: default
  labels:
    app: gitlab-runner-gitlab-runner
    release: prometheus
spec:
  selector:
    matchLabels:
      app: gitlab-runner-gitlab-runner # target gitlab service
  endpoints:
  - port: metrics
    interval: 15s
</code></pre>

<p>Nice to know: the <code>metrics</code> <code>targetPort</code> is defined in the gitlab runner chart.</p>
