<p>I want to use my already existing Prometheus and Grafana instances in the monitoring namespace to emulate what <a href="https://github.com/SeldonIO/seldon-core/tree/master/helm-charts/seldon-core-analytics" rel="nofollow noreferrer"><code>seldon-core-analytics</code></a> is doing. I'm using the prometheus community helm charts and installed <code>kube-prometheus-stack</code>  on k8s. Here's what I've done so far:</p>
<p>In the <code>values.yaml</code> file, under the prometheus config, I added the following annotations:</p>
<pre><code>annotations:
  prometheus.io/scrape: &quot;true&quot;
  prometheus.io/path: &quot;/prometheus
</code></pre>
<p>Next, I looked at the <code>prometheus-config.yaml</code> in their Github repo  and copied and pasted the configuration in a configmap file.</p>
<p>Also, created a ServiceMonitor</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: seldon-servicemonitor-default
  labels:
    seldon-monitor: seldon-default
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/managed-by: seldon-core
  endpoints:
    - interval: 15s
      path: /metrics
      port: http
    - interval: 15s
      path: /prometheus
      port: http
  namespaceSelector:
    matchNames:
      - seldon
      - default
      - monitoring
</code></pre>
<p>No errors with the above steps so far, but it doesn't appear as though the prometheus instance is able to scrape the metrics from a model I deployed on a different namespace. What other configuration do I need to do so that my own Prometheus and Grafana instances can gather and visualize the metrics from my seldon deployed models? The documentation doesn't really explain how to do this on your own instances, and the one they provide to you through <code>seldon-core-analytics</code> isn't production-ready.</p>

<p>Prometheus configuration in <code>seldon-core-analytics</code> is quite standard. It is based on built-in Kubernetes service discovery and it uses annotations to find scraping targets:</p>
<pre class="lang-yaml prettyprint-override"><code>annotations:
  prometheus.io/scrape: true
  prometheus.io/path: /metrics
  prometheus.io/scheme: http
  prometheus.io/port: 9100
</code></pre>
<p>In <a href="https://github.com/SeldonIO/seldon-core/blob/master/helm-charts/seldon-core-operator/values.yaml" rel="nofollow noreferrer">their example configuration</a> prometheus will target pods, services, and endpoints with <code>prometheus.io/scrape: true</code> annotation on them. The other three labels are used to override default scraping parameters per target. Thus if you have a config as in the example, you only need to put some of these annotations on pods.</p>
<p>The way <code>kube-prometheus-stack</code> works is different. It uses <a href="https://github.com/prometheus-operator/prometheus-operator" rel="nofollow noreferrer">prometheus operator</a> and CRDs to shape the configuration. <a href="https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md" rel="nofollow noreferrer">This</a> design document describes purpose of each CRD.</p>
<p>You need to create a <code>ServiceMonitor</code> resource in order to define a scraping rule for new services. <code>ServiceMonitor</code> itself should have labels as defined in prometheus resource (another CRD) under <code>serviceMonitorSelector</code> key. It is hard to provide you with a working example in these circumstances but <a href="https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md#related-resources" rel="nofollow noreferrer">this</a> short guide should be enough to understand what to do.</p>
<p>I suggest you describe one of the <code>ServiceMonitor</code>s that you have, then create a new one changing labels under <code>matchLabels</code>. Do not change the namespace in a new object, prometheus operator does not look for <code>ServiceMonitor</code>s in other namespaces by default. To make <code>ServiceMonitor</code> discover targets in all namespaces the <code>namespaceSelector</code> has to be empty:</p>
<pre class="lang-yaml prettyprint-override"><code>spec:
  namespaceSelector:
    any: true
</code></pre>
