<p>I want to monitor a couple applications running on a Kubernetes cluster in namespaces named <code>development</code> and <code>production</code> through prometheus-operator.</p>

<p>Installation command used (as per <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="noreferrer">Github</a>) is:</p>

<p><code>helm install prometheus-operator stable/prometheus-operator -n production --set prometheusOperator.enabled=true,prometheus.service.type=NodePort,prometheusOperator.service.type=NodePort,alertmanager.service.type=NodePort,grafana.service.type=NodePort,grafana.service.nodePort=30906</code></p>

<p>What parameters do I need to add to above command to have <code>prometheus-operator</code> discover and monitor all apps/services/pods running in all <code>namespaces</code>? </p>

<p>With this, <code>Service Discovery</code> only shows some <code>prometheus-operator</code> related <code>services</code>, but not the app that I am running within 'production' namespace even though <code>prometheus-operator</code> is installed in the same <code>namespace</code>. </p>

<p>Anything I am missing?</p>

<p>Note - Am running performing all actions using the same user (which uses the <code>$HOME/.kube/config file</code>), so I assume permissions are not an issue. </p>

<p>kubectl version - v1.17.3
helm version - 3.1.2</p>

<p>P.S.  There are numerous articles on this on different forums, but am still not finding simple and direct answers for this. </p>

<p>I had the same problem. After some investigation answering with more details.</p>
<p>I've installed Prometheus stack via <a href="https://github.com/prometheus-community/helm-charts" rel="noreferrer">Helm charts</a> which include Prometheus operator chart directly as a sub-project. Prometheus operator monitors namespaces specified by the following helm values:</p>
<pre><code>prometheusOperator:
  namespaces: ''
  denyNamespaces: ''
  prometheusInstanceNamespaces: ''
  alertmanagerInstanceNamespaces: ''
  thanosRulerInstanceNamespaces: ''
</code></pre>
<p>The <code>namespaces</code> value specifies monitored namespaces for ServiceMonitor and PodMonitor CRDs. Other CRDs have their own settings, which if not set, default to <code>namespaces</code>. Helm values are passed as command-line arguments to the operator. See <a href="https://github.com/prometheus-community/helm-charts/blob/7a8be84aae4fbe24562f367e7b1e7cb52fa70fd1/charts/kube-prometheus-stack/templates/prometheus-operator/deployment.yaml#L59" rel="noreferrer">here</a> and <a href="https://github.com/prometheus-community/helm-charts/blob/7a8be84aae4fbe24562f367e7b1e7cb52fa70fd1/charts/kube-prometheus-stack/templates/prometheus-operator/deployment.yaml#L81" rel="noreferrer">here</a>.</p>
<p>Prometheus CRDs are picked up by the operator from the mentioned namespaces, by default - everywhere. However, as the operator is designed with multiple simultaneous Prometheus releases in mind, what to pick up by a particular Prometheus app instance is controlled by the corresponding Prometheus CRD. CRDs selectors and corresponding namespaces selectors are controlled via the following Helm values:</p>
<pre><code>prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: true
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}
</code></pre>
<p>Similar values are present for other CRDs: alertmanagerConfigXXX, ruleNamespaceXXX, podMonitorXXX, probeXXX. <code>XXXSelectorNilUsesHelmValues</code> set to true, means to look for CRD with particular release label, e.g. <code>release=myrelease</code>. See <a href="https://github.com/prometheus-community/helm-charts/blob/7a8be84aae4fbe24562f367e7b1e7cb52fa70fd1/charts/kube-prometheus-stack/templates/prometheus/prometheus.yaml#L106" rel="noreferrer">here</a>.</p>
<p>Empty selector (for a namespace, CRD, or any other object) means no filtering. So for Prometheus object to pick up a ServiceMonitor from the other namespaces there are few options:</p>
<ul>
<li>Set <code>serviceMonitorSelectorNilUsesHelmValues: false</code>. This leaves serviceMonitorSelector empty.</li>
<li>Apply the release label, e.g. <code>release=myrelease</code>, to your ServiceMonitor CRD.</li>
<li>Set a non-empty serviceMonitorSelector that matches your ServiceMonitor.</li>
</ul>
<p>For the curious ones here are links to the operator sources:</p>
<ul>
<li><a href="https://github.com/prometheus-operator/prometheus-operator/blob/072f7be753332ee61f2ba998aceb15192677f5b8/pkg/prometheus/operator.go#L946" rel="noreferrer">Enqueue of Prometheus CRD processing</a></li>
<li><a href="https://github.com/prometheus-operator/prometheus-operator/blob/072f7be753332ee61f2ba998aceb15192677f5b8/pkg/prometheus/operator.go#L1023" rel="noreferrer">Processing of Prometheus CRD</a></li>
</ul>
