<p>I have installed kube-prometheus-stack as a <strong>dependency</strong> in my helm chart on a local docker for Mac Kubernetes cluster v1.19.7. I can view the default prometheus targets provided by the kube-prometheus-stack.</p>
<p>I have a python flask service that provides metrics which I can view successfully in the kubernetes cluster using <code>kubectl port forward</code>.</p>
<p>However, I am unable to get these metrics displayed on the prometheus targets web interface.</p>
<p>The <a href="https://hub.kubeapps.com/charts/prometheus-community/kube-prometheus-stack#!" rel="noreferrer">kube-prometheus-stack</a> documentation states that Prometheus.io/scrape does not support annotation-based discovery of services. Instead the the reader is referred to the concept of <code>ServiceMonitors</code> and <code>PodMonitors</code>.</p>
<p>So, I have configured my service as follows:</p>
<pre class="lang-yaml prettyprint-override"><code>---
kind:                       Service
apiVersion:                 v1  
metadata:
  name:                     flask-api-service                    
  labels:
    app:                    flask-api-service
spec:
  ports:
    - protocol:             TCP 
      port:                 4444
      targetPort:           4444
      name:                 web 
  selector:
    app:                    flask-api-service                    
    tier:                   backend 
  type:                     ClusterIP
---
apiVersion:                 monitoring.coreos.com/v1
kind:                       ServiceMonitor
metadata:
  name:                     flask-api-service
spec:
  selector:
    matchLabels:
      app:                  flask-api-service
  endpoints:
  - port:                   web 
</code></pre>
<p>Subsequently, I have setup a port forward to view the metrics:</p>
<pre class="lang-sh prettyprint-override"><code>Kubectl port-forward prometheus-flaskapi-kube-prometheus-s-prometheus-0 9090
</code></pre>
<p>Then visited prometheus web page at <code>http://localhost:9090</code></p>
<p>When I select the Status-&gt;Targets menu option, my flask-api-service is not displayed.</p>
<p>I know that the service is up and running and I have checked that I can view the metrics for a pod for my flask-api-service using <code>kubectl port-forward &lt;pod name&gt; 4444</code>.</p>
<p>Looking at a similar <a href="https://stackoverflow.com/a/65648944">issue</a> it looks as though there is a  configuration value <code>serviceMonitorSelectorNilUsesHelmValues</code> that defaults to true. Setting this to false makes the operator look outside it’s release labels in helm??</p>
<p>I tried adding this to the <code>values.yml</code> of my helm chart in addition to the <code>extraScrapeConfigs</code> configuration value. However, the <em>flask-api-service</em> still does not appear as an additional target on the prometheus web page when clicking the Status-&gt;Targets menu option.</p>
<pre class="lang-yaml prettyprint-override"><code>prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
  extraScrapeConfigs: |
    - job_name: 'flaskapi'
    static_configs:
      - targets: ['flask-api-service:4444']
</code></pre>
<p>How do I get my <em>flask-api-service</em> recognised on the prometheus targets page at <code>http://localhost:9090</code>?</p>
<p>I am installing Kube-Prometheus-Stack as a dependency via my helm chart with default values as shown below:</p>
<p><strong>Chart.yaml</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v2
appVersion: &quot;0.0.1&quot;
description: A Helm chart for flaskapi deployment
name: flaskapi
version: 0.0.1
dependencies:
- name: kube-prometheus-stack
  version: &quot;14.4.0&quot;
  repository: &quot;https://prometheus-community.github.io/helm-charts&quot;
- name: ingress-nginx
  version: &quot;3.25.0&quot;
  repository: &quot;https://kubernetes.github.io/ingress-nginx&quot;
- name: redis
  version: &quot;12.9.0&quot;
  repository: &quot;https://charts.bitnami.com/bitnami&quot;
</code></pre>
<p><strong>Values.yaml</strong></p>
<pre class="lang-yaml prettyprint-override"><code>docker_image_tag: dcs3spp/
hostname: flaskapi-service
redis_host: flaskapi-redis-master.default.svc.cluster.local 
redis_port: &quot;6379&quot;

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
  extraScrapeConfigs: |
    - job_name: 'flaskapi'
    static_configs:
      - targets: ['flask-api-service:4444']
</code></pre>

<p>Prometheus custom resource definition has a field called <code>serviceMonitorSelector</code>. Prometheus only listens to those matched serviceMonitor. In case of helm deployment it is your release name.</p>
<pre><code>release: {{ $.Release.Name | quote }}
</code></pre>
<p>So adding this field in your serviceMonitor  should solve the issue. Then you serviceMonitor manifest file will be:</p>
<pre><code>
apiVersion:                 monitoring.coreos.com/v1
kind:                       ServiceMonitor
metadata:
  name:                     flask-api-service
  labels:
      release: &lt;your_helm_realese_name_&gt;
spec:
  selector:
    matchLabels:
      app:                  flask-api-service
  endpoints:
  - port:                   web 
</code></pre>
