<p>I installed <code>kube-prometheus-stack</code> via helm:</p>
<pre class="lang-bash prettyprint-override"><code>kubectl create namespace monitoring &amp;&amp; \
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts &amp;&amp; \
helm repo update &amp;&amp; \
helm install -n monitoring prometheus-stack prometheus-community/kube-prometheus-stack
</code></pre>
<p><img src="https://user-images.githubusercontent.com/29863388/178033176-2a04e4ab-899d-4ee7-9735-712588058e8f.png" alt="image" /></p>
<p>Then proceeded to deploy a fastapi application which has a metrics endpoint that Prometheus is supposed to scrape. <code>/metrics</code> endpoint works fine as seen below:</p>
<pre class="lang-py prettyprint-override"><code>from starlette_prometheus import metrics, PrometheusMiddleware
from fastapi import FastAPI, Request


app = FastAPI()

# Add Prometheus metrics as middleware
app.add_middleware(PrometheusMiddleware)
app.add_route(&quot;/metrics&quot;, metrics)

</code></pre>
<p><img src="https://user-images.githubusercontent.com/29863388/178030474-0c3d8b76-9a50-4b36-b414-9012548c7a9e.png" alt="image" /></p>
<p>I expected both the target and service discovery to work but only service discovery appears to be working</p>
<ul>
<li><p>Prometheus Operator version:</p>
<p><code>2.36.1</code></p>
  
  
</li>
<li><p>Kubernetes version information:</p>
<p><code>Client Version: v1.24.2</code></p>
  
</li>
<li><p>Kubernetes cluster kind:</p>
<p><code>Minikube v1.25.2-1</code></p>
</li>
</ul>
<p>Here are the manifests for the deployed application:</p>
<ol>
<li>Namespace and deployment</li>
</ol>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Namespace
metadata:
  name: code-detector-demo

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: code-detector
  namespace: code-detector-demo
  labels:
    release: prometheus-stack
spec:
  replicas: 1
  selector:
    matchLabels:
      app: code-detector
      release: prometheus-stack
  template:
    metadata:
      labels:
        app: code-detector
        release: prometheus-stack
    spec:
      containers:
      - name: code-detector
        image: &lt;MY-IMAGE:TAG&gt;
        resources:
          limits:
            memory: 512Mi
            cpu: 1000m
        ports:
        - containerPort: 8000
</code></pre>
<ol start="2">
<li>Service</li>
</ol>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: &quot;true&quot;
  labels:
    release: prometheus-stack
  name: code-detector-service
  namespace: code-detector-demo
spec:
  selector:
    app: code-detector
    release: prometheus-stack
  ports:
    - name: code-detector-port
      port: 8000
      targetPort: 8000
</code></pre>
<ol start="3">
<li>ServiceMonitor</li>
</ol>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: code-detector-servicemonitor
  # same namespace that Prometheus is running in
  namespace: monitoring
  labels:
    app: code-detector
    release: prometheus-stack
spec:
  selector:
    matchLabels:
      app: code-detector
      release: prometheus-stack
  endpoints:
    - path: /metrics
      port: code-detector-port
      interval: 15s
  namespaceSelector:
    matchNames:
    - code-detector-demo # namespace where the app is running
</code></pre>
<p>Here's the labels in service discovery:
<img src="https://user-images.githubusercontent.com/29863388/178032152-f8384905-55db-4715-a84b-b53161af1eb7.png" alt="image" /></p>
<p>Targets pane don't show any picked up scraping job:
<img src="https://user-images.githubusercontent.com/29863388/178033037-323525cf-b9ab-46ec-a388-9c15dc178192.png" alt="image" /></p>
<p>What could I be doing wrong?</p>

Mysterio
<p>You have in your servicemonitor:</p>
<pre><code>spec:
  selector:
    matchLabels:
      app: code-detector
      release: prometheus-stack
</code></pre>
<p>which basicaly means &quot;get all endpoints from services with labels app=code-detector and release=prometheus-stack&quot;.</p>
<p>Your service don't have the label app=code-detector.</p>
