<p>I'm using <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus" rel="nofollow noreferrer">this prometheus chart</a>. In the documentation it says</p>
<blockquote>
<p>In order to get prometheus to scrape pods, you must add annotations to the the pods as below:</p>
<pre><code>metadata:   
  annotations:
    prometheus.io/scrape: &quot;true&quot;
    prometheus.io/path: /metrics
    prometheus.io/port: &quot;8080&quot;
</code></pre>
</blockquote>
<p>So I have created a service like this</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: nodejs-client-service
  labels:
    app: nodejs-client-app
  annotations:
    prometheus.io/scrape: &quot;true&quot;
    prometheus.io/path: /metrics
    prometheus.io/port: &quot;5000&quot;
spec:
  type: LoadBalancer
  selector:
    app: nodejs-client-app
  ports:
    - protocol: TCP
      name: http
      port: 80
      targetPort: 5000
</code></pre>
<p>But my Service won't show up in the prometheus targets. What am I missing?</p>

<p>I ran into the same problem with the <code>stable/prometheus-operator</code> chart.  I tried adding the annotations above both to the pods and to the service and neither worked.</p>
<p>For me, the solution was to add a <a href="https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md" rel="nofollow noreferrer">ServiceMonitor</a> object.  Once added, Prometheus dynamically discovered my service:</p>
<p><a href="https://i.stack.imgur.com/shgGp.png" rel="nofollow noreferrer">Fig 1: target list</a></p>
<p><a href="https://i.stack.imgur.com/sLRzq.png" rel="nofollow noreferrer">Fig 2: dynamically added scrape_config</a></p>
<h2>Solution Example</h2>
<p>This single command fixed the problem: <code>kubectl apply -f service-monitor.yml</code></p>
<pre class="lang-yaml prettyprint-override"><code># service-monitor.yml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: prom
  name: eztype
  namespace: default
spec:
  endpoints:
    - path: /actuator/prometheus
      port: management
  namespaceSelector:
    matchNames:
      - default
  selector:
    matchLabels:
      app.kubernetes.io/name: eztype
</code></pre>
<p>Here, my pods and service were annotated with the name <code>eztype</code> and were exposing metrics on port 8282 under the indicated path.  For completeness here's the relavant part of my service definition:</p>
<pre class="lang-yaml prettyprint-override"><code># service definition (partial)
spec:
  clusterIP: 10.128.156.246
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: management
    port: 8282
    protocol: TCP
    targetPort: 8282
</code></pre>
<p>It's worth noting that ServiceMonitor objects are used in the Prometheus chart itself:</p>
<pre><code>$ kubectl get servicemonitors -n monitor
NAME                                               AGE
prom-prometheus-operator-alertmanager              14d
prom-prometheus-operator-apiserver                 14d
prom-prometheus-operator-coredns                   14d
prom-prometheus-operator-grafana                   14d
prom-prometheus-operator-kube-controller-manager   14d
prom-prometheus-operator-kube-etcd                 14d
prom-prometheus-operator-kube-proxy                14d
prom-prometheus-operator-kube-scheduler            14d
prom-prometheus-operator-kube-state-metrics        14d
prom-prometheus-operator-kubelet                   14d
prom-prometheus-operator-node-exporter             14d
prom-prometheus-operator-operator                  14d
prom-prometheus-operator-prometheus                14d
</code></pre>
