<p>I want my prometheus server to scrape metrics from a pod.</p>
<p>I followed these steps:</p>
<ol>
<li>Created a pod using deployment - <code>kubectl apply -f sample-app.deploy.yaml</code></li>
<li>Exposed the same using <code>kubectl apply -f sample-app.service.yaml</code></li>
<li>Deployed Prometheus server using <code>helm upgrade -i prometheus prometheus-community/prometheus -f prometheus-values.yaml</code></li>
<li>created a serviceMonitor using <code>kubectl apply -f service-monitor.yaml</code> to add a target for prometheus.</li>
</ol>
<p>All pods are running, but when I open prometheus dashboard, <strong>I don't see <em>sample-app service</em> as prometheus target, under status&gt;targets in dashboard UI.</strong></p>
<p>I've verified following:</p>
<ol>
<li>I can see <code>sample-app</code> when I execute <code>kubectl get servicemonitors</code></li>
<li>I can see sample-app exposes metrics in prometheus format under at <code>/metrics</code></li>
</ol>
<p>At this point I debugged further, entered into the prometheus pod using
<code>kubectl exec -it pod/prometheus-server-65b759cb95-dxmkm -c prometheus-server sh</code>
, and saw that proemetheus configuration (/etc/config/prometheus.yml) didn't have sample-app as one of the jobs so I edited the configmap using</p>
<p><code>kubectl edit cm prometheus-server -o yaml</code>
Added</p>
<pre><code>    - job_name: sample-app
        static_configs:
        - targets:
          - sample-app:8080
</code></pre>
<p>Assuming all other fields such as <strong>scraping</strong> interval, scrape_timeout stays default.</p>
<p>I can see the same has been reflected in /etc/config/prometheus.yml, but still prometheus dashboard doesn't show <code>sample-app</code> as targets under status&gt;targets.</p>
<p>following are yamls for prometheus-server and service monitor.</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    autopilot.gke.io/resource-adjustment: '{&quot;input&quot;:{&quot;containers&quot;:[{&quot;name&quot;:&quot;prometheus-server-configmap-reload&quot;},{&quot;name&quot;:&quot;prometheus-server&quot;}]},&quot;output&quot;:{&quot;containers&quot;:[{&quot;limits&quot;:{&quot;cpu&quot;:&quot;500m&quot;,&quot;ephemeral-storage&quot;:&quot;1Gi&quot;,&quot;memory&quot;:&quot;2Gi&quot;},&quot;requests&quot;:{&quot;cpu&quot;:&quot;500m&quot;,&quot;ephemeral-storage&quot;:&quot;1Gi&quot;,&quot;memory&quot;:&quot;2Gi&quot;},&quot;name&quot;:&quot;prometheus-server-configmap-reload&quot;},{&quot;limits&quot;:{&quot;cpu&quot;:&quot;500m&quot;,&quot;ephemeral-storage&quot;:&quot;1Gi&quot;,&quot;memory&quot;:&quot;2Gi&quot;},&quot;requests&quot;:{&quot;cpu&quot;:&quot;500m&quot;,&quot;ephemeral-storage&quot;:&quot;1Gi&quot;,&quot;memory&quot;:&quot;2Gi&quot;},&quot;name&quot;:&quot;prometheus-server&quot;}]},&quot;modified&quot;:true}'
    deployment.kubernetes.io/revision: &quot;1&quot;
    meta.helm.sh/release-name: prometheus
    meta.helm.sh/release-namespace: prom
  creationTimestamp: &quot;2021-06-24T10:42:31Z&quot;
  generation: 1
  labels:
    app: prometheus
    app.kubernetes.io/managed-by: Helm
    chart: prometheus-14.2.1
    component: server
    heritage: Helm
    release: prometheus
  name: prometheus-server
  namespace: prom
  resourceVersion: &quot;6983855&quot;
  selfLink: /apis/apps/v1/namespaces/prom/deployments/prometheus-server
  uid: &lt;some-uid&gt;
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: prometheus
      component: server
      release: prometheus
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: prometheus
        chart: prometheus-14.2.1
        component: server
        heritage: Helm
        release: prometheus
    spec:
      containers:
      - args:
        - --volume-dir=/etc/config
        - --webhook-url=http://127.0.0.1:9090/-/reload
        image: jimmidyson/configmap-reload:v0.5.0
        imagePullPolicy: IfNotPresent
        name: prometheus-server-configmap-reload
        resources:
          limits:
            cpu: 500m
            ephemeral-storage: 1Gi
            memory: 2Gi
          requests:
            cpu: 500m
            ephemeral-storage: 1Gi
            memory: 2Gi
        securityContext:
          capabilities:
            drop:
            - NET_RAW
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/config
          name: config-volume
          readOnly: true
      - args:
        - --storage.tsdb.retention.time=15d
        - --config.file=/etc/config/prometheus.yml
        - --storage.tsdb.path=/data
        - --web.console.libraries=/etc/prometheus/console_libraries
        - --web.console.templates=/etc/prometheus/consoles
        - --web.enable-lifecycle
        image: quay.io/prometheus/prometheus:v2.26.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /-/healthy
            port: 9090
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 10
        name: prometheus-server
        ports:
        - containerPort: 9090
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /-/ready
            port: 9090
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 4
        resources:
          limits:
            cpu: 500m
            ephemeral-storage: 1Gi
            memory: 2Gi
          requests:
            cpu: 500m
            ephemeral-storage: 1Gi
            memory: 2Gi
        securityContext:
          capabilities:
            drop:
            - NET_RAW
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/config
          name: config-volume
        - mountPath: /data
          name: storage-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
        seccompProfile:
          type: RuntimeDefault
      serviceAccount: prometheus-server
      serviceAccountName: prometheus-server
      terminationGracePeriodSeconds: 300
      volumes:
      - configMap:
          defaultMode: 420
          name: prometheus-server
        name: config-volume
      - name: storage-volume
        persistentVolumeClaim:
          claimName: prometheus-server
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: &quot;2021-06-24T10:43:25Z&quot;
    lastUpdateTime: &quot;2021-06-24T10:43:25Z&quot;
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: &quot;True&quot;
    type: Available
  - lastTransitionTime: &quot;2021-06-24T10:42:31Z&quot;
    lastUpdateTime: &quot;2021-06-24T10:43:25Z&quot;
    message: ReplicaSet &quot;prometheus-server-65b759cb95&quot; has successfully progressed.
    reason: NewReplicaSetAvailable
    status: &quot;True&quot;
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
</code></pre>
<p>yaml for service Monitor</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {&quot;apiVersion&quot;:&quot;monitoring.coreos.com/v1&quot;,&quot;kind&quot;:&quot;ServiceMonitor&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;creationTimestamp&quot;:&quot;2021-06-24T07:55:58Z&quot;,&quot;generation&quot;:1,&quot;labels&quot;:{&quot;app&quot;:&quot;sample-app&quot;,&quot;release&quot;:&quot;prometheus&quot;},&quot;name&quot;:&quot;sample-app&quot;,&quot;namespace&quot;:&quot;prom&quot;,&quot;resourceVersion&quot;:&quot;6884573&quot;,&quot;selfLink&quot;:&quot;/apis/monitoring.coreos.com/v1/namespaces/prom/servicemonitors/sample-app&quot;,&quot;uid&quot;:&quot;34644b62-eb4f-4ab1-b9df-b22811e40b4c&quot;},&quot;spec&quot;:{&quot;endpoints&quot;:[{&quot;port&quot;:&quot;http&quot;}],&quot;selector&quot;:{&quot;matchLabels&quot;:{&quot;app&quot;:&quot;sample-app&quot;,&quot;release&quot;:&quot;prometheus&quot;}}}}
  creationTimestamp: &quot;2021-06-24T07:55:58Z&quot;
  generation: 2
  labels:
    app: sample-app
    release: prometheus
  name: sample-app
  namespace: prom
  resourceVersion: &quot;6904642&quot;
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/prom/servicemonitors/sample-app
  uid: &lt;some-uid&gt;
spec:
  endpoints:
  - port: http
  selector:
    matchLabels:
      app: sample-app
      release: prometheus 
</code></pre>

<p>You need to use the <code>prometheus-community/kube-prometheus-stack</code> chart, which includes the Prometheus operator, in order to have Prometheus' configuration update automatically based on ServiceMonitor resources.</p>
<p>The <code>prometheus-community/prometheus</code> chart you used does not include the Prometheus operator that watches for ServiceMonitor resources in the Kubernetes API and updates the Prometheus server's ConfigMap accordingly.</p>
<p>It seems that you have the necessary CustomResourceDefinitions (CRDs) installed in your cluster, otherwise you would not have been able to create a ServiceMonitor resource. These are not included in the <code>prometheus-community/prometheus</code> chart so perhaps they were added to your cluster previously.</p>
