<p>I have installed and configured Prometheus on my Azure Kubernetes as per the following configuration</p>
<pre><code>prometheus_values.yaml
======================
global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m
# Forcing Kubelet metrics scraping on http 
kubelet:
  enabled: true
  serviceMonitor:
    https: false
# Disabling scraping of Master Nodes Components
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
kubeProxy:
  enabled: false
scrape_configs:
- job_name: dev-go-app
  static_configs:
  - targets: ['40.x.x.x:8080']
- job_name: default-go-app
  static_configs:
  - targets: ['40.x.x.x:8080']
</code></pre>
<p>and installed prometheus as shown below</p>
<pre><code>Installing prometheus
=====================
helm upgrade --install prometheus \
--namespace monitoring \
stable/prometheus \
--values prometheus_values.yaml 
</code></pre>
<p>It looks like I have to manually update the Prometheus configuration and apply for each application which is a difficult task especially when I have 100+ applications in the same Azure Kubernetes cluster.</p>
<p>Is there a way to let Prometheus automatically discovery the Kubernetes services and start pulling the metrics?</p>

<p>Instead of using bare Prometheus, I suggest setting up a <code>kube-prometheus-stack</code> (formerly <code>prometheus-operator</code>): <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="noreferrer">https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack</a></p>
<p>This stack has a built-in discovery mechanism in it - a CRD called <code>ServiceMonitor</code>: <a href="https://coreos.com/operators/prometheus/docs/latest/user-guides/running-exporters.html" rel="noreferrer">https://coreos.com/operators/prometheus/docs/latest/user-guides/running-exporters.html</a>. This is a selector which will auto-discover your services thru labels, further discover pods behind these services, and load them into Prometheus scrape config.</p>
