<p>I have a Django project deployed in Kubernetes and I am trying to deploy Prometheus as a monitoring tool. I have successfully done all the steps needed to include <code>django_prometheus</code> in the project and locally I can go go <code>localhost:9090</code> and play around with querying the metrics.</p>
<p>I have also deployed Prometheus to my Kubernetes cluster and upon running a <code>kubectl port-forward ...</code> on the Prometheus pod I can see some metrics of my Kubernetes resources.</p>
<p>Where I am a bit confused is how to make the deployed Django app metrics available on the Prometheus dashboard just like the others.
I deployed my app in <code>default</code> namespace and  prometheus in a <code>monitoring</code> dedicated namespace. I am wondering what am I missing here. Do I need to expose the ports on the service and deployment from 8000 to 8005 according to the number of workers or something like that?</p>
<p>My Django app runs with gunicorn using <code>supervisord</code> like so:</p>
<pre><code>[program:gunicorn]
command=gunicorn --reload --timeout 200000 --workers=5 --limit-request-line 0 --limit-request-fields 32768 --limit-request-field_size 0 --chdir /code/ my_app.wsgi
</code></pre>
<ul>
<li><code>my_app</code> service:</li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: my_app
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: my-app
  sessionAffinity: None
  type: ClusterIP

</code></pre>
<ul>
<li>Trimmed version of the <code>deployment.yaml</code></li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-app
  name: my-app-deployment
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: my-app
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - image: ...
        imagePullPolicy: IfNotPresent
        name: my-app
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: regcred
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30

</code></pre>
<ul>
<li><code>prometheus configmap</code></li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
data:
  prometheus.rules: |-
    ... some rules
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    rule_files:
      - /etc/prometheus/prometheus.rules
    scrape_configs:
      - job_name: prometheus
        static_configs:
        - targets:
          - localhost:9090

      - job_name: my-app
        metrics_path: /metrics
        static_configs:
          - targets:
            - localhost:8000

      - job_name: 'node-exporter'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_endpoints_name]
          regex: 'node-exporter'
          action: keep

kind: ConfigMap
metadata:
  labels:
    name: prometheus-config
  name: prometheus-config
  namespace: monitoring
</code></pre>

<p>No need to expose the application outside the cluster.</p>
<p>Leveraging the Kubernetes service discovery, add the job to scrape Services, Pods, or both:</p>
<pre><code>- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
    regex: (.+)
  - regex: __meta_kubernetes_service_label_(.+)
    action: labelmap
  - regex: 'app_kubernetes_io_(.+)'
    action: labeldrop
  - regex: 'helm_sh_(.+)'
    action: labeldrop
</code></pre>
<pre><code>- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
    regex: (.+)
  - source_labels: [__meta_kubernetes_pod_node_name]
    action: replace
    target_label: host
    regex: (.+)
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod
    regex: (.+)
  - regex: __meta_kubernetes_pod_label_(.+)
    action: labelmap
  - regex: 'app_kubernetes_io_(.+)'
    action: labeldrop
  - regex: 'helm_sh_(.+)'
    action: labeldrop
</code></pre>
<p>Then, annotate the Service with:</p>
<pre><code>metadata:
  annotations:
    prometheus.io/scrape: &quot;true&quot;
    prometheus.io/port: &quot;80&quot;
    prometheus.io/path: &quot;/metrics&quot;
</code></pre>
<p>and the Deployment with:</p>
<pre><code>spec:
  template:
    metadata:
     annotations:
      prometheus.io/scrape: &quot;true&quot;
      prometheus.io/port: &quot;80&quot;
      prometheus.io/path: &quot;/metrics&quot;
</code></pre>
