<p>I have a Django project deployed in Kubernetes and I am trying to deploy Prometheus as a monitoring tool. I have successfully done all the steps needed to include <code>django_prometheus</code> in the project and locally I can go go <code>localhost:9090</code> and play around with querying the metrics.</p>
<p>I have also deployed Prometheus to my Kubernetes cluster and upon running a <code>kubectl port-forward ...</code> on the Prometheus pod I can see some metrics of my Kubernetes resources.</p>
<p>Where I am a bit confused is how to make the deployed Django app metrics available on the Prometheus dashboard just like the others.
I deployed my app in <code>default</code> namespace and  prometheus in a <code>monitoring</code> dedicated namespace. I am wondering what am I missing here. Do I need to expose the ports on the service and deployment from 8000 to 8005 according to the number of workers or something like that?</p>
<p>My Django app runs with gunicorn using <code>supervisord</code> like so:</p>
<pre><code>[program:gunicorn]
command=gunicorn --reload --timeout 200000 --workers=5 --limit-request-line 0 --limit-request-fields 32768 --limit-request-field_size 0 --chdir /code/ my_app.wsgi
</code></pre>
<ul>
<li><code>my_app</code> service:</li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: my_app
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: my-app
  sessionAffinity: None
  type: ClusterIP

</code></pre>
<ul>
<li>Trimmed version of the <code>deployment.yaml</code></li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-app
  name: my-app-deployment
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: my-app
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - image: ...
        imagePullPolicy: IfNotPresent
        name: my-app
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: regcred
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30

</code></pre>
<ul>
<li><code>prometheus configmap</code></li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
data:
  prometheus.rules: |-
    ... some rules
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    rule_files:
      - /etc/prometheus/prometheus.rules
    scrape_configs:
      - job_name: prometheus
        static_configs:
        - targets:
          - localhost:9090

      - job_name: my-app
        metrics_path: /metrics
        static_configs:
          - targets:
            - localhost:8000

      - job_name: 'node-exporter'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_endpoints_name]
          regex: 'node-exporter'
          action: keep

kind: ConfigMap
metadata:
  labels:
    name: prometheus-config
  name: prometheus-config
  namespace: monitoring
</code></pre>

<p>You do not have to expose services, if the promehteus is installed on the same cluster as your app. You can communicate with apps between namespaces by using Kubernetes DNS resolution, going by the rule:</p>
<pre><code>SERVICENAME.NAMESPACE.svc.cluster.local
</code></pre>
<p>so one way is to change your prometheus job target to something like this</p>
<pre class="lang-yaml prettyprint-override"><code>  - job_name: speedtest-ookla
    metrics_path: /metrics
    static_configs:
      - targets:
          - 'my_app.default.svc.cluster.local:9000'
</code></pre>
<p>And this is the &quot;manual&quot; way. A better approach will be to use prometheus <code>kubernetes_sd_config</code>. It will autodiscover your services and try to scrape them.</p>
<p>Reference: <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config" rel="nofollow noreferrer">https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config</a></p>
