<p>I have a simple pod with a nginx container which returns text <code>healthy</code> on path <code>/</code>. I have prometheus to scrape port 80 on path <code>/</code>. When I ran <code>up == 0</code> in the prometheus dashboard it showed this pod which means this pod is not healthy. But I tried ssh into the container, it was running fine and I saw in the nginx log prometheus was pinging <code>/</code> and getting 200 response. Any idea why?</p>
<p>deployment.yml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  ...
  template:
    metadata:
      labels:
        ...
      annotations:
        prometheus.io/scrape: &quot;true&quot;
        prometheus.io/path: &quot;/&quot;
        prometheus.io/port: &quot;80&quot;
    spec:
      containers:
        - name: nginx
          image: nginx
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx
              readOnly: true
          ports:
            - containerPort: 80
      volumes:
        - name: nginx-conf
          configMap:
            name: nginx-conf
            items:
              - key: nginx.conf
                path: nginx.conf


</code></pre>
<p>nginx.conf</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    http {
      server {
        listen 80;

        location / {
          return 200 'healthy\n';
        }
      }
    }
</code></pre>
<p>nginx access log</p>
<pre><code>192.168.88.81 - - [xxx +0000] &quot;GET / HTTP/1.1&quot; 200 8 &quot;-&quot; &quot;Prometheus/2.26.0&quot;
192.168.88.81 - - [xxx +0000] &quot;GET / HTTP/1.1&quot; 200 8 &quot;-&quot; &quot;Prometheus/2.26.0&quot;
192.168.88.81 - - [xxx +0000] &quot;GET / HTTP/1.1&quot; 200 8 &quot;-&quot; &quot;Prometheus/2.26.0&quot;
</code></pre>

<p>When you configure these annotations to pods, the Prometheus expects that the given path returns Prometheus-readable metrics. But <code>'healthy\n'</code> is not a valid Prometheus metrics type.</p>
<pre class="lang-yaml prettyprint-override"><code>      annotations:
        prometheus.io/scrape: &quot;true&quot;
        prometheus.io/path: &quot;/&quot;
        prometheus.io/port: &quot;80&quot;
</code></pre>
<p><strong>Recommended Fix:</strong></p>
<ul>
<li>Use <a href="https://github.com/nginxinc/nginx-prometheus-exporter" rel="nofollow noreferrer">nginx-prometheus-exporter</a> as sidecar.</li>
<li>Add sidecar info to annotation so that Prometheus can scrape metrics from it.</li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  ...
  template:
    metadata:
      labels:
        ...
      annotations:
        prometheus.io/scrape: &quot;true&quot;
        prometheus.io/path: &quot;/metrics&quot;
        prometheus.io/port: &quot;9113&quot;
    spec:
      containers:
        - name: nginx
          image: nginx
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx
              readOnly: true
          ports:
            - containerPort: 80
        - name: nginx-exporter
          args:
          - &quot;-nginx.scrape-uri=http://localhost:80/stub_status&quot; # nginx address
          image: nginx/nginx-prometheus-exporter:0.9.0
          ports:
            - containerPort: 9113
      volumes:
        - name: nginx-conf
          configMap:
            name: nginx-conf
            items:
              - key: nginx.conf
                path: nginx.conf
</code></pre>
<p>Now, try querying <code>nginx_up</code> from Prometheus. The nginx-prometheus-exporter also comes with a <a href="https://github.com/nginxinc/nginx-prometheus-exporter/tree/master/grafana" rel="nofollow noreferrer">grafana dashboard</a>, you can also give it a try.</p>
