<p>I'm building a dashboard in Grafana, using data from Prometheus, to monitor a namespace in a Kubernetes cluster. I need all this to see what happens during a load test.</p>
<p>Now I've spent half my day looking for information about the different metrics in Prometheus. I've read through <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/" rel="nofollow noreferrer">Prometheus docs</a> and <a href="https://github.com/kubernetes/kube-state-metrics/tree/master/docs" rel="nofollow noreferrer">kube state metrics docs</a> (which gets the data from our cluster) but I did not find any descriptions about which metric does what. I only can guess based on query results and examples found here and there, but that's slow and way more insecure than I'd like.</p>
<p>However, I've come upon <a href="https://stackoverflow.com/questions/40327062/how-to-calculate-containers-cpu-usage-in-kubernetes-with-prometheus-as-monitori">this</a> SO answer so I assume the quote must have been copied from somewhere. Anybody please?</p>

<p>I found the answer for my question. <a href="https://stackoverflow.com/a/57417515/7160820">yogesh's answer</a> gave me the hint to have a look at exporters and I found the other half of the answer <a href="https://prometheus.io/docs/guides/node-exporter/" rel="noreferrer">here</a>.</p>

<p>So on Prometheus UI, there is a list of exporters and their scraped endpoints (Status > Targets). If I call one of the endpoints, the response contains a description and the type of each metric offered by the endpoint.</p>

<p>Call:</p>

<pre class="lang-sh prettyprint-override"><code>curl http://exporter.endpoint:9100/metrics
</code></pre>

<p>Sample from a response:</p>

<pre><code># HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 16310
http_response_size_bytes{handler="prometheus",quantile="0.9"} 16326
http_response_size_bytes{handler="prometheus",quantile="0.99"} 16337
http_response_size_bytes_sum{handler="prometheus"} 1.46828673e+09
http_response_size_bytes_count{handler="prometheus"} 90835
# HELP node_arp_entries ARP entries by device
# TYPE node_arp_entries gauge
node_arp_entries{device="cali6b3dc03715a"} 1
node_arp_entries{device="eth0"} 15
</code></pre>

<p>It wasn't trivial for me how to get this done. I logged into the cluster and I found <code>curl</code> didn't get any response for any of the endpoints. The solution was to add the base of the endpoint urls to the <code>no_proxy</code> variable (my machine and the server are both sitting behind the corporate proxy).</p>

<p>But anyway, this is how one can read the description of the Prometheus metrics.</p>
