<p>I'm looking at Prometheus metrics in a Grafana dashboard, and I'm confused by a few panels that display metrics based on an ID that is unfamiliar to me. I assume that <code>/kubepods/burstable/pod99b2fe2a-104d-11e8-baa7-06145aa73a4c</code> points to a single pod, and I assume that <code>/kubepods/burstable/pod99b2fe2a-104d-11e8-baa7-06145aa73a4c/&lt;another-long-string&gt;</code> resolves to a container in the pod, but how do I resolve this ID to the pod name and a container i.e. how to do I map this ID to the pod name I see when I run <code>kubectl get pods</code>?</p>

<p>I already tried running <code>kubectl describe pods --all-namespaces | grep "99b2fe2a-104d-11e8-baa7-06145aa73a4c"</code> but that didn't turn up anything.</p>

<p>Furthermore, there are several subpaths in <code>/kubepods</code>, such as <code>/kubepods/burstable</code> and <code>/kubepods/besteffort</code>. What do these mean and how does a given pod fall into one or another of these subpaths?</p>

<p>Lastly, where can I learn more about what manages <code>/kubepods</code>?</p>

<p>Prometheus Query: </p>

<pre><code>sum (container_memory_working_set_bytes{id!="/",kubernetes_io_hostname=~"^$Node$"}) by (id)
</code></pre>

<p>/<a href="https://i.stack.imgur.com/NpQQL.png" rel="noreferrer"><img src="https://i.stack.imgur.com/NpQQL.png" alt="Grafana Screenshot"></a></p>

<p>Thanks for reading.</p>

<p>Eric</p>

<p>OK, now that I've done some digging around, I'll attempt to answer all 3 of my own questions. I hope this helps someone else.</p>

<p><strong>How to do I map this ID to the pod name I see when I run kubectl get pods?</strong></p>

<p>Given the following, <code>/kubepods/burstable/pod99b2fe2a-104d-11e8-baa7-06145aa73a4c</code>, the last bit is the pod UID, and can be resolved to a pod by looking at the <code>metadata.uid</code> property on the pod manifest:</p>

<pre><code>kubectl get pod --all-namespaces -o json | jq '.items[] | select(.metadata.uid == "99b2fe2a-104d-11e8-baa7-06145aa73a4c")'
</code></pre>

<p>Once you've resolved the UID to a pod, we can resolve the second UID (container ID) to a container by matching it with the <code>.status.containerStatuses[].containerID</code> in the pod manifest:</p>

<pre><code>~$ kubectl get pod my-pod-6f47444666-4nmbr -o json | jq '.status.containerStatuses[] | select(.containerID == "docker://5339636e84de619d65e1f1bd278c5007904e4993bc3972df8628668be6a1f2d6")'
</code></pre>

<p><strong>Furthermore, there are several subpaths in /kubepods, such as /kubepods/burstable and /kubepods/besteffort. What do these mean and how does a given pod fall into one or another of these subpaths?</strong></p>

<p>Burstable, BestEffort, and Guaranteed are Quality of Service (QoS) classes that Kubernetes assigns to pods based on the memory and cpu allocations in the pod spec. More information on QoS classes can be found here <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/" rel="noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/</a>.</p>

<p>To quote:</p>

<blockquote>
  <p>For a Pod to be given a QoS class of Guaranteed:</p>
  
  <ul>
  <li><p>Every Container in the Pod must have a memory limit and a memory request, and they must be the same.</p></li>
  <li><p>Every Container in the Pod must have a cpu limit and a cpu request, and they must be the same.</p></li>
  </ul>
  
  <p>A Pod is given a QoS class of Burstable if:</p>
  
  <ul>
  <li><p>The Pod does not meet the criteria for QoS class Guaranteed.</p></li>
  <li><p>At least one Container in the Pod has a memory or cpu request.</p></li>
  </ul>
  
  <p>For a Pod to be given a QoS class of BestEffort, the Containers in the
  Pod must not have any memory or cpu limits or requests.</p>
</blockquote>

<p><strong>Lastly, where can I learn more about what manages /kubepods?</strong></p>

<p><code>/kubepods/burstable</code>, <code>/kubepods/besteffort</code>, and <code>/kubepods/guaranteed</code> are all a part of the cgroups hierarchy, which is located in /sys/fs/cgroup directory. Cgroups is what manages resource usage for container processes such as CPU, memory, disk I/O, and network. Each resource has its own place in the cgroup hierarchy filesystem, and in each resource sub-directory are /kubepods subdirectories. More info on cgroups and Docker containers here: <a href="https://docs.docker.com/config/containers/runmetrics/#control-groups" rel="noreferrer">https://docs.docker.com/config/containers/runmetrics/#control-groups</a></p>
