<p>Given a kubernetes cluster with:</p>
<ol>
<li><a href="https://github.com/prometheus/prometheus" rel="nofollow noreferrer">prometheus</a></li>
<li><a href="https://github.com/prometheus/node_exporter" rel="nofollow noreferrer">node-exporter</a></li>
<li><a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">kube-state-metrics</a></li>
</ol>
<p>I like to use the metric <code>container_memory_usage_bytes</code> but select by <code>deployment_name</code> instead of <code>pod</code>.</p>
<p>Selectors like <code>container_memory_usage_bytes{pod_name=~&quot;foo-.+&quot;}</code> if the <code>deployment_name=foo</code> are great as long there is not a deployment with <code>deployment_name=foo-bar</code>.</p>
<p>The same I'd like to achieve with the metric <code>kube_pod_container_resource_limits_memory_bytes</code>.</p>
<p>Is there a way to achieve this?</p>

GreNodge
<p><strong>TL;DR</strong></p>
<p><strong>There is no straightforward way to query prometheus by a <code>deployment-name</code>.</strong></p>
<p>You can query memory usage of a specific deployment by using deployment's labels.</p>
<p>Used query:</p>
<pre><code>sum(
  kube_pod_labels{label_app=~&quot;ubuntu.*&quot;} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace=&quot;memory-testing&quot;, container=&quot;&quot;}
  ) 
  by (label_app)
</code></pre>
<p>There is an awesome article which explains the concepts behind this query. I encourage you to read it:</p>
<ul>
<li><em><a href="https://medium.com/@amimahloof/kubernetes-promql-prometheus-cpu-aggregation-walkthrough-2c6fd2f941eb" rel="noreferrer">Medium.com: Amimahloof: Kubernetes promql prometheus cpu aggregation walkthrough</a></em></li>
</ul>
<p>I've included an explanation with example below.</p>
<hr />
<p>The selector mentioned in the question:
<code>container_memory_usage_bytes{pod_name=~&quot;foo-.+&quot;}</code></p>
<blockquote>
<p><code>.+</code> - match any string but not an empty string</p>
</blockquote>
<p>with pods like:</p>
<ul>
<li><code>foo-12345678-abcde</code> - <strong>will match</strong> (deployment <code>foo</code>)</li>
<li><code>foo-deployment-98765432-zyxzy</code> - <strong>will match</strong> (deployment <code>foo-deployment</code>)</li>
</ul>
<p>As shown above it will match for both pods and for both deployments.</p>
<p>For more reference:</p>
<ul>
<li><em><a href="https://prometheus.io/docs/prometheus/latest/querying/basics/" rel="noreferrer">Prometheus.io: Docs: Prometheus: Querying: Basics</a></em></li>
</ul>
<hr />
<p>As mentioned earlier, you can use labels from your deployment to pinpoint the resource used by your specific deployment.</p>
<p>Assuming that:</p>
<ul>
<li>There are 2 deployments in <code>memory-testing</code> namespace:
<ul>
<li><code>ubuntu</code> with 3 replicas</li>
<li><code>ubuntu-additional</code> with 3 replicas</li>
</ul>
</li>
<li>Above deployments have labels the same as their names (they can be different):
<ul>
<li><code>app: ubuntu</code></li>
<li><code>app: ubuntu-additional</code></li>
</ul>
</li>
<li>Kubernetes cluster version <code>1.18.X</code></li>
</ul>
<blockquote>
<p><em>Why do I specify Kubernetes version?</em></p>
<blockquote>
<p>Kubernetes 1.16 will remove the duplicate <code>pod_name</code> and <code>container_name</code> metric labels from cAdvisor metrics. For the 1.14 and 1.15 release all <code>pod</code>, <code>pod_name</code>, <code>container</code> and <code>container_name</code> were available as a grace period.</p>
</blockquote>
<ul>
<li><em><a href="https://github.com/kubernetes/enhancements/issues/1206" rel="noreferrer">Github.com: Kubernetes: Metrics Overhaul</a></em></li>
</ul>
</blockquote>
<p>This means that you will need to substitute the parameters like:</p>
<ul>
<li><code>pod</code> with <code>pod_name</code></li>
<li><code>container</code> with <code>container_name</code></li>
</ul>
<p>To deploy Prometheus and other tools to monitor the cluster I used: <em><a href="https://github.com/coreos/kube-prometheus" rel="noreferrer">Github.com: Coreos: Kube-prometheus</a></em></p>
<p>The pods in <code>ubuntu</code> deployment are configured to generate artificial load (<code>stress-ng</code>). This is done to show how to avoid situation where the used resources are doubled.</p>
<p>The resources used by pods in <code>memory-testing</code> namespace:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl top pod --namespace=memory-testing
NAME                                 CPU(cores)   MEMORY(bytes)   
ubuntu-5b5d6c56f6-cfr9g              816m         280Mi           
ubuntu-5b5d6c56f6-g6vh9              834m         278Mi            
ubuntu-5b5d6c56f6-sxldj              918m         288Mi           
ubuntu-additional-84bdf9b7fb-b9pxm   0m           0Mi             
ubuntu-additional-84bdf9b7fb-dzt72   0m           0Mi             
ubuntu-additional-84bdf9b7fb-k5z6w   0m           0Mi     
</code></pre>
<p>If you were to query Prometheus with below query:</p>
<pre><code>container_memory_usage_bytes{namespace=&quot;memory-testing&quot;, pod=~&quot;ubuntu.*&quot;}
</code></pre>
<p>You would get output similar to one below (it's cut to show only one pod for example purposes, by default it would show all pods with <code>ubuntu</code> in name and in <code>memory-testing</code> namespace):</p>
<pre><code>container_memory_usage_bytes{endpoint=&quot;https-metrics&quot;,id=&quot;/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74&quot;,instance=&quot;192.168.0.117:10250&quot;,job=&quot;kubelet&quot;,metrics_path=&quot;/metrics/cadvisor&quot;,namespace=&quot;memory-testing&quot;,node=&quot;node1&quot;,pod=&quot;ubuntu-5b5d6c56f6-cfr9g&quot;,service=&quot;kubelet&quot;} 308559872
container_memory_usage_bytes{container=&quot;POD&quot;,endpoint=&quot;https-metrics&quot;,id=&quot;/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74/312980f90e6104d021c12c376e83fe2bfc524faa4d4cee6553182d0fa2e007a1&quot;,image=&quot;k8s.gcr.io/pause:3.2&quot;,instance=&quot;192.168.0.117:10250&quot;,job=&quot;kubelet&quot;,metrics_path=&quot;/metrics/cadvisor&quot;,name=&quot;k8s_POD_ubuntu-5b5d6c56f6-cfr9g_memory-testing_b96dea39-b388-471e-a789-8c74b1670c74_0&quot;,namespace=&quot;memory-testing&quot;,node=&quot;node1&quot;,pod=&quot;ubuntu-5b5d6c56f6-cfr9g&quot;,service=&quot;kubelet&quot;} 782336
container_memory_usage_bytes{container=&quot;ubuntu&quot;,endpoint=&quot;https-metrics&quot;,id=&quot;/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74/1b93889a3e7415ad3fa040daf89f3f6bc77e569d85069de518267666ede8e21c&quot;,image=&quot;ubuntu@sha256:55cd38b70425947db71112eb5dddfa3aa3e3ce307754a3df2269069d2278ce47&quot;,instance=&quot;192.168.0.117:10250&quot;,job=&quot;kubelet&quot;,metrics_path=&quot;/metrics/cadvisor&quot;,name=&quot;k8s_ubuntu_ubuntu-5b5d6c56f6-cfr9g_memory-testing_b96dea39-b388-471e-a789-8c74b1670c74_0&quot;,namespace=&quot;memory-testing&quot;,node=&quot;node1&quot;,pod=&quot;ubuntu-5b5d6c56f6-cfr9g&quot;,service=&quot;kubelet&quot;} 307777536
</code></pre>
<p>In this point you will need to choose which metric you will be using. In this example I used the first one. For more of a deep dive please take a look on this articles:</p>
<ul>
<li><em><a href="https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e66" rel="noreferrer">Blog.freshtracks.io: A deep dive into kubernetes metrics part 3 container resource metrics</a></em></li>
<li><em><a href="https://www.ianlewis.org/en/almighty-pause-container" rel="noreferrer">Ianlewis.org: Almighty pause container</a></em></li>
</ul>
<p>If we were to aggregate this metrics with <code>sum (QUERY) by (pod)</code> we would have in fact doubled our reported used resources.</p>
<p><strong>Dissecting the main query:</strong></p>
<pre><code>container_memory_usage_bytes{namespace=&quot;memory-testing&quot;, container=&quot;&quot;}
</code></pre>
<p>Above query will output records with used memory metric for each pod. The <code>container=&quot;&quot;</code>parameter is used to get only one record (mentioned before) which does not have <code>container</code> parameter.</p>
<pre><code>kube_pod_labels{label_app=~&quot;ubuntu.*&quot;}
</code></pre>
<p>Above query will output record with pods and it's labels with regexp of <code>ubuntu.*</code></p>
<pre><code>kube_pod_labels{label_app=~&quot;ubuntu.*&quot;} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace=&quot;memory-testing&quot;, container=&quot;&quot;}
</code></pre>
<p>Above query will match the <code>pod</code> from <code>kube_pod_labels</code> with <code>pod</code> of <code>container_memory_usage_bytes</code> and add the <code>label_app</code> to each of the records.</p>
<pre><code>sum (kube_pod_labels{label_app=~&quot;ubuntu.*&quot;} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace=&quot;memory-testing&quot;, container=&quot;&quot;}) by (label_app)
</code></pre>
<p>Above query will sum the records by <code>label_app</code>.</p>
<p>After that you should be able to get the query that will sum the used memory by a label (and in fact a Deployment).</p>
<p><a href="https://i.stack.imgur.com/cvW4W.png" rel="noreferrer"><img src="https://i.stack.imgur.com/cvW4W.png" alt="Grafana Prometheus query" /></a></p>
<hr />
<p>As for:</p>
<blockquote>
<p>The same I'd like to achieve with the metric
<code>kube_pod_container_resource_limits_memory_bytes</code>.</p>
</blockquote>
<p>You can use below query to get the memory limit for the deployment tagged with labels as in previous example, assuming that each pod in a deployment is having the same limits:</p>
<pre><code>kube_pod_labels{label_app=&quot;ubuntu-with-limits&quot;} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace=&quot;memory-testing&quot;, pod=~&quot;.*&quot;}
</code></pre>
<p>You can apply functions like <code>avg()</code>,<code>mean()</code>,<code>max()</code> on this query to get the single number that will be your memory limit:</p>
<pre><code>avg(kube_pod_labels{label_app=&quot;ubuntu-with-limits&quot;} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace=&quot;memory-testing&quot;, pod=~&quot;.*&quot;}) by (label_app)
</code></pre>
<p>Your memory limits  can vary if you use <code>VPA</code>. In that situation you could show all of them simultaneously or use the <code>avg()</code> to get the average for all of the &quot;deployment&quot;.</p>
<p><a href="https://i.stack.imgur.com/RR9Xy.png" rel="noreferrer"><img src="https://i.stack.imgur.com/RR9Xy.png" alt="Combined resources and limits" /></a></p>
<hr />
<p>As a <strong>workaround</strong> to above solutions you could try to work with regexp like below:</p>
<pre><code>container_memory_usage_bytes{pod=~&quot;^ubuntu-.{6,10}-.{5}&quot;}
</code></pre>
