<p>prometheus:v2.15.2
kubernetes:v1.14.9</p>

<p>I have a query where it shows exactly the maximum over time during the set period. 
But I would like to join with the metric already set in the kube_pod_container resource.</p>

<p>I would like to know if what is set is close to the percentage set or not, displaying the percentage.</p>

<p>I have other examples working with this same structure of metric</p>

<p><code>jvm_memory_bytes_used{instance="url.instance.com.br"} / jvm_memory_bytes_max{area="heap"} * 100 &gt; 80</code></p>

<p>but this one is not working.</p>

<p><code>max_over_time(sum(rate(container_cpu_usage_seconds_total{pod="pod-name-here",container_name!="POD", container_name!=""}[1m])) [1h:1s]) / kube_pod_container_resource_requests_cpu_cores * 100 &lt; 70</code></p>

<p>Well the first idea was to create a query to collect the maximum historical cpu usage of a container in a pod in a brief period:</p>

<p><code>max_over_time(sum(rate(container_cpu_usage_seconds_total{pod="xpto-92838241",container_name!="POD", container_name!=""}[1m])) [1h:1s])</code></p>

<p><strong>Element</strong>: {} <strong>Value:</strong> 0.25781324101515</p>

<p>If we execute it this way:</p>

<p><code>container_cpu_usage_seconds_total{pod="xpto-92838241",container_name!="POD", container_name!=""}</code></p>

<p><strong>Element:</strong> container_cpu_usage_seconds_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="t3.small",beta_kubernetes_io_os="linux",cluster="teste.k8s.xpto",container="xpto",container_name="xpto",cpu="total",failure_domain_beta_kubernetes_io_region="sa-east-1",failure_domain_beta_kubernetes_io_zone="sa-east-1c",generic="true",id="/kubepods/burstable/poda9999e9999e999e9-/99999e9999999e9",image="nginx",instance="kubestate-dev.internal.xpto",job="kubernetes-cadvisor",kops_k8s_io_instancegroup="nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="ip-99-999-9-99.sa-east-1.compute.internal",kubernetes_io_os="linux",kubernetes_io_role="node",name="k8s_nginx_nginx-99999e9999999e9",namespace="nmpc",pod="pod-92838241",pod_name="pod-92838241",spot="false"} <strong>Value:</strong> 22533.2</p>

<p>Now we have what is configured:</p>

<p><code>kube_pod_container_resource_requests_cpu_cores{pod="xpto-92838241"}</code></p>

<p><strong>Element:</strong> kube_pod_container_resource_requests_cpu_cores{container="xpto",instance="kubestate-dev.internal.xpto",job="k8s-http",namespace="nmpc",node="ip-99-999-999-99.sa-east-1.compute.internal",pod="pod-92838241"} <strong>Value:</strong> 1</p>

<p>Well, in my perception it would be to use these two metrics and get it close to the percentage like this:</p>

<p><code>max_over_time(sum(rate(container_cpu_usage_seconds_total{pod="xpto-dev-92838241",container_name!="POD", container_name!=""}[1m])) [1h:1s]) / kube_pod_container_resource_requests_cpu_cores * 100 &lt; 70</code></p>

<p><strong>Element:</strong> <em>no data</em> <strong>Value:</strong> </p>

<p>But these two metrics do not interact, I can not understand why and do not find in the documentation.</p>

<p>Regards</p>

<p>As you can see <a href="https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md#removed-metrics" rel="nofollow noreferrer">here</a>, only in <strong>Kubernetes 1.16</strong> <em>cadvisor metric labels <code>pod_name</code> and <code>container_name</code></em> were removed and substituted by <code>pod</code> and <code>container</code> respectively.
As you are using <strong>Kubernetes 1.14</strong>, you should still use <code>pod_name</code> and <code>container_name</code>.</p>

<p>Let me know if it helps.</p>
