<p>I have a Kubernetes cluster with an install of <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="nofollow noreferrer">kube-prometheus-stack</a> (Prometheus 2.27.1, kube-state-metrics v2.0.0)</p>
<p>I would like to have a query to return how much time each pod was running, over last 24 hours</p>
<ul>
<li>If a pod is still running, the time from its creation to now</li>
<li>If a post has terminated, the time from creation to completion</li>
</ul>
<p>Importantly, I need exactly the time the pod existed, as opposed to CPU usage.</p>
<p>I can do something like this with:</p>
<pre><code>kube_pod_completion_time - kube_pod_created
</code></pre>
<p>but it returns nothing for pods that are still running. And, since Prometheus does not return metrics that are more than 5 min old, it will not report anything for pods that were terminated and deleted.</p>
<p>How would I query Prometheus without these issues?</p>

<p>One working solution is this:</p>
<pre><code>sum by(namespace, pod) (

   (last_over_time(kube_pod_completion_time[1d]) 
   - last_over_time(kube_pod_created[1d])) 

  or 

    (time() - kube_pod_created)

)
</code></pre>
<p>The first part inside <code>sum</code> handles the case of pods that have terminated. We pick the last value of <code>kube_pod_completion_time</code> and <code>kube_pod_stared</code> and compute the difference.</p>
<p>The second part handles the pods that are still running. In that case, there is a fresh value of the <code>kube_pod_created</code> metric, and we can subtract it from the current time.</p>
