<p>I am learning and experimenting with Prometheus metrics and Grafana dashboards. The information is coming from a Kubernetes cluster.</p>
<p>I am struggling with figuring out how to collect together information about pods, that is coming from multiple metrics. I believe the metrics involved are all related in some way. They are all in the <code>kube_pod...</code> &quot;family&quot;.</p>
<h2>Background</h2>
<p>I've used a technique like the following that works for a simple metric to metric case:</p>
<pre><code>(metric_A) + on(&lt;common_label&gt;) group_left(&lt;metric_B_label&gt;, ...) (0 * metric_B)
</code></pre>
<p>This allows me to associate a label from the right side that is absent from the left side via a common label. There is no arithmetic involved, so the add and multiply really do nothing. The <code>on</code> (or <code>ignoring</code>) operator apparently requires a binary operator between the left and right sides. This seems really clunky, but it does work and I know of no other way to achieve this.</p>
<p>Here's a concrete example:</p>
<pre><code>(kube_pod_status_phase != 0) + on (pod) group_left (node) (0 * kube_pod_info)
</code></pre>
<p>The <code>kube_pod_status_phase</code> provides the <code>phase</code> (and, of course, <code>pod</code>) of each pod (Running, Failed, etc.), but does not have the node information. The <code>kube_pod_info</code> has the <code>node</code> label and a matching <code>pod</code> label. Using the query above provides a collection of pods, their current phase and which node they're associated with.</p>
<h2>Problem</h2>
<p>My current task is to collect the following information:</p>
<div class="s-table-container">
<table class="s-table">
<thead>
<tr>
<th>Node</th>
<th>Pod</th>
<th>Status</th>
<th>Created</th>
<th>Age</th>
</tr>
</thead>
<tbody>
<tr>
<td>node_1</td>
<td>pod_A_1</td>
<td>Running</td>
<td>mm/dd/yyyy hh:mm:ss</td>
<td>{x}d{y}h</td>
</tr>
<tr>
<td>node_1</td>
<td>pod_B_1</td>
<td>Running</td>
<td>mm/dd/yyyy hh:mm:ss</td>
<td>{x}d{y}h</td>
</tr>
<tr>
<td>node_2</td>
<td>pod_C_1</td>
<td>Pending</td>
<td>mm/dd/yyyy hh:mm:ss</td>
<td>{x}d{y}h</td>
</tr>
<tr>
<td>node_3</td>
<td>pod_A_2</td>
<td>Running</td>
<td>mm/dd/yyyy hh:mm:ss</td>
<td>{x}d{y}h</td>
</tr>
<tr>
<td>node_3</td>
<td>pod_B_2</td>
<td>Failed</td>
<td>mm/dd/yyyy hh:mm:ss</td>
<td>{x}d{y}h</td>
</tr>
<tr>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
</tr>
</tbody>
</table>
</div>
<p>My plan is to get the status (phase) from the <code>kube_pod_status_phase</code> metric, the created date/time and the age from the <code>kube_pod_start_time</code> metric and include the corresponding node from the <code>kube_pod_info</code> metric. The age is calculated as <code>time() - kube_pod_start_time</code>.</p>
<p>Another detail that complicates this is that the <code>phase</code> and <code>node</code> are <em>labels</em> in their respective metrics, while the created date/time and age are the &quot;result&quot; of running the queries (i.e. they are <em>not</em> labels). This has been causing me problems in several attempts.</p>
<p>I tried seeing if I could somehow chain together queries, but in addition to being incredibly ugly and complicated, I couldn't get the result values (created date and age) to be included in the results that I managed to get to work.</p>
<p>If anyone knows how this could be done, I would very much appreciate knowing about it.</p>

Joseph Gagnon
<p>I was able to find a method that does what I need. Thanks to comments from <a href="https://stackoverflow.com/users/21363224/markalex">https://stackoverflow.com/users/21363224/markalex</a> that got me going on the thought track.</p>
<p>I ended up creating 3 queries:</p>
<pre><code>(kube_pod_status_phase{namespace=~&quot;runner.*&quot;} != 0) + on (pod) group_left (node) (0 * kube_pod_info)

time() - kube_pod_start_time{namespace=~&quot;runner.*&quot;}

kube_pod_start_time{namespace=~&quot;runner.*&quot;}
</code></pre>
<p>Then joining them together with a &quot;Join by field&quot; transform on the <code>pod</code> label. Finally, used an &quot;Organize fields&quot; transform to hide the columns I don't care about as well as some re-ordering.</p>
