<p>We have a kubernetes cluster made up of 5 nodes.  2 of the nodes are only used for KIAM and the other 3 are for container deployments.</p>

<p>I have prometheus and grafana deployed and configured and I need to configure monitoring for CPU, memory and pod usage. However I want to totally exclude the nodes hosting KIAM from any stats or alert thresholds.</p>

<p>The only thing I can see being returned by prometheus that can identify the nodes i need is label_workload="gp" from the kube_node_labels metric.  What I don't know how to do is to get grafana to only use these nodes in it's calculations.</p>

<p>Perhaps it's possible to have some sort of query join or subselect to identify the node names to include??</p>

<p>I'd appreciate any help on this!!!</p>

<p>I believe the <code>node_uname_info</code> metrics is a better metric to get all your node infos. So I will explain using that metric.</p>

<p>You have two options:</p>

<p>Option 1: You hard-code the nodenames into your Grafana dashboard. Your query should look something like this <code>node_uname_info{nodename=~"node1|node2|node3"}</code> then. <code>node1-3</code> are the nodes which you want to have the metrics for.</p>

<p>Option 2: You create a variable and allow the user to select the nodes. Let's say the variable name is <code>$nodes</code> and the query should be <code>label_values(node_uname_info, nodename)</code>. It should be allowed to select multiple values. Next in your query you can then do <code>node_uname_info{nodename=~"$nodes"}</code> to only show metrics for selected nodes.</p>
