<p>I have Prometheus setup via Helm from Terraform and it's is configured to connect to my Kubernetes cluster. I open my Prometheus but I am not sure which metric to choose from the list to be able to view the CPU/MEM of running pods/jobs.
Here are all the pods running with the command (<strong>test1</strong> is the kube <strong>namespace</strong>):</p>
<pre><code>kubectl -n test1 get pods
</code></pre>
<p><a href="https://i.stack.imgur.com/BdNNe.png" rel="nofollow noreferrer">podsrunning</a></p>
<p>When, I am on Prometheus, I see many metrics related to CPU, but not sure which one to choose:</p>
<p><a href="https://i.stack.imgur.com/8wyQd.png" rel="nofollow noreferrer">prom1</a></p>
<p>I tried to choose one, but the namespace = prometheus and it uses <code>prometheus-node-exporter</code> and I don't see my cluster or my namespace <code>test1</code> anywhere here.</p>
<p><a href="https://i.stack.imgur.com/l1Mz2.png" rel="nofollow noreferrer">prom2</a></p>
<p>Could you please help me? Thank you very much in advance.</p>
<p><strong>UPDATE SCREENSHOT</strong>
<em><strong>UPDATE SCREENSHOT</strong></em>
I need to concentrate on this specific namespace, normally with the command:
<code>kubectl get pods --all-namespaces | grep hermatwin</code>
I see the first line with namespace = <code>jobs</code> I think this is namespace.
<a href="https://i.stack.imgur.com/BCwUI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BCwUI.png" alt="promQL1" /></a></p>
<p>No result when set calendar to last Friday:
<a href="https://i.stack.imgur.com/io6vI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/io6vI.png" alt="promQL2" /></a></p>
<p><strong>UPDATE SCREENSHOT April 20</strong>
I tried to select 2 days with starting date on last Saturday 17 April but I don't see any result:
<a href="https://i.stack.imgur.com/MG63W.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MG63W.png" alt="noResult1" /></a></p>
<p>ANd, if I remove (namespace=&quot;jobs&quot;) condition, I don't see any result either:
<a href="https://i.stack.imgur.com/RMo5s.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RMo5s.png" alt="noresult2" /></a></p>
<p>I tried to rerun the job (simulation jobs) again just now and tried to execute the prometheus query while the job was still running mode but I don't get any result :-( Here you can see my jobs where running.</p>
<p><a href="https://i.stack.imgur.com/Jza0o.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Jza0o.png" alt="jobsRunning" /></a></p>
<p>I don't get any result:
<a href="https://i.stack.imgur.com/j7tan.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/j7tan.png" alt="noresult3" /></a></p>
<p>When using simple filter, just <code>container_cpu_usage_seconds_total</code>, I can see the namespace=&quot;jobs&quot;
<a href="https://i.stack.imgur.com/W8AIZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/W8AIZ.png" alt="resultnamespacejobs" /></a></p>
<p><a href="https://i.stack.imgur.com/YRCmf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YRCmf.png" alt="iRate1" /></a></p>
<p><a href="https://i.stack.imgur.com/U0whc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/U0whc.png" alt="ResultJob" /></a></p>

<p><code>node_cpu_seconds_total</code> is a metric from <code>node-exporter</code>, the exporter that brings machine statistics and its metrics are prefixed with <code>node_</code>. You need metrics from <code>cAdvisor</code>, this one produces metrics related to containers and they are prefixed with <code>container_</code>:</p>
<pre><code>container_cpu_usage_seconds_total
container_cpu_load_average_10s
container_memory_usage_bytes
container_memory_rss
</code></pre>
<p>Here are some basic queries for you to get started. Be ready that they may require tweaking (you may have different label names):</p>
<h3>CPU Utilisation Per Pod</h3>
<pre><code>sum(irate(container_cpu_usage_seconds_total{container!=&quot;POD&quot;, container=~&quot;.+&quot;}[2m])) by (pod)
</code></pre>
<h3>RAM Usage Per Pod</h3>
<pre><code>sum(container_memory_usage_bytes{container!=&quot;POD&quot;, container=~&quot;.+&quot;}) by (pod)
</code></pre>
<h3>In/Out Traffic Rate Per Pod</h3>
<p>Beware that pods with <code>host</code> network mode (not isolated) show traffic rate for the whole node. <code>* 8</code> is to convert bytes to bits for convenience (MBit/s, GBit/s, etc).</p>
<pre><code># incoming
sum(irate(container_network_receive_bytes_total[2m])) by (pod) * 8
# outgoing
sum(irate(container_network_transmit_bytes_total[2m])) by (pod) * 8
</code></pre>
