<p><strong>My set up</strong></p>
<p>I have one physical node K8s cluster where I taint master node so it can also act a worker. the node has Centos7 with total of 512 GB memory. I am limiting my experiments to one node cluster; once a solution is found, I will test it on my small scale k8s cluster where master and worker services are on separate nodes.</p>
<p><strong>What I am trying to do</strong></p>
<p>I want to monitor pod level resource utilization (CPU and Memory). I am launching a pod which consumed memory at rate of 1GBPS; in around 100seconds, memory utilization reaches 100GB and at that point application reaches steady state. From that point, it keep running until killed with a trigger.</p>
<p><strong>Where I am right now with this</strong></p>
<p>After launching k8s metric server; I am able to do <code>kubectl top pods</code> and it shows per pod CPU and memory utilization. However these utilization numbers are not updated frequently. I tried to measure how long does k8s takes to update these telemetry and sampling interval appears to be close to 1 minute or 60 seconds.</p>
<p>I tired looking into <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/" rel="nofollow noreferrer">https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/</a> for figuring out various sampling intervals. there are a few parameters which could impact telemetry sampling rate; but they are set to ~20s (defaults) at max. I am not changing any Kubelet settings.</p>
<p><strong>My Question</strong></p>
<p>Why it takes around a minute for <code>kubectl top pods</code> to update resource utilization numbers.? how can I reduce this interval and have frequent updates.?</p>

<ul>
<li><p><strong>Why it takes around a minute for <code>kubectl top pods</code> to update resource utilization numbers?</strong></p>
<p>It's because metric server's default resolution which is set to 60s.</p>
</li>
<li><p><strong>How can I reduce this interval and have frequent updates?</strong></p>
<p>You can change the resolution with <code>--metric-resolution=&lt;duration&gt;</code> flag.
It's not however recommended settings values below 15s as this is the resolution of metrics calculated by Kubelet.</p>
<pre class="lang-yaml prettyprint-override"><code>spec:  
  containers:  
  - command:  
    - /metrics-server  
    - --metric-resolution=15s
</code></pre>
<p>Reference: <a href="https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md#how-often-metrics-are-scraped" rel="nofollow noreferrer">How often metrics are scraped</a></p>
</li>
</ul>
