<p>I deploy Prometheus on my minikube cluster which has 5 nodes. I also deploy a microservice on the cluster. Prometheus can collect the data normally.</p>
<p>I use <code>wrk2</code> which is a workload generator to send requests to my microservices. Jaeger shows that the requests are processed normally.</p>
<p>The following is what confuse me. After I test the service for <code>duration</code> seconds, I try to use <code>sum(irate(container_cpu_usage_seconds_total{{{constraint}}}[{duration}s])) by (container, pod)</code> to get the CPU usage of the pods. However, The vast majority of pods have zero CPU usage, which means no query results. I was very surprised by this because in <code>duration</code> seconds I increased the load on the service (i.e., sent a lot of requests to it), but it didn't increase the CPU usage compared to when there was no load.</p>
<p>Following is the python function I used to query Prometheus:</p>
<pre class="lang-py prettyprint-override"><code># endtime=starttime+duration, starttime is the time when I start wrk2 to generate workload
def get_cpu_usage(self, starttime, endtime, duration, diaplay=False):
        # Define Prometheus query to get CPU usage for each service
        constraint = f'namespace=&quot;{self.namespace}&quot;, container!=&quot;POD&quot;, container!=&quot;&quot;'
        prometheus_query = (
            f&quot;sum(irate(container_cpu_usage_seconds_total{{{constraint}}}[{duration}s])) by (container, pod)&quot;
            + &quot; / &quot; + f&quot;(sum(container_spec_cpu_quota{{{constraint}}}/({duration}*1000)) by (container, pod)) * 100&quot;
        )

        # Send query to Prometheus endpoint
        sleep(1)
        response = requests.get(self.prometheus_url + '/api/v1/query_range', params={
            'query': prometheus_query,
            'start': starttime,
            'end': endtime,
            'step': 1
        })

        # Parse response
        usage = response.json()
        cpu_result = pd.DataFrame(columns=[&quot;microservice&quot;, &quot;pod&quot;, &quot;usage&quot;])

</code></pre>
<p>Does there any bug in the code or the Prometheus setting?</p>

MissSirius
<p>Prometheus requires at least two <a href="https://docs.victoriametrics.com/keyConcepts.html#raw-samples" rel="nofollow noreferrer">raw samples</a> on the lookbehind window specified in square brackets for calculating the <a href="https://docs.victoriametrics.com/MetricsQL.html#irate" rel="nofollow noreferrer">irate()</a>. Otherwise it returns an empty result.</p>
<p>P.S. It is recommended to use <a href="https://docs.victoriametrics.com/MetricsQL.html#rate" rel="nofollow noreferrer">rate()</a> instead of <code>irate()</code>, since <code>irate()</code> doesn't catch spikes. See <a href="https://valyala.medium.com/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832" rel="nofollow noreferrer">this article</a> for details.</p>
