<p>We are using AKS version 1.19.11.</p>
<p>It is noticed that whenever a new rollout is in placed for our deployments or a new pod got created as part of the hpa settings or pod got restarted, We are getting high cpu usage alerts.</p>
<p>For example, -if a new pod got created  as part of any of the above activities, will this take up more CPU than the allowed Threshold ? [ the “Maximum limit” of 1 core specified in the deployment spec and the apps are light weight and  doesnt need thatmuch cpu anuyways ] ? its in turn makes sudden spike in the AzureMonitor for a short time and then it became normal.</p>
<p>Why the pods are taking more cpu during its startup or creation time?
if the pods are not using thatmuch cpu, what will be the reason for this repeating issues?</p>
<p>hpa settings as below</p>
<pre><code>Name:                                                  myapp
Namespace:                                             myapp
Labels:                                                app.kubernetes.io/managed-by=Helm
Annotations:                                           meta.helm.sh/release-name: myapp
                                                       meta.helm.sh/release-namespace: myapp
CreationTimestamp:                                     Mon, 26 Apr 2021 07:02:32 +0000
Reference:                                             Deployment/myapp
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  5% (17m) / 75%
Min replicas:                                          5
Max replicas:                                          12
Deployment pods:                                       1 current / 1 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
</code></pre>
<p>ading the events when a new rollout placed.</p>
<p>as per the events captured from the “myapp” Namespace , there were new deployment rolled out for myapp as below.</p>
<p>During the new pods creation its showing more CPU spikes as we are getting alert from the Azuremonitor that its exceeds the threshold of 80%.[the “Maximum limit” of 1 core specified in the deployment spec]</p>
<pre><code>30m         Normal    SuccessfulDelete    replicaset/myapp-1a2b3c4d5e   Deleted pod: myapp-1a2b3c4d5e-9fmrk
30m         Normal    SuccessfulDelete    replicaset/myapp-1a2b3c4d5e   Deleted pod: myapp-1a2b3c4d5e-hfr8w
29m         Normal    SuccessfulDelete    replicaset/myapp-1a2b3c4d5e   Deleted pod: myapp-1a2b3c4d5e-l2pnd
31m         Normal    ScalingReplicaSet   deployment/myapp              Scaled up replica set myapp-5ddc98fb69 to 1
30m         Normal    ScalingReplicaSet   deployment/myapp              Scaled down replica set myapp-1a2b3c4d5e to 2
30m         Normal    ScalingReplicaSet   deployment/myapp              Scaled up replica set myapp-5ddc98fb69 to 2
30m         Normal    ScalingReplicaSet   deployment/myapp              Scaled down replica set myapp-1a2b3c4d5e to 1
30m         Normal    ScalingReplicaSet   deployment/myapp              Scaled up replica set myapp-5ddc98fb69 to 3
29m         Normal    ScalingReplicaSet   deployment/myapp              Scaled down replica set myapp-1a2b3c4d5e to 0
</code></pre>
<p>Alert settings</p>
<pre><code>Period  Over the last 15 mins 
Value   100.274747 
Operator    GreaterThan 
Threshold   80 
</code></pre>

<p>i am not sure what metrics you are looking at in AKS monitoring specifically as you have not mentioned it but it could be possible,</p>
<p>when you are deploying the POD or HPA scaling the replicas your AKS showing the total resource of all replicas.</p>
<p>During the <strong>deployment</strong>, it's possible at a certain stage all PODs are in the running phase and taking &amp; consuming the resources.</p>
<blockquote>
<p>Are you checking specific resources of one single POD and it's going
above the threshold ?</p>
</blockquote>
<p>As you have mentioned application is lightweight however it is possible initially it taking resources to start the process, in that case, you might have to check resources using profiling.</p>
