<p>So this is happening on EKS K8s v1.15. You can see the api version in the describe output. The millicpu hovers between 80 and 120... which does not at ALL match the replica counts coming out of the HPA....</p>
<p>Here is the YAML:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: sequencer
  namespace: djin-content
spec:
  minReplicas: 1
  maxReplicas: 10
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sequencer
  metrics:
  - type: Pods
    pods:
      metricName: cpu_usage
      targetAverageValue: 500
</code></pre>
<p>Here is the kubectl describe:</p>
<pre class="lang-yaml prettyprint-override"><code>[root@ip-10-150-53-173 ~]# kubectl describe hpa -n djin-content
Name:                   sequencer
Namespace:              djin-content
Labels:                 &lt;none&gt;
Annotations:            kubectl.kubernetes.io/last-applied-configuration:
                          {&quot;apiVersion&quot;:&quot;autoscaling/v2beta1&quot;,&quot;kind&quot;:&quot;HorizontalPodAutoscaler&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;name&quot;:&quot;sequencer&quot;,&quot;namespace&quot;:&quot;djin-con...
CreationTimestamp:      Wed, 05 Aug 2020 20:40:37 +0000
Reference:              Deployment/sequencer
Metrics:                ( current / target )
  &quot;cpu_usage&quot; on pods:  122m / 500
Min replicas:           1
Max replicas:           10
Deployment pods:        7 current / 7 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 4
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from pods metric cpu_usage
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age                  From                       Message
  ----    ------             ----                 ----                       -------
  Normal  SuccessfulRescale  34m                  horizontal-pod-autoscaler  New size: 10; reason: pods metric cpu_usage above target
  Normal  SuccessfulRescale  15m (x2 over 34m)    horizontal-pod-autoscaler  New size: 6; reason: pods metric cpu_usage above target
  Normal  SuccessfulRescale  10m                  horizontal-pod-autoscaler  New size: 5; reason: All metrics below target
  Normal  SuccessfulRescale  9m51s (x2 over 23m)  horizontal-pod-autoscaler  New size: 3; reason: All metrics below target
  Normal  SuccessfulRescale  5m (x2 over 16m)     horizontal-pod-autoscaler  New size: 4; reason: pods metric cpu_usage above target
  Normal  SuccessfulRescale  4m45s (x2 over 15m)  horizontal-pod-autoscaler  New size: 5; reason: pods metric cpu_usage above target
  Normal  SuccessfulRescale  4m30s                horizontal-pod-autoscaler  New size: 7; reason: pods metric cpu_usage above target
</code></pre>
<p>The custom metric API is populated correctly/frequently and running well. The deployment targeting is working perfectly ... I have went through the entire k8s code base for this API and replica calculation and this makes NO sense ...</p>

<p>It seems like the metrics don't match, you have 122m (milicores) vs / 500 raw something.</p>
<pre class="lang-yaml prettyprint-override"><code>  &quot;cpu_usage&quot; on pods:  122m / 500
</code></pre>
<p>You didn't specify what's calculating your custom metrics, it could be that an extra <code>0</code> is being added to <code>122m</code> making it <code>1220 / 500</code> (I assume <code>cpu_usage</code> is the custom metric since the regular metrics server metric is just <code>cpu</code>) but you could try:</p>
<pre class="lang-yaml prettyprint-override"><code>targetAverageValue: 500m
</code></pre>
<p>The more common way to do HPA on CPU usage is to use CPU Utilization Percentage from the metrics server.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50
...
</code></pre>
<p>The scaling activities are managed by the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/" rel="nofollow noreferrer"><code>kube-controlller-manager</code></a> in your K8s control plane, if you have the EKS control plane logs enabled you could also take a look there to find more information. 👀</p>
<p>✌️</p>
