<p>I created HPA on our k8s cluster which should auto-scale on 90% memory utilization. However, it scales UP without hitting the target percentage. I use the following config:</p>
<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  namespace: {{ .Values.namespace }}
  name: {{ include &quot;helm-generic.fullname&quot; . }}
  labels:
    {{- include &quot;helm-generic.labels&quot; . | nindent 4 }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include &quot;helm-generic.fullname&quot; . }}
  minReplicas: 1
  maxReplicas: 2
  metrics:
    - type: Resource
      resource:
        name: memory
        targetAverageUtilization: 90
</code></pre>
<p>So for this config it creates 2 pods which is the maxReplicas number. If I add 4 for maxReplicas it will create 3.</p>
<p>This is what i get from kubectl describe hpa</p>
<pre><code>$ kubectl describe hpa -n trunkline

Name:                                                     test-v1
Namespace:                                                trunkline
Labels:                                                   app.kubernetes.io/instance=test-v1
                                                          app.kubernetes.io/managed-by=Helm
                                                          app.kubernetes.io/name=helm-generic
                                                          app.kubernetes.io/version=0.0.0
                                                          helm.sh/chart=helm-generic-0.1.3
Annotations:                                              meta.helm.sh/release-name: test-v1
                                                          meta.helm.sh/release-namespace: trunkline
CreationTimestamp:                                        Wed, 12 Oct 2022 17:36:54 +0300
Reference:                                                Deployment/test-v1
Metrics:                                                  ( current / target )
  **resource memory on pods  (as a percentage of request):  59% (402806784) / 90%**
  resource cpu on pods  (as a percentage of request):     11% (60m) / 80%
Min replicas:                                             1
Max replicas:                                             2
Deployment pods:                                          **2 current / 2** desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           &lt;none&gt;
</code></pre>
<p>As you see the pods memory % is 59 , with target 90 which I expect to produce only 1 pod.</p>

<p>The horizontal pod autoscaler has a <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details" rel="nofollow noreferrer">very specific formula</a> for calculating the target replica count:</p>
<pre class="lang-none prettyprint-override"><code>desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]
</code></pre>
<p>With the output you show, <code>currentMetricValue</code> is 59% and <code>desiredMetricValue</code> is 90%.  Multiplying that by the <code>currentReplicas</code> of 2, you get about 1.3 replicas, which gets rounded up to 2.</p>
<p>This formula, and especially the <code>ceil()</code> round-up behavior, can make HPA very slow to scale down, especially with a small number of replicas.</p>
<p>More broadly, autoscaling on Kubernetes-observable memory might not work the way you expect.  Most programming languages are garbage-collected (C, C++, and Rust are the most notable exceptions) and garbage collectors as a rule tend to allocate a large block of operating-system memory and reuse it, rather than return it to the operating system if load decreases.  If you have a pod that reaches 90% memory from the Kubernetes point of view, its possible that memory usage will never decrease.  You might need to autoscale on a different metric, or attach an external metrics system like Prometheus to get more detailed memory-manager statistics you can act on.</p>
