<p>I set up my HorizontalPodAutoscaler like described here <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling</a> to listen scale according to the number of unacked messages from my Pub/Sub. My desire is that the pods scale if there is more than 1 unacknowledged message. When I run <code>k describe hpa</code> I get:</p>

<pre><code>Namespace:                                                                               default
Labels:                                                                                  &lt;none&gt;
Annotations:                                                                             kubectl.kubernetes.io/last-applied-configuration:
                                                                                           {"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"foobar-gke-prod","namespace":"defau...
CreationTimestamp:                                                                       Mon, 25 May 2020 18:01:33 -0700
Reference:                                                                               Deployment/foobar-gke-prod
Metrics:                                                                                 ( current / target )
  "pubsub.googleapis.com|subscription|num_undelivered_messages" (target average value):  200m / 1
Min replicas:                                                                            3
Max replicas:                                                                            9
Deployment pods:                                                                         5 current / 5 desired
</code></pre>

<p>The metrics data returned is confusing me. When I ran that command the number of unacked knowledge messages was around 4 according to the console metrics. So I don't understand what does <code>200m</code> mean? Why wouldn't it say 4?</p>

<p>Here is my configuration for the HPA</p>

<pre><code># Template from https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: foobar-gke-prod
spec:
  minReplicas: 3
  maxReplicas: 9
  metrics:
  - external:
      metricName: pubsub.googleapis.com|subscription|num_undelivered_messages
      metricSelector:
        matchLabels:
          resource.labels.subscription_id: prod_foobar_subscription
      targetAverageValue: "1"
    type: External
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: foobar-gke-prod
</code></pre>

Daniel Kobe
<p><strong>Reference Example:</strong></p>

<pre><code>Name:                                                                                    pubsub
...
Metrics:                                                                                 ( current / target )
"pubsub.googleapis.com|subscription|num_undelivered_messages" (target average value):  2250m / 2
Min replicas:                                                                            1
Max replicas:                                                                            4
Conditions:
Type            Status  Reason            Message
----            ------  ------            -------
AbleToScale     True    SucceededRescale  the HPA controller was able to update the target scale to 4
ScalingLimited  True    TooManyReplicas   the desired replica count is more than the maximum replica count
Events:
Type    Reason             Age   From                       Message
----    ------             ----  ----                       -------
Normal  SuccessfulRescale  7s    horizontal-pod-autoscaler  New size: 4; reason: external metric pubsub.googleapis.com|subscription|num_undelivered_messages(&amp;LabelSelector{MatchLabels:map[string]string{resource.labels.subscription_id: echo-read,},MatchExpressions:[],}) above target
</code></pre>

<ul>
<li>The Metrics section gives the last value of metric observed by HPA. <strong>Fractional values are represented as milli-units</strong>. For example, in the output above there are 4 replicas of application and the current number of unacknowledged messages in Pub/Sub subscription is 9. So <strong>the average number of messages per replica is 2.25, or 2250m</strong>.</li>
</ul>

<hr>

<blockquote>
  <p>The metrics data returned is confusing me. When I ran that command the number of unacked knowledge messages was around 4 according to the console metrics. So I don't understand what does 200m mean? Why wouldn't it say 4?</p>
</blockquote>

<ul>
<li>That means that on your case <code>200m/1</code> means that at that moment the average number of undelivered messages <strong>per replica running</strong> is 0.2(20%) at the time HPA measured.</li>
</ul>

<p><strong>Considerations:</strong></p>

<ul>
<li>Make sure you make the readings on metrics console and HPA roughly at the same time to avoid discrepancies due to scaling running during reads.</li>
<li><p>a reading of 4 messages for 5 pods would result in a load of 800m but at that point the hpa could be already running another scale up event.</p></li>
<li><p>I encourage you to take a reading of the metrics console and hpa at the same time and verify again.</p></li>
</ul>

<p>If you still think the results don't match post here with the updated hpa describe and we can take another look.</p>

<hr>

<p><strong>EDIT:</strong></p>

<blockquote>
  <p>Is there anyway to make the metric not be an average across pods? I.e. if there are 5 unacked messages the metrics data would read 5000m?</p>
</blockquote>

<p>From Kubernetes API Reference <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#externalmetricsource-v2beta1-autoscaling" rel="nofollow noreferrer">ExternalMetricSource v2beta1 Autoscaling</a>:</p>

<ul>
<li><p><strong><code>targetAverageValue</code></strong> is the target per-pod value of global metric (as a quantity).</p></li>
<li><p><strong><code>targetValue</code></strong> is the target value of the metric (as a quantity). </p></li>
</ul>

<p>Note that <code>targetAverageValue</code> and <code>targetValue</code> are mutually exclusive.</p>

<p>So if you want the total instead of the average, just swap them on your HPA.</p>
