<p>I'm trying to configure a Horizontal Pod Autoscaler (HPA) on Google Kubernetes Engine (GKE) using External Metrics from an Ingress LoadBalancer, basing the configuration on instructions such as</p>

<p><a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling" rel="noreferrer">https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling</a> 
and
<a href="https://blog.doit-intl.com/autoscaling-k8s-hpa-with-google-http-s-load-balancer-rps-stackdriver-metric-92db0a28e1ea" rel="noreferrer">https://blog.doit-intl.com/autoscaling-k8s-hpa-with-google-http-s-load-balancer-rps-stackdriver-metric-92db0a28e1ea</a></p>

<p>With an HPA like </p>

<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: my-api
  namespace: production
spec:
  minReplicas: 1
  maxReplicas: 20
  metrics:
  - external:
      metricName: loadbalancing.googleapis.com|https|request_count
      metricSelector:
        matchLabels:
          resource.labels.forwarding_rule_name: k8s-fws-production-lb-my-api--63e2a8ddaae70
      targetAverageValue: "1"
    type: External
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-api

</code></pre>

<p>the autoscaler does kick in when the request count rises - but placing heavy load on the service, like 100 simultaneous requests per second, doesn't increase the external metric <code>request_count</code> much beyond 6 RPS, while the observed <code>backend_latencies</code> metric in Stackdriver does increase significantly; so I'd like to utilise that metric by adding to the HPA configuration, like so:</p>

<pre><code>  - external:
      metricName: loadbalancing.googleapis.com|https|backend_latencies
      metricSelector:
        matchLabels:
          resource.labels.forwarding_rule_name: k8s-fws-production-lb-my-api--63e2a8ddaae70
      targetValue: "3000"
    type: External
</code></pre>

<p>but that results in the error:</p>

<p><code>...unable to fetch metrics from external metrics API: googleapi: Error 400: Field aggregation.perSeriesAligner had an invalid value of "ALIGN_RATE": The aligner cannot be applied to metrics with kind DELTA and value type DISTRIBUTION., badRequest</code></p>

<p>which can be observed with the command</p>

<pre><code>$ kubectl describe hpa -n production
</code></pre>

<p>or by visiting</p>

<p><a href="http://localhost:8080/apis/external.metrics.k8s.io/v1beta1/namespaces/default/loadbalancing.googleapis.com%7Chttps%7Cbackend_latencies" rel="noreferrer">http://localhost:8080/apis/external.metrics.k8s.io/v1beta1/namespaces/default/loadbalancing.googleapis.com%7Chttps%7Cbackend_latencies</a></p>

<p>after setting up a proxy with</p>

<pre><code>$ kubectl proxy --port=8080
</code></pre>

<p>Are <code>https/backend_latencies</code> or <a href="https://cloud.google.com/monitoring/api/metrics_gcp#gcp-loadbalancing" rel="noreferrer"><code>https/total_latencies</code></a> not supported as External Stackdriver Metrics in an HPA configuration for GKE?</p>

Bjorn Thor Jonsson
<p>Maybe someone would find this helpful, though the question is old.</p>
<p>My working config looks like next:</p>
<pre><code>  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 95
  - type: External
    external:
      metric:
       name: loadbalancing.googleapis.com|https|backend_latencies
       selector:
         matchLabels:
           resource.labels.backend_name: frontend
           metric.labels.proxy_continent: Europe
           reducer: REDUCE_PERCENTILE_95
      target:
        type: Value
        value: &quot;79.5&quot;
</code></pre>
<p><code>type: Value</code> used because it's the only way to not divide the metric value by the replica number.</p>
<p><code>reducer: REDUCE_PERCENTILE_95</code> used to work only with a single value of the distribution (<a href="https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/pkg/adapter/translator/query_builder.go#L44" rel="nofollow noreferrer">source</a>).</p>
<p>Also, I edited <code>custom-metrics-stackdriver-adapter</code> deployment to look like this:</p>
<pre><code>  - image: gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0
    imagePullPolicy: Always
    name: pod-custom-metrics-stackdriver-adapter
    command:
    - /adapter
    - --use-new-resource-model=true
    - --fallback-for-container-metrics=true
    - --enable-distribution-support=true
</code></pre>
<p>The thing is this key <code>enable-distribution-support=true</code>, which enables working with distribution kind of metrics.</p>
