<p>I'm trying to define an Horizontal Pod Autoscaler for two Kubernetes services.</p>

<p>The Autoscaler strategy relies in 3 metrics:</p>

<ol>
<li>cpu </li>
<li>pubsub.googleapis.com|subscription|num_undelivered_messages</li>
<li>loadbalancing.googleapis.com|https|request_count</li>
</ol>

<p><em>CPU</em> and <em>num_undelivered_messages</em> are correctly obtained, but no matter what i do, i cannot get the <em>request_count</em> metric.</p>

<p>The first service is a backend service (Service A), and the other (Service B) is an API that uses an Ingress to manage the external access to the service.</p>

<p>The Autoscaling strategy is based on Google documentation: <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling" rel="nofollow noreferrer">Autoscaling Deployments with External Metrics</a>.</p>

<p>For service A, the following defines the metrics used for Autoscaling:</p>

<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: ServiceA
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: ServiceA
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 80
  - external:
      metricName: pubsub.googleapis.com|subscription|num_undelivered_messages
      metricSelector:
        matchLabels:
          resource.labels.subscription_id: subscription_id
      targetAverageValue: 100
    type: External
</code></pre>

<p>For service B, the following defines the metrics used for Autoscaling:</p>

<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: ServiceB
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: ServiceB
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 80
  - external:
      metricName: loadbalancing.googleapis.com|https|request_count
      metricSelector:
        matchLabels:
          resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--3a908157de956ba7
      targetAverageValue: 100
    type: External      
</code></pre>

<p>As defined in the above article, the metrics server is running, and the metrics server adapter is deployed:</p>

<pre><code>$ kubectl get apiservices |egrep metrics
v1beta1.custom.metrics.k8s.io          custom-metrics/custom-metrics-stackdriver-adapter   True        2h
v1beta1.external.metrics.k8s.io        custom-metrics/custom-metrics-stackdriver-adapter   True        2h
v1beta1.metrics.k8s.io                 kube-system/metrics-server                          True        2h
v1beta2.custom.metrics.k8s.io          custom-metrics/custom-metrics-stackdriver-adapter   True        2h
</code></pre>

<p>For service A, all metrics, CPU and num_undelivered_messages, are correctly obtained:</p>

<pre><code>$ kubectl get hpa ServiceA
NAME       REFERENCE             TARGETS               MINPODS   MAXPODS   REPLICAS   AGE
ServiceA   Deployment/ServiceA   0/100 (avg), 1%/80%   1         3         1          127m
</code></pre>

<p>For service B, HPA cannot obtain the Request Count:</p>

<pre><code>$ kubectl get hpa ServiceB
NAME                REFERENCE    TARGETS                              MINPODS   MAXPODS   REPLICAS   AGE
ServiceB   Deployment/ServiceB   &lt;unknown&gt;/100 (avg), &lt;unknown&gt;/80%   1         3         1          129m
</code></pre>

<p>When accessing the Ingress, i get this warning:</p>

<blockquote>
  <p>unable to get external metric default/loadbalancing.googleapis.com|https|request_count/&amp;LabelSelector{MatchLabels:map[string]string{resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--3a908157de956ba7,},MatchExpressions:[],}: no metrics returned from external metrics API </p>
</blockquote>

<p>The <strong>metricSelector</strong> for the forwarding-rule is correct, as confirmed when describing the ingress (only the relevant information is show):</p>

<pre><code>$ kubectl describe ingress serviceb
Annotations:
  ingress.kubernetes.io/https-forwarding-rule:  k8s-fws-default-serviceb--3a908157de956ba7
</code></pre>

<p>I've tried to use a different metric selector, for example, using <em>url_map_name</em>, to no avail, i've got a similar error.</p>

<p>I've followed the exact guidelines on Google Documentation, and checked with a few online tutorials that refer the exact same process, but i haven't been able to understand what i'm missing.
I'm probably lacking some configuration, or some specific detail, but i cannot find it documented anywhere.</p>

<p>What am i missing, that explains why i'm not being able to obtain the <em>loadbalancing.googleapis.com|https|request_count</em> metric?</p>

<p>It seems the metric that you're defining isn't available in the <a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/external-metrics-api.md#api" rel="nofollow noreferrer">External Metrics API</a>. To find out what's going on, you can inspect the External Metrics API directly:</p>

<pre><code>kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1" | jq
</code></pre>

<p>Is the <em>loadbalancing.googleapis.com|https|request_count</em> metric reported in the output?</p>

<p>You can then dig deeper by making requests <a href="http:///apis/external.metrics.k8s.io/v1beta1/namespaces/%3Cnamespace_name%3E/%3Cmetric_name%3E?labelSelector=%3Cselector%3E" rel="nofollow noreferrer">of the following form</a>:</p>

<pre><code>kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1/namespaces/&lt;namespace_name&gt;/&lt;metric_name&gt;?labelSelector=&lt;selector&gt;" | jq
</code></pre>

<p>And see what's returned given your metric name and a specific metric selector.</p>

<p>These are precisely the requests that the Horizontal Pod Autoscaler also makes at runtime. By replicating them manually, you should be able to pinpoint the source of the problem.</p>

<hr>

<p><strong>Comments about additional information:</strong></p>

<p><strong>1)</strong> 83m is the Kubernetes way of writing 0.083 (read as 83 "milli-units").</p>

<p><strong>2)</strong> In your HorizontalPodAutoscaler definition, you use a <code>targetAverageValue</code>. So, if there exist multiple targets with this metric, the HPA calculates their average. So, 83m might be an average of multiple targets. To make sure, you use only the metric of a single target, you can use the <code>targetValue</code> field (see <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#externalmetricsource-v2beta1-autoscaling" rel="nofollow noreferrer">API reference</a>). </p>

<p><strong>3)</strong> Not sure why the <code>items: []</code> array in the API response is empty. The documentation mentions that after sampling, the data is not visible for 210 seconds... You could try making the API request when the HPA is not running.</p>
