<p>I'm trying to define an Horizontal Pod Autoscaler for two Kubernetes services.</p>

<p>The Autoscaler strategy relies in 3 metrics:</p>

<ol>
<li>cpu </li>
<li>pubsub.googleapis.com|subscription|num_undelivered_messages</li>
<li>loadbalancing.googleapis.com|https|request_count</li>
</ol>

<p><em>CPU</em> and <em>num_undelivered_messages</em> are correctly obtained, but no matter what i do, i cannot get the <em>request_count</em> metric.</p>

<p>The first service is a backend service (Service A), and the other (Service B) is an API that uses an Ingress to manage the external access to the service.</p>

<p>The Autoscaling strategy is based on Google documentation: <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling" rel="nofollow noreferrer">Autoscaling Deployments with External Metrics</a>.</p>

<p>For service A, the following defines the metrics used for Autoscaling:</p>

<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: ServiceA
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: ServiceA
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 80
  - external:
      metricName: pubsub.googleapis.com|subscription|num_undelivered_messages
      metricSelector:
        matchLabels:
          resource.labels.subscription_id: subscription_id
      targetAverageValue: 100
    type: External
</code></pre>

<p>For service B, the following defines the metrics used for Autoscaling:</p>

<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: ServiceB
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: ServiceB
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 80
  - external:
      metricName: loadbalancing.googleapis.com|https|request_count
      metricSelector:
        matchLabels:
          resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--3a908157de956ba7
      targetAverageValue: 100
    type: External      
</code></pre>

<p>As defined in the above article, the metrics server is running, and the metrics server adapter is deployed:</p>

<pre><code>$ kubectl get apiservices |egrep metrics
v1beta1.custom.metrics.k8s.io          custom-metrics/custom-metrics-stackdriver-adapter   True        2h
v1beta1.external.metrics.k8s.io        custom-metrics/custom-metrics-stackdriver-adapter   True        2h
v1beta1.metrics.k8s.io                 kube-system/metrics-server                          True        2h
v1beta2.custom.metrics.k8s.io          custom-metrics/custom-metrics-stackdriver-adapter   True        2h
</code></pre>

<p>For service A, all metrics, CPU and num_undelivered_messages, are correctly obtained:</p>

<pre><code>$ kubectl get hpa ServiceA
NAME       REFERENCE             TARGETS               MINPODS   MAXPODS   REPLICAS   AGE
ServiceA   Deployment/ServiceA   0/100 (avg), 1%/80%   1         3         1          127m
</code></pre>

<p>For service B, HPA cannot obtain the Request Count:</p>

<pre><code>$ kubectl get hpa ServiceB
NAME                REFERENCE    TARGETS                              MINPODS   MAXPODS   REPLICAS   AGE
ServiceB   Deployment/ServiceB   &lt;unknown&gt;/100 (avg), &lt;unknown&gt;/80%   1         3         1          129m
</code></pre>

<p>When accessing the Ingress, i get this warning:</p>

<blockquote>
  <p>unable to get external metric default/loadbalancing.googleapis.com|https|request_count/&amp;LabelSelector{MatchLabels:map[string]string{resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--3a908157de956ba7,},MatchExpressions:[],}: no metrics returned from external metrics API </p>
</blockquote>

<p>The <strong>metricSelector</strong> for the forwarding-rule is correct, as confirmed when describing the ingress (only the relevant information is show):</p>

<pre><code>$ kubectl describe ingress serviceb
Annotations:
  ingress.kubernetes.io/https-forwarding-rule:  k8s-fws-default-serviceb--3a908157de956ba7
</code></pre>

<p>I've tried to use a different metric selector, for example, using <em>url_map_name</em>, to no avail, i've got a similar error.</p>

<p>I've followed the exact guidelines on Google Documentation, and checked with a few online tutorials that refer the exact same process, but i haven't been able to understand what i'm missing.
I'm probably lacking some configuration, or some specific detail, but i cannot find it documented anywhere.</p>

<p>What am i missing, that explains why i'm not being able to obtain the <em>loadbalancing.googleapis.com|https|request_count</em> metric?</p>

<p>Thank you very much for your detailed response.</p>

<p>When using the metricSelector to select the specific <em>forwarding_rule_name</em>, we need to use the exact <em>forwarding_rule_name</em> as defined by the ingress:</p>

<pre><code>metricSelector:
    matchLabels:
    resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--3a908157de956ba7
</code></pre>

<pre><code>$ kubectl describe ingress

Name: serviceb
...

Annotations:
  ingress.kubernetes.io/https-forwarding-rule:  k8s-fws-default-serviceb--9bfb478c0886702d
  ...
  kubernetes.io/ingress.allow-http:             false
  kubernetes.io/ingress.global-static-ip-name:  static-ip
</code></pre>

<p>The problem, is that the suffix of the <em>forwarding_rule_name</em> (3a908157de956ba7) changes for every deployment, and is created dynamically on Ingress creation:</p>

<ul>
<li>k8s-fws-default-serviceb--<strong>3a908157de956ba7</strong></li>
</ul>

<p>We have a fully automated deployment using Helm, and, as such, when the HPA is created, we don't know what the <em>forwarding_rule_name</em> will be.</p>

<p>And, it seems that the <em>matchLabels</em> does not accept regular expressions, or else we would simply do something like:</p>

<pre><code>metricSelector:
    matchLabels:
    resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--*
</code></pre>

<p>I've tried several approaches, all without success:</p>

<ol>
<li>Use Annotations to force the <em>forwarding_rule_name</em></li>
<li>Use a different machLabel, as <em>backend_target_name</em> </li>
<li>Obtain the <em>forwarding_rule_name</em> using a command, so i can insert it later in the yaml file.</li>
</ol>

<p><strong>Use Annotations to force the <em>forwarding_rule_name</em>:</strong></p>

<p>When creating the ingress, i can use specific annotations to change the default behavior, or define specific values, for example, on Ingress.yaml:</p>

<pre><code>  annotations:
    kubernetes.io/ingress.global-static-ip-name: static-ip
</code></pre>

<p>I tried to use the https-forwarding-rule annotation to force a specific "static" name, but this didn't work:</p>

<pre><code>  annotations:
    ingress.kubernetes.io/https-forwarding-rule: some_name


  annotations:
    kubernetes.io/https-forwarding-rule: some_name
</code></pre>

<p><strong>Use a different machLabel, as <em>backend_target_name</em></strong></p>

<pre><code>metricSelector:
        matchLabels:
          resource.labels.backend_target_name: serviceb
</code></pre>

<p>Also failed.</p>

<p><strong>Obtain the <em>forwarding_rule_name</em> using a command</strong></p>

<p>When executing the following command, i get the list of Forwarding Rules, but for all the clusters. And according to the <a href="https://cloud.google.com/sdk/gcloud/reference/compute/forwarding-rules/" rel="nofollow noreferrer">documentation</a>, is not possible to filter by cluster:</p>

<pre><code>gcloud compute forwarding-rules list
</code></pre>

<pre><code>NAME                                        P_ADDRESS   IP_PROTOCOL  TARGET
k8s-fws-default-serviceb--4e1c268b39df8462  xx          TCP          k8s-tps-default-serviceb--4e1c268b39df8462
k8s-fws-default-serviceb--9bfb478c0886702d  xx          TCP          k8s-tps-default-serviceb--9bfb478c0886702d
</code></pre>

<p>Is there any way to allow me to select the resource i need, in order to get the Requests count metric?</p>
