<p>I'm trying to scale a Kubernetes <code>Deployment</code> using a <code>HorizontalPodAutoscaler</code>, which listens to a custom metrics through Stackdriver.</p>

<p>I'm having a GKE cluster, with a Stackdriver adapter enabled.
I'm able to publish the custom metric type to Stackdriver, and following is the way it's being displayed in Stackdriver's Metric Explorer.</p>

<p><a href="https://i.stack.imgur.com/abCGr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/abCGr.png" alt="enter image description here"></a></p>

<p><a href="https://i.stack.imgur.com/tsdvJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/tsdvJ.png" alt="enter image description here"></a></p>

<p>This is how I have defined my <code>HPA</code>:</p>

<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: example-hpa
spec:
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metricName: custom.googleapis.com|worker_pod_metrics|baz
      targetValue: 400
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test-app-group-1-1
</code></pre>

<p>After successfully creating <code>example-hpa</code>, executing <code>kubectl get hpa example-hpa</code>, always shows <code>TARGETS</code> as <code>&lt;unknown&gt;</code>, and never detects the value from custom metrics.</p>

<pre><code>NAME          REFERENCE                       TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
example-hpa   Deployment/test-app-group-1-1   &lt;unknown&gt;/400   1         10        1          18m
</code></pre>

<p>I'm using a Java client which runs <em>locally</em> to publish my custom metrics.
I have given the appropriate resource labels as mentioned <a href="https://cloud.google.com/monitoring/api/resources#tag_gke_container" rel="nofollow noreferrer">here</a> (hard coded - so that it can run without a problem in local environment). I have followed <a href="https://cloud.google.com/monitoring/custom-metrics/creating-metrics#monitoring_write_timeseries-java" rel="nofollow noreferrer">this document</a> to create the Java client.</p>

<pre class="lang-java prettyprint-override"><code>private static MonitoredResource prepareMonitoredResourceDescriptor() {
        Map&lt;String, String&gt; resourceLabels = new HashMap&lt;&gt;();
        resourceLabels.put("project_id", "&lt;&lt;&lt;my-project-id&gt;&gt;&gt;);
        resourceLabels.put("pod_id", "&lt;my pod UID&gt;");
        resourceLabels.put("container_name", "");
        resourceLabels.put("zone", "asia-southeast1-b");
        resourceLabels.put("cluster_name", "my-cluster");
        resourceLabels.put("namespace_id", "mynamespace");
        resourceLabels.put("instance_id", "");

        return MonitoredResource.newBuilder()
                .setType("gke_container")
                .putAllLabels(resourceLabels)
                .build();
    }
</code></pre>

<p>What am I doing wrong in the above-mentioned steps please? Thank you in advance for any answers provided!</p>

<hr>

<p><strong>EDIT [RESOLVED]</strong>:
I think I have had some misconfigurations, since <code>kubectl describe hpa [NAME] --v=9</code> showed me some <code>403</code> status code, as well as I was using <code>type: External</code> instead of <code>type: Pods</code> (Thanks <a href="https://stackoverflow.com/users/11102471/mwz">MWZ</a> for your answer, pointing out this mistake).
<br/><br/>
I managed to fix it by creating a new project, a new service account, and a new GKE cluster (basically everything from the beginning again). Then I changed my yaml file as follows, exactly as <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling" rel="nofollow noreferrer">this document</a> explains.</p>

<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: test-app-group-1-1
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: test-app-group-1-1
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods                 # Earlier this was type: External
    pods:                      # Earlier this was external:
      metricName: baz                               # metricName: custom.googleapis.com|worker_pod_metrics|baz
      targetAverageValue: 20
</code></pre>

<p>I'm now exporting as <code>custom.googleapis.com/baz</code>, and NOT as <code>custom.googleapis.com/worker_pod_metrics/baz</code>. Also, now I'm explicitly specifying the <code>namespace</code> for my HPA in the yaml.</p>

Senthuran Ambalavanar
<p>It is a good practice to put some unique labels to target your metrics. Right now, based on metrics labelled in your java client, only <code>pod_id</code> looks unique which can't be used due to its stateless nature. </p>

<p>So, I would suggest you try introducing a deployment/metrics wide unqiue identifier.</p>

<pre><code>resourceLabels.put("&lt;identifier&gt;", "&lt;could-be-deployment-name&gt;");
</code></pre>

<p>After this, you can try modifying your HPA with something similar to following:</p>

<pre><code>kind: HorizontalPodAutoscaler
metadata:
  name: example-hpa
spec:
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metricName: custom.googleapis.com|worker_pod_metrics|baz
      metricSelector:
        matchLabels:
          # define labels to target
          metric.labels.identifier: &lt;deployment-name&gt;
      # scale +1 whenever it crosses multiples of mentioned value
      targetAverageValue: "400"
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test-app-group-1-1
</code></pre>

<p>Apart from this, this setup has no issues and should work smooth.</p>

<p>Helper command to see what metrics are exposed to HPA :</p>

<pre><code> kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|worker_pod_metrics|baz" | jq
</code></pre>
