title: HorizontalPodAutoscaler: Is the published documentation for deploying the custom metrics stackdriver adapter in GKE complete and correct?
body: <p>Google publishes a tutorial for using custom metrics to drive the <code>HorizontalPodAutoscaler</code> <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling" rel="nofollow noreferrer">here</a>, and this tutorial contains instructions for:</p>
  
  <ol>
  <li>Using a Kubernetes manifest to deploy the custom metrics adapter into a <code>custom-metrics</code> namespace.</li>
  <li>Deploying a dummy application to generate metrics.</li>
  <li>Configuring the HPA to use custom metrics.</li>
  </ol>
  
  <p>We are deploying into a default cluster without any special VPC rules, and we have roughly followed the tutorial's guidance, with a few exceptions:</p>
  
  <ul>
  <li>We're using Helm v2, and rather than grant cluster admin role to Tiller, we have granted all of the necessary cluster roles and role bindings to allow the custom-metrics-adapter-deploying Kubernetes manifest to work.  We see no issues there; at least the custom metrics adapter spins up and runs.</li>
  <li>We have defined some custom metrics that are based upon data extracted from a jsonPayload in Stackdriver logs.</li>
  <li>We have deployed a minute-by-minute CronJob that reads the above metrics and publishes a derived metric, which is the value we want to use to drive the autoscaler.  The CronJob is working, and we can see the metric in the derived metric, on a per-Pod basis, in the log metric explorer: <a href="https://i.stack.imgur.com/VmuNm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VmuNm.png" alt="log metrics"></a></li>
  </ul>
  
  <p>We're configuring the HPA to scale based on the average of the derived metric across all of the pods belonging to a stateful set (The HPA has a metrics entry with type Pods).  However, the HPA is unable to read our derived metric.  We see this error message: </p>
  
  <p><code>failed to get object metric value: unable to get metric xxx_scaling_metric: no metrics returned from custom metrics API</code></p>
  
  <p><strong>Update</strong></p>
  
  <p>We were seeing DNS errors, but these were apparently false alarms, perhaps in the log while the cluster was spinning up.  </p>
  
  <p>We restarted the Stackdriver metrics adapter with the command line option <code>--v=5</code> to get some more verbose debugging.  We see log entries like these:</p>
  
  <pre><code>I0123 20:23:08.069406       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/defaults/pods/%2A/xxx_scaling_metric: (56.16652ms) 200 [kubectl/v1.13.11 (darwin/amd64) kubernetes/2e298c7 10.44.1.1:36286]
  I0123 20:23:12.997569       1 translator.go:570] Metric 'xxx_scaling_metric' not found for pod 'xxx-0'
  I0123 20:23:12.997775       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta2/namespaces/default/pods/%2A/xxx_scaling_metric?labelSelector=app%3Dxxx: (98.101205ms) 200 [kube-controller-manager/v1.13.11 (linux/amd64) kubernetes/56d8986/system:serviceaccount:kube-system:horizontal-pod-autoscaler 10.44.1.1:36286]
  </code></pre>
  
  <p>So it <em>looks to us</em> as if the HPA is making the right query for pods-based custom metrics.  If we ask the custom metrics API what data it has, and filter with <code>jq</code> to our metric of interest, we see:</p>
  
  <pre><code>{"kind":"MetricValueList",
   "apiVersion":"custom.metrics.k8s.io/v1beta1",
   "metadata: {"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/xxx_scaling_metric"},
   "items":[]}
  </code></pre>
  
  <p>That the items array is empty is troubling.  Again, we can see data in the metrics explorer, so we're left to wonder if our CronJob app that publishes our scaling metric is supplying the right fields in order for the data to be saved in Stackdriver or exposed through the metrics adapter.  </p>
  
  <p>For what it's worth the resource.labels map for the time series that we're publishing in our CronJob looks like:</p>
  
  <pre><code>{'cluster_name': 'test-gke', 
   'zone': 'us-central1-f', 
   'project_id': 'my-project-1234', 
   'container_name': '', 
   'instance_id': '1234567890123456789',
   'pod_id': 'xxx-0', 
   'namespace_id': 'default'}
  </code></pre>