<p>Following the guide that google made for deploying an HPA in Google Kubernetes Engine: <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics</a></p>
<p>And adding the right permissions because I am using Workload Identity with this guide: <a href="https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/custom-metrics-stackdriver-adapter" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/custom-metrics-stackdriver-adapter</a></p>
<p>And also adding the firewall-rule commented here: <a href="https://github.com/kubernetes-sigs/prometheus-adapter/issues/134" rel="nofollow noreferrer">https://github.com/kubernetes-sigs/prometheus-adapter/issues/134</a></p>
<p>I am stuck in a point where the HPA returns me this error:</p>
<pre><code>kubectl describe hpa -n test-namespace
Name:                  my-hpa
Namespace:             test-namespace
Labels:                &lt;none&gt;
Annotations:           &lt;none&gt;
CreationTimestamp:     Tue, 13 Apr 2021 12:47:56 +0200
Reference:             StatefulSet/my-set
Metrics:               ( current / target )
  &quot;my-metric&quot; on pods:  &lt;unknown&gt; / 1
Min replicas:          1
Max replicas:          60
StatefulSet pods:      1 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric my-metric: no metrics returned from custom metrics API
Events:
  Type     Reason                        Age                   From                       Message
  ----     ------                        ----                  ----                       -------
  Warning  FailedGetPodsMetric           8m26s (x40 over 18m)  horizontal-pod-autoscaler  unable to get metric my-metric: no metrics returned from custom metrics API
  Warning  FailedComputeMetricsReplicas  3m26s (x53 over 18m)  horizontal-pod-autoscaler  failed to compute desired number of replicas based on listed metrics for StatefulSet/test-namespace/my-set: invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric my-metric: no metrics returned from custom metrics API
</code></pre>
<p>But the apiservices are in true,</p>
<pre><code>kubectl get apiservices
NAME                                     SERVICE                                             AVAILABLE   AGE
...
v1beta1.custom.metrics.k8s.io            custom-metrics/custom-metrics-stackdriver-adapter   True        24h
v1beta1.external.metrics.k8s.io          custom-metrics/custom-metrics-stackdriver-adapter   True        24h
v1beta2.custom.metrics.k8s.io            custom-metrics/custom-metrics-stackdriver-adapter   True        24h
...
</code></pre>
<p>And when I try to retrieve the metric data it returns ok,</p>
<pre><code>kubectl get --raw &quot;/apis/custom.metrics.k8s.io/v1beta2/namespaces/test-namespace/pods/*/my-metric&quot; | jq .
{
  &quot;kind&quot;: &quot;MetricValueList&quot;,
  &quot;apiVersion&quot;: &quot;custom.metrics.k8s.io/v1beta2&quot;,
  &quot;metadata&quot;: {
    &quot;selfLink&quot;: &quot;/apis/custom.metrics.k8s.io/v1beta2/namespaces/test-namespace/pods/%2A/my-metric&quot;
  },
  &quot;items&quot;: [
    {
      &quot;describedObject&quot;: {
        &quot;kind&quot;: &quot;Pod&quot;,
        &quot;namespace&quot;: &quot;test-namespace&quot;,
        &quot;name&quot;: &quot;my-metrics-api-XXXX&quot;,
        &quot;apiVersion&quot;: &quot;/__internal&quot;
      },
      &quot;metric&quot;: {
        &quot;name&quot;: &quot;my-metric&quot;,
        &quot;selector&quot;: null
      },
      &quot;timestamp&quot;: &quot;2021-04-13T11:15:30Z&quot;,
      &quot;value&quot;: &quot;5&quot;
    }
  ]
}
</code></pre>
<p>But the stackdriver gives me this error:</p>
<pre><code>2021-04-13T11:01:30.432634Z apiserver was unable to write a JSON response: http2: stream closed
2021-04-13T11:01:30.432679Z apiserver received an error that is not an metav1.Status: &amp;errors.errorString{s:&quot;http2: stream closed&quot;}
</code></pre>
<p>I had to configure the adapter that google provides like this:</p>
<pre><code>apiVersion: v1
kind: Namespace
metadata:
  name: custom-metrics
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: custom-metrics-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics-resource-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- kind: ServiceAccount
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
  labels:
    run: custom-metrics-stackdriver-adapter
    k8s-app: custom-metrics-stackdriver-adapter
spec:
  replicas: 1
  selector:
    matchLabels:
      run: custom-metrics-stackdriver-adapter
      k8s-app: custom-metrics-stackdriver-adapter
  template:
    metadata:
      labels:
        run: custom-metrics-stackdriver-adapter
        k8s-app: custom-metrics-stackdriver-adapter
        kubernetes.io/cluster-service: &quot;true&quot;
    spec:
      serviceAccountName: custom-metrics-stackdriver-adapter
      containers:
      - image: gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.0-gke.0
        imagePullPolicy: Always
        name: pod-custom-metrics-stackdriver-adapter
        command:
        - /adapter
        - --use-new-resource-model=true
        - --cert-dir=/tmp
        - --secure-port=4443
        resources:
          limits:
            cpu: 250m
            memory: 200Mi
          requests:
            cpu: 250m
            memory: 200Mi
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: custom-metrics-stackdriver-adapter
    k8s-app: custom-metrics-stackdriver-adapter
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Adapter
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 4443
  selector:
    run: custom-metrics-stackdriver-adapter
    k8s-app: custom-metrics-stackdriver-adapter
  type: ClusterIP
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  insecureSkipTLSVerify: true
  group: custom.metrics.k8s.io
  groupPriorityMinimum: 100
  versionPriority: 100
  service:
    name: custom-metrics-stackdriver-adapter
    namespace: custom-metrics
  version: v1beta1
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta2.custom.metrics.k8s.io
spec:
  insecureSkipTLSVerify: true
  group: custom.metrics.k8s.io
  groupPriorityMinimum: 100
  versionPriority: 200
  service:
    name: custom-metrics-stackdriver-adapter
    namespace: custom-metrics
  version: v1beta2
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  insecureSkipTLSVerify: true
  group: external.metrics.k8s.io
  groupPriorityMinimum: 100
  versionPriority: 100
  service:
    name: custom-metrics-stackdriver-adapter
    namespace: custom-metrics
  version: v1beta1
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-metrics-reader
rules:
- apiGroups:
  - &quot;external.metrics.k8s.io&quot;
  resources:
  - &quot;*&quot;
  verbs:
  - list
  - get
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-metrics-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-metrics-reader
subjects:
- kind: ServiceAccount
  name: horizontal-pod-autoscaler
  namespace: kube-system
</code></pre>
<p>Because it was disabled the port 443 and I had to change to 4443 and put also the --cert-dir=/tmp option because without that option, stackdriver returns me the error:</p>
<pre><code>&quot;unable to run custom metrics adapter: error creating self-signed certificates: mkdir apiserver.local.config: permission denied&quot;
</code></pre>
<p>I think that I explained all the steps that I did to configure it, without success. Any ideas?</p>

ourobor93
<p>Resolved for me!</p>
<p>After a several of test, changing in the HPA yaml,</p>
<p>the metric from <code>Pod</code> to <code>External</code>, and the metric name with <code>custom.google.apis/my-metric</code>, it works!</p>
<pre><code>apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
  namespace: test-namespace
spec:
  maxReplicas: 60
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: my-set
  metrics:
  - type: External
    external:
      metric: 
        name: custom.googleapis.com/my-metric #custom.googleapis.com/my-metric
      target:
        averageValue: 1
        type: AverageValue
</code></pre>
