<p>I'm using a HPA based on a custom metric on GKE.</p>
<p>The HPA is not working and it's showing me this error log:</p>
<blockquote>
<p>unable to fetch metrics from custom metrics API: the server is currently unable to handle the request</p>
</blockquote>
<p>When I run <code>kubectl get apiservices | grep custom</code> I get</p>
<blockquote>
<p>v1beta1.custom.metrics.k8s.io            services/prometheus-adapter      False (FailedDiscoveryCheck)   135d</p>
</blockquote>
<p>this is the HPA spec config :</p>
<pre><code>spec:
  scaleTargetRef:
    kind: Deployment
    name: api-name
    apiVersion: apps/v1
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Object
      object:
        target:
          kind: Service
          name: api-name
          apiVersion: v1
        metricName: messages_ready_per_consumer
        targetValue: '1'
</code></pre>
<p>and this is the service's spec config :</p>
<pre><code>spec:
  ports:
    - name: worker-metrics
      protocol: TCP
      port: 8080
      targetPort: worker-metrics
  selector:
    app.kubernetes.io/instance: api
    app.kubernetes.io/name: api-name
  clusterIP: 10.8.7.9
  clusterIPs:
    - 10.8.7.9
  type: ClusterIP
  sessionAffinity: None
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
</code></pre>
<p>What should I do to make it work ?</p>

<p>First of all, confirm that the Metrics Server POD is running in your <code>kube-system</code> namespace. Also, you can use the following manifest:</p>
<pre><code>---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        imagePullPolicy: Always
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
</code></pre>
<p>If so, take a look into the logs and look for any <em><strong>stackdriver adapter’s</strong></em> line. This issue is commonly caused due to a problem with the <code>custom-metrics-stackdriver-adapter</code>. It usually crashes in the <code>metrics-server</code> namespace. To solve that, use the resource from this <a href="https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml" rel="nofollow noreferrer">URL</a>, and for the deployment, use this image:</p>
<pre><code>gcr.io/google-containers/custom-metrics-stackdriver-adapter:v0.10.1
</code></pre>
<p>Another common root cause of this is an <strong>OOM</strong> issue. In this case, adding more memory solves the problem. To assign more memory, you can specify the new memory amount in the configuration file, as the following example shows:</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: memory-demo
  namespace: mem-example
spec:
  containers:
  - name: memory-demo-ctr
    image: polinux/stress
    resources:
      limits:
        memory: &quot;200Mi&quot;
      requests:
        memory: &quot;100Mi&quot;
    command: [&quot;stress&quot;]
    args: [&quot;--vm&quot;, &quot;1&quot;, &quot;--vm-bytes&quot;, &quot;150M&quot;, &quot;--vm-hang&quot;, &quot;1&quot;]
</code></pre>
<p>In the above example, the Container has a memory request of 100 MiB and a memory limit of 200 MiB. In the manifest, the &quot;--vm-bytes&quot;, &quot;150M&quot; argument tells the Container to attempt to allocate 150 MiB of memory. You can visit this Kubernetes Official <a href="https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/" rel="nofollow noreferrer">Documentation</a> to have more references about the Memory settings.</p>
<p>You can use the following threads for more reference <a href="https://stackoverflow.com/questions/61098043/gke-hpa-using-custom-metrics-unable-to-fetch-metrics">GKE - HPA using custom metrics - unable to fetch metrics</a>, <a href="https://stackoverflow.com/questions/60541105/stackdriver-metadata-agent-cluster-level-gets-oomkilled/60549732#60549732">Stackdriver-metadata-agent-cluster-level gets OOMKilled</a>, and <a href="https://github.com/GoogleCloudPlatform/k8s-stackdriver/issues/303" rel="nofollow noreferrer">Custom-metrics-stackdriver-adapter pod keeps crashing</a>.</p>
