<p>I am new to kubernetes and was trying to apply horizontal pod autoscaling to my existing application. and after following other stackoverflow details - got to know that I need to install metric-server - and I was able to - but some how it's not working and unable to handle request.</p>
<ul>
<li>Further I followed few more things but unable to resolve the issue - I will really appreciate any help here.
Please let me know for any further details you need for helping me :) Thanks in advance.</li>
</ul>
<p>Steps followed:</p>
<p>kubectl apply -f <a href="https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml" rel="noreferrer">https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml</a></p>
<pre><code>serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
</code></pre>
<p>kubectl get deploy,svc -n kube-system | egrep metrics-server</p>
<pre><code>deployment.apps/metrics-server   1/1     1            1           2m6s
service/metrics-server                       ClusterIP   10.32.0.32   &lt;none&gt;        443/TCP                        2m6s
</code></pre>
<p>kubectl get pods -n kube-system | grep metrics-server</p>
<pre><code>metrics-server-64cf6869bd-6gx88   1/1     Running   0          2m39s
</code></pre>
<p>vi ana_hpa.yaml</p>
<pre><code>apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: ana-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: common-services-auth
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 160
    
</code></pre>
<p>k apply -f ana_hpa.yaml</p>
<p>horizontalpodautoscaler.autoscaling/ana-hpa created</p>
<p>k get hpa</p>
<pre><code>NAME      REFERENCE                          TARGETS                         MINPODS   MAXPODS   REPLICAS   AGE
ana-hpa   StatefulSet/common-services-auth   &lt;unknown&gt;/160%, &lt;unknown&gt;/80%   1         10        0          4s
</code></pre>
<p>k describe hpa ana-hpa</p>
<pre><code>Name:                                                     ana-hpa
Namespace:                                                default
Labels:                                                   &lt;none&gt;
Annotations:                                              &lt;none&gt;
CreationTimestamp:                                        Tue, 12 Apr 2022 17:01:25 +0530
Reference:                                                StatefulSet/common-services-auth
Metrics:                                                  ( current / target )
  resource memory on pods  (as a percentage of request):  &lt;unknown&gt; / 160%
  resource cpu on pods  (as a percentage of request):     &lt;unknown&gt; / 80%
Min replicas:                                             1
Max replicas:                                             10
StatefulSet pods:                                         3 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get memory utilization: unable to get metrics for resource memory: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedGetResourceMetric       38s (x8 over 2m23s)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
  Warning  FailedComputeMetricsReplicas  38s (x8 over 2m23s)  horizontal-pod-autoscaler  invalid metrics (2 invalid out of 2), first error is: failed to get memory utilization: unable to get metrics for resource memory: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
  Warning  FailedGetResourceMetric       23s (x9 over 2m23s)  horizontal-pod-autoscaler  failed to get memory utilization: unable to get metrics for resource memory: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
  
  
</code></pre>
<p>kubectl get --raw /apis/metrics.k8s.io/v1beta1</p>
<pre><code>Error from server (ServiceUnavailable): the server is currently unable to handle the request
</code></pre>
<p>kubectl get --raw &quot;/apis/metrics.k8s.io/v1beta1/nodes&quot;</p>
<pre><code>Error from server (ServiceUnavailable): the server is currently unable to handle the request
</code></pre>
<p>kubectl edit deployments.apps -n kube-system metrics-server</p>
<p>Add hostNetwork: true</p>
<pre><code>deployment.apps/metrics-server edited
</code></pre>
<p>kubectl get pods -n kube-system | grep metrics-server</p>
<p>metrics-server-5dc6dbdb8-42hw9   1/1     Running   0          10m</p>
<p>k describe pod metrics-server-5dc6dbdb8-42hw9 -n kube-system</p>
<pre><code>Name:                 metrics-server-5dc6dbdb8-42hw9
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 pusntyn196.apac.avaya.com/10.133.85.196
Start Time:           Tue, 12 Apr 2022 17:08:25 +0530
Labels:               k8s-app=metrics-server
                      pod-template-hash=5dc6dbdb8
Annotations:          &lt;none&gt;
Status:               Running
IP:                   10.133.85.196
IPs:
  IP:           10.133.85.196
Controlled By:  ReplicaSet/metrics-server-5dc6dbdb8
Containers:
  metrics-server:
    Container ID:  containerd://024afb1998dce4c0bd5f4e58f996068ea37982bd501b54fda2ef8d5c1098b4f4
    Image:         k8s.gcr.io/metrics-server/metrics-server:v0.6.1
    Image ID:      k8s.gcr.io/metrics-server/metrics-server@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00
    Port:          4443/TCP
    Host Port:     4443/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --kubelet-use-node-status-port
      --metric-resolution=15s
    State:          Running
      Started:      Tue, 12 Apr 2022 17:08:26 +0530
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
      memory:     200Mi
    Liveness:     http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:https/readyz delay=20s timeout=1s period=10s #success=1 #failure=3
    Environment:  &lt;none&gt;
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g6p4g (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  &lt;unset&gt;
  kube-api-access-g6p4g:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       &lt;nil&gt;
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 2s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 2s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  2m31s  default-scheduler  Successfully assigned kube-system/metrics-server-5dc6dbdb8-42hw9 to pusntyn196.apac.avaya.com
  Normal  Pulled     2m32s  kubelet            Container image &quot;k8s.gcr.io/metrics-server/metrics-server:v0.6.1&quot; already present on machine
  Normal  Created    2m31s  kubelet            Created container metrics-server
  Normal  Started    2m31s  kubelet            Started container metrics-server
  
</code></pre>
<p>kubectl get --raw /apis/metrics.k8s.io/v1beta1</p>
<pre><code>Error from server (ServiceUnavailable): the server is currently unable to handle the request
</code></pre>
<p>kubectl get pods -n kube-system | grep metrics-server</p>
<pre><code>metrics-server-5dc6dbdb8-42hw9   1/1     Running   0          10m
</code></pre>
<p>kubectl logs -f metrics-server-5dc6dbdb8-42hw9 -n kube-system</p>
<pre><code>E0412 11:43:54.684784       1 configmap_cafile_content.go:242] kube-system/extension-apiserver-authentication failed with : missing content for CA bundle &quot;client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file&quot;
E0412 11:44:27.001010       1 configmap_cafile_content.go:242] key failed with : missing content for CA bundle &quot;client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file&quot;

k logs -f metrics-server-5dc6dbdb8-42hw9 -n kube-system
    I0412 11:38:26.447305       1 serving.go:342] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
    I0412 11:38:26.899459       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
    I0412 11:38:26.899477       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
    I0412 11:38:26.899518       1 configmap_cafile_content.go:201] &quot;Starting controller&quot; name=&quot;client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file&quot;
    I0412 11:38:26.899545       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
    I0412 11:38:26.899546       1 configmap_cafile_content.go:201] &quot;Starting controller&quot; name=&quot;client-ca::kube-system::extension-apiserver-authentication::client-ca-file&quot;
    I0412 11:38:26.899567       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
    I0412 11:38:26.900480       1 dynamic_serving_content.go:131] &quot;Starting controller&quot; name=&quot;serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key&quot;
    I0412 11:38:26.900811       1 secure_serving.go:266] Serving securely on [::]:4443
    I0412 11:38:26.900854       1 tlsconfig.go:240] &quot;Starting DynamicServingCertificateController&quot;
    W0412 11:38:26.900965       1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
    I0412 11:38:26.999960       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
    I0412 11:38:26.999989       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
    I0412 11:38:26.999970       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
    E0412 11:38:27.000087       1 configmap_cafile_content.go:242] kube-system/extension-apiserver-authentication failed with : missing content for CA bundle &quot;client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file&quot;
    E0412 11:38:27.000118       1 configmap_cafile_content.go:242] key failed with : missing content for CA bundle &quot;client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file&quot;
</code></pre>
<p>kubectl top nodes</p>
<pre><code>Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
</code></pre>
<p>kubectl top pods</p>
<pre><code>Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
</code></pre>
<p>Edit metrics server deployment yaml</p>
<pre><code>Add - --kubelet-insecure-tls
</code></pre>
<p>k apply -f metric-server-deployment.yaml</p>
<pre><code>serviceaccount/metrics-server unchanged
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/system:metrics-server unchanged
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server unchanged
service/metrics-server unchanged
deployment.apps/metrics-server configured
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged
</code></pre>
<p>kubectl get pods -n kube-system | grep metrics-server</p>
<pre><code>metrics-server-5dc6dbdb8-42hw9   1/1     Running   0          10m
</code></pre>
<p>kubectl top pods</p>
<pre><code>Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
</code></pre>
<p>Also tried by adding below to metrics server deployment</p>
<pre><code>    command:
    - /metrics-server
    - --kubelet-insecure-tls
    - --kubelet-preferred-address-types=InternalIP
</code></pre>

<p>First of all, execute the following command:</p>
<pre><code>kubectl get apiservices
</code></pre>
<p>And checkout the availablity (status) of <code>kube-system/metrics-server</code> service.</p>
<ul>
<li><p>In case the availability is <em>True</em>:
Add <code>hostNetwork: true</code> to the <code>spec</code> of your metrics-server deployment by executing the following command:</p>
<pre class="lang-bash prettyprint-override"><code>kubectl edit deployment -n kube-system metrics-server
</code></pre>
<p>It should look like the following:</p>
<pre class="lang-yaml prettyprint-override"><code>...
spec:
  hostNetwork: true
...

</code></pre>
<p>Setting hostNetwork to true means that Pod will have access to
the host where it's running.</p>
</li>
<li><p>In case the availability is <em>False (MissingEndpoints)</em>:</p>
<ol>
<li><p>Download metrics-server:</p>
<pre><code>wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml
</code></pre>
</li>
<li><p>Remove (legacy) metrics server:</p>
<pre><code>kubectl delete -f components.yaml  
</code></pre>
</li>
<li><p>Edit downloaded file and add <code>- --kubelet-insecure-tls</code> to args list:</p>
<pre><code>...
labels:
    k8s-app: metrics-server
spec:
  containers:
  - args:
    - --cert-dir=/tmp
    - --secure-port=443
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --kubelet-use-node-status-port
    - --metric-resolution=15s
    - --kubelet-insecure-tls # add this line
...
</code></pre>
</li>
<li><p>Create service once again:</p>
<pre><code>kubectl apply -f components.yaml
</code></pre>
</li>
</ol>
</li>
</ul>
