<p>In trying to securely install metrics-server on Kubernetes, I'm having problems.</p>
<p>It seems like the metric-server pod is unable to successfully make requests to the Kubelet API on it's <code>10250</code> port.</p>
<pre><code>NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   0/1     1            0           16h
</code></pre>
<p>The Metrics Server deployment never becomes ready and it repeats the same sequence of error logs:</p>
<pre class="lang-sh prettyprint-override"><code>I0522 01:27:41.472946       1 serving.go:342] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0522 01:27:41.798068       1 configmap_cafile_content.go:201] &quot;Starting controller&quot; name=&quot;client-ca::kube-system::extension-apiserver-authentication::client-ca-file&quot;
I0522 01:27:41.798092       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0522 01:27:41.798068       1 dynamic_cafile_content.go:156] &quot;Starting controller&quot; name=&quot;request-header::/front-ca/front-proxy-ca.crt&quot;
I0522 01:27:41.798107       1 dynamic_serving_content.go:131] &quot;Starting controller&quot; name=&quot;serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key&quot;
I0522 01:27:41.798240       1 secure_serving.go:266] Serving securely on [::]:4443
I0522 01:27:41.798265       1 tlsconfig.go:240] &quot;Starting DynamicServingCertificateController&quot;
W0522 01:27:41.798284       1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I0522 01:27:41.898439       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
E0522 01:27:55.297497       1 scraper.go:140] &quot;Failed to scrape node&quot; err=&quot;Get \&quot;https://192.168.1.106:10250/metrics/resource\&quot;: context deadline exceeded&quot; node=&quot;system76-pc&quot;
E0522 01:28:10.297872       1 scraper.go:140] &quot;Failed to scrape node&quot; err=&quot;Get \&quot;https://192.168.1.106:10250/metrics/resource\&quot;: context deadline exceeded&quot; node=&quot;system76-pc&quot;
I0522 01:28:10.325613       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
I0522 01:28:20.325231       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
E0522 01:28:25.297750       1 scraper.go:140] &quot;Failed to scrape node&quot; err=&quot;Get \&quot;https://192.168.1.106:10250/metrics/resource\&quot;: context deadline exceeded&quot; node=&quot;system76-pc&quot;
</code></pre>
<p>I'm running Kubernetes deployed with <code>kubeadm</code> version 1.23.4 and I'm trying to securely use metrics-server.</p>
<p>I'm looking for advice that could help with:</p>
<ol>
<li>How I can accurately diagnose the problem?</li>
<li>Or alternatively, what configuration seems most fruitful to check first?</li>
<li>Anything that will help with my mental model of which certificates and keys I need to configure explicitly and what is being handled automatically.</li>
</ol>
<p>So far, I have tried to validate that the I can retrieve API metrics:</p>
<p><code>kubectl get --raw /api/v1/nodes/system76-pc/proxy/stats/summary</code></p>
<pre class="lang-json prettyprint-override"><code>{
  &quot;node&quot;: {
    &quot;nodeName&quot;: &quot;system76-pc&quot;,
    &quot;systemContainers&quot;: [
      {
        &quot;name&quot;: &quot;kubelet&quot;,
        &quot;startTime&quot;: &quot;2022-05-20T01:51:28Z&quot;,
        &quot;cpu&quot;: {
          &quot;time&quot;: &quot;2022-05-22T00:48:40Z&quot;,
          &quot;usageNanoCores&quot;: 59453039,
          &quot;usageCoreNanoSeconds&quot;: 9768130002000
        },
        &quot;memory&quot;: {
          &quot;time&quot;: &quot;2022-05-22T00:48:40Z&quot;,
          &quot;usageBytes&quot;: 84910080,
          &quot;workingSetBytes&quot;: 84434944,
          &quot;rssBytes&quot;: 67149824,
          &quot;pageFaults&quot;: 893055,
          &quot;majorPageFaults&quot;: 290
        }
      },
      {
        &quot;name&quot;: &quot;runtime&quot;,
        &quot;startTime&quot;: &quot;2022-05-20T00:33:24Z&quot;,
        &quot;cpu&quot;: {
          &quot;time&quot;: &quot;2022-05-22T00:48:37Z&quot;,
          &quot;usageNanoCores&quot;: 24731571,
          &quot;usageCoreNanoSeconds&quot;: 3955659226000
        },
        &quot;memory&quot;: {
          &quot;time&quot;: &quot;2022-05-22T00:48:37Z&quot;,
          &quot;usageBytes&quot;: 484306944,
          &quot;workingSetBytes&quot;: 242638848,
          &quot;rssBytes&quot;: 84647936,
          &quot;pageFaults&quot;: 56994074,
          &quot;majorPageFaults&quot;: 428
        }
      },
      {
        &quot;name&quot;: &quot;pods&quot;,
        &quot;startTime&quot;: &quot;2022-05-20T01:51:28Z&quot;,
        &quot;cpu&quot;: {
          &quot;time&quot;: &quot;2022-05-22T00:48:37Z&quot;,
          &quot;usageNanoCores&quot;: 292818104,
          &quot;usageCoreNanoSeconds&quot;: 45976001446000
        },
        &quot;memory&quot;: {
          &quot;time&quot;: &quot;2022-05-22T00:48:37Z&quot;,
          &quot;availableBytes&quot;: 29648396288,
          &quot;usageBytes&quot;: 6108573696,

</code></pre>
<p><code>kubectl get --raw /api/v1/nodes/system76-pc/proxy/metrics/resource</code></p>
<pre class="lang-sh prettyprint-override"><code># HELP container_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the container in core-seconds
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{container=&quot;alertmanager&quot;,namespace=&quot;flux-system&quot;,pod=&quot;alertmanager-prometheus-stack-kube-prom-alertmanager-0&quot;} 108.399948 1653182143362
container_cpu_usage_seconds_total{container=&quot;calico-kube-controllers&quot;,namespace=&quot;kube-system&quot;,pod=&quot;calico-kube-controllers-56fcbf9d6b-n87ts&quot;} 206.442768 1653182144294
container_cpu_usage_seconds_total{container=&quot;calico-node&quot;,namespace=&quot;kube-system&quot;,pod=&quot;calico-node-p6pxk&quot;} 6147.643669 1653182155672
container_cpu_usage_seconds_total{container=&quot;cert-manager&quot;,namespace=&quot;cert-manager&quot;,pod=&quot;cert-manager-795d7f859d-8jp4f&quot;} 134.583294 1653182142601
container_cpu_usage_seconds_total{container=&quot;cert-manager&quot;,namespace=&quot;cert-manager&quot;,pod=&quot;cert-manager-cainjector-5fcddc948c-vw4zz&quot;} 394.286782 1653182151252
container_cpu_usage_seconds_total{container=&quot;cert-manager&quot;,namespace=&quot;cert-manager&quot;,pod=&quot;cert-manager-webhook-5b64f87794-pl7fb&quot;} 404.53758 1653182140528
container_cpu_usage_seconds_total{container=&quot;config-reloader&quot;,namespace=&quot;flux-system&quot;,pod=&quot;alertmanager-prometheus-stack-kube-prom-alertmanager-0&quot;} 6.01391 1653182139771
container_cpu_usage_seconds_total{container=&quot;config-reloader&quot;,namespace=&quot;flux-system&quot;,pod=&quot;prometheus-prometheus-stack-kube-prom-prometheus-0&quot;} 42.706567 1653182130750
container_cpu_usage_seconds_total{container=&quot;controller&quot;,namespace=&quot;flux-system&quot;,pod=&quot;sealed-secrets-controller-5884bbf4d6-mql9x&quot;} 43.814816 1653182144648
container_cpu_usage_seconds_total{container=&quot;controller&quot;,namespace=&quot;ingress-nginx&quot;,pod=&quot;ingress-nginx-controller-f9d6fc8d8-sgwst&quot;} 645.109711 1653182141169
container_cpu_usage_seconds_total{container=&quot;coredns&quot;,namespace=&quot;kube-system&quot;,pod=&quot;coredns-64897985d-crtd9&quot;} 380.682251 1653182141861
container_cpu_usage_seconds_total{container=&quot;coredns&quot;,namespace=&quot;kube-system&quot;,pod=&quot;coredns-64897985d-rpmxk&quot;} 365.519839 1653182140533
container_cpu_usage_seconds_total{container=&quot;dashboard-metrics-scraper&quot;,namespace=&quot;kubernetes-dashboard&quot;,pod=&quot;dashboard-metrics-scraper-577dc49767-cbq8r&quot;} 25.733362 1653182141877
container_cpu_usage_seconds_total{container=&quot;etcd&quot;,namespace=&quot;kube-system&quot;,pod=&quot;etcd-system76-pc&quot;} 4237.357682 1653182140459
container_cpu_usage_seconds_total{container=&quot;grafana&quot;,namespace=&quot;flux-system&quot;,pod=&quot;prometheus-stack-grafana-757f9b9fcc-9f58g&quot;} 345.034245 1653182154951
container_cpu_usage_seconds_total{container=&quot;grafana-sc-dashboard&quot;,namespace=&quot;flux-system&quot;,pod=&quot;prometheus-stack-grafana-757f9b9fcc-9f58g&quot;} 123.480584 1653182146757
container_cpu_usage_seconds_total{container=&quot;grafana-sc-datasources&quot;,namespace=&quot;flux-system&quot;,pod=&quot;prometheus-stack-grafana-757f9b9fcc-9f58g&quot;} 35.851112 1653182145702
container_cpu_usage_seconds_total{container=&quot;kube-apiserver&quot;,namespace=&quot;kube-system&quot;,pod=&quot;kube-apiserver-system76-pc&quot;} 14166.156638 1653182150749
container_cpu_usage_seconds_total{container=&quot;kube-controller-manager&quot;,namespace=&quot;kube-system&quot;,pod=&quot;kube-controller-manager-system76-pc&quot;} 4168.427981 1653182148868
container_cpu_usage_seconds_total{container=&quot;kube-prometheus-stack&quot;,namespace=&quot;flux-system&quot;,pod=&quot;prometheus-stack-kube-prom-operator-54d9f985c8-ml2qj&quot;} 28.79018 1653182155583
container_cpu_usage_seconds_total{container=&quot;kube-proxy&quot;,namespace=&quot;kube-system&quot;,pod=&quot;kube-proxy-gg2wd&quot;} 67.215459 1653182155156
container_cpu_usage_seconds_total{container=&quot;kube-scheduler&quot;,namespace=&quot;kube-system&quot;,pod=&quot;kube-scheduler-system76-pc&quot;} 579.321492 1653182147910
container_cpu_usage_seconds_total{container=&quot;kube-state-metrics&quot;,namespace=&quot;flux-system&quot;,pod=&quot;prometheus-stack-kube-state-metrics-56d4759d67-h6lfv&quot;} 158.343644 1653182153691
container_cpu_usage_seconds_total{container=&quot;kubernetes-dashboard&quot;,namespace=&quot;kubernetes-dashboard&quot;,pod=&quot;kubernetes-dashboard-69dc48777b-8cckh&quot;} 78.231809 1653182139263
container_cpu_usage_seconds_total{container=&quot;manager&quot;,namespace=&quot;flux-system&quot;,pod=&quot;helm-controller-dfb4b5478-7zgt6&quot;} 338.974637 1653182143679
container_cpu_usage_seconds_total{container=&quot;manager&quot;,namespace=&quot;flux-system&quot;,pod=&quot;image-automation-controller-77fd9657c6-lg44h&quot;} 280.841645 1653182154912
container_cpu_usage_seconds_total{container=&quot;manager&quot;,namespace=&quot;flux-system&quot;,pod=&quot;image-reflector-controller-86db8b6f78-5rz58&quot;} 2909.277578 1653182144081
container_cpu_usage_seconds_total{container=&quot;manager&quot;,namespace=&quot;flux-system&quot;,pod=&quot;kustomize-controller-cd544c8f8-hxvk6&quot;} 596.392781 1653182152714
container_cpu_usage_seconds_total{container=&quot;manager&quot;,namespace=&quot;flux-system&quot;,pod=&quot;notification-controller-d9cc9bf46-2jhbq&quot;} 244.387967 1653182142902
container_cpu_usage_seconds_total{container=&quot;manager&quot;,namespace=&quot;flux-system&quot;,pod=&quot;source-controller-84bfd77bf8-r827h&quot;} 541.650877 1653182148963
container_cpu_usage_seconds_total{container=&quot;metrics-server&quot;,namespace=&quot;flux-system&quot;,pod=&quot;metrics-server-55bc5f774-zznpb&quot;} 174.229886 1653182146946
container_cpu_usage_seconds_total{container=&quot;nfs-subdir-external-provisioner&quot;,namespace=&quot;flux-system&quot;,pod=&quot;nfs-subdir-external-provisioner-858745f657-zcr66&quot;} 244.061329 1653182139840
container_cpu_usage_seconds_total{container=&quot;node-exporter&quot;,namespace=&quot;flux-system&quot;,pod=&quot;prometheus-stack-prometheus-node-exporter-wj2fx&quot;} 29.852036 1653182148779
container_cpu_usage_seconds_total{container=&quot;prometheus&quot;,namespace=&quot;flux-system&quot;,pod=&quot;prometheus-prometheus-stack-kube-prom-prometheus-0&quot;} 7141.611234 1653182154042
# HELP container_memory_working_set_bytes [ALPHA] Current working set of the container in bytes
# TYPE container_memory_working_set_bytes gauge
container_memory_working_set_bytes{container=&quot;alertmanager&quot;,namespace=&quot;flux-system&quot;,pod=&quot;alertmanager-prometheus-stack-kube-prom-alertmanager-0&quot;} 2.152448e+07 1653182143362
</code></pre>
<p>metric-server config:</p>
<pre class="lang-yaml prettyprint-override"><code> spec:
      containers:
      - args:
        - --secure-port=4443
        - --cert-dir=/tmp
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-preferred-address-types=Hostname
        - --requestheader-client-ca-file=/front-ca/front-proxy-ca.crt
        - --kubelet-certificate-authority=/ca/ca.crt
        image: k8s.gcr.io/metrics-server/metrics-server:v0.6.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp
        - mountPath: /front-ca
          name: front-proxy-ca-dir
        - mountPath: /ca
          name: ca-dir
      dnsPolicy: ClusterFirst
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: metrics-server
      serviceAccountName: metrics-server
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: tmp
      - configMap:
          defaultMode: 420
          name: front-proxy-ca
        name: front-proxy-ca-dir
      - configMap:
          defaultMode: 420
          name: kubelet-ca
        name: ca-dir
</code></pre>
<p>kube-apiserver config:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.106:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.106
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.23.4
</code></pre>

<p>In my case I had the same issue with the metrics-server because of just 1 ocpu on master-node. Use at least 2.</p>
