<h2>Problem statement</h2>
<p>The kubernetes metrics server starts, but doesn't collect metrics.
When running <code>$ kubectl top pods</code> it returns <code>error: Metrics not available for pod &lt;namespace&gt;/&lt;deployment&gt;</code>.</p>
<h2>Artificats</h2>
<p>I'm using the following <code>metrics.yml</code></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: metrics-server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: &quot;true&quot;
    rbac.authorization.k8s.io/aggregate-to-edit: &quot;true&quot;
    rbac.authorization.k8s.io/aggregate-to-view: &quot;true&quot;
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - &quot;&quot;
  resources:
  - nodes/metrics
  verbs:
  - get
- apiGroups:
  - &quot;&quot;
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: metrics-server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: metrics-server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: metrics-server
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: metrics-server
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      hostNetwork: true
      containers:
      - args:
        - /metrics-server
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP
        - --kubelet-use-node-status-port
        - --kubelet-insecure-tls
        image: k8s.gcr.io/metrics-server/metrics-server:v0.6.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: metrics-server
  version: v1beta1
  versionPriority: 100
</code></pre>
<p>Once deployed on my local machine, the deployment looks like this:</p>
<pre><code>$ kubectl describe deployment.apps/metrics-server -n metrics-server
Name:                   metrics-server
Namespace:              metrics-server
CreationTimestamp:      Tue, 13 Dec 2022 12:38:39 +0100
Labels:                 k8s-app=metrics-server
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               k8s-app=metrics-server
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=metrics-server
  Service Account:  metrics-server
  Containers:
   metrics-server:
    Image:      k8s.gcr.io/metrics-server/metrics-server:v0.6.2
    Port:       4443/TCP
    Host Port:  4443/TCP
    Args:
      /metrics-server
      --cert-dir=/tmp
      --secure-port=4443
      --kubelet-preferred-address-types=InternalIP
      --kubelet-use-node-status-port
      --kubelet-insecure-tls
    Requests:
      cpu:        100m
      memory:     200Mi
    Liveness:     http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:https/readyz delay=20s timeout=1s period=10s #success=1 #failure=3
    Environment:  &lt;none&gt;
    Mounts:
      /tmp from tmp-dir (rw)
  Volumes:
   tmp-dir:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:          &lt;unset&gt;
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  &lt;none&gt;
NewReplicaSet:   metrics-server-5988cd75cb (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  2m2s  deployment-controller  Scaled up replica set metrics-server-5988cd75cb to 1
</code></pre>
<h2>Findings</h2>
<p>This exact same setup (without <code>--kubelet-insecure-tls</code>) is working on another DigitalOcean hosted cluster. It has the following kubernetes version:</p>
<pre><code>$ kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;23+&quot;, GitVersion:&quot;v1.23.14-dispatcher-dirty&quot;, GitCommit:&quot;35498c8a8d141664928467fda116cd500d09bc21&quot;, GitTreeState:&quot;dirty&quot;, BuildDate:&quot;2022-11-16T20:14:00Z&quot;, GoVersion:&quot;go1.17.13&quot;, Compiler:&quot;gc&quot;, Platform:&quot;windows/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;23&quot;, GitVersion:&quot;v1.23.9&quot;, GitCommit:&quot;c1de2d70269039fe55efb98e737d9a29f9155246&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2022-07-13T14:19:57Z&quot;, GoVersion:&quot;go1.17.11&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<p>My machine, where the metrics server isn't working has the version:</p>
<pre><code>$ kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;23+&quot;, GitVersion:&quot;v1.23.14-dispatcher-dirty&quot;, GitCommit:&quot;35498c8a8d141664928467fda116cd500d09bc21&quot;, GitTreeState:&quot;dirty&quot;, BuildDate:&quot;2022-11-16T20:14:00Z&quot;, GoVersion:&quot;go1.17.13&quot;, Compiler:&quot;gc&quot;, Platform:&quot;windows/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;24&quot;, GitVersion:&quot;v1.24.0&quot;, GitCommit:&quot;4ce5a8954017644c5420bae81d72b09b735c21f0&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2022-05-03T13:38:19Z&quot;, GoVersion:&quot;go1.18.1&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<h2>Possible solutions?</h2>
<ol>
<li>Try to install the same kubernetes version on my local machine? --&gt; Tried this but Docker desktop let me change the server version.</li>
</ol>

<p>I think I found the solution. I overlooked the following comment: <a href="https://github.com/kubernetes-sigs/metrics-server/issues/1061#issuecomment-1239227118" rel="nofollow noreferrer">https://github.com/kubernetes-sigs/metrics-server/issues/1061#issuecomment-1239227118</a></p>
<p>It turns out that Docker-Desktop 4.12 has this fixed. I was still on 4.9 but running a new Kubernetes version (1.24) where Dockershim has been removed. Several bugs have been fixed since 4.9. Updating Docker-Desktop to 4.15 (currently the latest version) fixed the issue for me.</p>
<p>So to summarize:</p>
<ol>
<li>Update docker desktop to 4.12 or later</li>
<li>Reset the Kubernetes cluster</li>
<li>Check if cluster was upgraded using <code>kubectl version</code></li>
<li>Apply <code>metrics.yml</code></li>
</ol>
<p><a href="https://i.stack.imgur.com/24L4m.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/24L4m.png" alt="enter image description here" /></a></p>
<p>This is the final metrics.yml</p>
<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: metrics-server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: &quot;true&quot;
    rbac.authorization.k8s.io/aggregate-to-edit: &quot;true&quot;
    rbac.authorization.k8s.io/aggregate-to-view: &quot;true&quot;
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - &quot;&quot;
  resources:
  - nodes/metrics
  verbs:
  - get
- apiGroups:
  - &quot;&quot;
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: metrics-server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: metrics-server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: metrics-server
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: metrics-server
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=40s
        - --kubelet-insecure-tls
        image: k8s.gcr.io/metrics-server/metrics-server:v0.6.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: metrics-server
  version: v1beta1
  versionPriority: 100
</code></pre>
