<p>I would like to install Kubernetes Metrics Server and try the Metrics API by following <a href="https://github.com/feiskyer/kubernetes-handbook/blob/73da37056c9c6a421be25c7b3ac42a481794ab9c/en/addons/metrics.md" rel="nofollow noreferrer">this recipe</a> (from Kubernetes Handbook). I currently have a Kubernetes 1.13 cluster that was installed with kubeadm.</p>

<p>The recipe's section <a href="https://github.com/feiskyer/kubernetes-handbook/blob/73da37056c9c6a421be25c7b3ac42a481794ab9c/en/addons/metrics.md#enable-api-aggregation" rel="nofollow noreferrer">Enable API Aggregation</a> recommends changes several settings in <code>/etc/kubernetes/manifests/kube-apiserver.yaml</code>. The current settings are as follows:</p>

<pre><code>--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
--requestheader-allowed-names=front-proxy-client
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
</code></pre>

<p>The suggested new settings are as follows:</p>

<pre><code>--requestheader-client-ca-file=/etc/kubernetes/certs/proxy-ca.crt
--proxy-client-cert-file=/etc/kubernetes/certs/proxy.crt
--proxy-client-key-file=/etc/kubernetes/certs/proxy.key
--requestheader-allowed-names=aggregator
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
</code></pre>

<p>If I install metrics-server without these changes its log contains errors like this: </p>

<blockquote>
  <p>unable to fully collect metrics: ... unable to fetch metrics from
  Kubelet ... x509: certificate signed by unknown authority</p>
</blockquote>

<p>Where do these credentials come from and what do they entail? I currently do not have a directory <code>/etc/kubernetes/certs</code>. </p>

<p><strong>UPDATE</strong> I've now tried adding the following at suitable places inside <code>metrics-server-deployment.yaml</code>, however the issue still persists (in the absence of <code>--kubelet-insecure-tls</code>):</p>

<pre><code>command:
- /metrics-server
- --client-ca-file
- /etc/kubernetes/pki/ca.crt

volumeMounts:
- mountPath: /etc/kubernetes/pki/ca.crt
  name: ca
  readOnly: true

volumens:
- hostPath:
    path: /etc/kubernetes/pki/ca.crt
    type: File
  name: ca
</code></pre>

<p><strong>UPDATE</strong> <a href="https://github.com/kubernetes-incubator/metrics-server/issues/146#issuecomment-472797066" rel="nofollow noreferrer">Here</a> is probably the reason why mounting the CA certificate into the container apparently did not help.</p>

<h2>About Kubernetes Certificates:</h2>

<p>Take a look on to how to <a href="https://kubernetes.io/docs/setup/certificates/" rel="nofollow noreferrer">Manage TLS Certificates in a Cluster</a>:</p>

<blockquote>
  <p>Every Kubernetes cluster has a cluster root Certificate Authority
  (CA). The CA is generally used by cluster components to validate the
  API server’s certificate, by the API server to validate kubelet client
  certificates, etc. To support this, the CA certificate bundle is
  distributed to every node in the cluster and is distributed as a
  secret attached to default service accounts.</p>
</blockquote>

<p>And also <a href="https://kubernetes.io/docs/setup/certificates/" rel="nofollow noreferrer">PKI Certificates and Requirements</a>:</p>

<blockquote>
  <p>Kubernetes requires PKI certificates for authentication over TLS. If
  you install Kubernetes with <code>kubeadm</code>, the certificates that your
  cluster requires are automatically generated.</p>
</blockquote>

<p><code>kubeadm</code>, by default, create the Kubernetes certificates at <strong><code>/etc/kubernetes/pki/</code></strong> directory.</p>

<h2>About the metrics-server error:</h2>

<p>It looks like the metrics-server is trying to validate the kubelet serving certs without having them be signed by the main Kubernetes CA. Installation tools like <code>kubeadm</code> may don't set up certificates properly.  </p>

<p>This problem can also happen in the case of your server have changed names/addresses after the Kubernetes installation, which causes a mismatch of the <code>apiserver.crt</code> <code>Subject Alternative Name</code> and your current names/addresses. Check it with:</p>

<pre><code>openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep DNS
</code></pre>

<p>The fastest/easy way to overcome this error is by using the <strong><code>--kubelet-insecure-tls</code></strong> flag for metrics-server. Something like this:</p>

<pre><code># metrics-server-deployment.yaml
[...]
- name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
        command:
        - /metrics-server
        - --kubelet-insecure-tls
</code></pre>

<p><strong>Note that this implies security concerns</strong>. If you are running for tests, ok. But for production, the best approach is to identify and fix the certificate issues (Take a look at this metrics-server issue for more information: <a href="https://github.com/kubernetes-incubator/metrics-server/issues/146" rel="nofollow noreferrer">#146</a>)</p>
