<p>I have installed the metric server on kubernetes, but its not working and logs </p>

<pre><code>unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:xxx: unable to fetch metrics from Kubelet ... (X.X): Get https:....: x509: cannot validate certificate for 1x.x.

x509: certificate signed by unknown authority
</code></pre>

<p>I was able to get metrics if modified the deployment yaml and added </p>

<pre><code> command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
</code></pre>

<p>this now collects metrics, and  kubectl top node returns results...</p>

<p>but logs still show</p>

<pre><code> E1120 11:58:45.624974       1 reststorage.go:144] unable to fetch pod metrics for pod dev/pod-6bffbb9769-6z6qz: no metrics known for pod
E1120 11:58:45.625289       1 reststorage.go:144] unable to fetch pod metrics for pod dev/pod-6bffbb9769-rzvfj: no metrics known for pod
E1120 12:00:06.462505       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-1x.x.x.eu-west-1.compute.internal: unable to get CPU for container ...discarding data: missing cpu usage metric, unable to fully scrape metrics from source
</code></pre>

<p>so questions</p>

<p>1) All this works on minikube, but not on my dev cluster, why would that be?</p>

<p>2) In production i dont want to do insecure-tls.. so can someone please explain why this issue is arising... or point me to some resource.</p>

<p>Kubeadm generates the kubelet certificate at <code>/var/lib/kubelet/pki</code> and those certificates (<code>kubelet.crt and kubelet.key</code>) are signed by different CA from the one which is used to generate all other certificates at <code>/etc/kubelet/pki</code>. </p>

<p>You need to regenerate the kubelet certificates which is signed by your root CA (<code>/etc/kubernetes/pki/ca.crt</code>)</p>

<p>You can use openssl or cfssl to generate the new certificates(I am using cfssl)</p>

<pre><code>$ mkdir certs; cd certs
$ cp /etc/kubernetes/pki/ca.crt ca.pem
$ cp /etc/kubernetes/pki/ca.key ca-key.pem
</code></pre>

<p>Create a file <code>kubelet-csr.json</code>:</p>

<pre><code>{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "&lt;node_name&gt;",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "US",
    "ST": "NY",
    "L": "City",
    "O": "Org",
    "OU": "Unit"
  }]
}
</code></pre>

<p>Create a ca-config.json file:</p>

<pre><code>{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}
</code></pre>

<p>Now generate the new certificates using above files:</p>

<pre><code>$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
--config=ca-config.json -profile=kubernetes \
kubelet-csr.json | cfssljson -bare kubelet
</code></pre>

<p>Replace the old certificates with newly generated one:</p>

<pre><code>$ scp kubelet.pem &lt;nodeip&gt;:/var/lib/kubelet/pki/kubelet.crt
$ scp kubelet-key.pem &lt;nodeip&gt;:/var/lib/kubelet/pki/kubelet.key
</code></pre>

<p>Now restart the kubelet so that new certificates will take effect on your node.</p>

<pre><code>$ systemctl restart kubelet
</code></pre>

<p>Look at the following tickets to get the context of issue:</p>

<p><a href="https://github.com/kubernetes-incubator/metrics-server/issues/146" rel="nofollow noreferrer">https://github.com/kubernetes-incubator/metrics-server/issues/146</a></p>

<p>Hope this helps.</p>
