<p>I need to generate my own SSL certificates for Kubernetes cluster components (apiserver, apiserver-kubelet-client, apiserver-etcd-client, front-proxy-client etc.). The reason for this is because Validity period for those certificates are set to 1 year by default and I need to have validity set to more than one year, because of my business reasons. When I generated my own set of certificates and initialized cluster, everything worked perfectly - PODs in kube-system namespaces started, comunication with apiserver worked. But I encoutered that some commands like <code>kubectl logs</code> or <code>kubectl port-forward</code> or <code>kubectl exec</code> stopped working and started throwing following erros:</p>

<pre><code>kubectl logs &lt;kube-apiserver-pod&gt; -n kube-system
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log &lt;kube-apiserver-pod&gt;))
</code></pre>

<p>or</p>

<pre><code>kubectl exec -it &lt;kube-apiserver-pod&gt; -n kube-system sh
error: unable to upgrade connection: Unauthorized`
</code></pre>

<p>however <code>docker exec</code> command to log to k8s_apiserver container is working properly.</p>

<p>During my debugging I found out that only self generated <code>apiserver-kubelet-client</code> key/cert file is causing this cluster behaviour.</p>

<p>Bellow is process I used to generate and use my own cert/key pair for apiserver-kubelet-client.</p>

<ol>
<li><p>I inicialized kubernetes cluster to set its own certificates into <code>/etc/kubernetes/pki</code> folder by running <code>kubeadm init ...</code></p></li>
<li><p>Make a backup of <code>/etc/kubernetes/pki</code> folder into <code>/tmp/pki_k8s</code></p></li>
<li><p>Open <code>apiserver-kubelet-client.crt</code> with openssl to check all set extentions, CN, O etc.</p>

<p><code>openssl x509 -noout -text -in /tmp/pki_k8s/apiserver-kubelet-client.crt</code></p></li>
<li><p>To ensure same extentions and CN,O parameters to appear in certificate generated by myself I created .conf file for extentions and .csr file for CN and O</p>

<p><code>cd /tmp/pki_k8s/
cat &lt;&lt;-EOF_api_kubelet_client-ext &gt; apiserver_kubelet_client-ext.conf
 [ v3_ca ]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
EOF_api_kubelet_client-ext</code></p>

<p><code>openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters,CN=kube-apiserver-kubelet-client"</code></p></li>
<li><p>Finally I generated my own apiserver-kubelet-client.crt. For its generation I reused existing apiserver-kubelet-client.key and ca.crt/ca.key generated by K8S initialization </p>

<p><code>openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -sha256 -out apiserver-kubelet-client.crt -extensions v3_ca -extfile apiserver_kubelet_client-ext.conf -days 3650</code></p></li>
<li><p>Once I had generated my own <code>apiserver-kubelet-client.crt</code> which overides the previous one generated by k8s initialization script itself, I reset kubernetes cluster by hitting <code>kubeadm reset</code>. This purged /etc/kubernetes folder</p></li>
<li><p>copy all certificates into <code>/etc/kubernetes/pki</code> from <code>/tmp/pki_k8s</code></p></li>
<li>and reinitialize K8S cluster <code>kubeadm init ...</code></li>
</ol>

<p>During that I saw that K8S cluster used already existing certificates stored in <code>/etc/kubernetes/pki</code> for setup.</p>

<pre><code>[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and key.
[certificates] Using the existing sa key.
[certificates] Using the existing front-proxy-ca certificate and key.
[certificates] Using the existing front-proxy-client certificate and key.
[certificates] Using the existing etcd/ca certificate and key.
[certificates] Using the existing etcd/server certificate and key.
[certificates] Using the existing etcd/peer certificate and key.
[certificates] Using the existing etcd/healthcheck-client certificate and key.
[certificates] Using the existing apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
</code></pre>

<p>After that, K8S cluster is UP, I can list pods, list description, make deployments etc. however not able to check logs, exec command as described above. </p>

<pre><code> kubectl get pods -n kube-system
NAME                                           READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-kjkp9                       1/1       Running   0          2m
coredns-78fcdf6894-q88lx                       1/1       Running   0          2m
...

kubectl  logs &lt;apiserver_pod&gt; -n kube-system -v 7
I0818 08:51:12.435494   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.436355   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.438413   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.447751   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.448109   12811 round_trippers.go:383] GET https://&lt;HOST_IP&gt;:6443/api/v1/namespaces/kube-system/pods/&lt;apiserver_pod&gt;
I0818 08:51:12.448126   12811 round_trippers.go:390] Request Headers:
I0818 08:51:12.448135   12811 round_trippers.go:393]     Accept: application/json, */*
I0818 08:51:12.448144   12811 round_trippers.go:393]     User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f
I0818 08:51:12.462931   12811 round_trippers.go:408] Response Status: 200 OK in 14 milliseconds
I0818 08:51:12.471316   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.471949   12811 round_trippers.go:383] GET https://&lt;HOST_IP&gt;:6443/api/v1/namespaces/kube-system/pods/&lt;apiserver_pod&gt;/log
I0818 08:51:12.471968   12811 round_trippers.go:390] Request Headers:
I0818 08:51:12.471977   12811 round_trippers.go:393]     Accept: application/json, */*
I0818 08:51:12.471985   12811 round_trippers.go:393]     User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f
I0818 08:51:12.475827   12811 round_trippers.go:408] Response Status: 401 Unauthorized in 3 milliseconds
I0818 08:51:12.476288   12811 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server has asked for the client to provide credentials ( pods/log &lt;apiserver_pod&gt;)",
  "reason": "Unauthorized",
  "details": {
    "name": "&lt;apiserver_pod&gt;",
    "kind": "pods/log"
  },
  "code": 401
}]
F0818 08:51:12.476325   12811 helpers.go:119] error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log &lt;apiserver_pod&gt;))
</code></pre>

<p>See kubelet service file below:</p>

<pre><code>[root@qa053 ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
Environment="CA_CLIENT_CERT=--client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELE=--rotate-certificates=true"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CERTIFICATE_ARGS $CA_CLIENT_CERT
</code></pre>

<p>Do you have any ideas ? :) 
Thanks</p>

<p>Best Regard</p>

<p>I found out reason why it did not worked.</p>

<p>When creating .csr file i used this:</p>

<pre><code>openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters,CN=kube-apiserver-kubelet-client"
</code></pre>

<p>But in -subj was wrong formatting which caused problems with parsing right CN from certificate. Instead of <code>"/O=system:masters,CN=kube-apiserver-kubelet-client"</code> it needs to be</p>

<pre><code>openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters/CN=kube-apiserver-kubelet-client"
</code></pre>

<p>Certificates generated by both .csr files looks same in terms of -text view. But they act differently.</p>
