<p>I am using <strong>KubeSpray</strong> to provision a two node cluster on AWS. By default, the <code>--kubelet-certificate-authority</code> parameter is not used. However, I would like to set it.</p>
<p>I do not know the correct setting for <code>--kubelet-certificate-authority</code>. When I set it to <code>/etc/kubernetes/pki/ca.crt</code> I see messages like the following in my logs:</p>
<pre><code>TLS handshake error from 10.245.207.223:59278: remote error: tls: unknown certificate authority
</code></pre>
<p>In order to create an isolated environment for testing, I SSH'ed to my controller node to run this command which runs apiserver on the non-standard port of <code>4667</code>. I copied these values directly from <code>/etc/kubernetes/manifests/kube-apiserver.yaml</code>. You'll need to adjust the value to match your own cluster. I purposely am running the container in interactive mode so that I can see the log messages.</p>
<pre class="lang-sh prettyprint-override"><code>sudo docker run \
  --name api-server-playground \
  -it \
  --rm \
  --network host \
  --volume /etc/kubernetes:/etc/kubernetes:ro \
  --volume /etc/pki:/etc/pki:ro \
  --volume /etc/ssl:/etc/ssl/:ro \
  k8s.gcr.io/kube-apiserver:v1.18.9 \
  kube-apiserver \
    --advertise-address=10.245.207.223 \
    --allow-privileged=true \
    --anonymous-auth=True \
    --apiserver-count=1 \
    --authorization-mode=Node,RBAC \
    --bind-address=0.0.0.0 \
    --client-ca-file=/etc/kubernetes/ssl/ca.crt \
    --cloud-config=/etc/kubernetes/cloud_config \
    --cloud-provider=aws \
    --enable-admission-plugins=NodeRestriction \
    --enable-aggregator-routing=False \
    --enable-bootstrap-token-auth=true \
    --endpoint-reconciler-type=lease \
    --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem \
    --etcd-certfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal.pem \
    --etcd-keyfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal-key.pem \
    --etcd-servers=https://10.245.207.119:2379 \
    --event-ttl=1h0m0s \
    --insecure-port=0 \
    --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt \
    --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key \
    --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP \
    --profiling=False \
    --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt \
    --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key \
    --request-timeout=1m0s \
    --requestheader-allowed-names=front-proxy-client \
    --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt \
    --requestheader-extra-headers-prefix=X-Remote-Extra- \
    --requestheader-group-headers=X-Remote-Group \
    --requestheader-username-headers=X-Remote-User \
    --secure-port=6447 \
    --service-account-key-file=/etc/kubernetes/ssl/sa.pub \
    --service-cluster-ip-range=10.233.0.0/18 \
    --service-node-port-range=30000-32767 \
    --storage-backend=etcd3 \
    --tls-cert-file=/etc/kubernetes/ssl/apiserver.crt \
    --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key
</code></pre>
<p>Now it's possible to SSH into the controller again and use <code>curl</code> to interact with the custom apiserver container.</p>
<ul>
<li>Set the APISERVER variable to point to the controller node. I'm using the value of the <code>advertise-address</code> parameter above.</li>
</ul>
<pre><code>APISERVER=https://10.245.207.223:6447
</code></pre>
<ul>
<li>Get a token.</li>
</ul>
<pre class="lang-sh prettyprint-override"><code>TOKEN=$(kubectl get secrets -o jsonpath=&quot;{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}&quot;|base64 --decode); echo &quot;TOKEN=$TOKEN&quot;
</code></pre>
<ul>
<li>Make an api request. This request will fail because &quot;Peer's Certificate issuer is not recognized.&quot;.</li>
</ul>
<pre class="lang-sh prettyprint-override"><code>curl --header &quot;Authorization: Bearer $TOKEN&quot; -X GET $APISERVER/api
</code></pre>
<p>An error message will appear in the docker log. It will look like this:</p>
<pre><code>I0921 14:39:07.662368       1 log.go:172] http: TLS handshake error 
from 10.245.207.223:59278: remote error: tls: unknown certificate authority
</code></pre>
<ul>
<li>Now use the <code>-k</code> curl parameter to bypass the recognition issue.</li>
</ul>
<pre><code>curl -k --header &quot;Authorization: Bearer $TOKEN&quot; -X GET $APISERVER/api
{
  &quot;kind&quot;: &quot;APIVersions&quot;,
  &quot;versions&quot;: [
    &quot;v1&quot;
  ],
  &quot;serverAddressByClientCIDRs&quot;: [
    {
      &quot;clientCIDR&quot;: &quot;0.0.0.0/0&quot;,
      &quot;serverAddress&quot;: &quot;10.245.207.223:6447&quot;
    }
  ]
}
</code></pre>
<p>You'll see the request works correctly. However, I don't want to use the <code>-k</code> parameter. So I tried to use the certificate authority from the apiserver.</p>
<ul>
<li>Get the certificate authority from the apiserver.</li>
</ul>
<pre class="lang-sh prettyprint-override"><code>echo | \
    openssl s_client -connect $APISERVER 2&gt;/dev/null | \
    openssl x509 -text | \
    sed -n &quot;/BEGIN CERTIFICATE/,/END CERTIFICATE/p&quot; \
    &gt; apiserver.ca.crt
</code></pre>
<ul>
<li>Use the certificate authority file for the api request.</li>
</ul>
<pre class="lang-sh prettyprint-override"><code>curl --cacert apiserver.ca.crt --header &quot;Authorization: Bearer $TOKEN&quot; -X GET $APISERVER/api
</code></pre>
<p><strong>UPDATE</strong></p>
<p>Following a thought prompted by Wiktor's response, I added /etc/kubernetes/ssl/ca.crt as the certificate authority. And used that file in my <code>curl</code> command.</p>
<pre class="lang-sh prettyprint-override"><code>curl --cacert /etc/kubernetes/ssl/ca.crt --header &quot;Authorization: Bearer $TOKEN&quot; -X GET $APISERVER/api
</code></pre>
<p>This worked.</p>

<p>In order to make the <code>--kubelet-certificate-authority</code> flag work you first need to make sure you got <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/#kubelet-authentication" rel="nofollow noreferrer">Kubelet authentication</a> and <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/#kubelet-authorization" rel="nofollow noreferrer">Kubelet authorization</a> enabled. After that you can follow the Kubernetes documentation and setup the <a href="https://kubernetes.io/docs/concepts/architecture/control-plane-node-communication/#apiserver-to-kubelet" rel="nofollow noreferrer">TLS connection between the apiserver and kubelet</a>. And finally, you can edit the API server pod specification file <code>/etc/kubernetes/manifests/kube-apiserver.yaml</code> on the master node and set the <code>--kubelet-certificate-authority</code> parameter to the path to the cert file for the certificate authority.</p>
<p>So, to sum up the steps to do are:</p>
<ol>
<li><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/#kubelet-authentication" rel="nofollow noreferrer">Kubelet authentication</a>:</li>
</ol>
<ul>
<li><p>start the kubelet with the <code>--anonymous-auth=false</code> flag</p>
</li>
<li><p>start the kubelet with the <code>--client-ca-file</code> flag, providing a CA bundle to verify client certificates with</p>
</li>
<li><p>start the apiserver with <code>--kubelet-client-certificate</code> and <code>--kubelet-client-key</code> flags</p>
</li>
<li><p>ensure the <code>authentication.k8s.io/v1beta1</code> API group is enabled in the API server</p>
</li>
<li><p>start the kubelet with the <code>--authentication-token-webhook</code> and <code>--kubeconfig flags</code></p>
</li>
<li><p>the kubelet calls the <code>TokenReview</code> API on the configured API server to determine user information from bearer tokens</p>
</li>
</ul>
<ol start="2">
<li><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/#kubelet-authorization" rel="nofollow noreferrer">Kubelet authorization</a>:</li>
</ol>
<ul>
<li><p>ensure the <code>authorization.k8s.io/v1beta1</code> API group is enabled in the API server</p>
</li>
<li><p>start the kubelet with the <code>--authorization-mode=Webhook</code> and the <code>--kubeconfig</code> flags</p>
</li>
<li><p>the kubelet calls the <code>SubjectAccessReview</code> API on the configured API server to determine whether each request is authorized</p>
</li>
</ul>
<ol start="3">
<li>Use the <code>--kubelet-certificate-authority</code> flag to provide the apiserver with a root certificate bundle to use to verify the kubelet's serving certificate.</li>
</ol>
<p>More details can be found in the linked documentation.</p>
