<p>I set up K3s on a server with:</p>
<pre><code>curl -sfL https://get.k3s.io | K3S_TOKEN={token} INSTALL_K3S_EXEC=&quot;server --cluster-init --disable=traefik --write-kubeconfig-mode 644&quot; sh -s -
</code></pre>
<p>Then I grabbed the kube config from <code>/etc/rancher/k3s/k3s.yaml</code> and copy it to my local machine so I can interact with the cluster from my machine rather than the server node I installed K3s on. I had to swap out references to 127.0.0.1 and change it to the actual hostname of the server I installed K3s on as well but other than that it worked.</p>
<p>I then hooked up 2 more server nodes to the cluster for a High Availability setup using:</p>
<pre><code>curl -sfL https://get.k3s.io | K3S_TOKEN={token} INSTALL_K3S_EXEC=&quot;server --server {https://{hostname or IP of server 1}:6443 --disable=traefik --write-kubeconfig-mode 644&quot; sh -s -
</code></pre>
<p>Now on my local machine again I run <code>kubectl get pods</code> (for example) and that works but I want a highly available setup so I placed a TCP Load Balancer (NGINX actually) in front of my cluster. Now I am trying to connect to the Kubernetes API through that proxy / load balancer and unfortunately, since my <code>~/.kube/config</code> has a client certificate for authentication, this no longer works because my load balancer / proxy that lives in front of that server cannot pass my client cert on to the K3s server.</p>
<p>My <code>~/.kube/config</code>:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: {omitted}
    server: https://my-cluster-hostname:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: {omitted}
    client-key-data: {omitted}
</code></pre>
<p>I also grabbed that client cert and key in my kube config, exported it to a file, and hit the API server with curl and it works when I directly hit the server nodes but NOT when I go through my proxy / load balancer.</p>
<p>What I would like to do instead of using the client certificate approach is use <code>token</code> authentication as my proxy would not interfere with that. However, I am not sure how to get such a token. I read the <a href="https://kubernetes.io/docs/reference/access-authn-authz/authentication/" rel="nofollow noreferrer">Kubernetes Authenticating guide</a> and specifically I tried creating a new service account and getting the token associated with it as described in the <a href="https://kubernetes.io/docs/reference/access-authn-authz/authentication/#service-account-tokens" rel="nofollow noreferrer">Service Account Tokens</a> section but that also did not work. I also dug through <a href="https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/" rel="nofollow noreferrer">K3s server config options</a> to see if there was any mention of static token file, etc. but didn't find anything that seemed likely.</p>
<p>Is this some limitation of K3s or am I just doing something wrong (likely)?</p>
<p>My <code>kubectl version</code> output:</p>
<pre><code>Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.7&quot;, GitCommit:&quot;132a687512d7fb058d0f5890f07d4121b3f0a2e2&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-05-12T12:40:09Z&quot;, GoVersion:&quot;go1.15.12&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;21&quot;, GitVersion:&quot;v1.21.7+k3s1&quot;, GitCommit:&quot;ac70570999c566ac3507d2cc17369bb0629c1cc0&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-11-29T16:40:13Z&quot;, GoVersion:&quot;go1.16.10&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>

<p>I figured out an approach that works for me by reading through the <a href="https://kubernetes.io/docs/reference/access-authn-authz/authentication/" rel="nofollow noreferrer">Kubernetes Authenticating Guide</a> in more detail. I settled on the <a href="https://kubernetes.io/docs/reference/access-authn-authz/authentication/#service-account-tokens" rel="nofollow noreferrer">Service Account Tokens</a> approach as it says:</p>
<blockquote>
<p>Normally these secrets are mounted into pods for in-cluster access to
the API server, but can be used from outside the cluster as well.</p>
</blockquote>
<p>My use is for outside the cluster.</p>
<p>First, I created a new <code>ServiceAccount</code> called <code>cluster-admin</code>:</p>
<pre><code>kubectl create serviceaccount cluster-admin
</code></pre>
<p>I then created a <code>ClusterRoleBinding</code> to assign cluster-wide permissions to my ServiceAccount (I named this <code>cluster-admin-manual</code> because K3s already had created one called <code>cluster-admin</code> that I didn't want to mess with):</p>
<pre><code>kubectl create clusterrolebinding cluster-admin-manual --clusterrole=cluster-admin --serviceaccount=default:cluster-admin
</code></pre>
<p>Now you have to get the <code>Secret</code> that is created for you when you created your <code>ServiceAccount</code>:</p>
<pre><code>kubectl get serviceaccount cluster-admin -o yaml
</code></pre>
<p>You'll see something like this returned:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: &quot;2021-12-20T15:55:55Z&quot;
  name: cluster-admin
  namespace: default
  resourceVersion: &quot;3973955&quot;
  uid: 66bab124-8d71-4e5f-9886-0bad0ebd30b2
secrets:
- name: cluster-admin-token-67jtw
</code></pre>
<p>Get the Secret content with:</p>
<pre><code>kubectl get secret cluster-admin-token-67jtw -o yaml
</code></pre>
<p>In that output you will see the <code>data/token</code> property. This is a base64 encoded JWT bearer token. Decode it with:</p>
<pre><code>echo {base64-encoded-token} | base64 --decode
</code></pre>
<p>Now you have your bearer token and you can add a user to your <code>~/.kube/config</code> with the following command. You can also paste that JWT into <a href="https://jwt.io/" rel="nofollow noreferrer">jwt.io</a> to take a look at the properties and make sure you base64 decoded it properly.</p>
<pre><code>kubectl config set-credentials my-cluster-admin --token={token}
</code></pre>
<p>Then make sure your existing <code>context</code> in your <code>~/.kube/config</code> has the user set appropriately (I did this manually by editing my kube config file but there's probably a <code>kubectl config</code> command for it). For example:</p>
<pre class="lang-yaml prettyprint-override"><code>- context:
    cluster: my-cluster
    user: my-cluster-admin
  name: my-cluster
</code></pre>
<p>My user in the kube config looks like this:</p>
<pre class="lang-yaml prettyprint-override"><code>- name: my-cluster-admin
  user:
    token: {token}
</code></pre>
<p>Now I can authenticate to the cluster using the token instead of relying on a transport layer specific mechanism (TLS with Mutual Auth) that my proxy / load-balancer does not interfere with.</p>
<p>Other resources I found helpful:</p>
<ul>
<li><a href="https://medium.com/devops-mojo/kubernetes-role-based-access-control-rbac-overview-introduction-rbac-with-kubernetes-what-is-2004d13195df" rel="nofollow noreferrer">Kubernetes — Role-Based Access Control (RBAC) Overview by Anish Patel</a></li>
</ul>
