<p>I have two kubernetes clusters that were set up by kops.  They are both running <code>v1.10.8</code>.  I have done by best to mirror the configuration between the two.  They both have RBAC enabled.  I have kubernetes-dashboard running on both.  They both have a <code>/srv/kubernetes/known_tokens.csv</code> with an <code>admin</code> and a <code>kube</code> user:</p>

<p><code>$ sudo cat /srv/kubernetes/known_tokens.csv
ABCD,admin,admin,system:masters
DEFG,kube,kube
(... other users ...)
</code></p>

<p>My question is how do these users get authorized with consideration to RBAC?  When authenticating to kubernetes-dashboard using tokens, the <code>admin</code> user's token works on both clusters and has full access.  But the <code>kube</code> user's token only has access on one of the clusters.  On one cluster, I get the following errors in the dashboard.</p>

<p><code>configmaps is forbidden: User "kube" cannot list configmaps in the namespace "default"
persistentvolumeclaims is forbidden: User "kube" cannot list persistentvolumeclaims in the namespace "default"
secrets is forbidden: User "kube" cannot list secrets in the namespace "default"
services is forbidden: User "kube" cannot list services in the namespace "default"
ingresses.extensions is forbidden: User "kube" cannot list ingresses.extensions in the namespace "default"
daemonsets.apps is forbidden: User "kube" cannot list daemonsets.apps in the namespace "default"
pods is forbidden: User "kube" cannot list pods in the namespace "default"
events is forbidden: User "kube" cannot list events in the namespace "default"
deployments.apps is forbidden: User "kube" cannot list deployments.apps in the namespace "default"
replicasets.apps is forbidden: User "kube" cannot list replicasets.apps in the namespace "default"
jobs.batch is forbidden: User "kube" cannot list jobs.batch in the namespace "default"
cronjobs.batch is forbidden: User "kube" cannot list cronjobs.batch in the namespace "default"
replicationcontrollers is forbidden: User "kube" cannot list replicationcontrollers in the namespace "default"
statefulsets.apps is forbidden: User "kube" cannot list statefulsets.apps in the namespace "default"
</code></p>

<p>As per the <a href="https://kubernetes.io/docs/reference/access-authn-authz/authentication/#users-in-kubernetes" rel="nofollow noreferrer">official docs</a>, "Kubernetes does not have objects which represent normal user accounts".  </p>

<p>I can't find anywhere on the working cluster that would give authorization to <code>kube</code>.  Likewise, I can't find anything that would restrict <code>kube</code> on the other cluster.  I've checked all <code>ClusterRoleBinding</code> resources in the <code>default</code> and <code>kube-system</code> namespace.  None of these reference the <code>kube</code> user.  So why the discrepancy in access to the dashboard and how can I adjust it?</p>

<p>Some other questions:</p>

<ul>
<li>How do I debug authorization issues such as this?  The dashboard logs just say this user doesn't have access.  Is there somewhere I can see which <code>serviceAccount</code> a particular request or token is mapped to?</li>
<li>What are <code>groups</code> in k8s?  The <a href="https://kubernetes.io/docs/reference/access-authn-authz/authentication/" rel="nofollow noreferrer">k8s docs</a> mention groups a lot. Even the static token users can be assigned a group such as <code>system:masters which looks like a</code>role<code>/</code>clusterrole<code>but there is no</code>system:masters<code>role in my cluster?  What exactly are</code>groups`?  As per <a href="https://stackoverflow.com/questions/44763590/create-user-group-using-rbac-api">Create user group using RBAC API?</a>, it appears groups are simply arbitrary labels that can be defined per user.  What's the point of them?  Can I map a group to a RBAC serviceAccount?</li>
</ul>

<p><strong>Update</strong></p>

<p>I restarted the working cluster and it no longer works.  I get the same authorization errors as the working cluster.  Looks like it was some sort of cached access.  Sorry for the bogus question.  I'm still curious on my follow-up questions but they can be made into separate questions.</p>

<p>Hard to tell without access to the cluster, but my guess is that you have a <code>Role</code> and a <code>RoleBinding</code> somewhere for the <code>kube</code> user on the cluster that works. Not a <code>ClusterRole</code> with <code>ClusterRoleBinding</code>.</p>

<p>Something like this:</p>

<pre><code>kind: Role
metadata:
  name: my-role
  namespace: default
rules:
- apiGroups: [""]
  Resources: ["services", "endpoints", "pods"]
  verbs: ["get", "list", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: kube-role-binding
  namespace: default
subjects:
- kind: User
  name: "kube"
  apiGroup: ""
roleRef:
  kind: Role
  name: my-role
  apiGroup: ""
</code></pre>

<blockquote>
  <p>How do I debug authorization issues such as this? The dashboard logs
  just say this user doesn't have access. Is there somewhere I can see
  which serviceAccount a particular request or token is mapped to?</p>
</blockquote>

<p>You can look at the kube-apiserver logs under <code>/var/log/kube-apiserver.log</code> on your leader master. Or if it's running in a container <code>docker logs &lt;container-id-of-kube-apiserver&gt;</code></p>
