<p>I am trying to deploy a <code>Kubernetes</code> cluster on <code>AWS EKS</code> using <code>Terraform</code>, run from a <code>Gitlab CI</code> pipeline. My code currently gets a full cluster up and running, except there is a step in which it tries to add the nodes (which are created separately) into the cluster.</p>
<p>When it tries to do this, this is the error I receive:</p>
<pre><code>│ Error: configmaps is forbidden: User &quot;system:serviceaccount:gitlab-managed-apps:default&quot; cannot create resource &quot;configmaps&quot; in API group &quot;&quot; in the namespace &quot;kube-system&quot;
│ 
│   with module.mastercluster.kubernetes_config_map.aws_auth[0],
│   on .terraform/modules/mastercluster/aws_auth.tf line 63, in resource &quot;kubernetes_config_map&quot; &quot;aws_auth&quot;:
│   63: resource &quot;kubernetes_config_map&quot; &quot;aws_auth&quot; {
│ 
</code></pre>
<p><code>Terraform</code> I believe is trying to edit the configmap <code>aws_auth</code> in the <code>kube-system</code> namespace, but for whatever reason, it doesn't have permission to do so?</p>
<p>I have found a different answer from years ago on Stackoverflow, that currently matches with what the documentation has to say about adding a <code>aws_eks_cluster_auth</code> data source and adding this to the <code>kubernetes</code> provider.</p>
<p>My configuration of this currently looks like this:</p>
<pre><code>data &quot;aws_eks_cluster&quot; &quot;mastercluster&quot; {
    name    = module.mastercluster.cluster_id
}

data &quot;aws_eks_cluster_auth&quot; &quot;mastercluster&quot; {
    name    = module.mastercluster.cluster_id
}

provider &quot;kubernetes&quot; {
    alias   = &quot;mastercluster&quot;

    host                    = data.aws_eks_cluster.mastercluster.endpoint
    cluster_ca_certificate  = base64decode(data.aws_eks_cluster.mastercluster.certificate_authority[0].data)
    token                   = data.aws_eks_cluster_auth.mastercluster.token

    load_config_file        = false
}
</code></pre>
<p>The weird thing is, this has worked for me before. I have successfully deployed multiple clusters using this method. This configuration is an almost identical copy to another one I had before, only the names of the clusters are different. I am totally lost as to why this can possibly go wrong.</p>

<h1>Use <code>semver</code> to lock <code>hashicorp provider</code> versions</h1>
<p>That's why is so important to use <a href="https://semver.org" rel="nofollow noreferrer"><code>semver</code></a> in <code>terraform</code> manifests.</p>
<p>As per <a href="https://learn.hashicorp.com/tutorials/terraform/provider-versioning" rel="nofollow noreferrer">Terraform documentation</a>:</p>
<blockquote>
<p>Terraform providers manage resources by communicating between Terraform and target APIs. Whenever the target APIs change or add functionality, provider maintainers may update and version the provider.</p>
<p>When multiple users or automation tools run the same Terraform configuration, they should all use the same versions of their required providers.</p>
</blockquote>
<h1>Use <code>RBAC</code> rules for <code>Kubernetes</code></h1>
<p>There is a Github issue filed about this: <a href="https://github.com/hashicorp/terraform-provider-kubernetes/issues/1127" rel="nofollow noreferrer">v2.0.1: Resources cannot be created. Does kubectl reference to kube config properly? · Issue #1127 · hashicorp/terraform-provider-kubernetes</a> with the same error message as in yours case.</p>
<p>And <a href="https://github.com/hashicorp/terraform-provider-kubernetes/issues/1127#issuecomment-765635852" rel="nofollow noreferrer">one of the comments answers</a>:</p>
<blockquote>
<p>Offhand, this looks related to <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole" rel="nofollow noreferrer">RBAC</a> rules in the cluster (which may have been installed by the helm chart). This command might help diagnose the permissions issues relating to the service account in the error message.</p>
<pre><code>$ kubectl auth can-i create namespace --as=system:serviceaccount:gitlab-prod:default
$ kubectl auth can-i --list --as=system:serviceaccount:gitlab-prod:default
</code></pre>
<p>You might be able to compare that list with other users on the cluster:</p>
<pre><code>kubectl auth can-i --list --namespace=default --as=system:serviceaccount:default:default
</code></pre>
<pre><code>$ kubectl auth can-i create configmaps
yes

$ kubectl auth can-i create configmaps --namespace=nginx-ingress --as=system:serviceaccount:gitlab-prod:default
no
</code></pre>
<p>And investigate related clusterroles:</p>
<pre><code>$ kube describe clusterrolebinding system:basic-user
Name:         system:basic-user
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
Role:
  Kind:  ClusterRole
  Name:  system:basic-user
Subjects:
  Kind   Name                  Namespace
  ----   ----                  ---------
  Group  system:authenticated


$ kubectl describe clusterrole system:basic-user
Name:         system:basic-user
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources                                      Non-Resource URLs  Resource Names  Verbs
  ---------                                      -----------------  --------------  -----
  selfsubjectaccessreviews.authorization.k8s.io  []                 []              [create]
  selfsubjectrulesreviews.authorization.k8s.io   []                 []              [create]
</code></pre>
<p>My guess is that the chart or Terraform config in question is responsible for creating the service account, and the [cluster] roles and rolebindings, but it might be doing so in the wrong order, or not idempotently (so you get different results on re-install vs the initial install). But we would need to see a configuration that reproduces this error. In my testing of version 2 of the providers on <a href="https://github.com/hashicorp/terraform-provider-kubernetes/tree/master/_examples/aks" rel="nofollow noreferrer">AKS</a>, <a href="https://github.com/hashicorp/terraform-provider-kubernetes/tree/master/_examples/eks" rel="nofollow noreferrer">EKS</a>, <a href="https://github.com/hashicorp/terraform-provider-kubernetes/tree/master/_examples/gke" rel="nofollow noreferrer">GKE</a>, and <a href="https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started#creating-your-first-kubernetes-resources" rel="nofollow noreferrer">minikube</a>, I haven't seen this issue come up.</p>
<p>Feel free to browse these working examples of building specific clusters and using them with Kubernetes and Helm providers. Giving the config a skim might give you some ideas for troubleshooting further.</p>
</blockquote>
<h1>Howto solve <code>RBAC</code> issues</h1>
<p>As for the error</p>
<pre><code>Error: configmaps is forbidden: User &quot;system:serviceaccount:kube-system:default&quot; cannot list
</code></pre>
<p>There is <a href="https://github.com/helm/helm/issues/5100#issuecomment-533787541" rel="nofollow noreferrer">great explanation by @m-abramovich</a>:</p>
<blockquote>
<p>First, some information for newbies.<br />
In Kubernetes there are:</p>
<ul>
<li>Account - something like your ID. Example: john</li>
<li>Role - some group in the project permitted to do something. Examples: cluster-admin, it-support, ...</li>
<li>Binding - joining Account to Role. &quot;John in it-support&quot; - is a binding.</li>
</ul>
<p>Thus, in our message above, we see that our Tiller acts as account &quot;default&quot; registered at namespace &quot;kube-system&quot;. Most likely you didn't bind him to a sufficient role.</p>
<p>Now back to the problem.<br />
How do we track it:</p>
<ul>
<li>check if you have specific <strong>account</strong> for tiller. Usually it has same name - &quot;tiller&quot;:<br />
<code>kubectl [--namespace kube-system] get serviceaccount</code><br />
create if not:<br />
<code>kubectl [--namespace kube-system] create serviceaccount tiller</code></li>
<li>check if you have <strong>role</strong> or <strong>clusterrole</strong> (cluster role is &quot;better&quot; for newbies - it is cluster-wide unlike namespace-wide role). If this is not a production, you can use highly privileged role &quot;cluster-admin&quot;:<br />
<code>kubectl [--namespace kube-system] get clusterrole</code><br />
you can check role content via:<br />
<code>kubectl [--namespace kube-system] get clusterrole cluster-admin -o yaml</code></li>
<li>check if <strong>account</strong> &quot;tiller&quot; in first clause has a binding to <strong>clusterrole</strong> &quot;cluster-admin&quot; that you deem sufficient:<br />
<code>kubectl [--namespace kube-system] get clusterrolebinding</code><br />
if it is hard to figure out based on names, you can simply create new:<br />
<code>kubectl [--namespace kube-system] create clusterrolebinding tiller-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:tiller</code></li>
<li>finally, when you have the account, the role and the binding between them, you can check if you really act as this account:<br />
<code>kubectl [--namespace kube-system] get deploy tiller-deploy -o yaml</code></li>
</ul>
<p>I suspect that your output will not have settings &quot;serviceAccount&quot; and &quot;serviceAccountName&quot;:</p>
<pre><code>dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
</code></pre>
<p>if yes, than add an account you want tiller to use:<br />
<code>kubectl [--namespace kube-system] patch deploy tiller-deploy -p '{&quot;spec&quot;:{&quot;template&quot;:{&quot;spec&quot;:{&quot;serviceAccount&quot;:&quot;tiller&quot;}}}}'</code><br />
(if you use PowerShell, then check below for post from <a href="https://github.com/snpdev" rel="nofollow noreferrer">@snpdev</a>)<br />
Now you repeat previous check command and see the difference:</p>
<pre><code>dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: tiller                     &lt;-- new line
serviceAccountName: tiller          &lt;-- new line
terminationGracePeriodSeconds: 30
</code></pre>
</blockquote>
<h1>Resources:</h1>
<ul>
<li><a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/" rel="nofollow noreferrer">Using RBAC Authorization | Kubernetes</a></li>
<li><a href="https://www.cncf.io/blog/2018/08/01/demystifying-rbac-in-kubernetes/" rel="nofollow noreferrer">Demystifying RBAC in Kubernetes | Cloud Native Computing Foundation</a></li>
<li><a href="https://helm.sh/docs/topics/rbac/" rel="nofollow noreferrer">Helm | Role-based Access Control</a></li>
<li><a href="https://learn.hashicorp.com/tutorials/terraform/provider-versioning" rel="nofollow noreferrer">Lock and Upgrade Provider Versions | Terraform - HashiCorp Learn</a></li>
</ul>
