<p>I install <a href="https://github.com/external-secrets/kubernetes-external-secrets" rel="nofollow noreferrer">kubernetes external secrets</a> with helm, on GKE.</p>
<ul>
<li>GKE: 1.16.15-gke.6000 on asia-northeast1</li>
<li>helm app version 6.2.0</li>
<li>using Workload Identity <a href="https://github.com/external-secrets/kubernetes-external-secrets#deploy-kubernetes-external-secrets-using-workload-identity" rel="nofollow noreferrer">as document written</a></li>
</ul>
<p>For workload identity,the service account I bind as below command (my-secrets-sa@$PROJECT.iam.gserviceaccount.com) has <strong>SecretManager.admin</strong> role, which seems necessary for using google secrets manager</p>
<blockquote>
<p>gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member &quot;serviceAccount:$CLUSTER_PROJECT.svc.id.goog[$SECRETS_NAMESPACE/kubernetes-external-secrets]&quot; my-secrets-sa@$PROJECT.iam.gserviceaccount.com</p>
</blockquote>
<p>Workload identity looks set correctly, because checking service account in pod on GKE shows correct serviceaccount</p>
<p><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_workload_identity_on_a_new_cluster" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_workload_identity_on_a_new_cluster</a></p>
<p>create a pod in cluster and check auth in it. it shows <code>my-secrets-sa@$PROJECT.iam.gserviceaccount.com</code></p>
<blockquote>
<p>$ kubectl run -it  --image google/cloud-sdk:slim --serviceaccount ksa-name --namespace k8s-namespace workload-identity-test</p>
<p>$ gcloud auth list</p>
</blockquote>
<p>But even if creating externalsecret, externalsecret shows error</p>
<blockquote>
<p>ERROR, 7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/project-id/secrets/my-gsm-secret-name/versions/latest' (or it may not exist).</p>
</blockquote>
<p>secret <code>my-gsm-secret-name</code> itself exist in secretmanager, so it should not &quot;not exist&quot;.
Also permission must be correctly set by workload identity.</p>
<p>it's the externalsecret I defined.</p>
<pre><code>apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: gcp-secrets-manager-example    # name of the k8s external secret and the k8s secret
spec:
  backendType: gcpSecretsManager
  projectId: my-gsm-secret-project
  data:
    - key: my-gsm-secret-name     # name of the GCP secret
      name: my-kubernetes-secret-name   # key name in the k8s secret
      version: latest    # version of the GCP secret
      property: value      # name of the field in the GCP secret
</code></pre>
<p>Has anyone had similar problem before ?
Thank you</p>
<H2> whole command </H2>
<ol>
<li>create a cluster with workload-pool.</li>
</ol>
<blockquote>
<p>$ gcloud container clusters create cluster --region asia-northeast1 --node-locations asia-northeast1-a --num-nodes 1 --preemptible --workload-pool=my-project.svc.id.goog</p>
</blockquote>
<ol start="2">
<li>create kubernetes service account.</li>
</ol>
<blockquote>
<p>$ kubectl create serviceaccount --namespace default ksa</p>
</blockquote>
<ol start="3">
<li>binding kubernetes service account &amp; service account</li>
</ol>
<blockquote>
<p>$ gcloud iam service-accounts add-iam-policy-binding <br />
--role roles/iam.workloadIdentityUser <br />
--member &quot;serviceAccount:my-project.svc.id.goog[default/ksa]&quot; <br />
my-secrets-sa@my-project.iam.gserviceaccount.com`</p>
</blockquote>
<ol start="4">
<li>add annotation</li>
</ol>
<blockquote>
<p>$ kubectl annotate serviceaccount <br />
--namespace default <br />
ksa <br />
iam.gke.io/gcp-service-account=my-secrets-sa@my-project.iam.gserviceaccount.com</p>
</blockquote>
<ol start="5">
<li>install with helm</li>
</ol>
<blockquote>
<p>$ helm install my-release external-secrets/kubernetes-external-secrets</p>
</blockquote>
<ol start="6">
<li>create external secret</li>
</ol>
<pre><code>apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: gcp-secrets-manager-example    # name of the k8s external secret and the k8s secret
spec:
  backendType: gcpSecretsManager
  projectId: my-gsm-secret-project
  data:
    - key: my-gsm-secret-name     # name of the GCP secret
      name: my-kubernetes-secret-name   # key name in the k8s secret
      version: latest    # version of the GCP secret
      property: value      # name of the field in the GCP secret
</code></pre>
<blockquote>
<p>$ kubectl apply -f excternal-secret.yaml</p>
</blockquote>

<p>I noticed that I had used different kubernetes service account.</p>
<p>When installing helm, new kubernetes service account <code>my-release-kubernetes-external-secrets</code> was created, and service/pods must be working on this service account.</p>
<p>So I should bind <code>my-release-kubernetes-external-secrets</code> &amp; google service account.</p>
<p>Now, it works well.</p>
<p>Thank you @matt_j @norbjd</p>
