<p>I'm using <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity" rel="nofollow noreferrer">Workload Identity with GKE</a> so that a namespaced pod can get access to a GCP resource. I noticed that it takes ~1 min for the binding between IAM and GKE accounts to take place which is way too slow for my application.</p>
<pre><code>curl -H &quot;Metadata-Flavor: Google&quot; http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email
# =&gt; myapp@myproject.iam.gserviceaccount.com


# Bind GKE service account to IAM service account
gcloud iam service-accounts add-iam-policy-binding \
      myapp@myproject.iam.gserviceaccount.com \
      --member &quot;myproject.svc.id.goog[user-1/myapp]&quot; \
      --role roles/iam.workloadIdentityUser


# ... Takes 1min of 403 until I can actually access the GCP resource
</code></pre>
<p>Is this normal behaviour? What's an alternative for instantly giving access to a pod?</p>

Pithikos
<p>Yes, this is normal.</p>
<p>Google Cloud IAM has various caching layers which means it takes time for <a href="https://cloud.google.com/iam/docs/access-change-propagation" rel="nofollow noreferrer">access change propagation</a> to complete. This makes applying IAM changes &quot;eventually consistent&quot;.</p>
<p>Additionally, there is a <a href="https://%3E%20https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity#restrictions" rel="nofollow noreferrer">known limitation</a> of several seconds for a metadata server using Workload Identity to become available to a pod:</p>
<blockquote>
<p>The GKE metadata server <strong>takes a few seconds</strong> to start accepting requests on a newly created Pod. Therefore, attempts to authenticate using Workload Identity within the first few seconds of a Pod's life might fail. Retrying the call will resolve the problem. See Troubleshooting for more details.</p>
</blockquote>
<p>You can use an <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/" rel="nofollow noreferrer">init container</a> in your pod to confirm the <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity#instance_metadata" rel="nofollow noreferrer">metadata server</a> is behaving properly (e.g. <code>curl -H 'Metadata-Flavor:google' http://metadata.google.internal/computeMetadata/v1</code>...) before your main application container starts. That way everything is ready when your app code starts running.</p>
<blockquote>
<p>What's an alternative for instantly giving access to a pod?</p>
</blockquote>
<p>This depends on your application needs. Why does each pod need a new service account? Some alternatives include:</p>
<ul>
<li>Using a pool of preexisting service accounts and cycling their permissions and pod usage (e.g. remove from pod, apply any changes, use on the next pod)</li>
<li>Pass in API keys or other needed values via <a href="https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod" rel="nofollow noreferrer">Kubernetes Secrets mounts</a></li>
<li>Mint and use <a href="https://cloud.google.com/iam/docs/create-short-lived-credentials-direct" rel="nofollow noreferrer">short lived access IAM tokens</a> for your Service Account</li>
</ul>
