<p>I'm using the latest version of google-kubernetes (1.22.8-gke.202) in a Kubernetes managed cluster. I also have a custom service account that has access to the &quot;Artifact Registry Reader&quot; scope that should grant it permission to pull private images from the repository - calling this <em>custom-service-account</em>.</p>
<p>I've validated that the nodes themselves have the <em>custom-service-account</em> service account linked to them within Compute Engine. Kubernetes is setup with a service account that is linked to the IAM service account with the same name through <a href="https://www.cobalt.io/blog/google-cloud-workload-identity-with-kubernetes-and-terraform" rel="nofollow noreferrer">workload identity</a>. However, when I try to spawn a pod that pulls from my private repo it fails indefinitely.</p>
<pre><code>Events:
  Type     Reason             Age                   From                Message
  ----     ------             ----                  ----                -------
  Warning  FailedScheduling   21m (x3 over 24m)     default-scheduler   0/2 nodes are available: 2 node(s) were unschedulable.
  Warning  FailedScheduling   19m                   default-scheduler   no nodes available to schedule pods
  Normal   NotTriggerScaleUp  18m (x25 over 24m)    cluster-autoscaler  pod didn't trigger scale-up: 1 node(s) had taint {reserved-pool: true}, that the pod didn't tolerate
  Normal   Scheduled          18m                   default-scheduler   Successfully assigned default/test-service-a-deployment-5757fc5797-b54gx to gke-personal-XXXX--personal-XXXX--ac9a05b6-16sb
  Normal   Pulling            17m (x4 over 18m)     kubelet             Pulling image &quot;us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest&quot;
  Warning  Failed             17m (x4 over 18m)     kubelet             Failed to pull image &quot;us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest&quot;: rpc error: code = Unknown desc = failed to pull and unpack image &quot;us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest&quot;: failed to resolve reference &quot;us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest&quot;: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed             17m (x4 over 18m)     kubelet             Error: ErrImagePull
  Warning  Failed             16m (x6 over 18m)     kubelet             Error: ImagePullBackOff
  Normal   BackOff            3m27s (x65 over 18m)  kubelet             Back-off pulling image &quot;us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest&quot;
</code></pre>
<p>I've also ssh'ed into the nodes themselves and at least by default with a regular <code>docker pull</code> or <code>crictl pull</code> see this same error.</p>
<p>So, the specific questions I have:</p>
<ul>
<li>How is GCP injecting the service account credentials into Kubernetes/Docker worker that tries to launch the images? Is it expected that the regular docker command doesn't seem to have these credentials?</li>
<li>Do I need to manually bootstrap some additional authentication for Kubernetes aside from just inheriting the service account on the pods?</li>
</ul>
<p>EDIT: Result of <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#troubleshooting" rel="nofollow noreferrer">here</a></p>
<pre><code>&gt; gcloud container clusters describe personal-XXXX-gke --zone us-central1-a --format=&quot;value(workloadIdentityConfig.workloadPool)&quot;
personal-XXXX.svc.id.goog

&gt; gcloud container node-pools describe personal-XXXX-gke-node-pool --cluster personal-XXXX-gke --format=&quot;value(config.workloadMetadataConfig.mode)&quot; --zone us-central1-a
GKE_METADATA

&gt; kubectl describe serviceaccount --namespace default be-service-account
Name:                be-service-account
Namespace:           default
Labels:              &lt;none&gt;
Annotations:         iam.gke.io/gcp-service-account: custom-service-account@personal-XXXX.iam.gserviceaccount.com
Image pull secrets:  &lt;none&gt;
Mountable secrets:   be-service-account-token-jmss9
Tokens:              be-service-account-token-jmss9
Events:              &lt;none&gt;

&gt; gcloud iam service-accounts get-iam-policy custom-service-account@personal-XXXX.iam.gserviceaccount.com
bindings:
- members:
  - serviceAccount:personal-XXXX.svc.id.goog[default/be-service-account]
  role: roles/iam.workloadIdentityUser
etag: BwXjqJ9DC6A=
version: 1
</code></pre>

<p>When checking for access to artifact registry, please check permission and scopes as per this <a href="https://cloud.google.com/artifact-registry/docs/access-control#gke" rel="nofollow noreferrer">documentation</a>.</p>
