<p>I have two projects in GCP. Let's call them project <code>A</code> and project <code>B</code>. Project <code>A</code> will contain the cluster for the deployment, while project <code>B</code> will contain the Artifact Registry (please note that this is not the same as the Container Registry).</p>
<p>The deployment is pulling a docker image from the Artifact Registry. Let's say the docker image is a stock standard Ubuntu image that our deployment is trying to deploy, and let's assume that the image is already in the Artifact Registry with no issues.</p>
<p>The deployment of the ubuntu image to the cluster is done via a GitLab CI/CD pipeline. It uses a service account that has the following roles:</p>
<ul>
<li>Artifact Registry Reader</li>
<li>Kubernetes Engine Developer</li>
<li>Viewer</li>
</ul>
<p>Additionally we also note that the cluster features two node pools. One is a custom node pool and the other is the default. They both have the following access scopes:</p>
<ul>
<li><a href="https://www.googleapis.com/auth/cloud-platform" rel="nofollow noreferrer">https://www.googleapis.com/auth/cloud-platform</a></li>
<li><a href="https://www.googleapis.com/auth/devstorage.read_only" rel="nofollow noreferrer">https://www.googleapis.com/auth/devstorage.read_only</a></li>
<li><a href="https://www.googleapis.com/auth/logging.write" rel="nofollow noreferrer">https://www.googleapis.com/auth/logging.write</a></li>
<li><a href="https://www.googleapis.com/auth/monitoring.write" rel="nofollow noreferrer">https://www.googleapis.com/auth/monitoring.write</a></li>
<li><a href="https://www.googleapis.com/auth/service.management.readonly" rel="nofollow noreferrer">https://www.googleapis.com/auth/service.management.readonly</a></li>
<li><a href="https://www.googleapis.com/auth/servicecontrol" rel="nofollow noreferrer">https://www.googleapis.com/auth/servicecontrol</a></li>
</ul>
<p>The <code>cloud-platform</code> scope is enabled to ensure that the node pools can pull from the Artifact Registry which is in a different project.</p>
<p>It is also important to note that both node pools use the default service account which has the roles:</p>
<ul>
<li>Artifact Registry Reader</li>
<li>Editor</li>
</ul>
<p>Upon deployment, the GitLab pipeline completes with no issues. However, the deployment workload fails in the cluster. The following events occur:</p>
<ol>
<li><code>Pulling image &quot;europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest&quot;</code></li>
<li><code>Failed to pull image &quot;europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest&quot;: rpc error: code = Unknown desc = failed to pull and unpack image &quot;europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest&quot;: failed to resolve reference &quot;europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest&quot;: failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden</code></li>
<li><code>Error: ErrImagePull</code></li>
<li><code>Error: ImagePullBackOff</code></li>
<li><code>Back-off pulling image &quot;europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest&quot;</code></li>
</ol>
<p>The deployment YAML file is as follows:</p>
<pre><code>---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu-build-deploy-demo
  labels:
    app: ubuntu
  namespace: customnamespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ubuntu
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      containers:
        - name: ubuntu
          image: europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest
          command: [&quot;sleep&quot;, &quot;123456&quot;]
</code></pre>
<p>Why is the image not pulled from correctly? Why am I getting an auth issue despite the correct service account roles and access scopes? How can I resolve this issue?</p>
<p>I have double checked that the image name, tag, and path are correct many times. I have also double checked that the service accounts have the correct roles (specified earlier). I have also ensured that the node pool access scopes are correct and do indeed have the correct access scopes (specified earlier).</p>
<p>I am at a loss of how to resolve this issue. Any help would be greatly appreciated.</p>

Harry Myburgh
<p>Putting in answer form. As @avinashjha said: the default service account from project <code>A</code> needs to have the role <code>Artifact Registry Reader</code> in project <code>B</code>. This allows for the default service account to oauth and pull the docker image from the registry.</p>
