<p>According to articles below, it seems we can pull container image to GKE from Artifact Registry without any additional authentication when these in same project.</p>
<p><a href="https://cloud.google.com/artifact-registry/docs/integrate-gke" rel="nofollow noreferrer">https://cloud.google.com/artifact-registry/docs/integrate-gke</a></p>
<p><a href="https://www.youtube.com/watch?v=BfS7mvPA-og" rel="nofollow noreferrer">https://www.youtube.com/watch?v=BfS7mvPA-og</a></p>
<p><a href="https://stackoverflow.com/questions/73205712/error-imagepullbackoff-and-error-errimagepull-errors-with-gke">Error: ImagePullBackOff and Error: ErrImagePull errors with GKE</a></p>
<p>But when I try it, I faced <code>ImagePullBackOff</code> error.<br />
Is there any mistake? misunderstanding? Or should I need use another authentication?</p>
<h2>Reproduce</h2>
<p>It's convenient to use Google Cloud Shell in some project on <a href="https://console.cloud.google.com" rel="nofollow noreferrer">https://console.cloud.google.com</a> .</p>
<h3>Create Artifact Registry</h3>
<pre class="lang-bash prettyprint-override"><code>gcloud artifacts repositories create test \
    --repository-format=docker \
    --location=asia-northeast2
</code></pre>
<h3>Push sample image</h3>
<pre class="lang-bash prettyprint-override"><code>gcloud auth configure-docker asia-northeast2-docker.pkg.dev
docker pull nginx
docker tag nginx asia-northeast2-docker.pkg.dev/${PROJECT_NAME}/test/sample-nginx-image
docker push asia-northeast2-docker.pkg.dev/${PROJECT_NAME}/test/sample-nginx-image
</code></pre>
<h3>Create GKE Autopilot cluster</h3>
<p>Create GKE Autopilot cluster by using GUI console.</p>
<p>Almost all options is default but I changed these 2.</p>
<ul>
<li>Set cluster name as test.</li>
<li>Set region same as registry's one. (In this case, asia-northeast2)</li>
<li>Enabled Anthos Service Mesh.</li>
</ul>
<h3>Deploy container image to GKE from Artifact Registry</h3>
<pre class="lang-bash prettyprint-override"><code>gcloud container clusters get-credentials test --zone asia-northeast2
kubectl run test --image asia-northeast2-docker.pkg.dev/${PROJECT_NAME}/test/sample-nginx-image
</code></pre>
<h3>Check Pod state</h3>
<pre class="lang-bash prettyprint-override"><code>kubectl describe po test
</code></pre>
<pre><code>Name:             test
Namespace:        default
Priority:         0
Service Account:  default
Node:             xxxxxxxxxxxxxxxxxxx
Start Time:       Wed, 08 Feb 2023 12:38:08 +0000
Labels:           run=test
Annotations:      autopilot.gke.io/resource-adjustment:
                    {&quot;input&quot;:{&quot;containers&quot;:[{&quot;name&quot;:&quot;test&quot;}]},&quot;output&quot;:{&quot;containers&quot;:[{&quot;limits&quot;:{&quot;cpu&quot;:&quot;500m&quot;,&quot;ephemeral-storage&quot;:&quot;1Gi&quot;,&quot;memory&quot;:&quot;2Gi&quot;},&quot;reque...
                  seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:           Pending
IP:               10.73.0.25
IPs:
  IP:  10.73.0.25
Containers:
  test:
    Container ID:
    Image:          asia-northeast2-docker.pkg.dev/${PROJECT_NAME}/test/sample-nginx-image
    Image ID:
    Port:           &lt;none&gt;
    Host Port:      &lt;none&gt;
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:                500m
      ephemeral-storage:  1Gi
      memory:             2Gi
    Requests:
      cpu:                500m
      ephemeral-storage:  1Gi
      memory:             2Gi
    Environment:          &lt;none&gt;
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-szq85 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-szq85:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       &lt;nil&gt;
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              &lt;none&gt;
Tolerations:                 kubernetes.io/arch=amd64:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age   From                                   Message
  ----     ------     ----  ----                                   -------
  Normal   Scheduled  19s   gke.io/optimize-utilization-scheduler  Successfully assigned default/test to xxxxxxxxxxxxxxxxxxx
  Normal   Pulling    16s   kubelet                                Pulling image &quot;asia-northeast2-docker.pkg.dev/${PROJECT_NAME}/test/sample-nginx-image&quot;
  Warning  Failed     16s   kubelet                                Failed to pull image &quot;asia-northeast2-docker.pkg.dev/${PROJECT_NAME}/test/sample-nginx-image&quot;: rpc error: code = Unknown desc = failed to pull and unpack image &quot;asia-northeast2-docker.pkg.dev/${PROJECT_NAME}/test/sample-nginx-image:latest&quot;: failed to resolve reference &quot;asia-northeast2-docker.pkg.dev/${PROJECT_NAME}/test/sample-nginx-image:latest&quot;: failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden
  Warning  Failed     16s   kubelet                                Error: ErrImagePull
  Normal   BackOff    15s   kubelet                                Back-off pulling image &quot;asia-northeast2-docker.pkg.dev/${PROJECT_NAME}/test/sample-nginx-image&quot;
  Warning  Failed     15s   kubelet                                Error: ImagePullBackOff
</code></pre>
<p>then, I got <code>ImagePullBackOff</code>.</p>

<p>This could be because the GKE Autopilot service account does not have the necessary permissions to access the Artifact Registry. You can grant the needed permissions by adding the <code>roles/artifactregistry.reader</code> role to the service account that the GKE Autopilot node pool is configured to use. Additionally, you may need to adjust the <a href="https://cloud.google.com/container-registry/docs/access-control#permissions_and_roles" rel="nofollow noreferrer">IAM permissions</a> for the service account so that it has access to the private Artifact Registry.</p>
<pre><code>gcloud artifacts repositories add-iam-policy-binding &lt;repository-name&gt; \
  --location=&lt;location&gt; \
  --member=serviceAccount:&lt;nnn&gt;-compute@developer.gserviceaccount.com \
  --role=&quot;roles/artifactregistry.reader&quot;
</code></pre>
<p>Can you try creating a new service account and granting it the necessary permissions to pull the image and try to pull the image once.</p>
<p>Simple <a href="https://cloud.google.com/container-registry/docs/troubleshooting#imagepullbackoff" rel="nofollow noreferrer">troubleshooting steps</a> are:</p>
<ol>
<li>you should ensure that your GKE cluster is configured to allow access to the Artifact Registry. You can do this by going to the GKE dashboard and making sure that the “Allow access to Artifact Registry” option is enabled.</li>
<li>The container image you are trying to pull does not exist in the Artifact Registry. You should check the registry to make sure that the container image is correctly uploaded and can be accessed.</li>
<li>you can look into the error logs to get more information on what is causing this issue. Additionally, you can check the GKE documentation for more information on troubleshooting this issue.</li>
</ol>
