<p>I cannot pull artifact registry images to a newly created GKE cluster with Terraform and a user-defined service account.</p>
<p>The terraform used to stand up the cluster is below.</p>
<pre><code>locals {
  service         = &quot;example&quot;
  resource_prefix = format(&quot;%s-%s&quot;, local.service, var.env)
  location        = format(&quot;%s-b&quot;, var.gcp_region)
}

resource &quot;google_service_account&quot; &quot;main&quot; {
  account_id   = format(&quot;%s-sa&quot;, local.resource_prefix)
  display_name = format(&quot;%s-sa&quot;, local.resource_prefix)
  project      = var.gcp_project
}

resource &quot;google_container_cluster&quot; &quot;main&quot; {
  name                     = local.resource_prefix
  description              = format(&quot;Cluster primarily servicing the service %s&quot;, local.service)
  location                 = local.location
  remove_default_node_pool = true
  initial_node_count       = 1
}

resource &quot;google_container_node_pool&quot; &quot;main&quot; {
  name       = format(&quot;%s-node-pool&quot;, local.resource_prefix)
  location   = local.location
  cluster    = google_container_cluster.main.name
  node_count = var.gke_cluster_node_count

  node_config {
    preemptible  = true
    machine_type = var.gke_node_machine_type
    # Google recommends custom service accounts that have cloud-platform scope and permissions granted via IAM Roles.
    service_account = google_service_account.main.email
    oauth_scopes = [
      &quot;https://www.googleapis.com/auth/logging.write&quot;,
      &quot;https://www.googleapis.com/auth/monitoring&quot;,
      &quot;https://www.googleapis.com/auth/cloud-platform&quot;,
      &quot;https://www.googleapis.com/auth/devstorage.read_only&quot;,
      &quot;https://www.googleapis.com/auth/servicecontrol&quot;,
      &quot;https://www.googleapis.com/auth/service.management.readonly&quot;,
      &quot;https://www.googleapis.com/auth/trace.append&quot;
    ]
  }

  autoscaling {
    min_node_count = var.gke_cluster_autoscaling_min_node_count
    max_node_count = var.gke_cluster_autoscaling_max_node_count
  }
}
</code></pre>
<p>I run a helm deployment to deploy an application and get the following issue.</p>
<pre><code>default       php-5996c7fbfd-d6xf5                                             0/1     ImagePullBackOff             0          37m
</code></pre>
<pre><code>Normal   Pulling    36m (x4 over 37m)      kubelet            Pulling image &quot;europe-docker.pkg.dev/example-999999/eu.gcr.io/example-php-fpm:latest&quot;
  Warning  Failed     36m (x4 over 37m)      kubelet            Failed to pull image &quot;europe-docker.pkg.dev/example-999999/eu.gcr.io/example-php-fpm:latest&quot;: rpc error: code = Unknown desc = failed to pull and unpack image &quot;europe-docker.pkg.dev/example-999999/eu.gcr.io/example-php-fpm:latest&quot;: failed to resolve reference &quot;europe-docker.pkg.dev/example-999999/eu.gcr.io/example-php-fpm:latest&quot;: failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden
  Warning  Failed     36m (x4 over 37m)      kubelet            Error: ErrImagePull
  Warning  Failed     35m (x6 over 37m)      kubelet            Error: ImagePullBackOff
</code></pre>
<p>Seems to me that I've missed something to do with the service account. Although using cloud ssh I am able to generate an oauth token, but that also does not work using <code>crictl</code></p>
<p><strong>UPDATE: issue resolved</strong></p>
<p>I have been able to resolve my problem with the following additional terraform code.</p>
<pre><code>resource &quot;google_project_iam_member&quot; &quot;artifact_role&quot; {
  role = &quot;roles/artifactregistry.reader&quot;
  member  = &quot;serviceAccount:${google_service_account.main.email}&quot;
  project = var.gcp_project
}
</code></pre>

<p>As error says : <strong>unexpected status: 403 Forbidden</strong></p>
<p>You might be having an issue with the Deployment <strong>imagepull</strong> <strong>secret</strong>.</p>
<p>For <strong>GKE</strong> you can use the service account <strong>JSON</strong></p>
<p>Ref doc : <a href="https://cloud.google.com/container-registry/docs/advanced-authentication#json-key" rel="nofollow noreferrer">https://cloud.google.com/container-registry/docs/advanced-authentication#json-key</a></p>
<p>Terraform create <strong>secret</strong> in <strong>GKE</strong> which you can use it to <em>deployment</em></p>
<pre><code>resource &quot;kubernetes_secret&quot; &quot;gcr&quot; {
    type = &quot;kubernetes.io/dockerconfigjson&quot;
    metadata {
        name = &quot;gcr-image-pull&quot;
        namespace = &quot;default&quot;
    }
    data = {
        &quot;.dockerconfigjson&quot; = jsonencode({
            auths = {
                &quot;gcr.io&quot; = {
                    username = &quot;_json_key&quot;
                    password = base64decode(google_service_account_key.myaccount.private_key)
                    email = google_service_account.main.email
                    auth = base64encode(&quot;_json_key:${ base64decode(google_service_account_key.myaccount.private_key) }&quot;)
                }
            }
        })
    }}
</code></pre>
<p>Or use the <strong>kubectl</strong> to create the <strong>secret</strong></p>
<pre><code>kubectl create secret docker-registry gcr \
    --docker-server=gcr.io \
    --docker-username=_json_key \
    --docker-password=&quot;$(cat google-service-account-key.json)&quot; \
    --docker-email=&lt;Email address&gt;
</code></pre>
<p>Now if you have the POD or deployment you can create <strong>YAML</strong> config like</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: uses-private-registry
spec:
  containers:
  - name: hello-app
    image: &lt;image URI&gt;
  imagePullSecrets:
  - name: secret-that-you-created
</code></pre>
<p><strong>Update:</strong></p>
<p>As per <strong>Guillaume's</strong> suggestion for <strong>GKE/GCP</strong> you can follow *<em><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity" rel="nofollow noreferrer">workload identity*</a></em> option as best practice with other extern repo it might could not work.</p>
<p>Create the <strong>IAM service account</strong> in GCP:</p>
<pre><code>gcloud iam service-accounts create gke-workload-indentity \
    --project=&lt;project-id&gt;
</code></pre>
<p>Create a service account in the <strong>K8s</strong> cluster :</p>
<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: gke-workload-indentity@PROJECT-ID.iam.gserviceaccount.com
  name: gke-sa-workload
  namespace: default
</code></pre>
<p>Policy binding run below <strong>Gcloud</strong> command :</p>
<pre><code>gcloud iam service-accounts add-iam-policy-binding gke-workload-indentity@PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member &quot;serviceAccount:PROJECT_ID.svc.id.goog[default/K8s_SANAME]&quot;
</code></pre>
<p>Now you can create the deployment POD with image in GCR/astifact repo just update the ServiceAccount</p>
<pre><code>spec:
      containers:
      - name: container
        image: IMAGE
        serviceAccountName: gke-sa-workload
</code></pre>
<p>Read more at : <a href="https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/</a></p>
