<p>So after googling a little bit (which is polluted by people having trouble with Pull Secrets) I am posting this here — and to GCP Support (will update as I hear).</p>

<p>I created a Cluster from GitLab Kubernetes integration (docs: <a href="https://about.gitlab.com/solutions/kubernetes" rel="noreferrer">https://about.gitlab.com/solutions/kubernetes</a>) within the same project as my GCR registry / images.  </p>

<p>When I add a new service / deployment to this Cluster using Kubectl (which relies on a private image within the GCR Registry in this project) the pods in the GitLab created cluster fail to pull from GCR with: ErrImagePull.</p>

<p>To be clear — I am NOT pulling from a GitLab private registry, I am attempting to pull from a GCR Registry within the same project as the GKE cluster created from GitLab (which should not require a Pull Secret).</p>

<p>Other Clusters (created from GCP console) within this project can properly access the same image so my thinking is that there is some difference between Clusters created via an API (in this case from GitLab) vs Clusters created from the GCP console.</p>

<p>I am hoping someone has run into this in the past — or can explain the differences in the Service Accounts etc that could be causing the problem.  </p>

<blockquote>
  <p>I am going to attempt to create a service account and manually grant it Project Viewer role to see if that solves the problem.</p>
</blockquote>

<p>Update: manually configured Service Account did not solve issue.</p>

<p><em>Note: I am trying to pull an image into the Cluster NOT into a GitLab Runner that is running on the Cluster.  Ie. I want a separate Service / Deployment to be running along side my GitLab infrastructure.</em></p>

<p><strong>TL;DR</strong> — Clusters created by GitLab-Ci Kubernetes Integration will not be able to pull an image from a GCR Registry in the same project as the container images — without modifying the Node(s) permissions (scopes).</p>

<blockquote>
  <p>While you CAN manually modify the permissions on an Individual Node machine(s) to grant the Application Default Credentials (see: <a href="https://developers.google.com/identity/protocols/application-default-credentials" rel="noreferrer">https://developers.google.com/identity/protocols/application-default-credentials</a>) the proper scopes in real time — doing it this way would mean that if your node is re-created at some point in the future it WOULD NOT have your modified scopes and things would break.</p>
</blockquote>

<p>Instead of modifying the permissions manually — create a new Node pool that has the proper Scope(s) to access your required GCP services.</p>

<p>Here are some resources I used for reference:</p>

<ol>
<li><a href="https://medium.com/google-cloud/updating-google-container-engine-vm-scopes-with-zero-downtime-50bff87e5f80" rel="noreferrer">https://medium.com/google-cloud/updating-google-container-engine-vm-scopes-with-zero-downtime-50bff87e5f80</a></li>
<li><a href="https://adilsoncarvalho.com/changing-a-running-kubernetes-cluster-permissions-a-k-a-scopes-3e90a3b95636" rel="noreferrer">https://adilsoncarvalho.com/changing-a-running-kubernetes-cluster-permissions-a-k-a-scopes-3e90a3b95636</a></li>
</ol>

<h2>Creating a properly Scoped Node Pool Generally looks like this</h2>

<pre><code>gcloud container node-pools create [new pool name] \
 --cluster [cluster name] \
 --machine-type [your desired machine type] \
 --num-nodes [same-number-nodes] \
 --scopes [your new set of scopes]
</code></pre>

<p>If you aren't sure what the names of your required Scopes are — You can see a full list of Scopes AND Scope Aliases over here: <a href="https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create" rel="noreferrer">https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create</a></p>

<p>For me I did  gke-default (same as my other cluster)  and sql-admin.  The reason for this being that I need to be able to access an SQL Database in Cloud SQL during part of my build and I don't want to have to connect to a pubic IP to do that.</p>

<h2>gke-default Scopes (for reference)</h2>

<ol>
<li><a href="https://www.googleapis.com/auth/devstorage.read_only" rel="noreferrer">https://www.googleapis.com/auth/devstorage.read_only</a> (allows you to pull)</li>
<li><a href="https://www.googleapis.com/auth/logging.write" rel="noreferrer">https://www.googleapis.com/auth/logging.write</a></li>
<li><a href="https://www.googleapis.com/auth/monitoring" rel="noreferrer">https://www.googleapis.com/auth/monitoring</a></li>
<li><a href="https://www.googleapis.com/auth/service.management.readonly" rel="noreferrer">https://www.googleapis.com/auth/service.management.readonly</a></li>
<li><a href="https://www.googleapis.com/auth/servicecontrol" rel="noreferrer">https://www.googleapis.com/auth/servicecontrol</a></li>
<li><a href="https://www.googleapis.com/auth/trace.append" rel="noreferrer">https://www.googleapis.com/auth/trace.append</a></li>
</ol>

<p>Contrast the above with more locked down permissions from a GitLab-Ci created cluster ( ONLY these two: <a href="https://www.googleapis.com/auth/logging.write" rel="noreferrer">https://www.googleapis.com/auth/logging.write</a>, <a href="https://www.googleapis.com/auth/monitoring" rel="noreferrer">https://www.googleapis.com/auth/monitoring</a>):</p>

<p>Obviosuly configuring your cluster to ONLY the minimum permissions needed is for sure the way to go here.  Once you figure out what that is and create your new properly scoped Node Pool...</p>

<p>List your nodes with:</p>

<pre><code>kubectl get nodes
</code></pre>

<p>The one you just created (most recent) is has the new settings while the older option is the default gitlab cluster that can pull from the GCR.</p>

<p>Then: </p>

<pre><code>kubectl cordon [your-node-name-here]
</code></pre>

<p>After that you want to drain:</p>

<pre><code>kubectl drain [your-node-name-here] --force
</code></pre>

<p>I ran into issues where the fact that I had a GitLab Runner installed meant that I couldn't drain the pods normally due to the local data / daemon set that was used to control it.   </p>

<p>For that reason once I cordon'd my Node I just deleted the node from Kubectl (not sure if this will cause problems — but it was fine for me).  Once your node is deleted you need to delete the 'default-pool' node pool created by GitLab.</p>

<p>List your node-pools:</p>

<pre><code>gcloud container node-pools list --cluster [CLUSTER_NAME]
</code></pre>

<p>See the old scopes created by gitlab:</p>

<pre><code>gcloud container node-pools describe default-pool \
    --cluster [CLUSTER_NAME]
</code></pre>

<p>Check to see if you have the correct new scopes (that you just added):</p>

<pre><code>gcloud container node-pools describe [NEW_POOL_NAME] \
    --cluster [CLUSTER_NAME]
</code></pre>

<p>If your new Node Pool has the right scopes your deployments can now delete the default pool with:</p>

<pre><code>gcloud container node-pools delete default-pool \
   --cluster &lt;YOUR_CLUSTER_NAME&gt; --zone &lt;YOUR_ZONE&gt;
</code></pre>

<p>In my personal case I am still trying to figure out how to allow access to the private network (ie. get to Cloud SQL via private IP) but I can pull my images now so I am half way there.  </p>

<p>I think that's it — hope it saved you a few minutes!</p>
