<p>This is a very wried thing.</p>

<p>I created a <strong>private</strong> GKE cluster with a node pool of 3 nodes. Then I have a replica set with 3 pods. some of these pods will be scheduled to one node. </p>

<p>So one of these pods always get <code>ImagePullBackOff</code>, I check the error </p>

<pre><code>Failed to pull image "bitnami/mongodb:3.6": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
</code></pre>

<p>And the pods scheduled to the remaining two nodes work well.</p>

<p>I ssh to that node, run <code>docker pull</code> and everything is fine. I cannot find another way to troubleshoot this error. </p>

<p>I tried to <code>drain</code> or <code>delete</code> that node and let the cluster to recreate the node. but it is still not working.</p>

<p>Help me, please.</p>

<p>Update:
From GCP <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#docker_hub" rel="nofollow noreferrer">documentation</a>, it will fail to pull images from the docker hub. </p>

<p>BUT the weirdest thing is ONLY ONE node is unable to pull the images.  </p>

<p>I recall seeing this before and finding an answer before.</p>
<p><a href="https://cloud.google.com/container-registry/docs/pulling-cached-images" rel="nofollow noreferrer">https://cloud.google.com/container-registry/docs/pulling-cached-images</a><br />
Talks about it a little, but I'll explain it so it's easy to follow.</p>
<p>If I spin up a private GKE cluster and I create 3 deployments:</p>
<ul>
<li>1st uses image: nginx:latest</li>
<li>2nd uses image: nginx:stable</li>
<li>3rd uses image: docker.io/busybox:1.36.0-glibc</li>
</ul>
<p>nginx:latest (common tag) will almost always work<br />
nginx:stable (popular tag) will work sometimes<br />
The super specific tag (rarely used tag) will almost always fail with ImagePullBackOff</p>
<p><strong>So why is this the case?</strong><br />
<strong>1. The ImagePullBackOff happens when the pods/nodes have no NAT Gateway/no Internet Access</strong><br />
<code>kubectl exec -it working-nginx-latest-pod -- curl yahoo.com</code><br />
^-- You can prove no internet with this, note <code>curl google.com</code> is a bad test on GKE, because it's still reachable via googles internal network / you'll get a response, because google's network can reach google.com without having to go through the internet, that's why I recommend testing with a non google URL like yahoo.com<br />
(Google's networking also occasionally does some counterintuitive / non-standard things, like route public IP Addresses over their internal network, so sometimes you can reach public IP addresses w/o internet access, it's usually google services with public IPs that are sometimes reachable w/o internet access.)</p>
<p><strong>2. So the next question is, but wait... how are <code>nginx:latest</code> and <code>nginx:stable</code> able to pull image that exists on the internet/on docker hub, when there's no internet access? Basically why is it working for some images and not others?</strong><br />
Answer boils down to popularity of the image:tag pair. Is it popular enough to get cached in mirror.gcr.io?</p>
<p>The initial link I shared at the top mentions &quot;Container Registry caches frequently-accessed public Docker Hub images on mirror.gcr.io&quot;, so basically if you reference a common tag of a popular image, you can sometimes get lucky enough to pull it even without internet, because the cache is reachable via private IP space / without internet access.</p>
<p>When a pod running on GKE private cluster gives you ImagePullBackOff, and you're like, what's going on? I know this image exists! <code>docker pull docker.io/busybox:1.36.0-glibc</code> pulls fine from my local machine, what's happening is that rarely used tag doesn't exist in their cache, that mirrors common tags of popular images.</p>
<p>Best way to fix it is to either pull all images from pkg.dev (GCP's Artifact Registry, which GKE should be able to access w/o internet access) or set up NAT gateway/ensure the private cluster has internet access. And you can use <code>kubectl exec -it working-nginx-latest-pod -- curl yahoo.com</code> as a feedback loop to check if the cluster has internet access as you tinker with VPC settings to add NAT GW.</p>
<p><a href="https://cloud.google.com/kubernetes-engine/docs/best-practices/networking#use-cloudnat" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/best-practices/networking#use-cloudnat</a><br />
mentions By default, (GKE) &quot;private clusters don't have internet access. In order to allow Pods to reach the internet, enable Cloud NAT for each region. At a minimum, enable Cloud NAT for the primary and secondary ranges in the GKE subnet.&quot;</p>
