<p>This is a very wried thing.</p>

<p>I created a <strong>private</strong> GKE cluster with a node pool of 3 nodes. Then I have a replica set with 3 pods. some of these pods will be scheduled to one node. </p>

<p>So one of these pods always get <code>ImagePullBackOff</code>, I check the error </p>

<pre><code>Failed to pull image "bitnami/mongodb:3.6": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
</code></pre>

<p>And the pods scheduled to the remaining two nodes work well.</p>

<p>I ssh to that node, run <code>docker pull</code> and everything is fine. I cannot find another way to troubleshoot this error. </p>

<p>I tried to <code>drain</code> or <code>delete</code> that node and let the cluster to recreate the node. but it is still not working.</p>

<p>Help me, please.</p>

<p>Update:
From GCP <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#docker_hub" rel="nofollow noreferrer">documentation</a>, it will fail to pull images from the docker hub. </p>

<p>BUT the weirdest thing is ONLY ONE node is unable to pull the images.  </p>

<p>There was a related reported bug in <a href="https://issuetracker.google.com/issues/119820482" rel="nofollow noreferrer">Kubernetes 1.11</a>  </p>

<p>Make sure it is not your case</p>
