<p>Running AWS &quot;Managed Nodes&quot; for an EKS Cluster across 2 AZ's.</p>
<p>3 Nodes in total. I get random timeouts when attempting to pull the containers down.</p>
<p>This has been so hard to trace because it does work (sometimes), so it's not like an ACL is blocking or a security group.</p>
<p>When I ssh into the nodes, sometimes I can pull down the image manually and sometimes I cannot. When I've run curl requests <code>curl -I https://hub.docker.com</code> it takes sometimes 2 minutes to get a response back. I'm guessing this is why the images are timing out.</p>
<p>I don't know of a way to increase the timeout for k8s to pull the image, but also can't figure out why the latency is so bad in doing the curl request.</p>
<p>Any suggestions are greatly appreciated.</p>
<p>FYI, worker nodes in Private Subnet, proper routes to NAT Gateway in place. VPC Flow logs are good.</p>

<p>Random is the hardest thing to trace 🤷.</p>
<p>🥼 You could move your images to a <a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html" rel="nofollow noreferrer">private ECR registry</a> or simply run a <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-top-of-digitalocean-spaces-and-use-it-with-digitalocean-kubernetes" rel="nofollow noreferrer">registry in your cluster</a> to discard that it's an issue with your Kubernetes networking. Running AWS CNI❓</p>
<p>It could also just be <a href="https://docs.docker.com/docker-hub/download-rate-limit/#:%7E:text=Docker%20will%20gradually%20impose%20download,Hub%20as%20an%20authenticated%20user." rel="nofollow noreferrer">rate-limiting from docker hub itself</a>. Are you using the same external NAT IP address to pull from multiple nodes/clusters❓:</p>
<blockquote>
<p>Docker will gradually impose download rate limits with an eventual limit of 300 downloads per six hours for anonymous users.
Logged in users will not be affected at this time. Therefore, we recommend that you log into Docker Hub as an authenticated user. For more information, see the following section How do I authenticate pull requests.</p>
</blockquote>
<p>✌️</p>
