<p>I am trying to make my kubernetes cluster pull from a container registry service running inside itself. I have kube dns setup, I have a registry deployment and service running. I can resolve the service internal name via host command on the node. I have added <code>--dns</code> flag to docker daemon with the address of kube dns service. I have kubelet running with <code>--cluster-dns</code> flag set to the same address as well. Yet somehow this is what I get when I try to create a pod using this registry.</p>
<p><code>Failed to pull image &quot;kube-registry.kube-system.svc.cluster.local/myuser/myimage&quot;: rpc error: code = Unknown desc = Error response from daemon: Get https://kube-registry.kube-system.svc.cluster.local/v1/_ping: dial tcp: lookup kube-registry.kube-system.svc.cluster.local: no such host </code></p>
<p>Somehow even with kube dns address explicitly given to both dockerd and kubelet, pulling images from the registry service fails because of name resolution. What am I missing?</p>

Mad Wombat
<p>Question: &quot;I am trying to make my kube-cluster pull from a registry running inside itself.&quot; (Note I plan to going to edit the title of your question to clarify slightly / make it easier to search)</p>
<p><strong>Short Answer: You can't</strong>*<br />
*<strong>Nuanced Answer: Technically it is possible with hacks and a solid understanding of Kubernetes fundamentals</strong>. You'll probably want to avoid doing this, unless you have a really good reason, and fully understand my explanation of the fundamental issue and workaround, as this is an advanced use case that will require debugging to force it to work. This is complicated and nuanced enough to make step by step directions difficult, but I can give you a solid idea of the fundamental issue you ran into that makes this challenging, and high level overview guidance on how to pull off what you're trying to do anyways.</p>
<p><strong>Why you can't / the fundamental issue you ran into:</strong><br />
In Kubernetes land 3 networks tend to exist: Internet, LAN, and Inner Cluster Network.<br />
(Resource that goes into more depth: <a href="https://oteemo.com/kubernetes-networking-and-services-101/" rel="nofollow noreferrer">https://oteemo.com/kubernetes-networking-and-services-101/</a>)<br />
AND these 3 networks each have their own DNS / there's 3 layers of DNS.</p>
<ul>
<li>Internet DNS: 8.8.8.8, 1.1.1.1, 9.9.9.9 (google, cloudflare, quad9 or whatever public internet DNS the router is configured to point to.)</li>
<li>LAN DNS: 192.168.1.1 (LAN DNS hosted on your router)</li>
<li>CoreDNS: 10.43.0.10 (10th IP of the CIDR range of the inner cluster network)</li>
</ul>
<p>Here's the gotcha you're running into:</p>
<ul>
<li>A pod can resolve DNS entries hosted at any of these 3 levels of DNS.</li>
<li>The OS hosting the Kubernetes Cluster can only resolve DNS entries hosted on LAN DNS or Internet DNS. (the OS isn't scoped to have visibility into the existence of CoreDNS/Inner Cluster Network.)</li>
<li>kubelet + docker/containerd/cri-o/other runtime, are responsible for pulling images from registries and these exist at the OS level in the form of systemd services and thus don't have scope to Inner Cluster DNS names. This is why what you're trying to do is failing.</li>
</ul>
<p><strong>Workaround options / hacks and nuances you can do to force what you're trying to do to work:</strong></p>
<p><strong>Option 1.) (I don't suggest this, has extra difficult chicken and egg issues, sharing for information purposes only)</strong></p>
<p>Host an additional instance of coredns as a LAN facing instance of
DNS on Kubernetes, Expose the registry and 2nd instance of coredns to
the LAN via explicit NodePorts (using static service manifests so
they'll come up with predictable/static NodePorts, vs random
NodePorts in the range of 30000 - 32768) so they're routable from the
LAN (I suggest NodePorts over LB's here as one less dependency/thing
that can go wrong). Have the 2nd instance of coredns use your LAN
router/LAN DNS as it's upstream DNS server. Reconfigure the OS to use
the LAN facing coredns as it's DNS server.</p>
<p><strong>Option 2.) More reasonable and what trow does:</strong></p>
<ul>
<li><p>What is trow:
<a href="https://thenewstack.io/trow-a-container-registry-to-run-inside-a-kubernetes-cluster/" rel="nofollow noreferrer">https://thenewstack.io/trow-a-container-registry-to-run-inside-a-kubernetes-cluster/</a></p>
</li>
<li><p>Proof they use the /etc/hosts method
<a href="https://github.com/ContainerSolutions/trow/blob/main/QUICK-INSTALL.md" rel="nofollow noreferrer">https://github.com/ContainerSolutions/trow/blob/main/QUICK-INSTALL.md</a></p>
</li>
</ul>
<ol>
<li><p>Pay $12 for a some-dns-name.tld</p>
</li>
<li><p>Use Cert Manager Kubernetes Operator or Cert Bot standalone docker container + proof you own the domain to get an <a href="https://registry.some-dns-name.tld" rel="nofollow noreferrer">https://registry.some-dns-name.tld</a> HTTPS cert from Lets Encrypt Free. And configure your inner cluster hosted registry to use this HTTPS cert.</p>
</li>
<li><p>Expose the registry hosted in the cluster to the LAN using an NodePort service with an explicitly pinned convention based port number, like 32443</p>
</li>
</ol>
<blockquote>
<p><strong>Why NodePort and not a LB?</strong>  There's 3 reason NP is better than LB for this scenario:</p>
<ul>
<li><p>1.) Service type LB's implementation differs
between Deployment Environment and Kubernetes Distribution, while type
NodePort is universal.</p>
</li>
<li><p>2.) If the LB changes you have to update every node's /etc/host
file to point to &quot;LB_IP   registry.some-dns-name.tld&quot; AND you have to
know the LB IP, that isn't always known in advance / which means you'd
have to follow some order of operations. If you use service type
NodePort you can add the localhost IP entry to every node's /etc/host,
so it looks like &quot;127.0.0.1  registry.some-dns-name.tld&quot;, it's well
known reusable and simplifies order of operations.</p>
</li>
<li><p>3.) If you ever need to change where your cluster is hosted, you
can arrange it so you can make the change in 1 centralized location
even in scenarios where you have no access to or control over LAN DNS.
You can craft services that point to a staticly defined IP or external
name (which could exist outside the cluster). and have the NodePort
service point to the staticly defined service.</p>
</li>
</ul>
</blockquote>
<ol start="4">
<li><p>Add &quot;127.0.0.1 registry.some-dns-name.tld&quot; to /etc/hosts of every node in the cluster.</p>
</li>
<li><p>Set your yaml manifests to pull from registry.some-dns-name.tls, or configure containerd/cri-o's registry mirroring feature to map registry.some-dns-name.tld:32443 to whatever entries are being mirrored on your local registry.</p>
</li>
<li><p>There's 2 more solvable chicken and egg problems to deal with. 1st chicken egg problem is that Kubernetes and the registry will both likely need access to container images to even get this far.</p>
</li>
</ol>
<ul>
<li>If you have internet access and if your internal registry is just for cacheing purposes this probably isn't a big deal.</li>
<li>If you don't have internet access you'd need to .tar up the images needed by your kube distro and registry and &quot;preload&quot; them into spot docker/containerd/cri-o expects.</li>
<li>If you don't have internet access you could alternatively have another /etc/hosts entry or LAN DNS entry for a non HA temporary docker compose based registry used for initial bootstrapping or one hosted outside the cluster.</li>
</ul>
<ol start="7">
<li>2nd chicken egg problem is the registry hosted on your cluster will need some way of being seeded with images.</li>
</ol>
<ul>
<li>If internet access should be easy to figure out and script</li>
<li>If no internet access you may need to come up with some kind of DR solution for backup and restoration of the registries backend persistent storage.</li>
<li>If no internet access you could alternatively use an &quot;emphemeral transport registry&quot; for seeding purposes. Basically do this docker compose to spin up non-HA registry:2 image with filesystem backing, use <a href="https://github.com/containers/skopeo" rel="nofollow noreferrer">skopeo</a> to seed that, tar up the filesystem backing, and import it into another computer, restore the file system backing, reload non-HA registry:2 from the pre-populated filesystem backing, then use skopeo to copy from your &quot;ephemeral transport registry&quot; to your registry hosted in the cluster.</li>
</ul>
