<p>I am running a small 3 node test kubernetes cluster (using kubeadm) running on Ubuntu Server 22.04, with Flannel as the network fabric.  I also have a separate gitlab private server, with container registry set up and working.</p>
<p>The problem I am running into is I have a simple test deployment, and when I apply the deployment yaml, it fails to pull the image from the gitlab private server.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: platform-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: platform-service
  template:
    metadata:
      labels:
        app: platform-service
    spec:
      containers:
        - name: platform-service
          image: registry.examle.com/demo/platform-service:latest
</code></pre>
<h4>Ubuntu Server: /etc/hosts (the relevant line)</h4>
<pre class="lang-bash prettyprint-override"><code>192.168.1.30 registry.example.com
</code></pre>
<h4>The Error</h4>
<pre class="lang-bash prettyprint-override"><code>Failed to pull image &quot;registry.example.com/demo/platform-service:latest&quot;: 
rpc error: code = Unknown desc = failed to pull and unpack image 
&quot;registry.example.com/deni/platform-service:latest&quot;: failed to resolve reference 
&quot;registry.example.com/demo/platform-service:latest&quot;: failed to do request: Head 
&quot;https://registry.example.com/v2/demo/platform-service/manifests/latest&quot;: dial tcp 
xxx.xxx.xxx.xxx:443: i/o timeout
</code></pre>
<p>The 'xxx.xxx.xxx.xxx' is related to my external network, to which exists a domain name in the DNS, however all of my internal networks are set up to attach to the internal network representation, and 'registry.example.com' is a representation of my own domains.</p>
<p>Also to note:</p>
<pre class="lang-bash prettyprint-override"><code>docker pull registry.example.com/demo/platform-service:latest
</code></pre>
<p>From the command line of the server, works perfectly fine, it is just not working from kubernetes deploy yaml.</p>
<h4>The problem</h4>
<p>While the network on the server, and the host files on the server are configured correctly, the docker image is not resolving because when I apply it is not using the correct  IP (that is configured in hosts), rather a public IP that is a different server. And the reason for the timeout is because the public facing server is not set up the same.</p>
<p>When I run <code>kubectl apply -f platform-service.yaml</code> why does it not respect the hosts file of the server, and is there a way configure hosts inside Kubernetes.</p>
<p>(If this problem is not clear, I apologize, I am quite new, and still learning terminology, maybe why google is not helping me with this problem.)</p>
<p>The closest S/O I could find is:</p>
<p><a href="https://stackoverflow.com/questions/62940403/kubernetes-not-able-pull-image-from-private-registry-having-private-domain-point">Kubernetes not able pull image from private registry having private domain pointed via /etc/hosts</a></p>
<p>(SO Answer #1): hostAliases (this is for the pod itself, not pulling the image), also, installed through apt/package manager rather than snap.  With the rest of the answer suggests changing the distribution, which I would rather go with my current setup than change it.</p>
<h4>Update</h4>
<p>Attempts to add hosts to coredns not working either:
(<a href="https://stackoverflow.com/questions/65283827/how-to-change-host-name-resolve-like-host-file-in-coredns">How to change host name resolve like host file in coredns</a>)</p>
<pre><code>kubectl -n kube-system edit configmap/coredns
</code></pre>
<pre><code>...
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        hosts custom.hosts registry.example.com {
            192.168.1.30 registry.example.com
            fallthrough
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
...
</code></pre>
<p>deleted the coredns pods (so they are recreated)</p>
<p>and still the docker pull on the deployment fails with the external ip address instead of the internal address.</p>

<p>After going through many different solutions and lots of research and testing.  The answer was actually very simple.</p>
<h3>Solution in my case</h3>
<p>The /etc/hosts file <em>MUST</em> contain the host for the registry (and possibly the entry for the gitlab instance as well) on <em>EVERY</em> node of the cluster including the master node.</p>
<pre><code>192.168.1.30 registry.example.com
192.168.1.30 gitlab.example.com    # Necessary in my case, not sure required
</code></pre>
<p>Once I included that on each of the 2 slaves, it attempted to pull the image, and failed with credential issues (which I was expecting to see once the hosts issue was resolved).  From there I was able to add the credentials and now the image pulls fine from the private registry rather than the public facing registry.</p>
<h4>Bonus: Fix for credentials error connecting to private registry (not part of the original question, but part of the setup process for connecting)</h4>
<p>After fixing the /etc/hosts issue, you will probably need to set up 'regcred' credentials to access the private registry, Kubernetes documentation provides the steps on that part:</p>
<p><a href="https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/</a></p>
