<p>I have issues with CoreDNS on some nodes are in Crashloopback state due to error trying to reach the kubernetes internal service.</p>

<p>This is a new K8s cluster deployed using Kubespray, the network layer is Weave with Kubernetes version 1.12.5 on Openstack.
I've already tested the connection to the endpoints and have no issue reaching to 10.2.70.14:6443 for example.
But telnet from the pods to 10.233.0.1:443 is failing.</p>

<p>Thanks in advance for the help</p>

<pre><code>kubectl describe svc kubernetes
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       &lt;none&gt;
Selector:          &lt;none&gt;
Type:              ClusterIP
IP:                10.233.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         10.2.70.14:6443,10.2.70.18:6443,10.2.70.27:6443 + 2 more...
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>

<p>And from CoreDNS logs:</p>

<pre><code>E0415 17:47:05.453762       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:311: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0415 17:47:05.456909       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:313: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0415 17:47:06.453258       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:318: Failed to list *v1.Namespace: Get https://10.233.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
</code></pre>

<p>Also, checking out the logs of kube-proxy from one of the problematic nodes revealed the following errors:</p>

<pre><code>I0415 19:14:32.162909       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.1:443/TCP/10.2.70.36:6443
I0415 19:14:32.162979       1 graceful_termination.go:171] Not deleting, RS 10.233.0.1:443/TCP/10.2.70.36:6443: 1 ActiveConn, 0 InactiveConn
I0415 19:14:32.162989       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.1:443/TCP/10.2.70.18:6443
I0415 19:14:32.163017       1 graceful_termination.go:171] Not deleting, RS 10.233.0.1:443/TCP/10.2.70.18:6443: 1 ActiveConn, 0 InactiveConn
E0415 19:14:32.215707       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
)
</code></pre>

<p>I had exactly the same problem, and it turned out that my kubespray config was wrong. Especially the nginx ingress setting <code>ingress_nginx_host_network</code></p>

<p>As it turns our you have to set <code>ingress_nginx_host_network: true</code> (defaults to false)</p>

<p><strong>If you do not want to rerun the whole kubespray script, edit the nginx ingress deamon set</strong></p>

<p><code>$ kubectl -n ingress-nginx edit ds ingress-nginx-controller</code></p>

<ol>
<li>Add <code>--report-node-internal-ip-address</code> to the commandline:</li>
</ol>

<pre><code>spec:
  container:
      args:
       - /nginx-ingress-controller
       - --configmap=$(POD_NAMESPACE)/ingress-nginx
       - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
       - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
       - --annotations-prefix=nginx.ingress.kubernetes.io
       - --report-node-internal-ip-address # &lt;- new
</code></pre>

<ol start="2">
<li>Set the following two properties on the same level as e.g <code>serviceAccountName: ingress-nginx</code>:</li>
</ol>

<pre><code>serviceAccountName: ingress-nginx
hostNetwork: true # &lt;- new
dnsPolicy: ClusterFirstWithHostNet  # &lt;- new
</code></pre>

<p>Then save and quit <code>:wq</code>, check the pod status <code>kubectl get pods --all-namespaces</code>.</p>

<p><em>Source:</em>
<a href="https://github.com/kubernetes-sigs/kubespray/issues/4357" rel="nofollow noreferrer">https://github.com/kubernetes-sigs/kubespray/issues/4357</a></p>
