<p>I'm working on installing a three node kubernetes cluster on a CentOS 7 with flannel for a some time, however the CoreDNS pods cannot connect to API server and constantly restarting.</p>

<p>The reference HowTo document I followed is <a href="https://www.howtoforge.com/tutorial/centos-kubernetes-docker-cluster/" rel="nofollow noreferrer">here</a>.</p>

<h2>What Have I Done so Far?</h2>

<ul>
<li>Disabled SELinux,</li>
<li>Disabled <code>firewalld</code>,</li>
<li>Enabled <code>br_netfilter</code>, <code>bridge-nf-call-iptables</code>,</li>
<li>Installed kubernetes on three nodes, set-up master's pod network with flannel default network (<code>10.244.0.0/16</code>),</li>
<li>Installed other two nodes, and joined the master.</li>
<li>Deployed flannel,</li>
<li>Configured Docker's BIP to use flannel default per-node subnet and network.</li>
</ul>

<h2>Current State</h2>

<ul>
<li>The kubelet works and the cluster reports nodes as ready.</li>
<li>The Cluster can schedule and migrate pods, so CoreDNS are spawned on nodes.</li>
<li>Flannel network is connected. No logs in containers and I can ping <code>10.244.0.0/24</code> networks from node to node.</li>
<li>Kubernetes can deploy and run arbitrary pods (Tried <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/" rel="nofollow noreferrer">shell demo</a>, and can access its shell via <code>kubectl</code> even if the container is on a different node.

<ul>
<li>However, since DNS is not working, they cannot resolve any IP addresses.</li>
</ul></li>
</ul>

<h2>What is the Problem?</h2>

<ul>
<li><p>CoreDNS pods report that they cannot connect to API server with error:</p>

<pre><code>Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
</code></pre></li>
<li><p>I cannot see <code>10.96.0.0</code> routes in routing tables:</p>

<pre><code>default via 172.16.0.1 dev eth0 proto static metric 100 
10.1.0.0/24 dev eth1 proto kernel scope link src 10.1.0.202 metric 101 
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink 
10.244.1.0/24 dev docker0 proto kernel scope link src 10.244.1.1 
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 
172.16.0.0/16 dev eth0 proto kernel scope link src 172.16.0.202 metric 100
</code></pre></li>
</ul>

<h2>Additional Info</h2>

<ul>
<li>Cluster init is done with the command <code>kubeadm init --apiserver-advertise-address=172.16.0.201 --pod-network-cidr=10.244.0.0/16</code>.</li>
<li>I have torn down the cluster and rebuilt with 1.12.0 The problem still persists.</li>
<li>The workaround in Kubernetes <a href="https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-pods-have-crashloopbackoff-or-error-state" rel="nofollow noreferrer">documentation</a> doesn't work.</li>
<li>Problem is present and same both with <code>1.11-3</code>and <code>1.12-0</code> CentOS7 packages.</li>
</ul>

<h2>Progress so Far</h2>

<ul>
<li>Downgraded Kubernetes to <code>1.11.3-0</code>.</li>
<li>Re-initialized Kubernetes with <code>kubeadm init --apiserver-advertise-address=172.16.0.201 --pod-network-cidr=10.244.0.0/16</code>, since the server has another external IP which cannot be accessed via other hosts, and Kubernetes tends to select that IP as API Server IP. <code>--pod-network-cidr</code> is mandated by <a href="https://coreos.com/flannel/docs/latest/kubernetes.html" rel="nofollow noreferrer">flannel</a>.</li>
<li><p>Resulting <code>iptables -L</code> output after initialization <em>with no joined nodes</em></p>

<pre><code>Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
DOCKER-USER  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination         

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000

Chain KUBE-SERVICES (1 references)
target     prot opt source               destination         
REJECT     udp  --  anywhere             10.96.0.10           /* kube-system/kube-dns:dns has no endpoints */ udp dpt:domain reject-with icmp-port-unreachable
REJECT     tcp  --  anywhere             10.96.0.10           /* kube-system/kube-dns:dns-tcp has no endpoints */ tcp dpt:domain reject-with icmp-port-unreachable
</code></pre></li>
<li><p>Looks like API Server is deployed as it should</p>

<pre><code>$ kubectl get svc kubernetes -o=yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-10-25T06:58:46Z
  labels:
    component: apiserver
    provider: kubernetes
  name: kubernetes
  namespace: default
  resourceVersion: "6"
  selfLink: /api/v1/namespaces/default/services/kubernetes
  uid: 6b3e4099-d823-11e8-8264-a6f3f1f622f3
spec:
  clusterIP: 10.96.0.1
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 6443
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
</code></pre></li>
<li><p>Then I've applied flannel network pod with </p>

<pre><code>kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
</code></pre></li>
<li><p>As soon as I apply the flannel network, CoreDNS pods start and start to give the same error:</p>

<pre><code>Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500\u0026resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
</code></pre></li>
<li><p>I've found out that <code>flanneld</code> is using the wrong network interface, and changed it in the <code>kube-flannel.yml</code> file before deployment. However the outcome is still the same.</p></li>
</ul>

<p>Any help is greatly appreciated.</p>

<p>This is basically saying that your coredns pod cannot talk to the kube-apiserver. The kube-apiserver is exposed in the pod through these environment variables: <code>KUBERNETES_SERVICE_HOST=10.96.0.1</code> and <code>KUBERNETES_SERVICE_PORT_HTTPS=443</code></p>

<p>I believe that the routes that you posted are routes on the host since this is what you get when you run <code>ip routes</code> in pod container:</p>

<pre><code>root@xxxx-xxxxxxxxxx-xxxxx:/# ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0  scope link
root@xxxx-xxxxxxxxxx-xxxxx:/#
</code></pre>

<p>In any case, you wouldn't see <code>10.96.0.1</code> since that's exposed in the cluster using iptables. So what is that address? It happens that is a <code>service</code> in the default namespace called <code>kubernetes</code>. That service's <code>ClusterIP</code> is <code>10.96.0.1</code> and it's listening on port <code>443</code>, it also maps to <code>targetPort</code> <code>6443</code> which is where your kube-apiserver is running.</p>

<p>Since you can deploy pods, etc. It seems like the kube-apiserver is not down and that's not your problem. So most likely you are missing that service (or there's some iptable rule not allowing you to connect to it). You can see it here, for example:</p>

<pre><code>$ kubectl get svc kubernetes
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    &lt;none&gt;        443/TCP   92d
</code></pre>

<p>The full output is something like this:</p>

<pre><code>$ kubectl get svc kubernetes -o=yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-07-23T21:10:22Z
  labels:
    component: apiserver
    provider: kubernetes
  name: kubernetes
  namespace: default
  resourceVersion: "24"
  selfLink: /api/v1/namespaces/default/services/kubernetes
  uid: xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
spec:
  clusterIP: 10.96.0.1
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 6443
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {} 
</code></pre>

<p>So if you are missing it, you can create it like this:</p>

<pre><code>cat &lt;&lt;EOF
apiVersion: v1
kind: Service
metadata:
  labels:
    component: apiserver
    provider: kubernetes
  name: kubernetes
  namespace: default
spec:
  clusterIP: 10.96.0.1
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 6443
  sessionAffinity: None
  type: ClusterIP
EOF | kubectl apply -f -
</code></pre>
