<p>I tried to install <code>Kubernetes</code> with <code>kubeadm</code> on 3 virtual machines with <code>Debian</code> OS on my laptop, one as master node and the other two as worker nodes. I did exactly as the tutorials on <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/" rel="nofollow noreferrer">kubernetes.io</a> suggests. I initialized cluster with command <code>kubeadm init --pod-network-cidr=10.244.0.0/16</code> and joined the workers with corresponding <code>kube join</code> command. I installed <code>Flannel</code> as the network overlay with command <code>kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml</code>.</p>

<p>The repsonse of command <code>kubectl get nodes</code> looks fine:</p>

<pre><code>NAME        STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE
k8smaster   Ready    master   20h   v1.18.3   192.168.1.100   &lt;none&gt;        Debian GNU/Linux 10 (buster)   4.19.0-9-amd64   docker://19.3.9
k8snode1    Ready    &lt;none&gt;   20h   v1.18.3   192.168.1.101   &lt;none&gt;        Debian GNU/Linux 10 (buster)   4.19.0-9-amd64   docker://19.3.9
k8snode2    Ready    &lt;none&gt;   20h   v1.18.3   192.168.1.102   &lt;none&gt;        Debian GNU/Linux 10 (buster)   4.19.0-9-amd64   docker://19.3.9
</code></pre>

<p>The response of command <code>kubectl get pods --all-namespaces</code> doesn't show any error:</p>

<pre><code>NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE    IP              NODE        NOMINATED NODE   READINESS GATES
kube-system   coredns-66bff467f8-7hlnp             1/1     Running   9          20h    10.244.0.22     k8smaster   &lt;none&gt;           &lt;none&gt;
kube-system   coredns-66bff467f8-wmvx4             1/1     Running   11         20h    10.244.0.23     k8smaster   &lt;none&gt;           &lt;none&gt;
kube-system   etcd-k8smaster                      1/1     Running   11         20h    192.168.1.100   k8smaster   &lt;none&gt;           &lt;none&gt;
kube-system   kube-apiserver-k8smaster            1/1     Running   9          20h    192.168.1.100   k8smaster   &lt;none&gt;           &lt;none&gt;
kube-system   kube-controller-manager-k8smaster   1/1     Running   11         20h    192.168.1.100   k8smaster   &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-amd64-9c5rr          1/1     Running   17         20h    192.168.1.102   k8snode2    &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-amd64-klw2p          1/1     Running   21         20h    192.168.1.101   k8snode1    &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-amd64-x7vm7          1/1     Running   11         20h    192.168.1.100   k8smaster   &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-jdfzg                    1/1     Running   11         19h    192.168.1.101   k8snode1    &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-lcdvb                    1/1     Running   6          19h    192.168.1.102   k8snode2    &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-w6jmf                    1/1     Running   11         20h    192.168.1.100   k8smaster   &lt;none&gt;           &lt;none&gt;
kube-system   kube-scheduler-k8smaster            1/1     Running   10         20h    192.168.1.100   k8smaster   &lt;none&gt;           &lt;none&gt;
</code></pre>

<p>Then i tried to create a <code>POD</code> with command <code>kubectl apply -f podexample.yml</code> with following content:</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: example 
spec:
  containers:
  - name: nginx 
    image: nginx
</code></pre>

<p>Command <code>kubectl get pods -o wide</code> shows that the <code>POD</code> is created on worker node1 and is in <code>Running</code> state.</p>

<pre><code>NAME      READY   STATUS    RESTARTS   AGE    IP            NODE       NOMINATED NODE   READINESS GATES
example   1/1     Running   0          135m   10.244.1.14   k8snode1   &lt;none&gt;           &lt;none&gt;
</code></pre>

<p>The thing is, when i try to connect to the pod with <code>curl -I 10.244.1.14</code> command i get the following response in master node:</p>

<pre><code>curl: (7) Failed to connect to 10.244.1.14 port 80: Connection timed out
</code></pre>

<p>but the same command on the worker node1 responds successfully with:</p>

<pre><code>HTTP/1.1 200 OK
Server: nginx/1.17.10
Date: Sat, 23 May 2020 19:45:05 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 14 Apr 2020 14:19:26 GMT
Connection: keep-alive
ETag: "5e95c66e-264"
Accept-Ranges: bytes
</code></pre>

<p>I thought maybe that's because somehow <code>kube-proxy</code> is not running on master node but command <code>ps aux | grep kube-proxy</code> shows that it's running.</p>

<pre><code>root     16747  0.0  1.6 140412 33024 ?        Ssl  13:18   0:04 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8smaster
</code></pre>

<p>Then i checked for kernel routing table with command <code>ip route</code> and it shows that packets destined for <code>10.244.1.0/244</code> get routed to flannel.</p>

<pre><code>default via 192.168.1.1 dev enp0s3 onlink 
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1 
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink 
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 
169.254.0.0/16 dev enp0s3 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.1.0/24 dev enp0s3 proto kernel scope link src 192.168.1.100 
</code></pre>

<p>Everything looks fine to me and i don't know what else should i check to see what's the problem. Am i missing something? </p>

<p>UPDATE1:</p>

<p>If i start an <code>NGINX</code> container on worker node1 and map it's 80 port to port 80 of the worker node1 host, then i can connect to it via command <code>curl -I 192.168.1.101</code> from master node. Also, i didn't add any iptable rule and there is no firewall daemon like <code>UFW</code> installed on the machines. So, i think it's not a firewall issue.</p>

<p>UPDATE2:</p>

<p>I recreated the cluster and used <code>canal</code> instead of <code>flannel</code>, still no luck.</p>

<p>UPDATE3:</p>

<p>I took a look at canal and flannel logs with following commands and everything seems fine:</p>

<pre><code>kubectl logs -n kube-system canal-c4wtk calico-node
kubectl logs -n kube-system canal-c4wtk kube-flannel
kubectl logs -n kube-system canal-b2fkh calico-node
kubectl logs -n kube-system canal-b2fkh kube-flannel 
</code></pre>

<p>UPDATE4:</p>

<p>for the sake of completeness, <a href="https://gofile.io/d/rZIjLa" rel="nofollow noreferrer">here are the logs of mentioned containers</a>.</p>

<p>UPDATE5:</p>

<p>I tried to install specific version of kubernetes components and docker, to check if there is an issue related to versioning mismatch with following commands:</p>

<pre><code>sudo apt-get install docker-ce=18.06.1~ce~3-0~debian
sudo apt-get install -y kubelet=1.12.2-00 kubeadm=1.12.2-00 kubectl=1.12.2-00 kubernetes-cni=0.6.0-00
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
</code></pre>

<p>but nothing changed.</p>

<p>i even updated file <code>/etc/bash.bashrc</code> on all nodes to clear any proxy settings just to make sure it's not about proxy:</p>

<pre><code>export HTTP_PROXY=
export http_proxy=
export NO_PROXY=127.0.0.0/8,192.168.0.0/16,172.0.0.0/8,10.0.0.0/8
</code></pre>

<p>and also added following environments to docker systemd file <code>/lib/systemd/system/docker.service</code> on all nodes:</p>

<pre><code>Environment="HTTP_PROXY="
Environment="NO_PROXY="
</code></pre>

<p>Then rebooted all nodes and when i logged in, still got <code>curl: (7) Failed to connect to 10.244.1.12 port 80: Connection timed out</code> </p>

<p>UPDATE6:</p>

<p>i event tried to setup the cluster in <code>CentOS</code> machines. thought maybe there is something related to <code>Debian</code>. i also stopped and disabled <code>firewalld</code> to make sure that firewall is not causing problem, but i got the exact same result again: <code>Failed to connect to 10.244.1.2 port 80: Connection timed out</code>.</p>

<p>The only thing that now i'm suspicious about is that maybe it's all because of <code>VirtualBox</code> and virtual machines network configuration? The virtual machines are attched to a <code>Bridge Adapter</code> connected to my Wireless network interface. </p>

<p>UPDATE7:</p>

<p>I went inside the created POD and figured out there is no internet connectivity inside the POD. So, I created another POD from a <code>NGINX</code> image that has commands like <code>curl</code>, <code>wget</code>, <code>ping</code> and <code>traceroute</code> and tried <code>curl https://www.google.com -I</code> and got result: <code>curl: (6) Could not resolve host: www.google.com</code>. I checked <code>/etc/resolv.conf</code> file and found that the DNS server address inside the POD is <code>10.96.0.10</code>. Changed the DNS to <code>8.8.8.8</code> still <code>curl https://www.google.com -I</code> results in <code>curl: (6) Could not resolve host: www.google.com</code>. Tried to ping <code>8.8.8.8</code> and the result is <code>56 packets transmitted, 0 received, 100% packet loss, time 365ms</code>. For the last step i tried <code>traceroute 8.8.8.8</code> and got the following result:</p>

<pre><code> 1  10.244.1.1 (10.244.1.1)  0.116 ms  0.056 ms  0.052 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
</code></pre>

<p>I don't know the fact that there is no internet connectivity in POD has anything to do with the problem that i can't connect to POD within the cluster from nodes other than the one that POD is deployed on.</p>

<p>Debian system uses <code>nftables</code> for the <code>iptables</code> backend which is not compatible with Kubernetes network setup. So you have to set it to use iptables-legacy instead of nftables with the following commands: </p>

<pre><code>sudo update-alternatives --set iptables /usr/sbin/iptables-legacy 
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
</code></pre>
