<p>I've deployed several different services and always get the same error.</p>

<p>The service is reachable on the node port from the machine where the pod is running. On the two other nodes I get timeouts.</p>

<p>The kube-proxy is running on all worker nodes and I can see in the logfiles from kube-proxy that the service port was added and the node port was opened.
In this case I've deployed the stars demo from calico</p>

<p>Kube-proxy log output:</p>

<pre><code>Mar 11 10:25:10 kuben1 kube-proxy[659]: I0311 10:25:10.229458     659 service.go:309] Adding new service port "management-ui/management-ui:" at 10.32.0.133:9001/TCP
Mar 11 10:25:10 kuben1 kube-proxy[659]: I0311 10:25:10.257483     659 proxier.go:1427] Opened local port "nodePort for management-ui/management-ui:" (:30002/tcp)
</code></pre>

<p>The kube-proxy is listening on the port 30002</p>

<pre><code>root@kuben1:/tmp# netstat -lanp | grep 30002
tcp6       0      0 :::30002                :::*                    LISTEN      659/kube-proxy   
</code></pre>

<p>There are also some iptable rules defined:</p>

<pre><code>root@kuben1:/tmp# iptables -L -t nat | grep management-ui
KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* management-ui/management-ui: */ tcp dpt:30002
KUBE-SVC-MIYW5L3VT4JVLCIZ  tcp  --  anywhere             anywhere             /* management-ui/management-ui: */ tcp dpt:30002
KUBE-MARK-MASQ  tcp  -- !10.200.0.0/16        10.32.0.133          /* management-ui/management-ui: cluster IP */ tcp dpt:9001
KUBE-SVC-MIYW5L3VT4JVLCIZ  tcp  --  anywhere             10.32.0.133          /* management-ui/management-ui: cluster IP */ tcp dpt:9001
</code></pre>

<p>The interesting part is that I can reach the service IP from any worker node</p>

<pre><code>root@kubem1:/tmp# kubectl get svc -n management-ui
NAME            TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
management-ui   NodePort   10.32.0.133   &lt;none&gt;        9001:30002/TCP   52m
</code></pre>

<p>The service IP/port can be accessed from any worker node if I do a "curl <a href="http://10.32.0.133:9001" rel="nofollow noreferrer">http://10.32.0.133:9001</a>"</p>

<p>I don't understand why kube-proxy does not "route" this properly...<br>
Has anyone a hint where I can find the error?</p>

<hr>

<p>Here some cluster specs:</p>

<p>This is a hand build cluster inspired by Kelsey Hightower's "kubernetes the hard way" guide.</p>

<ul>
<li>6 Nodes (3 master: 3 worker) local vms</li>
<li>OS: Ubuntu 18.04</li>
<li>K8s: v1.13.0</li>
<li>Docker: 18.9.3</li>
<li>Cni: calico</li>
</ul>

<p>Component status on the master nodes looks okay</p>

<pre><code>root@kubem1:/tmp# kubectl get componentstatus
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}     
</code></pre>

<p>The worker nodes are looking okay if I trust kubectl</p>

<pre><code>root@kubem1:/tmp# kubectl get nodes -o wide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kuben1   Ready    &lt;none&gt;   39d   v1.13.0   192.168.178.77   &lt;none&gt;        Ubuntu 18.04.2 LTS   4.15.0-46-generic   docker://18.9.3
kuben2   Ready    &lt;none&gt;   39d   v1.13.0   192.168.178.78   &lt;none&gt;        Ubuntu 18.04.2 LTS   4.15.0-46-generic   docker://18.9.3
kuben3   Ready    &lt;none&gt;   39d   v1.13.0   192.168.178.79   &lt;none&gt;        Ubuntu 18.04.2 LTS   4.15.0-46-generic   docker://18.9.3
</code></pre>

<p>As asked by P Ekambaram:</p>

<pre><code>root@kubem1:/tmp# kubectl get po -n kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
calico-node-bgjdg                      1/1     Running   5          40d
calico-node-nwkqw                      1/1     Running   5          40d
calico-node-vrwn4                      1/1     Running   5          40d
coredns-69cbb76ff8-fpssw               1/1     Running   5          40d
coredns-69cbb76ff8-tm6r8               1/1     Running   5          40d
kubernetes-dashboard-57df4db6b-2xrmb   1/1     Running   5          40d
</code></pre>

w4ri0r
<p>I've found a solution for my "Problem".<br>
This behavior was caused by a change in Docker v1.13.x and the issue was fixed in kubernetes with version 1.8.</p>

<p>The easy solution was to change the forward rules via iptables.<br>
Run the following cmd on all worker nodes: "iptables -A FORWARD -j ACCEPT"</p>

<p>To fix it the right way i had to tell the kube-proxy the cidr for the pods.
Theoretical that could be solved in two ways:</p>

<ul>
<li>Add "--cluster-cidr=10.0.0.0/16" as argument to the kube-proxy command line(in my case  in the systemd service file)</li>
<li>Add 'clusterCIDR: "10.0.0.0/16"' to the kubeconfig file for kube-proxy</li>
</ul>

<p>In my case the cmd line argument doesn't had any effect.<br>
As i've added the line to my kubeconfig file and restarted the kube-proxy on all worker nodes everything works well.</p>

<p>Here is the github merge request for this "FORWARD" issue: <a href="https://github.com/kubernetes/kubernetes/pull/52569" rel="nofollow noreferrer">link</a></p>
