<p>I am trying to install kubernetes.
There was a problem when checking the container executed on kubernetes.
I set the type of service to Nodeport, but I could not access from a node other than the one on which the container is running.
Want to make it accessible from other computers, please tell me where it is different.
I tried externalIPs and LoadBarancer, but it was impossible.</p>

<p>Enviroment</p>

<ul>
<li>OS:Ubuntu 16.04 LTS</li>
<li>Kubernetes:1.8</li>
<li>Docker:17.09.0-ce</li>
<li>etcd:3.2.8</li>
<li>flannel:0.9.0</li>
</ul>

<p>Network</p>

<ul>
<li>Physical:10.1.1.0/24</li>
<li>flannel:172.16.0.0/16</li>
<li>docker:192.168.0.0/16</li>
</ul>

<p>Machines</p>

<ul>
<li>Master Node(2nodes):10.1.1.24,10.1.1.25</li>
<li>Worker Node(2nodes):10.1.1.26,10.1.1.27</li>
</ul>

<p>kubectl describe svc nginx-cluster</p>

<pre><code>Name:              nginx-cluster
Namespace:         default
Labels:            app=nginx-demo
Annotations:       &lt;none&gt;
Selector:          app=nginx-demo
Type:              ClusterIP
IP:                172.16.236.159
Port:              &lt;unset&gt;  8090/TCP
TargetPort:        80/TCP
Endpoints:         192.168.24.2:80
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>

<p>kubectl describe svc nginx-service</p>

<pre><code>Name:                     nginx-service
Namespace:                default
Labels:                   app=nginx-demo
Annotations:              &lt;none&gt;
Selector:                 app=nginx-demo
Type:                     NodePort
IP:                       172.16.199.69
Port:                     &lt;unset&gt;  8090/TCP
TargetPort:               80/TCP
NodePort:                 &lt;unset&gt;  31659/TCP
Endpoints:                192.168.24.2:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;
</code></pre>

<p>running container worker node(10.1.1.27)</p>

<p>curl 10.1.1.27:31659</p>

<pre><code>&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
...
</code></pre>

<p>worker node(10.1.1.26)</p>

<p>curl 10.1.1.27:31659</p>

<pre><code>curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.
</code></pre>

<p>other machine(10.1.1.XX)</p>

<p>curl 10.1.1.27:31659</p>

<pre><code>curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.
</code></pre>

<p>kubectl get pods -o wide</p>

<pre><code>NAME                          READY     STATUS    RESTARTS   AGE       IP             NODE
echoserver-848b75d85-9fx7r    1/1       Running   3          6d        192.168.70.2   k8swrksv01
nginx-demo-85cc49574c-wv2b9   1/1       Running   3          6d        192.168.2.2    k8swrksv02
</code></pre>

<p>kubectl get svc -o wide</p>

<pre><code>NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE       SELECTOR
clusterip       ClusterIP   172.16.39.77     &lt;none&gt;        80/TCP           6d        run=echoserver
kubernetes      ClusterIP   172.16.0.1       &lt;none&gt;        443/TCP          10d       &lt;none&gt;
nginx-cluster   ClusterIP   172.16.236.159   &lt;none&gt;        8090/TCP         6d        app=nginx-demo
nginx-service   NodePort    172.16.199.69    &lt;none&gt;        8090:31659/TCP   6d        app=nginx-demo
nodeport        NodePort    172.16.38.40     &lt;none&gt;        80:31317/TCP     6d        run=echoserver
</code></pre>

<p>netstat -ntlp</p>

<pre><code>tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1963/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      2202/kube-proxy
tcp        0      0 127.0.0.1:4243          0.0.0.0:*               LISTEN      1758/dockerd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      996/sshd
tcp6       0      0 :::4194                 :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::10250                :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::31659                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::10255                :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::10256                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::31317                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::22                   :::*                    LISTEN      996/sshd
</code></pre>

<p>iptables-save</p>

<pre><code>*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-AZ4EGFEAU4RTSLJO - [0:0]
:KUBE-SEP-C7HQKKO26GIFOZZM - [0:0]
:KUBE-SEP-EWKNS2YCPXGJCXDC - [0:0]
:KUBE-SEP-LQVPUPFGW6BWATIP - [0:0]
:KUBE-SEP-OMMOFZ27GPKZ4OPA - [0:0]
:KUBE-SEP-UD3HOGDD5NDLNY74 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-CQNAS6RSUGJF2C2D - [0:0]
:KUBE-SVC-GKN7Y2BSGW4NJTYL - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-XP7QDA4CRQ2QA33W - [0:0]
:KUBE-SVC-Z5P6OMNAEVLAQUTS - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 192.168.2.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.0.0/16 -d 192.168.0.0/16 -j RETURN
-A POSTROUTING -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.2.0/24 -j RETURN
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -s 192.168.70.2/32 -m comment --comment "default/clusterip:" -j KUBE-MARK-MASQ
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -p tcp -m comment --comment "default/clusterip:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-C7HQKKO26GIFOZZM -s 192.168.70.2/32 -m comment --comment "default/nodeport:" -j KUBE-MARK-MASQ
-A KUBE-SEP-C7HQKKO26GIFOZZM -p tcp -m comment --comment "default/nodeport:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-EWKNS2YCPXGJCXDC -s 10.1.1.25/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-EWKNS2YCPXGJCXDC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.25:6443
-A KUBE-SEP-LQVPUPFGW6BWATIP -s 192.168.2.2/32 -m comment --comment "default/nginx-service:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m comment --comment "default/nginx-service:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -s 10.1.1.24/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.24:6443
-A KUBE-SEP-UD3HOGDD5NDLNY74 -s 192.168.2.2/32 -m comment --comment "default/nginx-cluster:" -j KUBE-MARK-MASQ
-A KUBE-SEP-UD3HOGDD5NDLNY74 -p tcp -m comment --comment "default/nginx-cluster:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SERVICES -d 172.16.236.159/32 -p tcp -m comment --comment "default/nginx-cluster: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-Z5P6OMNAEVLAQUTS
-A KUBE-SERVICES -d 172.16.199.69/32 -p tcp -m comment --comment "default/nginx-service: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-SERVICES -d 172.16.38.40/32 -p tcp -m comment --comment "default/nodeport: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-SERVICES -d 172.16.39.77/32 -p tcp -m comment --comment "default/clusterip: cluster IP" -m tcp --dport 80 -j KUBE-SVC-CQNAS6RSUGJF2C2D
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-CQNAS6RSUGJF2C2D -m comment --comment "default/clusterip:" -j KUBE-SEP-AZ4EGFEAU4RTSLJO
-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m comment --comment "default/nginx-service:" -j KUBE-SEP-LQVPUPFGW6BWATIP
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-XP7QDA4CRQ2QA33W -m comment --comment "default/nodeport:" -j KUBE-SEP-C7HQKKO26GIFOZZM
-A KUBE-SVC-Z5P6OMNAEVLAQUTS -m comment --comment "default/nginx-cluster:" -j KUBE-SEP-UD3HOGDD5NDLNY74
COMMIT
*filter
:INPUT ACCEPT [40:14606]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [42:6275]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:DOCKER-USER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
</code></pre>

<p>It's caused by default DROP on FORWARD chain (which in turn is caused by docker).</p>

<p>You can see it working again if you add <code>iptables -A FORWARD -j ACCEPT</code> rule to your node.</p>

<p>The k8s issue is here: <a href="https://github.com/kubernetes/kubernetes/issues/39823" rel="noreferrer">https://github.com/kubernetes/kubernetes/issues/39823</a> but the actual fix is here <a href="https://github.com/kubernetes/kubernetes/pull/52569" rel="noreferrer">https://github.com/kubernetes/kubernetes/pull/52569</a> (expected to be in 1.9).</p>
