<p>I have two identical <code>pod</code> running on two working nodes, which are served externally via a <code>svc</code>. as follows: </p>

<pre><code>root@master1:~# kubectl get pod -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP            NODE
kubia-nwjcc   1/1     Running   0          33m   10.244.1.27   worker1
kubia-zcpbb   1/1     Running   0          33m   10.244.2.11   worker2

root@master1:~# kubectl get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1     &lt;none&gt;        443/TCP   26h
kubia        ClusterIP   10.98.41.49   &lt;none&gt;        80/TCP    34m
</code></pre>

<p>But when I try to access the <code>svc</code> in one of the <code>pod</code>, I can only get the response of the <code>pod</code> on the same node. When svc accesses the <code>pod</code> on the other node. When <code>svc</code> tries to access the <code>pod</code> on other nodes, it will return <code>command terminated with exit code 7</code>. correct output and bad output seem to be randomly generated, as follows</p>

<p><strong>correct output</strong></p>

<pre><code>root@master1:~# k exec kubia-nwjcc -- curl http://10.98.41.49
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    23    0    23    0     0   8543      0 --:--:-- --:--:-- --:--:-- 11500
You've hit kubia-nwjcc
</code></pre>

<p><strong>bad output</strong></p>

<pre><code>root@master1:~# kubectl exec kubia-nwjcc -- curl http://10.98.41.49   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 10.98.41.49 port 80: No route to host
command terminated with exit code 7
</code></pre>

<p>The following is the software version I am using:</p>

<ul>
<li>ubuntu: <code>v18.04</code></li>
<li>kubelet / kubeadm / kubectl: <code>v1.15.0</code></li>
<li>docker: <code>v18.09.5</code></li>
</ul>

<p>The following is <code>svc</code> describe:</p>

<pre><code>root@master1:~# kubectl describe svc kubia
Name:              kubia
Namespace:         default
Labels:            &lt;none&gt;
Annotations:       &lt;none&gt;
Selector:          app=kubia
Type:              ClusterIP
IP:                10.98.41.49
Port:              &lt;unset&gt;  80/TCP
TargetPort:        8080/TCP
Endpoints:         10.244.1.27:8080,10.244.2.11:8080
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>

<p>The following is return results with using <code>-v=9</code>:</p>

<pre><code>root@master1:~# kubectl exec kubia-nwjcc -v=9 -- curl -s http://10.98.41.49
I0702 11:45:52.481239   23171 loader.go:359] Config loaded from file:  /root/.kube/config
I0702 11:45:52.501154   23171 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" 'https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc'
I0702 11:45:52.525926   23171 round_trippers.go:438] GET https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc 200 OK in 24 milliseconds
I0702 11:45:52.525980   23171 round_trippers.go:444] Response Headers:
I0702 11:45:52.525992   23171 round_trippers.go:447]     Content-Type: application/json
I0702 11:45:52.526003   23171 round_trippers.go:447]     Content-Length: 2374
I0702 11:45:52.526012   23171 round_trippers.go:447]     Date: Tue, 02 Jul 2019 11:45:52 GMT
I0702 11:45:52.526063   23171 request.go:947] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"kubia-nwjcc","generateName":"kubia-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/kubia-nwjcc","uid":"2fd67789-c48d-4459-8b03-ac562b4a3f5c","resourceVersion":"188689","creationTimestamp":"2019-07-02T10:51:34Z","labels":{"app":"kubia"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"kubia","uid":"f3a4c457-dee4-4aec-ad73-1f0ca41628aa","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-6pgh8","secret":{"secretName":"default-token-6pgh8","defaultMode":420}}],"containers":[{"name":"kubia","image":"luksa/kubia","ports":[{"containerPort":8080,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"default-token-6pgh8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"worker1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:15Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:20Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:20Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-02T10:51:34Z"}],"hostIP":"192.168.56.21","podIP":"10.244.1.27","startTime":"2019-07-03T01:35:15Z","containerStatuses":[{"name":"kubia","state":{"running":{"startedAt":"2019-07-03T01:35:19Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"luksa/kubia:latest","imageID":"docker-pullable://luksa/kubia@sha256:3f28e304dc0f63dc30f273a4202096f0fa0d08510bd2ee7e1032ce600616de24","containerID":"docker://27da556930baf857e5af92b13934dcb1b2b2f001ecab5e7b952b2bda5aa27f0b"}],"qosClass":"BestEffort"}}
I0702 11:45:52.543108   23171 round_trippers.go:419] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" 'https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc/exec?command=curl&amp;command=-s&amp;command=http%3A%2F%2F10.98.41.49&amp;container=kubia&amp;stderr=true&amp;stdout=true'
I0702 11:45:52.591166   23171 round_trippers.go:438] POST https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc/exec?command=curl&amp;command=-s&amp;command=http%3A%2F%2F10.98.41.49&amp;container=kubia&amp;stderr=true&amp;stdout=true 101 Switching Protocols in 47 milliseconds
I0702 11:45:52.591208   23171 round_trippers.go:444] Response Headers:
I0702 11:45:52.591217   23171 round_trippers.go:447]     Connection: Upgrade
I0702 11:45:52.591221   23171 round_trippers.go:447]     Upgrade: SPDY/3.1
I0702 11:45:52.591225   23171 round_trippers.go:447]     X-Stream-Protocol-Version: v4.channel.k8s.io
I0702 11:45:52.591229   23171 round_trippers.go:447]     Date: Wed, 03 Jul 2019 02:29:33 GMT
F0702 11:45:53.783725   23171 helpers.go:114] command terminated with exit code 7
</code></pre>

<p>The status of <code>kube-system</code> pod and the two pods providing the service is all <code>Running</code>, as following:</p>

<pre><code>root@master1:~/k8s-yaml# kubectl get --all-namespaces pod -o wide
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE     IP              NODE      NOMINATED NODE   READINESS GATES
default       kubia-6pjz9                       1/1     Running   0          5m35s   10.244.2.12     worker2   &lt;none&gt;           &lt;none&gt;
default       kubia-nwjcc                       1/1     Running   0          16h     10.244.1.27     worker1   &lt;none&gt;           &lt;none&gt;
kube-system   coredns-bccdc95cf-792px           1/1     Running   4          5d19h   10.244.0.11     master1   &lt;none&gt;           &lt;none&gt;
kube-system   coredns-bccdc95cf-bc76j           1/1     Running   4          5d19h   10.244.0.10     master1   &lt;none&gt;           &lt;none&gt;
kube-system   etcd-master1                      1/1     Running   8          5d19h   192.168.56.11   master1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-apiserver-master1            1/1     Running   7          5d19h   192.168.56.11   master1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-controller-manager-master1   1/1     Running   7          5d18h   192.168.56.11   master1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-amd64-9trbq       1/1     Running   3          5d18h   192.168.56.21   worker1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-amd64-btt74       1/1     Running   5          5d18h   192.168.56.11   master1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-amd64-swzzd       1/1     Running   0          3d20h   192.168.56.22   worker2   &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-27zfk                  1/1     Running   7          5d19h   192.168.56.11   master1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-79kt9                  1/1     Running   0          3d20h   192.168.56.22   worker2   &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-lx4gk                  1/1     Running   3          5d18h   192.168.56.21   worker1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-scheduler-master1            1/1     Running   7          5d19h   192.168.56.11   master1   &lt;none&gt;           &lt;none&gt;
</code></pre>

<p>My expected return is: when using <code>curl</code> access <code>svc</code>, <code>svc</code> can randomly access the <code>pod</code> on different nodes as follows:</p>

<pre><code>root@master1:~# kubectl get po
NAME          READY   STATUS    RESTARTS   AGE
kubia-nwjcc   1/1     Running   0          50m
kubia-zcpbb   1/1     Running   0          50m
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-zcpbb
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-zcpbb
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
</code></pre>

<p>And now the return results of five tests are as follows:</p>

<pre><code>root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
command terminated with exit code 7
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
command terminated with exit code 7
</code></pre>

hopgoldy
<p>This problem is solved. The official documentation of the <code>flannel</code> mentions that need to use <code>--iface</code> to specify the network card to be used when running in a <code>vagrant</code> type virtual machine. You can use the command <code>kubectl edit daemonset kube-flannel-ds-amd64 -n kube-system</code> to edit the <code>fllannel</code> configuration file. Then use <code>kubectl delete pod -n kube-system &lt;pod-name&gt;</code> to  all flannel pods. K8s will rebuild them. </p>

<p>You can find detailed answers in <a href="https://medium.com/@anilkreddyr/kubernetes-with-flannel-understanding-the-networking-part-1-7e1fe51820e4" rel="nofollow noreferrer">Kubernetes with Flannel</a> and <a href="https://coreos.com/flannel/docs/latest/troubleshooting.html" rel="nofollow noreferrer">flannel - troubleshooting</a>.</p>
