{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/flannel/issues/533","repository_url":"https://api.github.com/repos/coreos/flannel","labels_url":"https://api.github.com/repos/coreos/flannel/issues/533/labels{/name}","comments_url":"https://api.github.com/repos/coreos/flannel/issues/533/comments","events_url":"https://api.github.com/repos/coreos/flannel/issues/533/events","html_url":"https://github.com/coreos/flannel/issues/533","id":184035148,"number":533,"title":"kube-flannel: host->remote-pod networking does not work on first node","user":{"login":"aaronlevy","id":374924,"avatar_url":"https://avatars.githubusercontent.com/u/374924?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlevy","html_url":"https://github.com/aaronlevy","followers_url":"https://api.github.com/users/aaronlevy/followers","following_url":"https://api.github.com/users/aaronlevy/following{/other_user}","gists_url":"https://api.github.com/users/aaronlevy/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlevy/subscriptions","organizations_url":"https://api.github.com/users/aaronlevy/orgs","repos_url":"https://api.github.com/users/aaronlevy/repos","events_url":"https://api.github.com/users/aaronlevy/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlevy/received_events","type":"User","site_admin":false},"labels":[{"id":468892382,"url":"https://api.github.com/repos/coreos/flannel/labels/component/kubernetes","name":"component/kubernetes","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-10-19T18:11:41Z","updated_at":"2016-11-23T23:57:19Z","closed_at":null,"body":"I've been trying to use kube-flannel as part of the bootkube, but I'm still seeing an issue on bootstrap.\r\n\r\nWhat I'm seeing is that on some nodes host-network --> remote pod (seemingly) hits some kind of race condition when bootstrapping using kube-flannel.\r\n\r\npod-to-pod (on same node and to remote nodes) works fine, this just affects host to remote pod, and it begins working after restarting kube-flannel on the remote node.\r\n## Repro steps:\r\n### Launch a cluster:\r\n\r\n```\r\ngo get github.com/kubernetes-incubator/bootkube\r\ngit remote add aaronlevy https://github.com/aaronlevy/bootkube\r\ncd $GOPATH/src/github.com/kubernetes-incubator/bootkube\r\ngit fetch aaronlevy\r\ngit checkout -b v1.4.0+flannel aaronlevy/v1.4.0+flannel\r\nmake clean all\r\ncd hack/multi-node\r\n./bootkube-up\r\n```\r\n### Start nginx pods for testing:\r\n\r\n```\r\nkubectl --kubeconfig=cluster/auth/kubeconfig run n1 --image=nginx --replicas=2\r\n```\r\n\r\nYou should will have one nginx pod on each node, but can verify by checking nodeIP of each pod:\r\n\r\n```\r\nkubectl --kubeconfig=cluster/auth/kubeconfig get pods -owide\r\n```\r\n\r\nMake note of the podIP for each of the pods above, and which node they are assigned to. E.g.:\r\n\r\n```\r\nNAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE       IP             NODE\r\ndefault       n1-1110676790-e0w0c                        1/1       Running   0          12m       10.2.1.2       172.17.4.201\r\ndefault       n1-1110676790-s3qfi                        1/1       Running   0          12m       10.2.0.5       172.17.4.101\r\n```\r\n- The 10.2.0.5 is assigned to the \"controller\" (c1) node (172.17.4.101)\r\n- The 10.2.1.2 is assinged to the \"worker\" (w1) node (172.17.4.201)\r\n### Test the routability from each host:\r\n- note: This seems to fail in different ways sometimes (e.g. both sides, or just one side). Typically from what I've seen the common case is master (host) --> worker (pod) fails.\r\n\r\nFrom controller node (route to remote pod likely doesn't work):\r\n\r\n```\r\nvagrant ssh c1\r\n$ curl 10.2.0.5 # same node, should work\r\n$ curl 10.2.1.2 # different node, probably doesn't work\r\n```\r\n\r\nFrom worker node (all routes should work):\r\n\r\n```\r\nvagrant ssh w1\r\n$ curl 10.2.0.5 # different node, probably works\r\n$ curl 10.2.1.2 # same node, should work\r\n```\r\n### Now to resolve the issue:\r\n\r\nKill the kube-flannel pod on the failing \"remote destination\" side (in this case, worker)\r\n\r\n```\r\nkubectl --kubeconfig=cluster/auth/kubeconfig --namespace=kube-system delete pod kube-flannel-abcde\r\n```\r\n\r\nWait until the kube-flannel daemonset re-launches a replacment pod, then you should be able to re-do the tests above and all networking should work.\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/flannel/issues/comments/262658656","html_url":"https://github.com/coreos/flannel/issues/533#issuecomment-262658656","issue_url":"https://api.github.com/repos/coreos/flannel/issues/533","id":262658656,"user":{"login":"philips","id":33544,"avatar_url":"https://avatars.githubusercontent.com/u/33544?v=3","gravatar_id":"","url":"https://api.github.com/users/philips","html_url":"https://github.com/philips","followers_url":"https://api.github.com/users/philips/followers","following_url":"https://api.github.com/users/philips/following{/other_user}","gists_url":"https://api.github.com/users/philips/gists{/gist_id}","starred_url":"https://api.github.com/users/philips/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philips/subscriptions","organizations_url":"https://api.github.com/users/philips/orgs","repos_url":"https://api.github.com/users/philips/repos","events_url":"https://api.github.com/users/philips/events{/privacy}","received_events_url":"https://api.github.com/users/philips/received_events","type":"User","site_admin":false},"created_at":"2016-11-23T23:57:19Z","updated_at":"2016-11-23T23:57:19Z","body":"Changing the backend to UDP causes the same behavior! This is useful information!\r\n\r\nIt means we can likely point our fingers at @mikedanese :-P (love you mike)"}}
{"actor":{"display_login":"philips"}}