{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19029","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19029/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19029/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19029/events","html_url":"https://github.com/kubernetes/kubernetes/issues/19029","id":123559811,"number":19029,"title":"iptables kube-proxy could handle UDP backend changes better","user":{"login":"joshk0","id":5368926,"avatar_url":"https://avatars.githubusercontent.com/u/5368926?v=3","gravatar_id":"","url":"https://api.github.com/users/joshk0","html_url":"https://github.com/joshk0","followers_url":"https://api.github.com/users/joshk0/followers","following_url":"https://api.github.com/users/joshk0/following{/other_user}","gists_url":"https://api.github.com/users/joshk0/gists{/gist_id}","starred_url":"https://api.github.com/users/joshk0/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joshk0/subscriptions","organizations_url":"https://api.github.com/users/joshk0/orgs","repos_url":"https://api.github.com/users/joshk0/repos","events_url":"https://api.github.com/users/joshk0/events{/privacy}","received_events_url":"https://api.github.com/users/joshk0/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/networking","name":"area/networking","color":"006b75"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/help-wanted","name":"help-wanted","color":"006b75"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"freehan","id":1332699,"avatar_url":"https://avatars.githubusercontent.com/u/1332699?v=3","gravatar_id":"","url":"https://api.github.com/users/freehan","html_url":"https://github.com/freehan","followers_url":"https://api.github.com/users/freehan/followers","following_url":"https://api.github.com/users/freehan/following{/other_user}","gists_url":"https://api.github.com/users/freehan/gists{/gist_id}","starred_url":"https://api.github.com/users/freehan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freehan/subscriptions","organizations_url":"https://api.github.com/users/freehan/orgs","repos_url":"https://api.github.com/users/freehan/repos","events_url":"https://api.github.com/users/freehan/events{/privacy}","received_events_url":"https://api.github.com/users/freehan/received_events","type":"User","site_admin":false},"milestone":null,"comments":9,"created_at":"2015-12-22T21:31:48Z","updated_at":"2016-03-04T01:04:02Z","closed_at":null,"body":"I encountered a strange issue after opting-in to kube-proxy iptables support.\r\n\r\nSteps to repro (I think):\r\n1. Create a Service for a UDP port (say 8125; in my case, this was statsd)\r\n2. Create pods that bind to this Service\r\n3. Start a client Pod that uses the Service, e.g. via environment variable which specifies the VIP for the Service. This client writes a UDP packet to the Service every 10 seconds using the same socket (e.g. DialUDP is only called once)\r\n4. Log in to the node. Use [ngrep](http://conntrack-tools.netfilter.org/) -q '' udp port 8125  to view the outgoing UDP traffic from the box to the service port. Observe that rewriting has occurred, and the packet is destined to one of the concrete endpoints specified in the Kubernetes model.\r\n5. Delete that endpoint by, for example, killing the corresponding Pod which the client is communicating with\r\n6. Observe that the Pod's IP no longer appears in endpoints (e.g. using kubectl)\r\n7. BUG: Observe that ngrep continues to report that the packets are being rewritten to the old endpoint.\r\n8. BUG (corollary): Observe that [conntrack](http://conntrack-tools.netfilter.org/) -L -d SERVICE_VIP shows that same socket being routed to the old endpoint.\r\n9. WORKAROUND: Restart the client Pod, or have the client Pod call DialUDP each time it needs to send data.\r\n\r\nThe only solution i can see is that kube-proxy, when it rewrites iptables rules based on endpoints changes, needs to reset connections between local sockets and destroyed endpoints.\r\n\r\nThis didn't happen with the userspace kube-proxy, because kube-proxy was accepting the packets locally regardless of the endpoints, and would always use the latest endpoints information to forward the packet on.\r\n\r\nSorry for the long bug report, but I think it should be pretty clear by now if you've made it here. :)"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/192043591","html_url":"https://github.com/kubernetes/kubernetes/issues/19029#issuecomment-192043591","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19029","id":192043591,"user":{"login":"thockin","id":5595220,"avatar_url":"https://avatars.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"created_at":"2016-03-04T01:04:02Z","updated_at":"2016-03-04T01:04:02Z","body":"oh, fantastic.  Way better answer than I expected.\n\nOn Thu, Mar 3, 2016 at 5:01 PM, Minhan Xia <notifications@github.com> wrote:\n\n> I will submit PR shortly.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/kubernetes/kubernetes/issues/19029#issuecomment-192042904>\n> .\n>\n"}}