{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/59811","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/59811/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/59811/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/59811/events","html_url":"https://github.com/kubernetes/kubernetes/issues/59811","id":296798473,"node_id":"MDU6SXNzdWUyOTY3OTg0NzM=","number":59811,"title":"Service fails to estabilish connection (timeouts) after endpoint pod is recreated by deployment.","user":{"login":"OJezu","id":3941091,"node_id":"MDQ6VXNlcjM5NDEwOTE=","avatar_url":"https://avatars2.githubusercontent.com/u/3941091?v=4","gravatar_id":"","url":"https://api.github.com/users/OJezu","html_url":"https://github.com/OJezu","followers_url":"https://api.github.com/users/OJezu/followers","following_url":"https://api.github.com/users/OJezu/following{/other_user}","gists_url":"https://api.github.com/users/OJezu/gists{/gist_id}","starred_url":"https://api.github.com/users/OJezu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OJezu/subscriptions","organizations_url":"https://api.github.com/users/OJezu/orgs","repos_url":"https://api.github.com/users/OJezu/repos","events_url":"https://api.github.com/users/OJezu/events{/privacy}","received_events_url":"https://api.github.com/users/OJezu/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":778118402,"node_id":"MDU6TGFiZWw3NzgxMTg0MDI=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/lifecycle/rotten","name":"lifecycle/rotten","color":"604460","default":false},{"id":621158926,"node_id":"MDU6TGFiZWw2MjExNTg5MjY=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/azure","name":"sig/azure","color":"d2b48c","default":false},{"id":116712108,"node_id":"MDU6TGFiZWwxMTY3MTIxMDg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/network","name":"sig/network","color":"d2b48c","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2018-02-13T16:22:21Z","updated_at":"2018-07-16T12:38:52Z","closed_at":"2018-07-16T12:38:52Z","author_association":"NONE","body":"<!-- This form is for bug reports and feature requests ONLY! \r\n\r\nIf you're looking for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).\r\n-->\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?**:\r\n\r\n/kind bug\r\n\r\n**What happened**:\r\n\r\nAfter replacement by deployment of a pod which is the only endpoint for a service, the service stopped serving traffic to and from the endpoint. The service was used by one other pod. The issue resolved itself after 2 hours or trying to use the service from some other pods then the one that had the original connection problems (?).\r\n\r\n**What you expected to happen**:\r\n\r\nService still serving from the endpoint, as pod is replaced by deployment.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nI don't know, we have problems with this one particular service and pod, but everything else seems to be working in order, including virtually identical configurations for other tiers of our services. It's not the first time the service cannot reach the endpoint pod after endpoint pod restart.\r\n\r\n**Anything else we need to know?**:\r\nThe service dns name is resolvable, the service is not usable by it's IP address, but the endpoint pod is working fine when using it's IP directly.\r\n\r\n```\r\n~ $ kubectl exec -ti (kubectl get pod -l \"name=canary-samus-varnish\" -o name | sed -E 's%pods?/%%' | tail -n 1) -- bash\r\nroot@canary-samus-varnish-7cb56f97c6-5cnzp:/# host canary-kabuto                                                                                                                                                  \r\ncanary-kabuto.default.svc.cluster.local has address 10.0.121.77\r\nroot@canary-samus-varnish-7cb56f97c6-5cnzp:/# curl -sL http://canary-kabuto/ \r\n^C (timeout)\r\nroot@canary-samus-varnish-7cb56f97c6-5cnzp:/# curl -sL http://10.0.121.77/  \r\n^C (timeout)\r\nroot@canary-samus-varnish-7cb56f97c6-5cnzp:/# curl -sL http://10.240.0.66/\r\n<!doctype html><html><body>Hello world!</body></html>\r\n```\r\n\r\nService metadata from kubectl:\r\n```\r\n~ $ kubectl get service canary-kabuto\r\nNAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE\r\ncanary-kabuto   ClusterIP   10.0.121.77   <none>        80/TCP    5d\r\n\r\n~ $ kubectl get endpoints canary-kabuto\r\nNAME            ENDPOINTS        AGE\r\ncanary-kabuto   10.240.0.66:80   5d\r\n```\r\n\r\nBoth pods (the pod behind service, and one that's trying to use that service) are by chance on the same node:\r\n```\r\n~ $ kubectl describe pod canary-samus-varnish-7cb56f97c6-5cnzp | grep Node\r\nNode:           k8s-agent-10531284-1/10.240.0.65\r\nNode-Selectors:  <none>\r\n\r\n~ $ kubectl describe pod canary-kabuto-6bc965c4fb-v2l8t | grep Node\r\nNode:           k8s-agent-10531284-1/10.240.0.65\r\nNode-Selectors:  <none>\r\n```\r\n\r\nNode's iptables pertaining to the service look fine to me:\r\n```\r\nroot@k8s-agent-10531284-1:~# iptables-save | grep canary-kabuto\r\n-A KUBE-SEP-CG6CX7OLHU57F3EX -s 10.240.0.66/32 -m comment --comment \"default/canary-kabuto:http\" -j KUBE-MARK-MASQ\r\n-A KUBE-SEP-CG6CX7OLHU57F3EX -p tcp -m comment --comment \"default/canary-kabuto:http\" -m tcp -j DNAT --to-destination 10.240.0.66:80\r\n-A KUBE-SERVICES ! -s 10.240.0.0/12 -d 10.0.121.77/32 -p tcp -m comment --comment \"default/canary-kabuto:http cluster IP\" -m tcp --dport 80 -j KUBE-MARK-MASQ\r\n-A KUBE-SERVICES -d 10.0.121.77/32 -p tcp -m comment --comment \"default/canary-kabuto:http cluster IP\" -m tcp --dport 80 -j KUBE-SVC-MMLUSW4QS4ZMIPKO\r\n-A KUBE-SVC-MMLUSW4QS4ZMIPKO -m comment --comment \"default/canary-kabuto:http\" -j KUBE-SEP-CG6CX7OLHU57F3EX\r\n```\r\n\r\nIssue resolved itself as I was trying to check for the sake of this issue report, if the same service behaviour can be observed from other pods/nodes. Not only service is behaving as expected on other nodes and pods, after using it from other pods it started working on the originally affected pod. I'm not sure if there is causation there, as I was not testing for a few minutes prior. That happened roughly 2 hours from the onset of the problem.\r\n\r\nOutputs above were gathered as issue was ongoing, and I did not try to actively resolve the issue by changing configuration, restarting pods or services etc.\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n```\r\nClient Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.2\", GitCommit:\"bdaeafa71f6c7c04636251031f93464384d54963\", GitTreeState:\"clean\", BuildDate:\"2017-10-24T19:48:57Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.4\", GitCommit:\"9befc2b8928a9426501d3bf62f72849d5cbcd5a3\", GitTreeState:\"clean\", BuildDate:\"2017-11-20T05:17:43Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n- Cloud provider or hardware configuration: \r\nAzure, DSv2 VMs\r\n- OS (e.g. from /etc/os-release):\r\nUbuntu 16.04.3 LTS\r\n- Kernel (e.g. `uname -a`):\r\nLinux k8s-agent-10531284-1 4.13.0-1007-azure #9-Ubuntu SMP Thu Jan 25 10:47:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n- Install tools:\r\nACS Engine, run from local machine, to have k8s 1.8.4 and managed disks\r\n- Others:\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/405233735","html_url":"https://github.com/kubernetes/kubernetes/issues/59811#issuecomment-405233735","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/59811","id":405233735,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTIzMzczNQ==","user":{"login":"OJezu","id":3941091,"node_id":"MDQ6VXNlcjM5NDEwOTE=","avatar_url":"https://avatars2.githubusercontent.com/u/3941091?v=4","gravatar_id":"","url":"https://api.github.com/users/OJezu","html_url":"https://github.com/OJezu","followers_url":"https://api.github.com/users/OJezu/followers","following_url":"https://api.github.com/users/OJezu/following{/other_user}","gists_url":"https://api.github.com/users/OJezu/gists{/gist_id}","starred_url":"https://api.github.com/users/OJezu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OJezu/subscriptions","organizations_url":"https://api.github.com/users/OJezu/orgs","repos_url":"https://api.github.com/users/OJezu/repos","events_url":"https://api.github.com/users/OJezu/events{/privacy}","received_events_url":"https://api.github.com/users/OJezu/received_events","type":"User","site_admin":false},"created_at":"2018-07-16T12:38:52Z","updated_at":"2018-07-16T12:38:52Z","author_association":"NONE","body":"Issue ceased after migration to AKS from ACS."}}
{"actor":{"display_login":"OJezu"}}