{"action":"created","issue":{"url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/521","repository_url":"https://api.github.com/repos/cloudnativelabs/kube-router","labels_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/521/labels{/name}","comments_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/521/comments","events_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/521/events","html_url":"https://github.com/cloudnativelabs/kube-router/issues/521","id":354015178,"node_id":"MDU6SXNzdWUzNTQwMTUxNzg=","number":521,"title":"TCP Connection reset after connection was idle for more that 16 minutes","user":{"login":"tanapoln","id":1481232,"node_id":"MDQ6VXNlcjE0ODEyMzI=","avatar_url":"https://avatars0.githubusercontent.com/u/1481232?v=4","gravatar_id":"","url":"https://api.github.com/users/tanapoln","html_url":"https://github.com/tanapoln","followers_url":"https://api.github.com/users/tanapoln/followers","following_url":"https://api.github.com/users/tanapoln/following{/other_user}","gists_url":"https://api.github.com/users/tanapoln/gists{/gist_id}","starred_url":"https://api.github.com/users/tanapoln/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tanapoln/subscriptions","organizations_url":"https://api.github.com/users/tanapoln/orgs","repos_url":"https://api.github.com/users/tanapoln/repos","events_url":"https://api.github.com/users/tanapoln/events{/privacy}","received_events_url":"https://api.github.com/users/tanapoln/received_events","type":"User","site_admin":false},"labels":[{"id":585287813,"node_id":"MDU6TGFiZWw1ODUyODc4MTM=","url":"https://api.github.com/repos/cloudnativelabs/kube-router/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-08-25T14:08:13Z","updated_at":"2018-11-29T01:41:52Z","closed_at":null,"author_association":"NONE","body":"My applications communicates via gRPC and the connections was not active all time. When a specific connection was idle for more than 16 minutes, it would get RST packet after try to write to that connection.\r\n\r\nA tcpdump on client-side (a sender) shows that RST packet comes from remote IP but tcpdump on server-side (a receiver) show nothing about RST packet. I also notice that TTL of RST packet is not decrease (default TTL=64 in my server).\r\n\r\nI suspect that it was not sent from remote but in a local itself. You can check with this tcpdump capture: [tcpdump.zip](https://github.com/cloudnativelabs/kube-router/files/2320742/tcpdump.zip). \r\n\r\nMore Informations about tcpdump packet:\r\n- server pod's IP is 100.96.25.248\r\n- client pod's IP is 100.96.12.176\r\n- service's IP is 100.65.151.237\r\n- kube-bridge's mac address is 0a:58:64:60:0c:01\r\n\r\nThe problem I found can reproduce reliably, you can use attached Kubernetes YAML to test and see result: [grpc-test.yml.zip](https://github.com/cloudnativelabs/kube-router/files/2320741/grpc-test.yml.zip). \r\n\r\nThe code of grpc-test do simply task: client send RPC to server every specific interval which can be configure via environment variable `INTERVAL_MS`. The connection will establish at the beginning when client was started.\r\n\r\nIt works fine if I switch to headless service or using other CNI (tested with kubenet).\r\n\r\n### Environments:\r\nCloud: AWS\r\nKubernetes: 1.9.3\r\nDocker: 17.3.2\r\nLinux Kernel: 4.4.121-k8s (Debian jessie)\r\nKube-router: 0.1.0\r\n\r\nMy cluster was provisioned using Kops with this command:\r\n```\r\nexport NAME=testcluster.k8s.local\r\nexport KOPS_STATE_STORE=s3://my-testcluster-kops-xxx\r\nkops create cluster --dns private --api-loadbalancer-type public --kubernetes-version=1.9.3 --zones ap-southeast-1b --node-count 1 --node-size t2.medium --master-size t2.medium --networking kube-router -t public ${NAME}\r\n```"},"comment":{"url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/comments/442673738","html_url":"https://github.com/cloudnativelabs/kube-router/issues/521#issuecomment-442673738","issue_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/521","id":442673738,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MjY3MzczOA==","user":{"login":"znoodl","id":3964419,"node_id":"MDQ6VXNlcjM5NjQ0MTk=","avatar_url":"https://avatars3.githubusercontent.com/u/3964419?v=4","gravatar_id":"","url":"https://api.github.com/users/znoodl","html_url":"https://github.com/znoodl","followers_url":"https://api.github.com/users/znoodl/followers","following_url":"https://api.github.com/users/znoodl/following{/other_user}","gists_url":"https://api.github.com/users/znoodl/gists{/gist_id}","starred_url":"https://api.github.com/users/znoodl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/znoodl/subscriptions","organizations_url":"https://api.github.com/users/znoodl/orgs","repos_url":"https://api.github.com/users/znoodl/repos","events_url":"https://api.github.com/users/znoodl/events{/privacy}","received_events_url":"https://api.github.com/users/znoodl/received_events","type":"User","site_admin":false},"created_at":"2018-11-29T01:41:52Z","updated_at":"2018-11-29T01:41:52Z","author_association":"NONE","body":"@roffe Thank you, I figure it out. Now I set tcp_keepalive in my applications, it's working well."}}
{"actor":{"display_login":"znoodl"}}