{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57841","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57841/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57841/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57841/events","html_url":"https://github.com/kubernetes/kubernetes/issues/57841","id":286005437,"node_id":"MDU6SXNzdWUyODYwMDU0Mzc=","number":57841,"title":"ipvs proxier doesn't respect graceful termination","user":{"login":"jsravn","id":522123,"node_id":"MDQ6VXNlcjUyMjEyMw==","avatar_url":"https://avatars3.githubusercontent.com/u/522123?v=4","gravatar_id":"","url":"https://api.github.com/users/jsravn","html_url":"https://github.com/jsravn","followers_url":"https://api.github.com/users/jsravn/followers","following_url":"https://api.github.com/users/jsravn/following{/other_user}","gists_url":"https://api.github.com/users/jsravn/gists{/gist_id}","starred_url":"https://api.github.com/users/jsravn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsravn/subscriptions","organizations_url":"https://api.github.com/users/jsravn/orgs","repos_url":"https://api.github.com/users/jsravn/repos","events_url":"https://api.github.com/users/jsravn/events{/privacy}","received_events_url":"https://api.github.com/users/jsravn/received_events","type":"User","site_admin":false},"labels":[{"id":755527763,"node_id":"MDU6TGFiZWw3NTU1Mjc3NjM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/ipvs","name":"area/ipvs","color":"0052cc","default":false},{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":116712108,"node_id":"MDU6TGFiZWwxMTY3MTIxMDg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/network","name":"sig/network","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"m1093782566","id":4953962,"node_id":"MDQ6VXNlcjQ5NTM5NjI=","avatar_url":"https://avatars1.githubusercontent.com/u/4953962?v=4","gravatar_id":"","url":"https://api.github.com/users/m1093782566","html_url":"https://github.com/m1093782566","followers_url":"https://api.github.com/users/m1093782566/followers","following_url":"https://api.github.com/users/m1093782566/following{/other_user}","gists_url":"https://api.github.com/users/m1093782566/gists{/gist_id}","starred_url":"https://api.github.com/users/m1093782566/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/m1093782566/subscriptions","organizations_url":"https://api.github.com/users/m1093782566/orgs","repos_url":"https://api.github.com/users/m1093782566/repos","events_url":"https://api.github.com/users/m1093782566/events{/privacy}","received_events_url":"https://api.github.com/users/m1093782566/received_events","type":"User","site_admin":false},"assignees":[{"login":"m1093782566","id":4953962,"node_id":"MDQ6VXNlcjQ5NTM5NjI=","avatar_url":"https://avatars1.githubusercontent.com/u/4953962?v=4","gravatar_id":"","url":"https://api.github.com/users/m1093782566","html_url":"https://github.com/m1093782566","followers_url":"https://api.github.com/users/m1093782566/followers","following_url":"https://api.github.com/users/m1093782566/following{/other_user}","gists_url":"https://api.github.com/users/m1093782566/gists{/gist_id}","starred_url":"https://api.github.com/users/m1093782566/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/m1093782566/subscriptions","organizations_url":"https://api.github.com/users/m1093782566/orgs","repos_url":"https://api.github.com/users/m1093782566/repos","events_url":"https://api.github.com/users/m1093782566/events{/privacy}","received_events_url":"https://api.github.com/users/m1093782566/received_events","type":"User","site_admin":false}],"milestone":null,"comments":24,"created_at":"2018-01-04T14:39:33Z","updated_at":"2018-07-03T04:09:13Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"<!-- This form is for bug reports and feature requests ONLY! \r\n\r\nIf you're looking for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).\r\n-->\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?**:\r\n/kind bug\r\n\r\n**What happened**:\r\nUpon removing an endpoint, the ipvs proxier immediately deletes the ipvs real server, causing all connections to get dropped.\r\n\r\n**What you expected to happen**:\r\nIt should allow the terminating pod to gracefully close connections, just like the iptables proxier.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n1. Enable ipvs proxier\r\n2. Create a keepalive / long lived connection to a pod (e.g. `while :; do echo -e \"GET / HTTP/1.1\\nhost: $host\\n\\n\"; sleep 5; echo; done | telnet $serviceip 80`)\r\n3. Delete that pod - observe the connection gets closed immediately, further requests will fail. On iptables proxier, it will continue to work (until the pod itself stops or closes the connection).\r\n\r\n**Anything else we need to know?**:\r\nThe ipvs proxier should instead be setting weight to 0, then reaping the stale real servers after some time period (that should be greater than any pod's graceful termination time). This may also fix the existing bug around UDP connections getting dropped prematurely (https://github.com/kubernetes/kubernetes/issues/45976).\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`): tested on 1.8, but same issue in 1.9 afaict\r\n- Cloud provider or hardware configuration: AWS\r\n- OS (e.g. from /etc/os-release): Ubuntu 16.04\r\n- Kernel (e.g. `uname -a`): 4.4.0\r\n- Install tools:\r\n- Others:\r\n\r\n  "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/402007534","html_url":"https://github.com/kubernetes/kubernetes/issues/57841#issuecomment-402007534","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57841","id":402007534,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMjAwNzUzNA==","user":{"login":"Lion-Wei","id":28696384,"node_id":"MDQ6VXNlcjI4Njk2Mzg0","avatar_url":"https://avatars1.githubusercontent.com/u/28696384?v=4","gravatar_id":"","url":"https://api.github.com/users/Lion-Wei","html_url":"https://github.com/Lion-Wei","followers_url":"https://api.github.com/users/Lion-Wei/followers","following_url":"https://api.github.com/users/Lion-Wei/following{/other_user}","gists_url":"https://api.github.com/users/Lion-Wei/gists{/gist_id}","starred_url":"https://api.github.com/users/Lion-Wei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lion-Wei/subscriptions","organizations_url":"https://api.github.com/users/Lion-Wei/orgs","repos_url":"https://api.github.com/users/Lion-Wei/repos","events_url":"https://api.github.com/users/Lion-Wei/events{/privacy}","received_events_url":"https://api.github.com/users/Lion-Wei/received_events","type":"User","site_admin":false},"created_at":"2018-07-03T04:09:13Z","updated_at":"2018-07-03T04:09:13Z","author_association":"CONTRIBUTOR","body":"@jsravn @m1093782566 @rramkumar1 @jhorwit2 Hi, guys. Recently I have been testing this issue, and got some result. Here is my test step:\r\n\r\n1. Create a pod with `liseCycle/preStop` specified and `terminationGracePeriodSeconds=300`. Expose this pod to a service.\r\n\r\n   ```yaml\r\n   apiVersion: v1\r\n   kind: Pod\r\n   ...\r\n       lifecycle:\r\n         preStop:\r\n           exec:\r\n             command:\r\n             - /bin/sh\r\n             - -c\r\n             - sleep 300\r\n   ...\r\n     terminationGracePeriodSeconds: 300\r\n   ```\r\n\r\n2. Check this pod and service.\r\n\r\n   ```shell\r\n   # k get svc\r\n   NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE\r\n   sourceip     ClusterIP   10.0.0.175   <none>        8080/TCP   8s\r\n   # k get pod -owide\r\n   NAME           READY     STATUS    RESTARTS   AGE       IP           NODE\r\n   sourceip       1/1       Running   0          45s       172.17.0.4   127.0.0.1\r\n   ```\r\n\r\n3. Use `telnet` test service clusterIP connection.\r\n\r\n   ```shell\r\n   # time telnet 10.0.0.133 8080\r\n   Trying 10.0.0.133...\r\n   Connected to 10.0.0.133.\r\n   Escape character is '^]'.\r\n   Connection closed by foreign host.\t\r\n   ```\r\n\r\n4. Use `kubectl` delete pod, considering what we specified in pod `preStop`, this pod should be in `terminating` status for 300s.\r\n\r\n   ```shell\r\n   # k delete pod sourceip\r\n   pod \"sourceip\" deleted\r\n   # k get pod\r\n   NAME           READY     STATUS        RESTARTS   AGE\r\n   sourceip       1/1       Terminating   0          3m\r\n   ```\r\n\r\n5. Check ipvs rules.\r\n\r\n   ```shell\r\n   # ipvs real server have been deleted\r\n   # ipvsadm -l\r\n   IP Virtual Server version 1.2.1 (size=4096)\r\n   Prot LocalAddress:Port Scheduler Flags\r\n     -> RemoteAddress:Port           Forward Weight ActiveConn InActConn\r\n   TCP  10.0.0.1:https rr\r\n     -> 10.162.199.82:6443           Masq    1      1          0\r\n   TCP  10.0.0.10:domain rr\r\n     -> 172.17.0.4:domain            Masq    1      0          0\r\n   TCP  10.0.0.133:http-alt rr\r\n   UDP  10.0.0.10:domain rr\r\n     -> 172.17.0.4:domain            Masq    1      0          0\r\n   # check ipvs connections\r\n   # ipvsadm -lnc\r\n   IPVS connection entries\r\n   pro expire state       source             virtual            destination\r\n   TCP 14:06  ESTABLISHED 10.0.0.133:58441   10.0.0.133:8080    172.17.0.5:80\r\n   ```\r\n\r\n   We can see ipvs realserver have been deleted, but `telnet` still in connection, and the connection is in state `ESTABLISHED` with an expire time 14m6s. \r\n\r\n6. After 5min, the container be deleted, telnet connection closed. Check ipvsadm connection.\r\n\r\n   ```shell\r\n   # telnet returned\r\n   # time telnet 10.0.0.133 8080\r\n   Trying 10.0.0.133...\r\n   Connected to 10.0.0.133.\r\n   Escape character is '^]'.\r\n   Connection closed by foreign host.\r\n\r\n   real    5m12.514s\r\n   user    0m0.000s\r\n   sys     0m0.003s\r\n\r\n   # ipvs connection in FIN_WAIT status, with expire time.\r\n   # ipvsadm -lnc\r\n   IPVS connection entries\r\n   pro expire state       source             virtual            destination\r\n   TCP 03:56  FIN_WAIT    10.0.0.133:59648   10.0.0.133:8080    172.17.0.5:80\r\n   ```\r\n\r\n\r\n\r\nSo, according to my test, I think ipvs proxier should have graceful termination for lone lived connection. If you have any question or suggestion about the test process, please let me know. And I'd like people introduce other test process. "}}
{"actor":{"display_login":"Lion-Wei"}}