{"action":"created","issue":{"url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/544","repository_url":"https://api.github.com/repos/cloudnativelabs/kube-router","labels_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/544/labels{/name}","comments_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/544/comments","events_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/544/events","html_url":"https://github.com/cloudnativelabs/kube-router/issues/544","id":364706059,"node_id":"MDU6SXNzdWUzNjQ3MDYwNTk=","number":544,"title":"Apache Bench can fill up ipvs service proxy in seconds","user":{"login":"neeseius","id":35146131,"node_id":"MDQ6VXNlcjM1MTQ2MTMx","avatar_url":"https://avatars1.githubusercontent.com/u/35146131?v=4","gravatar_id":"","url":"https://api.github.com/users/neeseius","html_url":"https://github.com/neeseius","followers_url":"https://api.github.com/users/neeseius/followers","following_url":"https://api.github.com/users/neeseius/following{/other_user}","gists_url":"https://api.github.com/users/neeseius/gists{/gist_id}","starred_url":"https://api.github.com/users/neeseius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/neeseius/subscriptions","organizations_url":"https://api.github.com/users/neeseius/orgs","repos_url":"https://api.github.com/users/neeseius/repos","events_url":"https://api.github.com/users/neeseius/events{/privacy}","received_events_url":"https://api.github.com/users/neeseius/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2018-09-28T01:16:41Z","updated_at":"2018-11-22T23:39:38Z","closed_at":"2018-11-22T20:42:01Z","author_association":"NONE","body":"I am not sure if I have something configured wrong but here is my Centos7 physical node and kube-router agent setup:\r\n\r\n[ipvsadm package]\r\n$ rpm -q ipvsadm\r\nipvsadm-1.27-7.el7.x86_64\r\n\r\n[kube router process and options]\r\n$ ps -ocommand= -C kube-router\r\n/usr/local/bin/kube-router --run-router=true --run-firewall=true --run-service-proxy=true --kubeconfig=/etc/kubernetes/kube-router.kubeconfig --hostname-override=node6 --enable-overlay=true\r\n\r\n[service]\r\nkubectl get svc\r\nNAME        TYPE           CLUSTER-IP       EXTERNAL-IP                            PORT(S)        AGE\r\ntest-svc    NodePort       172.30.176.114   <none>                               80:30530/TCP   7h\r\n\r\n[ipvs]\r\n$ ipvsadm -ln | head -n 1\r\nIP Virtual Server version 1.2.1 (size=4096)\r\n\r\n[ipvs service]\r\n$ ipvsadm -ln | grep -A1 30530\r\nTCP  10.200.1.146:30530 rr\r\n  -> 172.32.9.68:80               Masq    1      0          0\r\n\r\n\r\nIf I use apache bench with tcp keepalive all is swell and absurdly fast, posting over 10,000 requests per second and ipvsadm will show stats like below during such a test:\r\n$ ipvsadm -ln | grep -A1 30530\r\nTCP  10.200.1.146:30530 rr\r\n  -> 172.32.9.68:80               Masq    1      0          757\r\n\r\nHowever if I run the same test without keep-alive then \"InActConn\" jumps up to 14000 within a few seconds and up until that point things are very fast, but after that point the virtual server just completely hangs up and stops responding to requests until \"InActConn\" drops back below 14000. This happens if I run apache bench on the node itself and hit the clusterIp, or if I run it from a random server and hit the nodeport.\r\n\r\n---ipvs\r\n$ ipvsadm -ln | grep -A1 30530\r\nTCP  10.200.1.146:30530 rr\r\n  -> 172.32.9.68:80               Masq    1      0          14115\r\n\r\n--- apache bench output\r\nab -c 100 -n 20000 http://node6:30530\r\nThis is ApacheBench, Version 2.3 <$Revision: 1826891 $>\r\nCopyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/\r\nLicensed to The Apache Software Foundation, http://www.apache.org/\r\n\r\nBenchmarking node6 (be patient)\r\nCompleted 2000 requests\r\nCompleted 4000 requests\r\nCompleted 6000 requests\r\nCompleted 8000 requests\r\nCompleted 10000 requests\r\nCompleted 12000 requests\r\nCompleted 14000 requests\r\nCompleted 16000 requests\r\nCompleted 18000 requests\r\nCompleted 20000 requests\r\nFinished 20000 requests\r\n\r\n\r\nServer Software:        Apache/2.4.34\r\nServer Hostname:        node6\r\nServer Port:            30530\r\n\r\nDocument Path:          /\r\nDocument Length:        2512 bytes\r\n\r\nConcurrency Level:      100\r\nTime taken for tests:   63.914 seconds\r\nComplete requests:      20000\r\nFailed requests:        0\r\nTotal transferred:      55860000 bytes\r\nHTML transferred:       50240000 bytes\r\nRequests per second:    312.92 [#/sec] (mean)\r\nTime per request:       319.569 [ms] (mean)\r\nTime per request:       3.196 [ms] (mean, across all concurrent requests)\r\nTransfer rate:          853.50 [Kbytes/sec] received\r\n\r\nConnection Times (ms)\r\n              min  mean[+/-sd] median   max\r\nConnect:        1  311 456.6     10    1005\r\nProcessing:     1    7   4.5      8      36\r\nWaiting:        0    7   4.5      8      36\r\nTotal:          2  318 453.1     19    1010\r\n\r\nPercentage of the requests served within a certain time (ms)\r\n  50%     19\r\n  66%     21\r\n  75%   1003\r\n  80%   1004\r\n  90%   1004\r\n  95%   1005\r\n  98%   1005\r\n  99%   1006\r\n 100%   1010 (longest request)\r\n"},"comment":{"url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/comments/441134166","html_url":"https://github.com/cloudnativelabs/kube-router/issues/544#issuecomment-441134166","issue_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/544","id":441134166,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTEzNDE2Ng==","user":{"login":"igoratencompass","id":3914396,"node_id":"MDQ6VXNlcjM5MTQzOTY=","avatar_url":"https://avatars0.githubusercontent.com/u/3914396?v=4","gravatar_id":"","url":"https://api.github.com/users/igoratencompass","html_url":"https://github.com/igoratencompass","followers_url":"https://api.github.com/users/igoratencompass/followers","following_url":"https://api.github.com/users/igoratencompass/following{/other_user}","gists_url":"https://api.github.com/users/igoratencompass/gists{/gist_id}","starred_url":"https://api.github.com/users/igoratencompass/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/igoratencompass/subscriptions","organizations_url":"https://api.github.com/users/igoratencompass/orgs","repos_url":"https://api.github.com/users/igoratencompass/repos","events_url":"https://api.github.com/users/igoratencompass/events{/privacy}","received_events_url":"https://api.github.com/users/igoratencompass/received_events","type":"User","site_admin":false},"created_at":"2018-11-22T23:39:38Z","updated_at":"2018-11-22T23:39:38Z","author_association":"NONE","body":"> Special sauce for me seems to be:\r\n> \r\n> net.ipv4.vs.conntrack=1\r\n> net.ipv4.vs.conn_reuse_mode=0\r\n> net.ipv4.vs.expire_nodest_conn=1\r\n\r\nDon't understand how can the last two be used in the same time when the kernel docs about `conn_reuse_mode` clearly says:\r\n\r\n```\r\n       0: disable any special handling on port reuse. The new\r\n\tconnection will be delivered to the same real server that was\r\n\tservicing the previous connection. **This will effectively\r\n\tdisable expire_nodest_conn**\r\n```\r\nso by setting `net.ipv4.vs.conn_reuse_mode=0` you disable `net.ipv4.vs.expire_nodest_conn`."}}
{"actor":{"display_login":"igoratencompass"}}