{"action":"created","issue":{"url":"https://api.github.com/repos/k8sp/issues/issues/27","repository_url":"https://api.github.com/repos/k8sp/issues","labels_url":"https://api.github.com/repos/k8sp/issues/issues/27/labels{/name}","comments_url":"https://api.github.com/repos/k8sp/issues/issues/27/comments","events_url":"https://api.github.com/repos/k8sp/issues/issues/27/events","html_url":"https://github.com/k8sp/issues/issues/27","id":158108295,"number":27,"title":"Kubernetes Network performance test","user":{"login":"Yancey1989","id":1426912,"avatar_url":"https://avatars.githubusercontent.com/u/1426912?v=3","gravatar_id":"","url":"https://api.github.com/users/Yancey1989","html_url":"https://github.com/Yancey1989","followers_url":"https://api.github.com/users/Yancey1989/followers","following_url":"https://api.github.com/users/Yancey1989/following{/other_user}","gists_url":"https://api.github.com/users/Yancey1989/gists{/gist_id}","starred_url":"https://api.github.com/users/Yancey1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yancey1989/subscriptions","organizations_url":"https://api.github.com/users/Yancey1989/orgs","repos_url":"https://api.github.com/users/Yancey1989/repos","events_url":"https://api.github.com/users/Yancey1989/events{/privacy}","received_events_url":"https://api.github.com/users/Yancey1989/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"typhoonzero","id":13348433,"avatar_url":"https://avatars.githubusercontent.com/u/13348433?v=3","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false},"assignees":[{"login":"typhoonzero","id":13348433,"avatar_url":"https://avatars.githubusercontent.com/u/13348433?v=3","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false}],"milestone":null,"comments":15,"created_at":"2016-06-02T10:27:15Z","updated_at":"2016-07-26T13:30:33Z","closed_at":null,"body":"Kubernetes 应该如何在flannel，Calico等底层网络模块中去选择，性能如何？\r\n- [x] test perf under flannel(vxlan mode)\r\n- [x] test network ping latency and bandwidth under GCE mode\r\n- [x] test network perf under flanner(host-gw mode)\r\n- [x] test network perf under calico\r\n- [ ] test network perf under L2 network"},"comment":{"url":"https://api.github.com/repos/k8sp/issues/issues/comments/235267576","html_url":"https://github.com/k8sp/issues/issues/27#issuecomment-235267576","issue_url":"https://api.github.com/repos/k8sp/issues/issues/27","id":235267576,"user":{"login":"typhoonzero","id":13348433,"avatar_url":"https://avatars.githubusercontent.com/u/13348433?v=3","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false},"created_at":"2016-07-26T13:30:33Z","updated_at":"2016-07-26T13:30:33Z","body":"使用下面的脚本卸载iptables的内核模块\r\n```\r\n#!/bin/bash\r\n\r\n#/sbin/modprobe --version 2>&1 | grep -q module-init-tools \\\r\n#    && NEW_MODUTILS=1 \\\r\n#    || NEW_MODUTILS=0\r\n\r\nNEW_MODUTILS=1\r\nIPTABLES=iptables\r\nIPV＝ip\r\nPROC_IPTABLES_NAMES=/proc/net/${IPV}_tables_names\r\nNF_TABLES=$(cat \"$PROC_IPTABLES_NAMES\" 2>/dev/null)\r\n\r\necho $PROC_IPTABLES_NAMES\r\necho $NF_TABLES\r\n\r\nrmmod_r() {\r\n    # Unload module with all referring modules.\r\n    # At first all referring modules will be unloaded, then the module itself.\r\n    local mod=$1\r\n    local ret=0\r\n    local ref=\r\n\r\n    # Get referring modules.\r\n    # New modutils have another output format.\r\n    [ $NEW_MODUTILS = 1 ] \\\r\n        && ref=$(lsmod | awk \"/^${mod}/ { print \\$4; }\" | tr ',' ' ') \\\r\n        || ref=$(lsmod | grep ^${mod} | cut -d \"[\" -s -f 2 | cut -d \"]\" -s -f 1)\r\n\r\n    # recursive call for all referring modules\r\n    for i in $ref; do\r\n        rmmod_r $i\r\n        let ret+=$?;\r\n    done\r\n\r\n    # Unload module.\r\n    # The extra test is for 2.6: The module might have autocleaned,\r\n    # after all referring modules are unloaded.\r\n    if grep -q \"^${mod}\" /proc/modules ; then\r\n        modprobe -r $mod > /dev/null 2>&1\r\n        res=$?\r\n        [ $res -eq 0 ] || echo -n \" $mod\"\r\n        let ret+=$res;\r\n    fi\r\n\r\n    return $ret\r\n}\r\n\r\nflush_n_delete() {\r\n    # Flush firewall rules and delete chains.\r\n    [ ! -e \"$PROC_IPTABLES_NAMES\" ] && return 0\r\n\r\n    # Check if firewall is configured (has tables)\r\n    [ -z \"$NF_TABLES\" ] && return 1\r\n\r\n    echo -n $\"${IPTABLES}: Flushing firewall rules: \"\r\n    ret=0\r\n    # For all tables\r\n    for i in $NF_TABLES; do\r\n        # Flush firewall rules.\r\n\t$IPTABLES -t $i -F;\r\n\tlet ret+=$?;\r\n\r\n        # Delete firewall chains.\r\n\t$IPTABLES -t $i -X;\r\n\tlet ret+=$?;\r\n\r\n\t# Set counter to zero.\r\n\t$IPTABLES -t $i -Z;\r\n\tlet ret+=$?;\r\n    done\r\n\r\n    #[ $ret -eq 0 ] && success || failure\r\n    echo\r\n    return $ret\r\n}\r\n\r\nflush_n_delete\r\n\r\nrmmod_r iptable_nat\r\nrmmod_r nf_nat_ipv4\r\nrmmod_r iptable_filter\r\nrmmod_r ip_tables\r\nrmmod_r x_tables\r\nrmmod_r nf_nat\r\nrmmod_r nf_conntrack\r\n\r\n```\r\n在docker.service中，增加下面的drop-in配置，在服务启动之后卸载netfilter和conntrack的内核模块，就可以完全禁止docker使用iptables：\r\n```\r\n# vim /etc/systemd/system/docker.service.d/modprobe.conf\r\n\r\n[Service]\r\nExecStartPost=-/bin/systemctl restart iptables\r\nExecStartPost=-/sbin/modprobe -r nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_conntrack\r\n```\r\n然后重启docker服务：\r\n```\r\n# systemctl daemon-reload\r\n# systemctl restart docker\r\n# lsmod | grep conntrack\r\n```\r\n发现相关的内核模块已经卸载掉了。之后再执行ab测试，结果就可以达到物理机性能的100%！！！\r\n```\r\n[root@k8s-node-1 ~]# ab -c 50 -n 90000 http://172.24.103.2/\r\nThis is ApacheBench, Version 2.3 <$Revision: 1706008 $>\r\nCopyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/\r\nLicensed to The Apache Software Foundation, http://www.apache.org/\r\n\r\nBenchmarking 172.24.103.2 (be patient)\r\nCompleted 9000 requests\r\nCompleted 18000 requests\r\nCompleted 27000 requests\r\nCompleted 36000 requests\r\nCompleted 45000 requests\r\nCompleted 54000 requests\r\nCompleted 63000 requests\r\nCompleted 72000 requests\r\nCompleted 81000 requests\r\nCompleted 90000 requests\r\nFinished 90000 requests\r\n\r\n\r\nServer Software:        nginx/1.11.1\r\nServer Hostname:        172.24.103.2\r\nServer Port:            80\r\n\r\nDocument Path:          /\r\nDocument Length:        612 bytes\r\n\r\nConcurrency Level:      50\r\nTime taken for tests:   6.055 seconds\r\nComplete requests:      90000\r\nFailed requests:        0\r\nTotal transferred:      76050000 bytes\r\nHTML transferred:       55080000 bytes\r\nRequests per second:    14864.62 [#/sec] (mean)\r\nTime per request:       3.364 [ms] (mean)\r\nTime per request:       0.067 [ms] (mean, across all concurrent requests)\r\nTransfer rate:          12266.21 [Kbytes/sec] received\r\n\r\nConnection Times (ms)\r\n              min  mean[+/-sd] median   max\r\nConnect:        0    0   0.7      0       6\r\nProcessing:     0    3   1.4      3      14\r\nWaiting:        0    3   1.4      3      14\r\nTotal:          2    3   1.5      3      15\r\n\r\nPercentage of the requests served within a certain time (ms)\r\n  50%      3\r\n  66%      3\r\n  75%      3\r\n  80%      3\r\n  90%      5\r\n  95%      5\r\n  98%      6\r\n  99%     14\r\n 100%     15 (longest request)\r\n```\r\n\r\n但此时会引来其他的问题:\r\n\r\n![9cb209a1-6fcb-4183-afe6-466ed284bfad](https://cloud.githubusercontent.com/assets/13348433/17139705/28518480-5378-11e6-95ec-4c0dfb2b0cec.png)\r\n\r\nkube-proxy设置Service的相关命令都会失败。此时需要更换kube-proxy的方式，或者提交Service都用LoadBalancer方式？"}}
{"actor":{"display_login":"typhoonzero"}}