{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/35082","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/35082/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/35082/comments","events_url":"https://api.github.com/repos/moby/moby/issues/35082/events","html_url":"https://github.com/moby/moby/issues/35082","id":262719991,"number":35082,"title":"[SWARM] Very poor performance for ingress network with lots of parallel requests","user":{"login":"vide","id":270267,"avatar_url":"https://avatars3.githubusercontent.com/u/270267?v=4","gravatar_id":"","url":"https://api.github.com/users/vide","html_url":"https://github.com/vide","followers_url":"https://api.github.com/users/vide/followers","following_url":"https://api.github.com/users/vide/following{/other_user}","gists_url":"https://api.github.com/users/vide/gists{/gist_id}","starred_url":"https://api.github.com/users/vide/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vide/subscriptions","organizations_url":"https://api.github.com/users/vide/orgs","repos_url":"https://api.github.com/users/vide/repos","events_url":"https://api.github.com/users/vide/events{/privacy}","received_events_url":"https://api.github.com/users/vide/received_events","type":"User","site_admin":false},"labels":[{"id":237472034,"url":"https://api.github.com/repos/moby/moby/labels/area/networking","name":"area/networking","color":"fbca04","default":false},{"id":392678049,"url":"https://api.github.com/repos/moby/moby/labels/area/swarm","name":"area/swarm","color":"fbca04","default":false},{"id":335124477,"url":"https://api.github.com/repos/moby/moby/labels/kind/performance","name":"kind/performance","color":"006b75","default":false},{"id":687939503,"url":"https://api.github.com/repos/moby/moby/labels/version/17.09","name":"version/17.09","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2017-10-04T09:22:46Z","updated_at":"2018-04-19T16:02:42Z","closed_at":null,"author_association":"NONE","body":"**Description**\r\n\r\nExecuting a large number of parallel connections against plain Docker and Docker Swarms leads to 2 completely different perfomance results, with Swarm being the slowest one by a **50x** factor!\r\nThe test is reproducible (at least on my VMs) easily with Siege and the official Nginx image, but I'm actually experiencing the problem in production with our custom java-based HTTP microservice. I cannot see any obvious error message in Docker logs or kernel logs.\r\n\r\n**Steps to reproduce the issue:**\r\nRun the nginx container:\r\n```\r\n[root@stresstest01 ~]# docker run -d --rm --net bridge -m 0b  -p 80:80  --name test nginx\r\n35c231e361d7e5ca73fb1bcfbeeaf57a066da057b708055477855e6d16af575d\r\n```\r\n\r\nSiege the container, and the results are good, over 13k trans/sec, and CPU in `stresstest01` is 100% used by the nginx process.\r\n\r\n```\r\n[root@siege01 ~]# siege -b -c 250 -t 20s -f test_vm_docker.txt >/dev/null\r\n** SIEGE 4.0.2\r\n** Preparing 250 concurrent users for battle.\r\nThe server is now under siege...\r\n\r\nLifting the server siege...\r\nTransactions:\t\t      260810 hits\r\nAvailability:\t\t      100.00 %\r\nElapsed time:\t\t       19.03 secs\r\nData transferred:\t      140.03 MB\r\nResponse time:\t\t        0.02 secs\r\nTransaction rate:\t    13705.20 trans/sec\r\nThroughput:\t\t        7.36 MB/sec\r\nConcurrency:\t\t      245.51\r\nSuccessful transactions:      231942\r\nFailed transactions:\t           0\r\nLongest transaction:\t        7.03\r\nShortest transaction:\t        0.00\r\n\r\n```\r\n\r\nNow, lets try with Docker Swarm (1 node swarm, 1 container stack)\r\n```\r\n[root@stresstest01 ~]# cat docker-compose.yml \r\nservices:\r\n  server:\r\n    deploy:\r\n      replicas: 1\r\n    image: nginx:latest\r\n    ports:\r\n    - published: 80\r\n      target: 80\r\nversion: '3.3'\r\n```\r\n```\r\n[root@stresstest01 ~]# docker stack deploy test --compose-file docker-compose.yml \r\nCreating network test_default\r\nCreating service test_server\r\n```\r\nAfter the first run, the results are already far worse than with plain Docker, but after the second it's\r\njust a disaster :( Moreover, host CPU is only slightly used, and only by the nginx process. No docker related processes (`dockerd`,`cointanerd` etc) seem to be contained by CPU.\r\n\r\n```\r\n[root@siege01 ~]# siege -b -c 250 -t 20s -f test_vm_docker.txt >/dev/null\r\n** SIEGE 4.0.2\r\n** Preparing 250 concurrent users for battle.\r\nThe server is now under siege...\r\n\r\nLifting the server siege...\r\nTransactions:\t\t       65647 hits\r\nAvailability:\t\t      100.00 %\r\nElapsed time:\t\t       19.44 secs\r\nData transferred:\t       35.28 MB\r\nResponse time:\t\t        0.07 secs\r\nTransaction rate:\t     3376.90 trans/sec\r\nThroughput:\t\t        1.81 MB/sec\r\nConcurrency:\t\t      246.66\r\nSuccessful transactions:       58469\r\nFailed transactions:\t           0\r\nLongest transaction:\t        3.02\r\nShortest transaction:\t        0.00\r\n \r\n[root@siege01 ~]# siege -b -c 250 -t 20s -f test_vm_docker.txt >/dev/null\r\n** SIEGE 4.0.2\r\n** Preparing 250 concurrent users for battle.\r\nThe server is now under siege...\r\n\r\nLifting the server siege...\r\nTransactions:\t\t        4791 hits\r\nAvailability:\t\t      100.00 %\r\nElapsed time:\t\t       19.47 secs\r\nData transferred:\t        2.59 MB\r\nResponse time:\t\t        1.00 secs\r\nTransaction rate:\t      246.07 trans/sec\r\nThroughput:\t\t        0.13 MB/sec\r\nConcurrency:\t\t      245.61\r\nSuccessful transactions:        4291\r\nFailed transactions:\t           0\r\nLongest transaction:\t        1.20\r\nShortest transaction:\t        0.00\r\n```\r\n\r\n**Describe the results you received:**\r\nGood performances with plain Docker.\r\nVery bad performances with Docker Swarm enabled.\r\n\r\n**Describe the results you expected:**\r\nSimilar performances for the two Docker flavour on the same machine\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      17.09.0-ce\r\n API version:  1.32\r\n Go version:   go1.8.3\r\n Git commit:   afdb6d4\r\n Built:        Tue Sep 26 22:41:23 2017\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      17.09.0-ce\r\n API version:  1.32 (minimum version 1.12)\r\n Go version:   go1.8.3\r\n Git commit:   afdb6d4\r\n Built:        Tue Sep 26 22:42:49 2017\r\n OS/Arch:      linux/amd64\r\n Experimental: false\r\n\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 0\r\n Running: 0\r\n Paused: 0\r\n Stopped: 0\r\nImages: 2\r\nServer Version: 17.09.0-ce\r\nStorage Driver: overlay\r\n Backing Filesystem: xfs\r\n Supports d_type: false\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\nSwarm: active\r\n NodeID: s2ei2tx1nbf6lgn6d2yi9k782\r\n Is Manager: true\r\n ClusterID: s2dwwy929baleeoyk943wh2r9\r\n Managers: 1\r\n Nodes: 1\r\n Orchestration:\r\n  Task History Retention Limit: 5\r\n Raft:\r\n  Snapshot Interval: 10000\r\n  Number of Old Snapshots to Retain: 0\r\n  Heartbeat Tick: 1\r\n  Election Tick: 3\r\n Dispatcher:\r\n  Heartbeat Period: 5 seconds\r\n CA Configuration:\r\n  Expiry Duration: 3 months\r\n  Force Rotate: 0\r\n Autolock Managers: false\r\n Root Rotation In Progress: false\r\n Node Address: 192.168.10.187\r\n Manager Addresses:\r\n  192.168.10.187:2377\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0\r\nrunc version: 3f2f8b84a77f73d38244dd690525642a72156c64\r\ninit version: 949e6fa\r\nSecurity Options:\r\n seccomp\r\n  Profile: default\r\nKernel Version: 3.10.0-693.2.2.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 16\r\nTotal Memory: 7.609GiB\r\nName: stresstest01\r\nID: 4XPS:KBEY:W53L:YAK6:4MZL:4HDN:DMUR:DD4T:5RWA:IUK6:522E:TCAL\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\nIt's a KVM virtual machine (under oVirt) but the same happens when using a physical machine."},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/382790576","html_url":"https://github.com/moby/moby/issues/35082#issuecomment-382790576","issue_url":"https://api.github.com/repos/moby/moby/issues/35082","id":382790576,"user":{"login":"tmarti","id":2405414,"avatar_url":"https://avatars2.githubusercontent.com/u/2405414?v=4","gravatar_id":"","url":"https://api.github.com/users/tmarti","html_url":"https://github.com/tmarti","followers_url":"https://api.github.com/users/tmarti/followers","following_url":"https://api.github.com/users/tmarti/following{/other_user}","gists_url":"https://api.github.com/users/tmarti/gists{/gist_id}","starred_url":"https://api.github.com/users/tmarti/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tmarti/subscriptions","organizations_url":"https://api.github.com/users/tmarti/orgs","repos_url":"https://api.github.com/users/tmarti/repos","events_url":"https://api.github.com/users/tmarti/events{/privacy}","received_events_url":"https://api.github.com/users/tmarti/received_events","type":"User","site_admin":false},"created_at":"2018-04-19T16:02:42Z","updated_at":"2018-04-19T16:02:42Z","author_association":"NONE","body":"Finally found the problem.\r\n\r\nAs a last-resort option, and knowing that swarm relies on netfilter facilites to do its internal load balancing for overlay networks (as the very simple case for the service in the prevous post which by default uses an overlay network), I downloaded the Linux Kernel and tampered a little bit with the files.\r\n\r\nThe sources of interest reside in the following folder:\r\n\r\n`[kernel source dir]/net/netfilter`\r\n\r\nThat TIME_WAIT timeout is hardcoded inside this file from the **ip_vs** module, inside:\r\n\r\n`[kernel source dir]/net/netfilter/ipvs/ip_vs_proto_tcp.c`\r\n\r\nYou can check the latest version of this file (which suffers from the same problem) here:\r\n\r\nhttps://github.com/torvalds/linux/blob/master/net/netfilter/ipvs/ip_vs_proto_tcp.c\r\n\r\nInside this file, you can see following piece of code:\r\n\r\n```\r\nstatic const int tcp_timeouts[IP_VS_TCP_S_LAST+1] = {\r\n\t[IP_VS_TCP_S_NONE]\t\t=\t2*HZ,\r\n\t[IP_VS_TCP_S_ESTABLISHED]\t=\t15*60*HZ,\r\n\t[IP_VS_TCP_S_SYN_SENT]\t\t=\t2*60*HZ,\r\n\t[IP_VS_TCP_S_SYN_RECV]\t\t=\t1*60*HZ,\r\n\t[IP_VS_TCP_S_FIN_WAIT]\t\t=\t2*60*HZ,\r\n\t[IP_VS_TCP_S_TIME_WAIT]\t\t=\t2*60*HZ,\r\n\t[IP_VS_TCP_S_CLOSE]\t\t=\t10*HZ,\r\n\t[IP_VS_TCP_S_CLOSE_WAIT]\t=\t60*HZ,\r\n\t[IP_VS_TCP_S_LAST_ACK]\t\t=\t30*HZ,\r\n\t[IP_VS_TCP_S_LISTEN]\t\t=\t2*60*HZ,\r\n\t[IP_VS_TCP_S_SYNACK]\t\t=\t120*HZ,\r\n\t[IP_VS_TCP_S_LAST]\t\t=\t2*HZ,\r\n};\r\n```\r\n\r\nSo the guilty of that high timeout is:\r\n\r\n`[IP_VS_TCP_S_TIME_WAIT]\t\t=\t2*60*HZ,`\r\n\r\nIf the previous one is changed to:\r\n\r\n`[IP_VS_TCP_S_TIME_WAIT]\t\t=\t2*HZ,`\r\n\r\nThe TIME_WAIT timeout is lowered from 120s to 2s.\r\n\r\nThen recompiling the module, replacing the system module with the compiled one, rebooting the swarm machine, restarting the service and repeating the load test yields incredibly good results. No more flooding of connections in TIME_WAIT state is observed for moderately high loads (2000 req/s).\r\n\r\nIf the code from the rest of the file is inspected, there is really no way (or I don't see it) to reload those timeouts. That **tcp_timeouts** vector seems to be used to initialize the internal timeout table that will be used for the connection management, (without apparently any way to tune it) in this function:\r\n\r\n```\r\n/* ---------------------------------------------\r\n *   timeouts is netns related now.\r\n * ---------------------------------------------\r\n */\r\nstatic int __ip_vs_tcp_init(struct netns_ipvs *ipvs, struct ip_vs_proto_data *pd)\r\n{\r\n\tip_vs_init_hash_table(ipvs->tcp_apps, TCP_APP_TAB_SIZE);\r\n\tpd->timeout_table = ip_vs_create_timeout_table((int *)tcp_timeouts,\r\n\t\t\t\t\t\t\tsizeof(tcp_timeouts));\r\n\tif (!pd->timeout_table)\r\n\t\treturn -ENOMEM;\r\n\tpd->tcp_state_table =  tcp_states;\r\n\treturn 0;\r\n}\r\n``` \r\n\r\nThe file **ip_vs_ctl.c,** which seems to be in charge of updating the tuning for the module, exposes following systctl parameters:\r\n\r\n```\r\nIPVS sysctl table (under the /proc/sys/net/ipv4/vs/)\r\n\r\n.procname       = \"amemthresh\",\r\n.procname       = \"am_droprate\",\r\n.procname       = \"drop_entry\",\r\n.procname       = \"drop_packet\",\r\n.procname       = \"conntrack\",\r\n.procname       = \"secure_tcp\",\r\n.procname       = \"snat_reroute\",\r\n.procname       = \"sync_version\",\r\n.procname       = \"sync_ports\",\r\n.procname       = \"sync_persist_mode\",\r\n.procname       = \"sync_qlen_max\",\r\n.procname       = \"sync_sock_size\",\r\n.procname       = \"cache_bypass\",\r\n.procname       = \"expire_nodest_conn\",\r\n.procname       = \"sloppy_tcp\",\r\n.procname       = \"sloppy_sctp\",\r\n.procname       = \"expire_quiescent_template\",\r\n.procname       = \"sync_threshold\",\r\n.procname       = \"sync_refresh_period\",\r\n.procname       = \"sync_retries\",\r\n.procname       = \"nat_icmp_send\",\r\n.procname       = \"pmtu_disc\",\r\n.procname       = \"backup_only\",\r\n.procname       = \"conn_reuse_mode\",\r\n.procname       = \"schedule_icmp\",\r\n.procname       = \"ignore_tunneled\",\r\n.procname       = \"debug_level\",\r\n```\r\n\r\nNothing like timeouts exposed here.\r\n\r\nSo there is no effective way to update the TIME_WAIT timeout parameter for this module once it has started (nor to tweak it so the module reads the tuned value during init).\r\n\r\nIf anybody has an idea of how this problem could be workarounded (s)he will deserve big hugs.\r\n\r\nBy the moment, in a cul-de-sac again. (it is not very practical to recompile kernel modules after each kernel image upgrade)"}}
{"actor":{"display_login":"tmarti"}}