{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/26465","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/26465/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/26465/comments","events_url":"https://api.github.com/repos/docker/docker/issues/26465/events","html_url":"https://github.com/docker/docker/issues/26465","id":176181026,"number":26465,"title":"docker swarm scale problem","user":{"login":"yejune","id":3735511,"avatar_url":"https://avatars.githubusercontent.com/u/3735511?v=3","gravatar_id":"","url":"https://api.github.com/users/yejune","html_url":"https://github.com/yejune","followers_url":"https://api.github.com/users/yejune/followers","following_url":"https://api.github.com/users/yejune/following{/other_user}","gists_url":"https://api.github.com/users/yejune/gists{/gist_id}","starred_url":"https://api.github.com/users/yejune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yejune/subscriptions","organizations_url":"https://api.github.com/users/yejune/orgs","repos_url":"https://api.github.com/users/yejune/repos","events_url":"https://api.github.com/users/yejune/events{/privacy}","received_events_url":"https://api.github.com/users/yejune/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/area/swarm","name":"area/swarm","color":"fbca04"},{"url":"https://api.github.com/repos/docker/docker/labels/version/1.12","name":"version/1.12","color":"ededed"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-09-10T13:58:13Z","updated_at":"2016-10-20T15:14:37Z","closed_at":null,"body":"**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.7\r\n Git commit:   23cf638\r\n Built:        Thu Aug 18 22:32:50 UTC 2016\r\n OS/Arch:      darwin/amd64\r\n\r\nServer:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   23cf638\r\n Built:        Thu Aug 18 05:28:14 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 3\r\n Running: 1\r\n Paused: 0\r\n Stopped: 2\r\nImages: 1\r\nServer Version: 1.12.1\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 26\r\n Dirperm1 Supported: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null overlay bridge host\r\nSwarm: active\r\n NodeID: 7wm3mmh10a2cb24q112kc0pc3\r\n Is Manager: true\r\n ClusterID: dkpahczmnenifcm34annpotga\r\n Managers: 3\r\n Nodes: 6\r\n Orchestration:\r\n  Task History Retention Limit: 5\r\n Raft:\r\n  Snapshot Interval: 10000\r\n  Heartbeat Tick: 1\r\n  Election Tick: 3\r\n Dispatcher:\r\n  Heartbeat Period: 5 seconds\r\n CA Configuration:\r\n  Expiry Duration: 3 months\r\n Node Address: 172.31.51.106\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: apparmor seccomp\r\nKernel Version: 4.2.0-18-generic\r\nOperating System: Ubuntu 15.10\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 1\r\nTotal Memory: 991.1 MiB\r\nName: cluster-manager1\r\nID: 2SK7:GIM3:4ZB5:47WH:KQRX:L5VA:I66F:YH2F:GJDP:OH6M:AHCF:XISF\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: No swap limit support\r\nLabels:\r\n provider=amazonec2\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nAWS\r\n\r\n\r\n**Steps to reproduce the issue:**\r\n\r\n1\\. execute test code on OSX\r\n\r\n```bash\r\n#!/bin/bash\r\n\r\ndocker-machine create --driver amazonec2  --amazonec2-zone=b cluster-manager1\r\ndocker-machine create --driver amazonec2  --amazonec2-zone=b cluster-manager2\r\ndocker-machine create --driver amazonec2  --amazonec2-zone=b cluster-manager3\r\ndocker-machine create --driver amazonec2  --amazonec2-zone=b cluster-worker1\r\ndocker-machine create --driver amazonec2  --amazonec2-zone=b cluster-worker2\r\ndocker-machine create --driver amazonec2  --amazonec2-zone=b cluster-worker3\r\n\r\n# assignment all elastic ip \r\n# security group, inbound rules All traffic anywhere\r\n\r\ndocker-machine regenerate-certs cluster-manager1 cluster-manager2 cluster-manager3 cluster-worker1 cluster-worker2 cluster-worker3\r\n\r\ndocker $(docker-machine config cluster-manager1) swarm init --advertise-addr eth0\r\ndocker $(docker-machine config cluster-manager2) swarm join --token `docker $(docker-machine config cluster-manager1) swarm join-token manager -q` $(docker-machine ssh cluster-manager1 ifconfig eth0 | perl -lane 'print $1 if /inet addr:(\\S+)/'):2377\r\ndocker $(docker-machine config cluster-manager3) swarm join --token `docker $(docker-machine config cluster-manager1) swarm join-token manager -q` $(docker-machine ssh cluster-manager1 ifconfig eth0 | perl -lane 'print $1 if /inet addr:(\\S+)/'):2377\r\ndocker $(docker-machine config cluster-worker1) swarm join --token `docker $(docker-machine config cluster-manager1) swarm join-token worker -q` $(docker-machine ssh cluster-manager1 ifconfig eth0 | perl -lane 'print $1 if /inet addr:(\\S+)/'):2377\r\ndocker $(docker-machine config cluster-worker2) swarm join --token `docker $(docker-machine config cluster-manager1) swarm join-token worker -q` $(docker-machine ssh cluster-manager1 ifconfig eth0 | perl -lane 'print $1 if /inet addr:(\\S+)/'):2377\r\ndocker $(docker-machine config cluster-worker3) swarm join --token `docker $(docker-machine config cluster-manager1) swarm join-token worker -q` $(docker-machine ssh cluster-manager1 ifconfig eth0 | perl -lane 'print $1 if /inet addr:(\\S+)/'):2377\r\n\r\ndocker $(docker-machine config cluster-manager1) network rm cluster-network\r\ndocker $(docker-machine config cluster-manager1) network create --driver overlay cluster-network\r\n\r\necho \"\r\nupstream cluster {\r\n    server $(docker-machine ip cluster-manager1):80 max_fails=1 fail_timeout=1s;\r\n    server $(docker-machine ip cluster-manager2):80 max_fails=1 fail_timeout=1s;\r\n    server $(docker-machine ip cluster-manager3):80 max_fails=1 fail_timeout=1s;\r\n}\r\n\r\nserver {\r\n    listen 80;\r\n    server_name localhost;\r\n\r\n    error_page 502 @502page;\r\n    error_page 504 @504page;\r\n\r\n    location @502page {\r\n        default_type text/plain;\r\n        return 502 '502 Bad Gateway';\r\n    }\r\n    \r\n    location @504page {\r\n        default_type text/plain;\r\n        return 504 '504 Gateway Time-out';\r\n    }   \r\n\r\n    location / {\r\n        proxy_pass http://cluster;\r\n        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;\r\n        proxy_connect_timeout 1;\r\n\r\n        access_log /tmp/docker_cluster_access_log;\r\n        error_log  /tmp/docker_cluster_error_log;\r\n    }\r\n}\r\n\r\n\" > /usr/local/etc/nginx/servers/docker-cluster-localhost.upstream\r\n\r\nsudo nginx -s reload\r\n\r\ndocker $(docker-machine config cluster-manager1) service create --network cluster-network --name web --env DOMAIN=_ --env STAGE_NAME=local --env DEBUG=0 --publish 80:80  yejune/web\r\n```\r\n\r\n2\\. test\r\n\r\n```bash\r\n$ echo \"HOST_IP      CONTAINER_IP\";   for i in {1..10}; do curl  http://localhost; echo \"\"; done\r\nHOST_IP      CONTAINER_IP\r\n10.255.0.3   10.255.0.10\r\n10.255.0.4   10.255.0.10\r\n10.255.0.5   10.255.0.10\r\n10.255.0.3   10.255.0.10\r\n10.255.0.4   10.255.0.10\r\n10.255.0.5   10.255.0.10\r\n10.255.0.3   10.255.0.10\r\n10.255.0.4   10.255.0.10\r\n10.255.0.5   10.255.0.10\r\n10.255.0.3   10.255.0.10\r\n```\r\n\r\n3\\. web scaled to 10, test\r\n\r\n```bash\r\n$ docker $(docker-machine config cluster-manager1) service scale web=10\r\n\r\n$ echo \"HOST_IP      CONTAINER_IP\";   for i in {1..10}; do curl  http://localhost; echo \"\"; done\r\nHOST_IP      CONTAINER_IP\r\n10.255.0.5   10.255.0.12\r\n10.255.0.3   10.255.0.12\r\n10.255.0.4   10.255.0.12\r\n10.255.0.5   10.255.0.11\r\n10.255.0.3   10.255.0.17\r\n10.255.0.4   10.255.0.17\r\n10.255.0.5   10.255.0.17\r\n10.255.0.3   10.255.0.11\r\n10.255.0.4   10.255.0.11\r\n10.255.0.5   10.255.0.16\r\n```\r\n\r\n4\\. web scaled to 1, test\r\n\r\n```bash\r\n$ docker $(docker-machine config cluster-manager1) service scale web=1\r\n\r\n$ echo \"HOST_IP      CONTAINER_IP\";   for i in {1..100}; do curl  http://localhost; echo \"\"; done\r\nHOST_IP      CONTAINER_IP\r\n10.255.0.5   10.255.0.14\r\n504 Gateway Time-out\r\n10.255.0.4   10.255.0.14\r\n504 Gateway Time-out\r\n504 Gateway Time-out\r\n504 Gateway Time-out\r\n504 Gateway Time-out\r\n10.255.0.5   10.255.0.14\r\n504 Gateway Time-out\r\n10.255.0.4   10.255.0.14\r\n504 Gateway Time-out\r\n504 Gateway Time-out\r\n504 Gateway Time-out\r\n10.255.0.5   10.255.0.14\r\n504 Gateway Time-out\r\n10.255.0.4   10.255.0.14\r\n504 Gateway Time-out\r\n504 Gateway Time-out\r\n504 Gateway Time-out\r\n10.255.0.4   10.255.0.14\r\n504 Gateway Time-out\r\n10.255.0.5   10.255.0.14\r\n504 Gateway Time-out\r\n502 Bad Gateway\r\n10.255.0.4   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n504 Gateway Time-out\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n10.255.0.5   10.255.0.14\r\n10.255.0.4   10.255.0.14\r\n504 Gateway Time-out\r\n502 Bad Gateway\r\n10.255.0.5   10.255.0.14\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n10.255.0.5   10.255.0.14\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n10.255.0.5   10.255.0.14\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n502 Bad Gateway\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n10.255.0.5   10.255.0.14\r\n...\r\n```\r\n\r\n```bash\r\n...\r\n2016/09/11 13:33:26 [error] 11436#0: *1281 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 127.0.0.1, server: localhost, request: \"GET / HTTP/1.1\", upstream: \"http://xxx.xxx.xxx.144:80/\", host: \"localhost\"\r\n2016/09/11 13:33:27 [error] 11436#0: *1284 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 127.0.0.1, server: localhost, request: \"GET / HTTP/1.1\", upstream: \"http://xxx.xxx.xxx.230:80/\", host: \"localhost\"\r\n2016/09/11 13:33:29 [error] 11436#0: *1291 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 127.0.0.1, server: localhost, request: \"GET / HTTP/1.1\", upstream: \"http://xxx.xxx.xxx.144:80/\", host: \"localhost\"\r\n2016/09/11 13:33:30 [error] 11436#0: *1294 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 127.0.0.1, server: localhost, request: \"GET / HTTP/1.1\", upstream: \"http://xxx.xxx.xxx.230:80/\", host: \"localhost\"\r\n2016/09/11 13:33:32 [error] 11436#0: *1303 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 127.0.0.1, server: localhost, request: \"GET / HTTP/1.1\", upstream: \"http://xxx.xxx.xxx.144:80/\", host: \"localhost\"\r\n...\r\n```\r\n\r\n**Virtualbox also have the same problem. Is there a problem with my test? :)**"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/255135950","html_url":"https://github.com/docker/docker/issues/26465#issuecomment-255135950","issue_url":"https://api.github.com/repos/docker/docker/issues/26465","id":255135950,"user":{"login":"pslestang","id":2951321,"avatar_url":"https://avatars.githubusercontent.com/u/2951321?v=3","gravatar_id":"","url":"https://api.github.com/users/pslestang","html_url":"https://github.com/pslestang","followers_url":"https://api.github.com/users/pslestang/followers","following_url":"https://api.github.com/users/pslestang/following{/other_user}","gists_url":"https://api.github.com/users/pslestang/gists{/gist_id}","starred_url":"https://api.github.com/users/pslestang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pslestang/subscriptions","organizations_url":"https://api.github.com/users/pslestang/orgs","repos_url":"https://api.github.com/users/pslestang/repos","events_url":"https://api.github.com/users/pslestang/events{/privacy}","received_events_url":"https://api.github.com/users/pslestang/received_events","type":"User","site_admin":false},"created_at":"2016-10-20T15:14:37Z","updated_at":"2016-10-20T15:14:37Z","body":"Same problem here using docker 1.12.2, after scaling down to 1 I randomly get timeout through nginx acting as a reverse proxy, even if I force request on docker host running the container.\r\n\r\n```\r\n$ docker version\r\nClient:\r\n Version:      1.12.2\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   bb80604\r\n Built:        Tue Oct 11 17:43:41 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.2\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   bb80604\r\n Built:        Tue Oct 11 17:43:41 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n```\r\n$ docker info\r\nContainers: 6\r\n Running: 2\r\n Paused: 0\r\n Stopped: 4\r\nImages: 9\r\nServer Version: 1.12.2\r\nStorage Driver: aufs\r\n Root Dir: /data/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 58\r\n Dirperm1 Supported: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null overlay bridge host\r\nSwarm: active\r\n NodeID: 39kb2origwfswk2tvwaypph7i\r\n Is Manager: true\r\n ClusterID: 8werus3koylasmr9ku5e21kjw\r\n Managers: 3\r\n Nodes: 3\r\n Orchestration:\r\n  Task History Retention Limit: 5\r\n Raft:\r\n  Snapshot Interval: 10000\r\n  Heartbeat Tick: 1\r\n  Election Tick: 3\r\n Dispatcher:\r\n  Heartbeat Period: 5 seconds\r\n CA Configuration:\r\n  Expiry Duration: 3 months\r\n Node Address: 192.168.102.35\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options:\r\nKernel Version: 3.16.0-4-amd64\r\nOperating System: Debian GNU/Linux 8 (jessie)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 1.957 GiB\r\nName: vdockerdev001\r\nID: NX3R:QVYA:4IAM:NFBJ:REOR:BYL3:PYD5:HX4W:2GR6:I674:HGXD:P2NE\r\nDocker Root Dir: /data/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: No memory limit support\r\nWARNING: No swap limit support\r\nWARNING: No kernel memory limit support\r\nWARNING: No oom kill disable support\r\nWARNING: No cpu cfs quota support\r\nWARNING: No cpu cfs period support\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n```\r\n"}}
{"actor":{"display_login":"pslestang"}}