{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/24078","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/24078/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/24078/comments","events_url":"https://api.github.com/repos/docker/docker/issues/24078/events","html_url":"https://github.com/docker/docker/issues/24078","id":162808804,"number":24078,"title":"Docker no starting on WORKER in swarm mode","user":{"login":"rvernica","id":241185,"avatar_url":"https://avatars.githubusercontent.com/u/241185?v=3","gravatar_id":"","url":"https://api.github.com/users/rvernica","html_url":"https://github.com/rvernica","followers_url":"https://api.github.com/users/rvernica/followers","following_url":"https://api.github.com/users/rvernica/following{/other_user}","gists_url":"https://api.github.com/users/rvernica/gists{/gist_id}","starred_url":"https://api.github.com/users/rvernica/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rvernica/subscriptions","organizations_url":"https://api.github.com/users/rvernica/orgs","repos_url":"https://api.github.com/users/rvernica/repos","events_url":"https://api.github.com/users/rvernica/events{/privacy}","received_events_url":"https://api.github.com/users/rvernica/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/area/swarm","name":"area/swarm","color":"fbca04"},{"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94"},{"url":"https://api.github.com/repos/docker/docker/labels/priority/P2","name":"priority/P2","color":"fad8c7"},{"url":"https://api.github.com/repos/docker/docker/labels/version/1.12","name":"version/1.12","color":"ededed"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/docker/docker/milestones/53","html_url":"https://github.com/docker/docker/milestones/1.12.0","labels_url":"https://api.github.com/repos/docker/docker/milestones/53/labels","id":1657301,"number":53,"title":"1.12.0","description":"feature/code freeze: June 14th, 2016","creator":{"login":"thaJeztah","id":1804568,"avatar_url":"https://avatars.githubusercontent.com/u/1804568?v=3","gravatar_id":"","url":"https://api.github.com/users/thaJeztah","html_url":"https://github.com/thaJeztah","followers_url":"https://api.github.com/users/thaJeztah/followers","following_url":"https://api.github.com/users/thaJeztah/following{/other_user}","gists_url":"https://api.github.com/users/thaJeztah/gists{/gist_id}","starred_url":"https://api.github.com/users/thaJeztah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thaJeztah/subscriptions","organizations_url":"https://api.github.com/users/thaJeztah/orgs","repos_url":"https://api.github.com/users/thaJeztah/repos","events_url":"https://api.github.com/users/thaJeztah/events{/privacy}","received_events_url":"https://api.github.com/users/thaJeztah/received_events","type":"User","site_admin":false},"open_issues":134,"closed_issues":648,"state":"open","created_at":"2016-03-21T13:47:53Z","updated_at":"2016-07-15T18:11:20Z","due_on":"2016-07-28T07:00:00Z","closed_at":null},"comments":30,"created_at":"2016-06-28T23:17:30Z","updated_at":"2016-07-15T19:26:02Z","closed_at":null,"body":"**Output of `docker version` on MANAGER:**\r\n\r\n```\r\n> docker version\r\nClient:\r\n Version:      1.12.0-rc2\r\n API version:  1.24\r\n Go version:   go1.6.2\r\n Git commit:   906eacd\r\n Built:        \r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.0-rc2\r\n API version:  1.24\r\n Go version:   go1.6.2\r\n Git commit:   906eacd\r\n Built:        \r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n**Output of `docker info` on MANAGER:**\r\n\r\n```\r\n> docker info\r\nContainers: 0\r\n Running: 0\r\n Paused: 0\r\n Stopped: 0\r\nImages: 2\r\nServer Version: 1.12.0-rc2\r\nStorage Driver: devicemapper\r\n Pool Name: centos-dockerpool\r\n Pool Blocksize: 65.54 kB\r\n Base Device Size: 10.74 GB\r\n Backing Filesystem: xfs\r\n Data file: \r\n Metadata file: \r\n Data Space Used: 2.541 GB\r\n Data Space Total: 42.95 GB\r\n Data Space Available: 40.41 GB\r\n Metadata Space Used: 3.67 MB\r\n Metadata Space Total: 41.94 MB\r\n Metadata Space Available: 38.27 MB\r\n Thin Pool Minimum Free Space: 4.295 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: false\r\n Deferred Deletion Enabled: false\r\n Deferred Deleted Device Count: 0\r\n Library Version: 1.02.107-RHEL7 (2016-06-09)\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null overlay bridge host\r\nSwarm: active\r\n NodeID: bcunjya7hjuhdlyn3odnx3e1q\r\n IsManager: Yes\r\n Managers: 1\r\n Nodes: 1\r\n CACertHash: sha256:03af27975890f7c5b9b64abb2723fec74dd565ea97a4e3559f7695c282e7a71a\r\nRuntimes: default\r\nDefault Runtime: default\r\nSecurity Options: seccomp\r\nKernel Version: 3.10.0-327.22.2.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 4\r\nTotal Memory: 23.37 GiB\r\nName: blu-03\r\nID: CBXH:UD64:UUH7:DK4P:D2AC:Y627:3EGT:OTYB:YQTC:QH5C:TZKV:Q7X7\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nHttp Proxy: http://...\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nThree physical machines (MANAGER and two WORKERS) running CentOS 7.2 with `firewalld` disabled. They have the same Docker version and setup.\r\n```\r\n> cat /etc/centos-release\r\nCentOS Linux release 7.2.1511 (Core) \r\n> uname -a\r\nLinux blu-03 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux\r\n> sudo service firewalld status\r\nRedirecting to /bin/systemctl status  firewalld.service\r\n● firewalld.service - firewalld - dynamic firewall daemon\r\n   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)\r\n   Active: inactive (dead)\r\n```\r\n\r\n**Steps to reproduce the issue:**\r\n1. On one of the machines (MANAGER), do `docker swarm init`\r\n2. On the other two machines (WORKER), do `docker swarm join <MANAGER-IP>`\r\n3. Restart all three machines\r\n\r\n**Describe the results you received:**\r\nDocker no longer starts on the WORKER machines. (Docker starts fine on MANAGER machine.)\r\n```\r\n> sudo service docker status -l\r\nRedirecting to /bin/systemctl status  -l docker.service\r\n● docker.service - Docker Application Container Engine\r\n   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)\r\n  Drop-In: /etc/systemd/system/docker.service.d\r\n           └─http-proxy.conf, storage.conf\r\n   Active: failed (Result: exit-code) since Tue 2016-06-28 16:08:42 PDT; 13min ago\r\n     Docs: https://docs.docker.com\r\n  Process: 980 ExecStart=/usr/bin/dockerd -H fd:// --storage-driver devicemapper --storage-opt dm.thinpooldev=/dev/mapper/centos-dockerpool (code=exited, status=1/FAILURE)\r\n Main PID: 980 (code=exited, status=1/FAILURE)\r\n\r\nJun 28 16:08:32 blu-04 dockerd[980]: time=\"2016-06-28T16:08:32.336448954-07:00\" level=warning msg=\"Running modprobe bridge br_netfilter failed with message: modprobe: WARNING: Module br_netfilter not found.\\ninsmod /lib/modules/3.10.0-327.22.2.el7.x86_64/kernel/net/llc/llc.ko \\ninsmod /lib/modules/3.10.0-327.22.2.el7.x86_64/kernel/net/802/stp.ko \\ninsmod /lib/modules/3.10.0-327.22.2.el7.x86_64/kernel/net/bridge/bridge.ko \\n, error: exit status 1\"\r\nJun 28 16:08:32 blu-04 dockerd[980]: time=\"2016-06-28T16:08:32.364210543-07:00\" level=info msg=\"Firewalld running: false\"\r\nJun 28 16:08:32 blu-04 dockerd[980]: time=\"2016-06-28T16:08:32.603355256-07:00\" level=info msg=\"Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address\"\r\nJun 28 16:08:32 blu-04 dockerd[980]: time=\"2016-06-28T16:08:32.698269276-07:00\" level=info msg=\"Loading containers: done.\"\r\nJun 28 16:08:42 blu-04 dockerd[980]: time=\"2016-06-28T16:08:42.069860930-07:00\" level=error msg=\"cluster exited with error: can't create raft node: rpc error: code = 2 desc = grpc: the client connection is closing\"\r\nJun 28 16:08:42 blu-04 dockerd[980]: time=\"2016-06-28T16:08:42.069987542-07:00\" level=fatal msg=\"Error creating cluster component: swarm component could not be started\"\r\nJun 28 16:08:42 blu-04 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE\r\nJun 28 16:08:42 blu-04 systemd[1]: Failed to start Docker Application Container Engine.\r\nJun 28 16:08:42 blu-04 systemd[1]: Unit docker.service entered failed state.\r\nJun 28 16:08:42 blu-04 systemd[1]: docker.service failed.\r\n```\r\n\r\n**Describe the results you expected:**\r\nWORKER machines can join the swarm and Docker starts on WORKER machines.\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\nInitially `firewalld` was running. Once I stopped `firewalld`, one of the WORKERS was able to join the swarm, `docker node ls` on MANAGER listed the WORKER. This happened once and I am not able to reproduce it. I disabled `firewalld` and rebooted the machines before reporting this."},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/233048341","html_url":"https://github.com/docker/docker/issues/24078#issuecomment-233048341","issue_url":"https://api.github.com/repos/docker/docker/issues/24078","id":233048341,"user":{"login":"aaronlehmann","id":10601430,"avatar_url":"https://avatars.githubusercontent.com/u/10601430?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlehmann","html_url":"https://github.com/aaronlehmann","followers_url":"https://api.github.com/users/aaronlehmann/followers","following_url":"https://api.github.com/users/aaronlehmann/following{/other_user}","gists_url":"https://api.github.com/users/aaronlehmann/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlehmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlehmann/subscriptions","organizations_url":"https://api.github.com/users/aaronlehmann/orgs","repos_url":"https://api.github.com/users/aaronlehmann/repos","events_url":"https://api.github.com/users/aaronlehmann/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlehmann/received_events","type":"User","site_admin":false},"created_at":"2016-07-15T19:26:02Z","updated_at":"2016-07-15T19:26:02Z","body":"Relevant grpc-go issue is https://github.com/grpc/grpc-go/issues/622."}}
{"actor":{"display_login":"aaronlehmann"}}