title: Docker 1.12.1: after swarm init, workers unable to join swarm
body: <p>I am seeing the same problem as <a href="https://stackoverflow.com/questions/38693560/docker1-12-worker-not-able-to-join-in-clusterswarm-pending">described here</a> and <a href="https://stackoverflow.com/questions/38825792/node-cannot-join-swarm-cluster">here</a>. I have tried everything that worked in those two cases to no avail - I still see the same behavior.  Can someone offer alternatives I might try?</p>
  
  <p>My setup:</p>
  
  <p>I am running 3 Centos 7.2 boxes.  Network Time Protocol (ntpd) running on all machines.  All have been yum updated.  Here is some detailed info:</p>
  
  <pre><code>Linux version 3.10.0-327.28.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) )
  </code></pre>
  
  <p>Docker version:</p>
  
  <pre><code># docker version
  Client:
   Version:      1.12.1
   API version:  1.24
   Go version:   go1.6.3
   Git commit:   23cf638
   Built:        
   OS/Arch:      linux/amd64
  
  Server:
   Version:      1.12.1
   API version:  1.24
   Go version:   go1.6.3
   Git commit:   23cf638
   Built:        
   OS/Arch:      linux/amd64
  </code></pre>
  
  <p>Setup the swarm manager:</p>
  
  <pre><code>&gt;docker swarm init --advertise-addr 10.1.1.40:2377 --force-new-cluster
  // on some retry attempts (after 'docker swarm leave --force') I ran:
  &gt;docker swarm init --advertise-addr 10.1.1.40:2377 --force-new-cluster
  </code></pre>
  
  <p>Manager status:</p>
  
  <pre><code>&gt;docker node inspect self
  [
  {
      "ID": "3x5q1n9v956g3ptdle2eve856",
      "Version": {
          "Index": 10
      },
      "CreatedAt": "2016-08-27T13:01:13.400345797Z",
      "UpdatedAt": "2016-08-27T13:01:13.580143388Z",
      "Spec": {
          "Role": "manager",
          "Availability": "active"
      },
      "Description": {
          "Hostname": "mymanagerhost.mycompany.com",
          "Platform": {
              "Architecture": "x86_64",
              "OS": "linux"
          },
          "Resources": {
              "NanoCPUs": 4000000000,
              "MemoryBytes": 16659128320
          },
          "Engine": {
              "EngineVersion": "1.12.1",
              "Plugins": [
                  {
                      "Type": "Network",
                      "Name": "bridge"
                  },
                  {
                      "Type": "Network",
                      "Name": "host"
                  },
                  {
                      "Type": "Network",
                      "Name": "null"
                  },
                  {
                      "Type": "Network",
                      "Name": "overlay"
                  },
                  {
                      "Type": "Volume",
                      "Name": "local"
                  }
              ]
          }
      },
      "Status": {
          "State": "ready"
      },
      "ManagerStatus": {
          "Leader": true,
          "Reachability": "reachable",
          "Addr": "10.1.1.40:2377"
      }
  }
  ]
  </code></pre>
  
  <p>On the worker node (I have two, but they both behave the same).</p>
  
  <p>Join Swarm:</p>
  
  <pre><code>&gt;docker swarm join     --token SWMTKN-1-4fjh7kncdpwjvxnxisamhldgenmmnqyvhnx9qdi8d4hkkfuacv-168gs9okd5ck0r4lokdgpef92     10.1.1.40:2377
  
  Error response from daemon: Timeout was reached before node was joined. Attempt to join the cluster will continue in the background. Use "docker info" command to see the current swarm status of your node.
  </code></pre>
  
  <p>Output of Docker info command:</p>
  
  <pre><code>&gt;docker info
  Plugins:
   Volume: local
   Network: null host bridge overlay
  Swarm: pending
   NodeID: 
   Error: rpc error: code = 1 desc = context canceled
   Is Manager: false
   Node Address: 10.1.1.50
  Runtimes: runc
  Default Runtime: runc
  Security Options: seccomp
  Kernel Version: 3.10.0-327.28.2.el7.x86_64
  Operating System: CentOS Linux 7 (Core)
  OSType: linux
  Architecture: x86_64
  CPUs: 4
  Total Memory: 15.52 GiB
  Name: myWorkerNode.mycompany.com
  ID: DAWE:VDRA:ZUVS:P7PH:ADCP:MFNU:2LOS:C6TG:XSIS:Y7EX:I46S:KFXT
  Docker Root Dir: /var/lib/docker
  Debug Mode (client): false
  Debug Mode (server): false
  Registry: https://index.docker.io/v1/
  WARNING: bridge-nf-call-iptables is disabled
  WARNING: bridge-nf-call-ip6tables is disabled
  Insecure Registries:
   127.0.0.0/8
  </code></pre>
  
  <p><strong>Edit per first answer below</strong></p>
  
  <p>So I tried leaving with stop/start surrounding commands.  I did:</p>
  
  <pre><code># docker swarm leave --force
  Node left the swarm.
  # service docker stop
  Redirecting to /bin/systemctl stop  docker.service
  # 
  # service docker start
  Redirecting to /bin/systemctl start  docker.service
  
  # docker swarm init --advertise-addr 10.1.1.40:2377
  Swarm initialized: current node (0e0y2k2hngnwyeg86ilzbrjmu) is now a manager.
  
  To add a worker to this swarm, run the following command:
  docker swarm join \
      --token SWMTKN-1-2ggj60tnbppgjlg63a58oe5pqtv0vfrpj81hheawanf76x7cjc-7v48qak22wd03y3jyv903a9if \
  10.1.1.40:2377
  </code></pre>
  
  <p>Then on the worker I did:</p>
  
  <pre><code># docker swarm leave
  Node left the swarm.
  # service docker stop
  Redirecting to /bin/systemctl stop  docker.service
  # service docker start
  Redirecting to /bin/systemctl start  docker.service
  # docker swarm join \
  &gt;     --token SWMTKN-1-2ggj60tnbppgjlg63a58oe5pqtv0vfrpj81hheawanf76x7cjc-    7v48qak22wd03y3jyv903a9if \
      &gt;     10.1.1.40:2377
  Error response from daemon: Timeout was reached before node was joined.     Attempt to join the cluster will continue in the background. Use "docker info" command to see the current swarm status of your node.
  </code></pre>
  
  <p>Which is obviously the same behavior...</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I have tried all the steps outlined by @Miad Abrin.  I still get the same behavior.  I am guessing the cause is related to the CERTS errors I see when I do:</p>
  
  <pre><code># journalctl -xe
  Aug 29 12:26:15 dockerd[6577]: time="2016-08-29T12:26:15.554904435-04:00" level=warning msg="failed to retrieve remote root CA certificate: rpc
  Aug 29 12:26:15 dockerd[6577]: time="2016-08-29T12:26:15.555400400-04:00" level=warning msg="failed to retrieve remote root CA certificate: rpc
  Aug 29 12:26:15 dockerd[6577]: time="2016-08-29T12:26:15.555478782-04:00" level=warning msg="failed to retrieve remote root CA certificate: rpc
  Aug 29 12:26:15 dockerd[6577]: time="2016-08-29T12:26:15.555528929-04:00" level=warning msg="failed to retrieve remote root CA certificate: rpc
  Aug 29 12:26:15 dockerd[6577]: time="2016-08-29T12:26:15.555685464-04:00" level=warning msg="failed to retrieve remote root CA certificate: rpc
  </code></pre>
  
  <p>Does anyone know the cause of this and how to correct?</p>