title: Docker swarm mode load balancing not working as described
body: <h2>Update</h2>
  
  <p>I believe the culprit is the master who does not appear to be listening on port 7946. <code>netstat</code> shows that 7946 is listening on the nodes, but not the master. When I check the syslogs for the nodes I see the following error</p>
  
  <pre><code>level=error msg="Failed to join memberlist [10.0.0.12] on retry: 1 error(s) occurred:\n\n* Failed to join 10.0.0.12: dial tcp 10.0.0.12:7946: getsockopt: connection refused"
  </code></pre>
  
  <h2>Original Post</h2>
  
  <p>I am running a three node Swarm Mode cluster in AWS; one master and two workers. This is <strong>swarm mode</strong> not to be confused with <strong>docker swarm</strong> from pre 1.12. </p>
  
  <p>I created all of the services with docker-machine. Each machine is running Ubuntu 15.10 with Docker 1.12.3.</p>
  
  <pre><code>Linux swarm-master-01 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  </code></pre>
  
  <p>Using the master node I have created a service with the following</p>
  
  <pre><code>docker service create --replicas 1 --name myapp -p 3000 myapp
  </code></pre>
  
  <p>When I run <code>docker service ps myapp</code> I get the following output</p>
  
  <pre><code>ID                         NAME     IMAGE         NODE             DESIRED STATE  CURRENT STATE            ERROR
  02awst8p9pezgpkfzqgz8z79t  myapp.1  myapp:latest  swarm-node-01    Running        Running 19 minutes ago
  </code></pre>
  
  <p>The running task is deployed to swarm-node-01.</p>
  
  <p>I checked the auto-selected port which was published publicly</p>
  
  <pre><code>$ docker service inspect myapp | jq .[].Endpoint.Ports[].PublishedPort
  30000
  </code></pre>
  
  <p>According to the <a href="https://docs.docker.com/engine/swarm/key-concepts/#load-balancing" rel="nofollow noreferrer">documentation</a>:</p>
  
  <blockquote>
    <p>External components, such as cloud load balancers, can access the service on the PublishedPort of <strong>any node in the cluster whether or not the node is currently running the task for the service</strong>. All nodes in the swarm route ingress connections to a running task instance.</p>
  </blockquote>
  
  <p>But when I try to curl the nodes who do not have the task running I'm getting <code>connection refused</code>. </p>
  
  <pre><code>$ curl $(docker-machine ip swarm-node-01):30000/stats
  {"uptime":"2016-11-09T14:48:35Z","requestCount":7,"statuses":{"200":7},"pid":1,"open_db_conns":0}
  
  $ curl $(docker-machine ip swarm-node-02):30000/stats
  curl: (7) Failed to connect to [the IP] port 30000: Connection refused
  </code></pre>
  
  <p><strong><em>note: I scrubbed the IP of node-02</em></strong></p>
  
  <hr>
  
  <p>My Troubleshooting:</p>
  
  <ul>
  <li>The nodes are both properly connected to the swarm</li>
  <li>Scaling the service up to 5 (which inherently deploys the task to every node) makes curl work on every node, because the task is deployed to every node. </li>
  </ul>
  
  <p>UPDATE 1</p>
  
  <p>I initialized the swarm with </p>
  
  <pre><code>docker swarm init --advertise-addr 10.0.0.12:2377 --listen-addr 10.0.0.12:2377
  </code></pre>
  
  <p>I checked the syslogs from the nodes and I'm seeing the following errors</p>
  
  <pre><code>level=error msg="Failed to join memberlist [10.0.0.12] on retry: 1 error(s) occurred:\n\n* Failed to join 10.0.0.12: dial tcp 10.0.0.12:7946: getsockopt: connection refused"
  </code></pre>
  
  <p>I checked to see if the ingress port was listening and it doesn't seem to be</p>
  
  <pre><code>ubuntu@swarm-master-01:~$ sudo lsof -i :7946
  ubuntu@swarm-master-01:~$ cat &lt; /dev/tcp/10.0.0.12/7946
  -bash: connect: Connection refused
  -bash: /dev/tcp/10.0.0.12/7946: Connection refused
  ubuntu@swarm-master-01:~$ cat &lt; /dev/tcp/0.0.0.0/7946
  -bash: connect: Connection refused
  -bash: /dev/tcp/0.0.0.0/7946: Connection refused
  </code></pre>