title: Flink TaskManager cannot connect to JobManager in docker swarm task
body: <p>I cannot manage to make a TaskManager communicate with the JobManager on a docker swarm stack.</p>
  
  <p>The content of the <code>stack.yml</code> file I use to <code>docker stack deploy</code> is:</p>
  
  <pre><code>version: "3"
  services:
    jobmanager:
      image: affo/flink:1.1.3
      ports:
        - "48081:8081"
      command: jobmanager
      networks:
        - my-net
      deploy:
          mode: replicated
          replicas: 1
          restart_policy:
              condition: none
          placement:
              constraints:
                  - node.role == manager
  
    taskmanager:
      image: affo/flink:1.1.3
      depends_on:
        - jobmanager
      command: taskmanager
      networks:
        - my-net
      deploy:
          mode: replicated
          replicas: 4
          restart_policy:
              condition: none
          placement:
              constraints:
                  - node.role != manager
  
  networks:
      my-net:
          external: true
  </code></pre>
  
  <p>Docker image <code>affo/flink:1.1.3</code> is a push on dockerhub of the image built following the README @ <a href="https://github.com/apache/flink/tree/release-1.1.3-rc2/flink-contrib/docker-flink" rel="nofollow noreferrer">https://github.com/apache/flink/tree/release-1.1.3-rc2/flink-contrib/docker-flink</a>.</p>
  
  <p>Network <code>my-net</code> is an overlay attachable network.</p>
  
  <p>I tried to ping every container from others using DNS resolution and everything works correctly.</p>
  
  <p>However no TaskManager can make it through to the JobManager.</p>
  
  <p>I report the JobManager log: <a href="http://pastebin.com/Ai5s4Xvr" rel="nofollow noreferrer">http://pastebin.com/Ai5s4Xvr</a></p>
  
  <p>And the log of one TaskManager: <a href="http://pastebin.com/ty5pZhSp" rel="nofollow noreferrer">http://pastebin.com/ty5pZhSp</a></p>
  
  <p>The JM has VIP 10.0.42.7. And <code>jobmanager.rpc.address</code> is set to <code>jobmanager</code> which resolves to 10.0.42.7.</p>
  
  <p>Any help or hint on where to start solving the problem would be appreciated.</p>
  
  <p>Thanks a lot!</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I add the output of <code>docker exec &lt;jobmanager&gt; netstat -tulpn</code>:</p>
  
  <pre><code>Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
  tcp        0      0 127.0.0.11:40762        0.0.0.0:*               LISTEN      -
  tcp        0      0 ::ffff:10.0.42.7:6123   :::*                    LISTEN      218/java
  tcp        0      0 :::8081                 :::*                    LISTEN      218/java
  tcp        0      0 :::34963                :::*                    LISTEN      218/java
  udp        0      0 127.0.0.11:57000        0.0.0.0:*                           -
  </code></pre>
  
  <p>And of <code>docker exec &lt;a_taskmanager&gt; telnet jobmanager 6123</code>:</p>
  
  <pre><code>telnet: can't connect to remote host (10.0.42.7): Connection refused
  </code></pre>
  
  <p>I also put a link to a maybe-related issue on github: <a href="https://github.com/docker/docker/issues/28795" rel="nofollow noreferrer">https://github.com/docker/docker/issues/28795</a>.</p>
  
  <p>Thanks again</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I recently managed to change the <code>jobmanager.rpc.address</code> to <code>0.0.0.0</code> only at the JobManager's and now it is effectively listening:</p>
  
  <pre><code>Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
  tcp        0      0 127.0.0.11:56218        0.0.0.0:*               LISTEN      -
  tcp        0      0 :::6123                 :::*                    LISTEN      218/java
  tcp        0      0 :::8081                 :::*                    LISTEN      218/java
  tcp        0      0 :::55231                :::*                    LISTEN      218/java
  udp        0      0 127.0.0.11:47549        0.0.0.0:*                           -
  </code></pre>
  
  <p>I can even <code>nc</code> or <code>telnet</code> from TaskManagers.</p>
  
  <p>However, now the problem is (on the JobManager):</p>
  
  <pre><code>2017-02-09 10:31:20,794 ERROR akka.remote.EndpointWriter                
  - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient
  [Actor[akka.tcp://flink@10.0.42.7:6123/]] arriving at [akka.tcp://flink@10.0.42.7:6123]
  inbound addresses are [akka.tcp://flink@0.0.0.0:6123]
  </code></pre>
  
  <p>Any help would be appreciated, thank you!</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I think I isolated the problem. Issue opened on github: <a href="https://github.com/docker/docker/issues/30874" rel="nofollow noreferrer">https://github.com/docker/docker/issues/30874</a></p>