title: Multiple IP addresses and Host Names used by Spark Driver and Master
body: <p>Spark Master listens on several ports. Unfortunately the IP address / hostname scheme used differs among them - and it often happens that connections failed.  </p>
  
  <p>Then we are left to wonder: how to fix the connection problems:  Spark decides on its own how to translate among:</p>
  
  <ul>
  <li>hostname</li>
  <li>hostname.local  (on mac os/x)</li>
  <li>hostname.domain</li>
  <li>localhost</li>
  <li>localhost.localdomain</li>
  <li>127.0.0.1</li>
  <li>external IP address</li>
  <li>internal ip address (on AWS)</li>
  </ul>
  
  <p>The important consideration:  some of the networking clients/connections need an <strong>exact string match</strong>  to successfully contact the master. So in that case <code>127.0.0.1</code> is not the same as <code>hostname</code>.  I have seen in cases where <code>hostname</code> works and <code>hostname.local</code> does not: that one is a Mac-centric problem. But .. then the former stops working - and I lack the tools to troubleshoot why.  </p>
  
  <p>The <code>--master</code> provides opportunities for confusion on the Linux when you have an internal and external IP address. </p>
  
  <p>Below is an example on my Mac.  I see other patterns on AWS and yet other ones on standalone clusters.  It is all perplexing and time consuming since it is not clearly documented either:</p>
  
  <ul>
  <li>where the mappings occur</li>
  <li>how to achieve a consistent master address string across:
  
  <ul>
  <li>master </li>
  <li>master web ui</li>
  <li>akka address for master</li>
  </ul></li>
  </ul>
  
  <p>Below we see output when the <code>--master</code> option were provided to spark-submit. </p>
  
  <pre><code>--master spark://mellyrn:7077 
  </code></pre>
  
  <p>Notice the variety of ip addresses</p>
  
  <pre><code>http://25.x.x.x:4040
  akka.tcp://sparkMaster@mellyrn:7077
  mellyrn/127.0.0.1:707
  </code></pre>
  
  <p>Here is the output on MAC:</p>
  
  <pre><code>15/07/31 12:21:34 INFO SparkEnv: Registering OutputCommitCoordinator
  15/07/31 12:21:34 INFO Utils: Successfully started service 'SparkUI' on port 4040.
  15/07/31 12:21:34 INFO SparkUI: Started SparkUI at http://25.101.19.24:4040
  15/07/31 12:21:34 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@mellyrn:7077/user/Master...
  15/07/31 12:21:35 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@mellyrn:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@mellyrn:7077
  15/07/31 12:21:35 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@mellyrn:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: mellyrn/127.0.0.1:7077
  15/07/31 12:21:54 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@mellyrn:7077/user/Master...
  15/07/31 12:21:54 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@mellyrn:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@mellyrn:7077
  15/07/31 12:21:54 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@mellyrn:7077]. Address is now gated for 5000 
  </code></pre>
  
  <p>On Linux the spark connection with --master option <strong>does</strong> work (though .setMaster() does not reliably).  Yet even on linux there is a variety of master/driver strings generated:</p>