title: spark-shell cannot connect to remote master
body: <p>I have two same version spark.
  <code>192.168.2.230</code> as master and <code>192.168.2.5</code> as slave.<br>
  I tried <code>./spark-submit --version</code> on <code>2.5</code> and <code>2.230</code>:</p>
  
  <pre><code>Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
        /_/
  
  Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_121
  Branch
  Compiled by user jenkins on 2016-12-16T02:04:48Z
  Revision
  Url
  Type --help for more information.
  </code></pre>
  
  <p>their spark version is same.<br>
  Then I  changed the master config(<code>2.230</code>) <code>conf/spark-env.sh</code> set:</p>
  
  <pre><code>SPARK_LOCAL_IP=192.168.2.230 
  SPARK_MASTER_HOST=192.168.2.230
  </code></pre>
  
  <p>and run master(<code>2.230</code>):</p>
  
  <pre><code>sbin/start-all.sh
  </code></pre>
  
  <p>I can see the master management ui on port <code>http://192.168.2.230:8080</code>.</p>
  
  <p>Then I try to use spark shell on <code>2.5</code>:</p>
  
  <pre><code>./spark-shell --master=spark://192.168.2.230:7077
  </code></pre>
  
  <p>there is some errors occur and cannot connect to <code>2.230</code>, it likes version incompatible, but these two spark copy from the same <code>tar.gz</code>, here is the errors:</p>
  
  <pre><code>[root@localhost bin]# ./spark-shell --master=spark://192.168.2.230:7077
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  Setting default log level to "WARN".
  To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
  17/04/19 03:04:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  17/04/19 03:04:20 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.2.5 instead (on interface enp3s0)
  17/04/19 03:04:20 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
  17/04/19 03:04:20 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 192.168.2.230:7077
  org.apache.spark.SparkException: Exception thrown in awaitResult
      at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
      at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
      at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
      at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
      at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
      at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
      at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.lang.RuntimeException: java.io.InvalidClassException: org.apache.spark.rpc.netty.RequestMessage; local class incompatible: stream classdesc serialVersionUID = -5447855329526097695, local class serialVersionUID = -2221986757032131007
      at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)
      at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
      at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
      at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
      at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:109)
      at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1$$anonfun$apply$1.apply(NettyRpcEnv.scala:258)
      at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
      at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:310)
      at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1.apply(NettyRpcEnv.scala:257)
      at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
      at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:256)
      at org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:588)
      at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:570)
      at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:149)
      at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:102)
      at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:104)
      at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
      at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
      at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
      at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
      at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:86)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
      at java.lang.Thread.run(Thread.java:745)
  
      at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:189)
      at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:121)
      at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
      at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)
      at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)
      at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)
      at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)
      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
      at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
      ... 1 more
  17/04/19 03:04:40 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
  17/04/19 03:04:40 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
  17/04/19 03:04:40 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
  17/04/19 03:04:40 ERROR SparkContext: Error initializing SparkContext.
  java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
      at scala.Predef$.require(Predef.scala:224)
      at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
      at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:524)
      at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
      at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
      at $line3.$read$$iw$$iw.&lt;init&gt;(&lt;console&gt;:15)
      at $line3.$read$$iw.&lt;init&gt;(&lt;console&gt;:42)
      at $line3.$read.&lt;init&gt;(&lt;console&gt;:44)
      at $line3.$read$.&lt;init&gt;(&lt;console&gt;:48)
      at $line3.$read$.&lt;clinit&gt;(&lt;console&gt;)
      at $line3.$eval$.$print$lzycompute(&lt;console&gt;:7)
      at $line3.$eval$.$print(&lt;console&gt;:6)
      at $line3.$eval.$print(&lt;console&gt;)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
      at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
      at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
      at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
      at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
      at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
      at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
      at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
      at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)
      at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:105)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
      at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
      at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
      at org.apache.spark.repl.Main$.doMain(Main.scala:68)
      at org.apache.spark.repl.Main$.main(Main.scala:51)
      at org.apache.spark.repl.Main.main(Main.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
    at scala.Predef$.require(Predef.scala:224)
    at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
    at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:524)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
    at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
    ... 47 elided
  &lt;console&gt;:14: error: not found: value spark
         import spark.implicits._
            ^
  &lt;console&gt;:14: error: not found: value spark
         import spark.sql
            ^
  Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
        /_/
  
  Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_121)
  Type in expressions to have them evaluated.
  Type :help for more information.
  
  scala&gt;
  </code></pre>
  
  <p>The reason why <code>2.5</code> cannot connect to <code>2.230</code> likes class incompatible:<br>
  <code>Caused by: java.lang.RuntimeException: java.io.InvalidClassException: org.apache.spark.rpc.netty.RequestMessage; local class incompatible: stream classdesc serialVersionUID = -5447855329526097695, local class serialVersionUID = -2221986757032131007</code>  </p>
  
  <p>But i tried locally on <code>2.230</code>: <code>./spark-shell --master spark://192.168.2.230:7077</code> it success...   </p>
  
  <p>Why i cannot use remote spark-shell (<code>2.5</code>) to connect master (<code>2.230</code>)?</p>