title: failing to connect to spark driver when submitting job to spark in yarn mode
body: <p>When I submit a spark job to the cluster it failed with the following exeption in the shell:</p>
  
  <pre><code>&gt; Exception in thread "main" org.apache.spark.SparkException:
  &gt; Application application_1497125798633_0065 finished with failed status
  &gt;         at org.apache.spark.deploy.yarn.Client.run(Client.scala:1244)
  &gt;         at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1290)
  &gt;         at org.apache.spark.deploy.yarn.Client.main(Client.scala)
  &gt;         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  &gt;         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  &gt;         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  &gt;         at java.lang.reflect.Method.invoke(Method.java:498)
  &gt;         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:750)
  &gt;         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
  &gt;         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
  &gt;         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
  &gt;         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 17/06/29 10:25:36 INFO ShutdownHookManager: Shutdown hook called
  </code></pre>
  
  <p>This is what it gives in Yarn logs:</p>
  
  <pre><code>&gt; Caused by: java.io.IOException: Failed to connect to /0.0.0.0:35994 at
  &gt; org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
  &gt; at
  &gt; org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
  &gt; at
  &gt; org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
  &gt; at org.apache.spark.rpc.netty.Outbox$anon$1.call(Outbox.scala:194) at
  &gt; org.apache.spark.rpc.netty.Outbox$anon$1.call(Outbox.scala:190) at
  &gt; java.util.concurrent.FutureTask.run(FutureTask.java:266) at
  &gt; java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  &gt; at
  &gt; java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  &gt; at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>Which I guess means it failed to connect to the driver. I tried to increase "spark.yarn.executor.memoryOverhead" parameter but that didn't work.</p>
  
  <p>This is the submit command I use:</p>
  
  <pre><code>/bin/spark-submit \
    --class example.Hello \
    --jars ... \
    --master yarn \
    --deploy-mode cluster \
    --supervise \
    --conf spark.yarn.driver.memoryOverhead=1024 ...(jar file path)
  </code></pre>
  
  <p>I am using HDP-2.6.1.0 and spark 2.1.1</p>