title: Spark Remote execution to Cluster fails - HDFS connection Refused at 8020
body: <p>I am having issues submitting a spark-submit remote job from a machine outside from the Spark Cluster running on YARN.</p>
  
  <pre><code>Exception in thread "main" java.net.ConnectionException: Call from remote.dev.local/192.168.10.65 to target.dev.local:8020 failed on connection exception: java.net.ConnectionException: Connection Refused
  </code></pre>
  
  <p>In my core-site.xml:</p>
  
  <pre><code>&lt;property&gt;
    &lt;name&gt;fs.defaultFS&lt;/name&gt;
    &lt;value&gt;hdfs://target.dev.local:8020&lt;/value&gt;
  &lt;property&gt;
  </code></pre>
  
  <p>Also at my hdfs-site.xml in the cluster I have disbled permissions checking for HDFS:</p>
  
  <pre><code>&lt;property&gt;
    &lt;name&gt;dfs.permissions.enabled&lt;/name&gt;
    &lt;value&gt;false&lt;/value&gt;
  &lt;property&gt;
  </code></pre>
  
  <p>Also, when I telnet from the machine outside the cluster:</p>
  
  <pre><code>telnet target.dev.local 8020
  </code></pre>
  
  <p>I am getting</p>
  
  <pre><code>telnet: connect to address 192.168.10.186: Connection Refused
  </code></pre>
  
  <p>But, when I </p>
  
  <pre><code>telnet target.dev.local 9000
  </code></pre>
  
  <p>it says <strong>Connected</strong>.</p>
  
  <p>Also when I <code>ping target.dev.local</code> it works.</p>
  
  <p>My spark-submit script from the remote machine is:</p>
  
  <pre><code>export HADOOP_CONF_DIR=/&lt;path_to_conf_dir_copied_from_cluster&gt;/
  
  spark-submit --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode cluster \
  --driver-memory 5g \
  --executor-memory 50g \
  --executor-cores 5 \
  --queue default \
  &lt;path to jar&gt;.jar \
  10
  </code></pre>
  
  <p>What am I missing here?</p>