title: Apache Spark Running Locally Giving Refused Connection Error
body: <p>I have Spark and Hadoop installed on OS X. I successfully worked through an example where Hadoop ran locally, had files stored in HDFS and I ran spark with</p>
  
  <pre class="lang-sh prettyprint-override"><code>spark-shell --master yarn-client
  </code></pre>
  
  <p>and from within the shell worked with HDFS. I'm having problems, however, trying to get Spark to run without HDFS, just locally on my machine. I looked at <a href="https://stackoverflow.com/questions/22129321/connection-refused-when-running-sparkpi-locally">this answer</a> but it doesn't feel right messing around with environment variables when the <a href="http://spark.apache.org/docs/latest/index.html" rel="nofollow noreferrer">Spark documentation</a> says </p>
  
  <blockquote>
    <p>It’s easy to run locally on one machine — all you need is to have java
    installed on your system PATH, or the JAVA_HOME environment variable
    pointing to a Java installation.</p>
  </blockquote>
  
  <p>If I run the basic <code>SparkPi</code> example I get the correct output.</p>
  
  <p>If I try run the sample Java app <a href="http://spark.apache.org/docs/latest/quick-start.html#tab_java_3" rel="nofollow noreferrer">they provide</a>, again, I get output, but this time with <strong>connection refused</strong> errors relating to <strong>port 9000</strong>, which sounds like it's trying to connect to <strong>Hadoop</strong>, but I don't know why because I'm not specifying that</p>
  
  <pre class="lang-sh prettyprint-override"><code>    $SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local[4] ~/study/scala/sampleJavaApp/target/simple-project-1.0.jar
      Exception in thread "main" java.net.ConnectException: Call From 37-2-37-10.tssg.org/10.37.2.37 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  ...
  ...
  ...
  org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:604)
          at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:699)
          at org.apache.hadoop.ipc.Client$Connection.access(Client.java:367)
          at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462)
          at org.apache.hadoop.ipc.Client.call(Client.java:1381)
          ... 51 more
      15/07/31 11:05:06 INFO spark.SparkContext: Invoking stop() from shutdown hook
      15/07/31 11:05:06 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
  ...
  ...
  ...
      15/07/31 11:05:06 INFO ui.SparkUI: Stopped Spark web UI at http://10.37.2.37:4040
      15/07/31 11:05:06 INFO scheduler.DAGScheduler: Stopping DAGScheduler
      15/07/31 11:05:06 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
      15/07/31 11:05:06 INFO util.Utils: path = /private/var/folders/cg/vkq1ghks37lbflpdg0grq7f80000gn/T/spark-c6ba18f5-17a5-4da9-864c-509ec855cadf/blockmgr-b66cc31e-7371-472f-9886-4cd33d5ba4b1, already present as root for deletion.
      15/07/31 11:05:06 INFO storage.MemoryStore: MemoryStore cleared
      15/07/31 11:05:06 INFO storage.BlockManager: BlockManager stopped
      15/07/31 11:05:06 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
      15/07/31 11:05:06 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
      15/07/31 11:05:06 INFO spark.SparkContext: Successfully stopped SparkContext
      15/07/31 11:05:06 INFO util.Utils: Shutdown hook called
      15/07/31 11:05:06 INFO util.Utils: Deleting directory /private/var/folders/cg/vkq1ghks37lbflpdg0grq7f80000gn/T/spark-c6ba18f5-17a5-4da9-864c-509ec855cadf
  </code></pre>
  
  <p>Any pointers/explanations as to where I'm going wrong would be much appreciated!</p>
  
  <hr>
  
  <h1>UPDATE</h1>
  
  <p>It seems that the fact I have the environment variable <code>HADOOP_CONF_DIR</code> set is causing some issues. Under that directory, I have <code>core-site.xml</code> which contains the following</p>
  
  <pre class="lang-xml prettyprint-override"><code>&lt;property&gt;
       &lt;name&gt;fs.default.name&lt;/name&gt;                                     
       &lt;value&gt;hdfs://localhost:9000&lt;/value&gt;                             
  &lt;/property&gt; 
  </code></pre>
  
  <p>If I change the value e.g. <code>&lt;value&gt;hdfs://localhost:9100&lt;/value&gt;</code> then when I attempt to run the spark job, the connection refused error refers to this changed port</p>
  
  <pre class="lang-sh prettyprint-override"><code>Exception in thread "main" java.net.ConnectException: Call From 37-2-37-10.tssg.org/10.37.2.37 to localhost:9100 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused 
  </code></pre>
  
  <p>So for some reason, despite instructing it to run locally, it is trying to connect to HDFS. If I remove the <code>HADOOP_CONF_DIR</code> environment variable, the job works fine. </p>