title: Local Spark talking to remote HDFS?
body: <p>I have a file in HDFS inside my HortonWorks HDP 2.3_1 VirtualBox VM. </p>
  
  <p>If I go into the guest spark-shell and refer to the file thus, it works fine </p>
  
  <p>val words=sc.textFile("hdfs:///tmp/people.txt") 
    words.count </p>
  
  <p>However if I try to access it from a local Spark app on my Windows host, it doesn't work </p>
  
  <pre><code> val conf = new SparkConf().setMaster("local").setAppName("My App") 
    val sc = new SparkContext(conf) 
  
    val words=sc.textFile("hdfs://localhost:8020/tmp/people.txt") 
    words.count 
  </code></pre>
  
  <p>Emits </p>
  
  <p>Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-452094660-10.0.2.15-1437494483194:blk_1073742905_2098 file=/tmp/people.txt
      at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:838)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:526)</p>
  
  <p>The port 8020 is open, and if I choose the wrong file name, it will tell me </p>
  
  <pre><code>Input path does not exist: hdfs://localhost:8020/tmp/people.txt!!
  </code></pre>
  
  <p>localhost:8020 should be correct as the guest HDP VM hat NAT port tunneling to my host Windows box.</p>
  
  <p>And I it's telling that if I give it the wrong name I get an appropriate exception</p>
  
  <p>My pom has </p>
  
  <pre><code>&lt;dependency&gt;
                  &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
                  &lt;artifactId&gt;spark-core_2.11&lt;/artifactId&gt;
                  &lt;version&gt;1.4.1&lt;/version&gt;
                  &lt;scope&gt;provided&lt;/scope&gt;
          &lt;/dependency&gt;
  </code></pre>
  
  <p>Am I doing something wrong? And what is the BlockMissingException trying to tell me?</p>