title: Spark submission on DCOS cluster fails with java.net.UnknownHostException: hdfs
body: <p>I am running a <code>spark-submit</code> in cluster/rest mode on a DCOS cluster:</p>
  
  <pre><code>$ ./spark-submit  --deploy-mode cluster --master mesos://localhost:7077 --conf spark.master.rest.enabled=true --conf spark.mesos.uris=http://api.hdfs.marathon.l4lb.thisdcos.directory/v1/endpoints/hdfs-site.xml,http://api.hdfs.marathon.l4lb.thisdcos.directory/v1/endpoints/core-site.xml --conf spark.mesos.executor.docker.image=someregistry:5000/someimage:2.0.0-rc3 --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://hdfs/history --conf spark.executor.extraClassPath=/opr/spark/dist/elasticsearch-spark-20_2.11-6.4.2.jar  --conf spark.mesos.driverEnv.SPARK_HDFS_CONFIG_URL=http://api.hdfs.marathon.l4lb.thisdcos.directory/v1/endpoints/hdfs-site.xml --conf spark.executor.memory=42G --conf spark.driver.memory=8G --conf  spark.executor.cores=8 --driver-class-path /opt/spark/dist/elasticsearch-spark-20_2.11-6.4.2.jar http://hostname/somescript.py
  </code></pre>
  
  <p>The task fails as follows:</p>
  
  <pre><code>java.lang.IllegalArgumentException: java.net.UnknownHostException: hdfs
      at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
      at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
      at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
      at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:668)
      at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:604)
      at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2598)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
      at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1853)
      at org.apache.spark.scheduler.EventLoggingListener.&lt;init&gt;(EventLoggingListener.scala:68)
      at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:530)
      at org.apache.spark.api.java.JavaSparkContext.&lt;init&gt;(JavaSparkContext.scala:58)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:236)
      at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
      at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
      at java.lang.Thread.run(Thread.java:748)
  Caused by: java.net.UnknownHostException: hdfs
      ... 26 more
  </code></pre>
  
  <p>I have created a tunnel from my localhost:7707 to master 7707 because the internal uri:7707 was directly reachable;</p>
  
  <p><strong>edit</strong>: I think it could be related to my executors not being able to read <code>core-site.xml</code> where the following declaration resides</p>
  
  <pre><code>&lt;property&gt;
      &lt;name&gt;fs.defaultFS&lt;/name&gt;
      &lt;value&gt;hdfs://hdfs&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p>Do I need to point my local <code>spark</code> installation at these files somehow?</p>