title: Accesing Hdfs from Spark gives TokenCache error Can't get Master Kerberos principal for use as renewer
body: <p>I'm trying to run a test Spark script in order to connect Spark to hadoop.
  The script is the following</p>
  
  <pre><code>from pyspark import SparkContext
  
  sc = SparkContext("local", "Simple App")
  file = sc.textFile("hdfs://hadoop_node.place:9000/errs.txt")
  errors = file.filter(lambda line: "ERROR" in line)
  errors.count()
  </code></pre>
  
  <p>When I run it with pyspark I get </p>
  
  <blockquote>
    <p>py4j.protocol.Py4JJavaError: An error occurred while calling
    o21.collect. : java.io.IOException: <strong>Can't get Master Kerberos
    principal for use as renewer</strong>
            at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:116)
            at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
            at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
            at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:187)
            at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:251)
            at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:140)
            at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
            at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
            at scala.Option.getOrElse(Option.scala:120)
            at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
            at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
            at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
            at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
            at scala.Option.getOrElse(Option.scala:120)
            at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
            at org.apache.spark.api.python.PythonRDD.getPartitions(PythonRDD.scala:46)
            at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
            at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
            at scala.Option.getOrElse(Option.scala:120)
            at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
            at org.apache.spark.SparkContext.runJob(SparkContext.scala:898)
            at org.apache.spark.rdd.RDD.collect(RDD.scala:608)
            at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:243)
            at org.apache.spark.api.java.JavaRDD.collect(JavaRDD.scala:27)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
            at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
            at py4j.Gateway.invoke(Gateway.java:259)
            at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
            at py4j.commands.CallCommand.execute(CallCommand.java:79)
            at py4j.GatewayConnection.run(GatewayConnection.java:207)
            at java.lang.Thread.run(Thread.java:744)</p>
  </blockquote>
  
  <p>This happens despite the facts that</p>
  
  <ul>
  <li>I've done a kinit and a klist shows I have the correct tokens</li>
  <li>when I issue a ./bin/hadoop fs -ls hdfs://hadoop_node.place:9000/errs.txt
  it shows the file </li>
  <li>Both the local hadoop client and spark have the same configuration file</li>
  </ul>
  
  <p>The core-site.xml  in the spark/conf and hadoop/conf folders is the following
  (got it from one of the hadoop nodes)</p>
  
  <pre><code>&lt;configuration&gt;
      &lt;property&gt;
  
          &lt;name&gt;hadoop.security.auth_to_local&lt;/name&gt;
          &lt;value&gt;
              RULE:[1:$1](.*@place)s/@place//
              RULE:[2:$1/$2@$0](.*/node1.place@place)s/^([a-zA-Z]*).*/$1/
              RULE:[2:$1/$2@$0](.*/node2.place@place)s/^([a-zA-Z]*).*/$1/
              RULE:[2:$1/$2@$0](.*/node3.place@place)s/^([a-zA-Z]*).*/$1/
              RULE:[2:$1/$2@$0](.*/node4.place@place)s/^([a-zA-Z]*).*/$1/
              RULE:[2:$1/$2@$0](.*/node5.place@place)s/^([a-zA-Z]*).*/$1/
              RULE:[2:$1/$2@$0](.*/node6.place@place)s/^([a-zA-Z]*).*/$1/
              RULE:[2:$1/$2@$0](.*/node7.place@place)s/^([a-zA-Z]*).*/$1/
              RULE:[2:nobody]
              DEFAULT
          &lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;net.topology.node.switch.mapping.impl&lt;/name&gt;
          &lt;value&gt;org.apache.hadoop.net.TableMapping&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;net.topology.table.file.name&lt;/name&gt;
          &lt;value&gt;/etc/hadoop/conf/topology.table.file&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;fs.defaultFS&lt;/name&gt;
          &lt;value&gt;hdfs://server.place:9000/&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;hadoop.security.authentication&lt;/name&gt;
        &lt;value&gt;kerberos&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
        &lt;name&gt;hadoop.security.authorization&lt;/name&gt;
        &lt;value&gt;true&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
        &lt;name&gt;hadoop.proxyuser.hive.hosts&lt;/name&gt;
        &lt;value&gt;*&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
        &lt;name&gt;hadoop.proxyuser.hive.groups&lt;/name&gt;
        &lt;value&gt;*&lt;/value&gt;
      &lt;/property&gt;
  
  &lt;/configuration&gt;
  </code></pre>
  
  <p>Can someone point out what am I missing?</p>