<p>I'm currently in the process of setting up a Kerberized environment for submitting Spark Jobs using Livy in Kubernetes.</p>

<p>What I've achieved so far:</p>

<ul>
<li>Running Kerberized HDFS Cluster</li>
<li>Livy using SPNEGO</li>
<li>Livy submitting Jobs to k8s and spawning Spark executors</li>
<li>KNIME is able to interact with Namenode and Datanodes from outside the k8s Cluster</li>
</ul>

<p>To achieve this I used the following Versions for the involved components:</p>

<ul>
<li>Spark 2.4.4</li>
<li>Livy 0.5.0 (The currently only supported version by KNIME)</li>
<li>Namenode and Datanode 2.8.1</li>
<li>Kubernetes 1.14.3</li>
</ul>

<p>What I'm currently struggling with:</p>

<ul>
<li>Accessing HDFS from the Spark executors</li>
</ul>

<p>The error message I'm currently getting, when trying to access HDFS from the executor is the following:</p>

<pre><code>org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "livy-session-0-1575455179568-exec-1/10.42.3.242"; destination host is: "hdfs-namenode-0.hdfs-namenode.hdfs.svc.cluster.local":8020;
</code></pre>

<p>The following is the current state:</p>

<ol>
<li>KNIME connects to HDFS after having successfully challenged against the KDC (using Keytab + Principal) --> Working</li>
<li>KNIME puts staging jars to HDFS --> Working</li>
<li>KNIME requests new Session from Livy (SPNEGO challenge) --> Working</li>
<li>Livy submits Spark Job with k8s master / spawns executors --> Working</li>
<li>KNIME submits tasks to Livy which should be executed by the executors --> Basically working</li>
<li>When trying to access HDFS to read a file the error mentioned before occurs --> The problem</li>
</ol>

<p>Since KNIME is placing jar files on HDFS which have to be included in the dependencies for the Spark Jobs it is important to be able to access HDFS. (KNIME requires this to be able to retrieve preview data from DataSets for example)</p>

<p>I tried to find a solution to this but unfortunately, haven't found any useful resources yet.
I had a look at the code an checked <code>UserGroupInformation.getCurrentUser().getTokens()</code>.
But that collection seems to be empty. That's why I assume that there are not Delegation Tokens available.</p>

<p>Has anybody ever achieved running something like this and can help me with this?</p>

<p>Thank you all in advance!</p>

denglai
<p>For everybody struggeling with this:
It took a while to find the reason on why this is not working, but basically it is related to Spark's Kubernetes implementation as of 2.4.4.
There is no override defined for <code>CoarseGrainedSchedulerBackend</code>'s <a href="https://github.com/apache/spark/blob/branch-2.4/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala" rel="nofollow noreferrer"><code>fetchHadoopDelegationTokens</code></a> in <a href="https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L687" rel="nofollow noreferrer"><code>KubernetesClusterSchedulerBackend</code></a>.</p>

<p>There has been the <a href="https://github.com/apache/spark/pull/21669" rel="nofollow noreferrer">pull request</a> which will solve this by passing secrets to executors containing the delegation tokens.
It was already pulled into master and is available in Spark 3.0.0-preview but is not, at least not yet, available in the Spark 2.4 branch.</p>
