<p>So I have a K8s cluster up and running and I want to run Spark jobs on top of it.</p>

<p><strong>Kubernetes is v1.15.3 and Spark v2.4.5</strong>.</p>

<p>Now for data storage I am thinking of using HDFS but I do not want to install the entire Hadoop library which includes YARN and MapReduce (pls correct me if I am wrong).</p>

<p>I have seen <a href="https://github.com/apache-spark-on-k8s/kubernetes-HDFS" rel="nofollow noreferrer">this</a> repository as the only direct solution available online but it's not working for me currently. </p>

<p>When I try to deploy it same as how it's mentioned in the ReadMe on the repo, I see that multiple pods are created and as soon as all of them go to <code>Running</code> state, the <code>my-hdfs-namenode-0</code> pod goes into <code>Error</code> state and a lot of pods start crashing. </p>

<p>This is the error I get in the log from <code>kubectl logs pod/my-hdfs-namenode-0</code>:</p>

<pre><code>20/05/11 09:47:57 ERROR namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Unable to construct journal, qjournal://my-hdfs-journalnode-1.my-hdfs-journalnode.default.svc.cluster.local:8485;my-hdfs-journalnode-2.my-hdfs-journalnode.default.svc.cluster.local:8485;my-hdfs-journalnode-0.my-hdfs-journalnode.default.svc.cluster.local:8485/hdfs-k8s
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1638)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:282)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournalsForWrite(FSEditLog.java:247)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:985)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1429)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1636)
    ... 5 more
Caused by: java.lang.NullPointerException
    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.getName(IPCLoggerChannelMetrics.java:107)
    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.create(IPCLoggerChannelMetrics.java:91)
    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel.&lt;init&gt;(IPCLoggerChannel.java:178)
    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$1.createLogger(IPCLoggerChannel.java:156)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:367)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:149)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.&lt;init&gt;(QuorumJournalManager.java:116)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.&lt;init&gt;(QuorumJournalManager.java:105)
    ... 10 more
</code></pre>

<p>I'm guessing it's a name resolution (DNS) related error? <a href="https://pastebin.com/yefbEH8f" rel="nofollow noreferrer">Here</a> is the complete log for reference.</p>

<p>Now, this repository isn't actively maintained so if someone can suggest either how I can resolve this error or another way how I can deploy HDFS to my Kubernetes cluster. </p>

Dr. Ameto
<p>In general, I suggest you don't use HDFS within k8s... (although, <a href="https://www.uber.com/blog/hadoop-container-blog/" rel="nofollow noreferrer">companies like Uber have done it</a>, they also have large teams to maintain that effort)</p>
<ol>
<li>NameNode HA would need to be containerized, and NameNode filesystem must be stateful.</li>
<li>You need Zookeeper QJM, which competes with etcd, in a way, for leader election purposes.</li>
</ol>
<p>HDFS was designed before k8s persistent volumes were really thought about. <a href="https://ozone.apache.org/" rel="nofollow noreferrer">Hadoop Ozone project is now generally available</a>, and meant to work around these limitations. It has k8s deployment instructions, as well.</p>
<p>Alternatively, I suggest you look into using MinIO, or Project Rook (on CephFS), both of which offer a Hadoop-compatible file system (HCFS)</p>
<hr />
<p>If you must use HDFS, then you can set it up outside k8s, then make requests to it from within the containers.</p>
<p>Regarding YARN, make sure to watch the Yunikorn project (YARN on k8s)</p>
