<p>Revisiting the data locality for Spark on Kubernetes question: if the Spark pods are colocated on the same nodes as the HDFS data node pods then does data locality work ?</p>
<p>The Q&amp;A session here: <a href="https://www.youtube.com/watch?v=5-4X3HylQQo" rel="nofollow noreferrer">https://www.youtube.com/watch?v=5-4X3HylQQo</a> seems to suggest it doesn't.</p>

<p>Locality is an issue Spark on Kubernetes. Basic Data locality does work if the Kubernetes provider provides a network topology plugins that are required to resolve where the data is and where the spark nodes should be run.  <em>and you have built kubernetes to include the <a href="https://github.com/apache-spark-on-k8s/kubernetes-HDFS" rel="nofollow noreferrer">code here</a></em></p>
<p>There is a method to test <a href="https://github.com/apache-spark-on-k8s/kubernetes-HDFS/blob/master/topology/README.md" rel="nofollow noreferrer">this data locality</a>. I have copied it here for completeness:</p>
<p>Here's how one can check if data locality in the namenode works.</p>
<p>Launch a HDFS client pod and go inside the pod.</p>
<pre><code>$ kubectl run -i --tty hadoop --image=uhopper/hadoop:2.7.2
--generator=&quot;run-pod/v1&quot; --command -- /bin/bash 
</code></pre>
<p>Inside the pod, create a simple text file on HDFS.</p>
<pre><code>$ hadoop fs
-fs hdfs://hdfs-namenode-0.hdfs-namenode.default.svc.cluster.local
-cp file:/etc/hosts /hosts
</code></pre>
<p>Set the number of replicas for the file to the number of your cluster nodes. This ensures that there will be a copy of the file in the cluster node that your client pod is running on. Wait some time until this happens.</p>
<pre><code> `$ hadoop fs -setrep NUM-REPLICAS /hosts` 
</code></pre>
<p>Run the following hdfs cat command. From the debug messages, see which datanode is being used. Make sure it is your local datanode. (You can get this from $ kubectl get pods hadoop -o json | grep hostIP. Do this outside the pod)</p>
<pre><code>$ hadoop --loglevel DEBUG fs
-fs hdfs://hdfs-namenode-0.hdfs-namenode.default.svc.cluster.local
-cat /hosts ... 17/04/24 20:51:28 DEBUG hdfs.DFSClient: Connecting to datanode 10.128.0.4:50010 ... 
</code></pre>
<p>If no, you should check if your local datanode is even in the list from the debug messsages above. If it is not, then this is because step (3) did not finish yet. Wait more. (You can use a smaller cluster for this test if that is possible)</p>
<pre><code>`17/04/24 20:51:28 DEBUG hdfs.DFSClient: newInfo = LocatedBlocks{ fileLength=199 underConstruction=false blocks=[LocatedBlock{BP-347555225-10.128.0.2-1493066928989:blk_1073741825_1001; getBlockSize()=199; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.128.0.4:50010,DS-d2de9d29-6962-4435-a4b4-aadf4ea67e46,DISK], DatanodeInfoWithStorage[10.128.0.3:50010,DS-0728ffcf-f400-4919-86bf-af0f9af36685,DISK], DatanodeInfoWithStorage[10.128.0.2:50010,DS-3a881114-af08-47de-89cf-37dec051c5c2,DISK]]}] lastLocatedBlock=LocatedBlock{BP-347555225-10.128.0.2-1493066928989:blk_1073741825_1001;`
</code></pre>
<p>Repeat the hdfs cat command multiple times. Check if the same datanode is being consistently used.</p>
