<p>Hadoop server is in Kubernetes. And the Hadoop client is located on an external network. So I try to use the Hadoop server using a kubernetes-service. But <code>hadoop fs -put</code> does not work for the Hadoop client. As I know, the namenode gives the datanode IP to Hadoop client. If yes, where does the namenode get IP from?</p>

K.k
<p>You can check my <a href="https://stackoverflow.com/a/53137701/2989261">other answer</a>. HDFS is not production ready in K8s yet (as of this writing)</p>

<p>The namenode gives the client the IP addresses of the datanodes and it knows those when they join the cluster as shown below:</p>

<p><a href="https://i.stack.imgur.com/BjMtd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BjMtd.png" alt="datanodes"></a></p>

<p>The issue in K8s is that you have to expose each data node as a service or external IP, but the namenode sees the datanodes with their pod IP addresses that are not available to the outside world. Also, HDFS doesn't provide a <a href="https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html" rel="nofollow noreferrer">publish IP for each datanode config</a> where you could force to use a service IP, so you'll have to do fancy custom networking or your client has to be inside the podCidr (Which kind of defeats the purpose of HDFS being a distributed filesystem).</p>
