title: Write to HDFS running in Docker from another Docker container running Spark
body: <p>I have a docker image for spark + jupyter (<a href="https://github.com/zipfian/spark-install" rel="noreferrer">https://github.com/zipfian/spark-install</a>)</p>
  
  <p>I have another docker image for hadoop. (<a href="https://github.com/kiwenlau/hadoop-cluster-docker" rel="noreferrer">https://github.com/kiwenlau/hadoop-cluster-docker</a>)</p>
  
  <p>I am running 2 containers from the above 2 images in Ubuntu.
  For the first container: 
    I am able to successfully launch jupyter and run python code:</p>
  
  <pre><code>   import pyspark
     sc = pyspark.sparkcontext('local[*]')
     rdd = sc.parallelize(range(1000))
     rdd.takeSample(False,5)
  </code></pre>
  
  <p>For the second container:</p>
  
  <p>In the host Ubuntu OS, I am able to successfully go to the </p>
  
  <ul>
  <li>web browser localhost:8088 : And browse the Hadoop all applications       </li>
  <li>localhost:50070: and browse the HDFS file system.</li>
  </ul>
  
  <p><a href="https://i.stack.imgur.com/fZehH.png" rel="noreferrer"><img src="https://i.stack.imgur.com/fZehH.png" alt="enter image description here"></a></p>
  
  <p>Now I want to write to the HDFS file system (running in the 2nd container) from jupyter (running in the first container).</p>
  
  <p>So I add the additional line </p>
  
  <pre><code>rdd.saveAsTextFile("hdfs:///user/root/input/test")
  </code></pre>
  
  <p>I get the error:</p>
  
  <p><strong><em>HDFS URI, no host: hdfs:///user/root/input/test</em></strong></p>
  
  <p>Am I giving the hdfs path incorrectly ?</p>
  
  <p>My understanding is that, I should be able to talk to a docker container running hdfs from another container running spark. Am I missing anything ?</p>
  
  <p>Thanks for your time.</p>
  
  <p>I haven't tried docker compose yet.</p>