title: How does Hadoop distribute the data/tasks for MapReduce jobs?
body: <p>I've setup a Hadoop cluster with 4 nodes, one of which serves as the NameNode for HDFS as well as the Yarn master. This node is also the most powerful.</p>
  
  <p>Now, I've distributed 2 text files, one on the node01 (namenode) and one on node03 (datanode). When running the basic WordCount MapReduce job, I can see in the logs that only node01 was doing any calculations.</p>
  
  <p>My question is why Hadoop didn't decide to do MapReduce on node03 and transfer the result instead of transferring the entire book to node01. I also checked, duplication is disabled and the book is only available on node03.</p>
  
  <p>So, how does Hadoop decide between transferring the data and setting up the jobs and in this decision, does it check which machine has more compute power (e.g. did it decide to transfer to node01 because node01 is a 4 core 4gig ram machine vs 2core 1 gig on node03)?</p>
  
  <p>I couldn't find anything on this topic, so any guidance would be appreciated.</p>
  
  <p>Thank you!</p>
  
  <hr>
  
  <p>Some more clarifications:
  node01 is running a NameNode as well as a DataNode and a ResourceManager as well as a NodeManager. Thus, it serves as "main node" as well as a "compute node".</p>
  
  <p>I made sure to put one file on node01 and one file on node03 by running:
  <code>hdfs dfs -put sample1.txt samples</code> on node01 and <code>hdfs dfs -put sample02.txt samples</code> on node03. As replication is disabled, this leads to the data - that was available locally on node01 respective node03 - only being stored there.</p>
  
  <p>I verified this using the HDFS Webinterface. For sample1.txt, it says the blocks are only available on node01; for sample2.txt, it says the blocks are only available on node03.</p>
  
  <p>Regarding @cricket_007:
  My concern is that sample2.txt is only available on node03. The YARN Webinterface tells me that that for the Application Attempt, only one container was allocated on node01. If the map task for file sample2.txt, there would have been a container on node03 as well.</p>
  
  <p>Thus, node01 needs to have fetched the sample2.txt file from node03.</p>
  
  <p>Yes, I know Hadoop is not running well on 1gig of RAM, but I am working with a Raspberry Pi cluster just to fiddle around and learn a little. This is not for production usage.</p>