title: What is YARN (Hadoop) - very simple example
body: <p>I am new to Hadoop and trying to understand it. I found a nice explanation
  of HDFS and MapReduce with very simple examples (see below). But I cannot
  google any similar simple example for YARN. <strong>Could someone please explain it
  (like for a layman)?</strong></p>
  
  <p><a href="https://www.ibm.com/analytics/hadoop/hdfs" rel="nofollow noreferrer">HDFS</a></p>
  
  <blockquote>
    <p>Think of a file that contains the phone numbers for everyone in the United
    States; the people with a last name starting with A might be stored on server
    1, B on server 2, and so on.</p>
    
    <p>In a Hadoop world, pieces of this phonebook would be stored across the cluster,
    and to reconstruct the entire phonebook, your program would need the blocks
    from every server in the cluster. To achieve availability as components fail,
    HDFS replicates these smaller pieces onto two additional servers by default.
    (This redundancy can be increased or decreased on a per-file basis or for a
    whole environment; for example, a development Hadoop cluster typically doesn’t
    need any data redundancy.) This redundancy offers multiple benefits, the most
    obvious being higher availability.</p>
    
    <p>In addition, this redundancy allows the Hadoop cluster to break work up into
    smaller chunks and run those jobs on all the servers in the cluster for better
    scalability. Finally, you get the benefit of data locality, which is critical
    when working with large data sets. We detail these important benefits later in
    this chapter.</p>
  </blockquote>
  
  <p><a href="https://www.ibm.com/analytics/hadoop/mapreduce" rel="nofollow noreferrer">MapReduce</a></p>
  
  <blockquote>
    <p>Let’s look at a simple example. Assume you have five files, and each file
    contains two columns (a key and a value in Hadoop terms) that represent a city
    and the corresponding temperature recorded in that city for the various
    measurement days. Of course we’ve made this example very simple so it’s easy to
    follow. You can imagine that a real application won’t be quite so simple, as
    it’s likely to contain millions or even billions of rows, and they might not be
    neatly formatted rows at all; in fact, no matter how big or small the amount of
    data you need to analyze, the key principles we’re covering here remain the
    same. Either way, in this example, city is the key and temperature is the
    value.</p>
    
    <p>Toronto, 20
    Whitby, 25
    New York, 22
    Rome, 32
    Toronto, 4
    Rome, 33
    New York, 18</p>
    
    <p>Out of all the data we have collected, we want to find the maximum temperature
    for each city across all of the data files (note that each file might have the
    same city represented multiple times). Using the MapReduce framework, we can
    break this down into five map tasks, where each mapper works on one of the five
    files and the mapper task goes through the data and returns the maximum
    temperature for each city. For example, the results produced from one mapper
    task for the data above would look like this:</p>
    
    <p>(Toronto, 20) (Whitby, 25) (New York, 22) (Rome, 33)</p>
    
    <p>Let’s assume the other four mapper tasks (working on the other four files not
    shown here) produced the following intermediate results:</p>
    
    <p>(Toronto, 18) (Whitby, 27) (New York, 32) (Rome, 37)(Toronto, 32) (Whitby, 20)
    (New York, 33) (Rome, 38)(Toronto, 22) (Whitby, 19) (New York, 20) (Rome,
    31)(Toronto, 31) (Whitby, 22) (New York, 19) (Rome, 30)</p>
    
    <p>All five of these output streams would be fed into the reduce tasks, which
    combine the input results and output a single value for each city, producing a
    final result set as follows:</p>
    
    <p>(Toronto, 32) (Whitby, 27) (New York, 33) (Rome, 38)</p>
    
    <p>As an analogy, you can think of map and reduce tasks as the way a census was
    conducted in Roman times, where the census bureau would dispatch its people to
    each city in the empire. Each census taker in each city would be tasked to
    count the number of people in that city and then return their results to the
    capital city.</p>
    
    <p>There, the results from each city would be reduced to a single count (sum of
    all cities) to determine the overall population of the empire. This mapping of
    people to cities, in parallel, and then combining the results (reducing) is
    much more efficient than sending a single person to count every person in the
    empire in a serial fashion.</p>
  </blockquote>