title: Amazon EMR job failed: Shut down as step failed
body: <p>I'm writing a simple streaming map reduce job using Python to be run on Amazon EMR. It's basically an aggregator of user records which groups the entries for each user id together.</p>
  
  <p>Mapper</p>
  
  <pre><code>#!/usr/bin/env python
  import sys
  
  def main(argv):
  line = sys.stdin.readline()
  try:
      while line:
          line = line.rstrip()
          elements = line.split()
          print '%s\t%s' % (elements[0] , (elements[1],elements[2]) )
          line = sys.stdin.readline()
  except "end of file":
      return None
  
  if __name__ == '__main__':
      main(sys.argv)
  </code></pre>
  
  <p>Reducer:</p>
  
  <pre><code>#!/usr/bin/env python
  import sys
  
  def main(argv):
  users=dict()
  for line in sys.stdin:
      elements=line.split('\t',1)
      if elements[0] in users:
          users[elements[0]].append(elements[1])
      else:
          users[elements[0]]=elements[1]
  
  for user in users:
      print '%s\t%s'% ( user, users[user] )
  
  if __name__ == '__main__':
      main(sys.argv)
  </code></pre>
  
  <p>This job should run on a directory with five text files. The parameters of the EMR job are:</p>
  
  <p>Input: [Bucket name]/[Input folder name]</p>
  
  <p>Output: [Bucket name]/output</p>
  
  <p>Mapper: [Bucket name]/mapper.py</p>
  
  <p>Reducer: [Bucket name]/reducer.py</p>
  
  <p>The job keeps failing with the reason : Shut down as step failed. Here's a copy of the log</p>
  
  <pre><code>2013-01-01 12:06:16,270 INFO org.apache.hadoop.mapred.JobClient (main): Default number      of map tasks: null
  
  2013-01-01 12:06:16,271 INFO org.apache.hadoop.mapred.JobClient (main): Setting default number of map tasks based on cluster size to : 8
  
  2013-01-01 12:06:16,271 INFO org.apache.hadoop.mapred.JobClient (main): Default number of reduce tasks: 3
  
  2013-01-01 12:06:18,392 INFO org.apache.hadoop.security.ShellBasedUnixGroupsMapping (main): add hadoop to shell userGroupsCache
  
  2013-01-01 12:06:18,393 INFO org.apache.hadoop.mapred.JobClient (main): Setting group to hadoop
  
  2013-01-01 12:06:18,647 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
  
  2013-01-01 12:06:18,670 WARN com.hadoop.compression.lzo.LzoCodec (main): Could not find build properties file with revision hash
  
  2013-01-01 12:06:18,670 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded &amp; initialized native-lzo library [hadoop-lzo rev UNKNOWN]
  
  2013-01-01 12:06:18,695 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy (main): Snappy native library is available
  
  2013-01-01 12:06:18,695 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy (main): Snappy native library loaded
  
  2013-01-01 12:06:19,050 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input paths to process : 5
  
  2013-01-01 12:06:20,688 INFO org.apache.hadoop.streaming.StreamJob (main): getLocalDirs(): [/mnt/var/lib/hadoop/mapred]
  
  2013-01-01 12:06:20,688 INFO org.apache.hadoop.streaming.StreamJob (main): Running job: job_201301011204_0001
  
  2013-01-01 12:06:20,688 INFO org.apache.hadoop.streaming.StreamJob (main): To kill this job, run:
  
  2013-01-01 12:06:20,688 INFO org.apache.hadoop.streaming.StreamJob (main): /home/hadoop/bin/hadoop job  -Dmapred.job.tracker=10.255.131.225:9001 -kill job_201301011204_0001
  
  2013-01-01 12:06:20,689 INFO org.apache.hadoop.streaming.StreamJob (main): Tracking URL: http://domU-12-31-39-01-7C-13.compute-1.internal:9100/jobdetails.jsp?jobid=job_201301011204_0001
  
  2013-01-01 12:06:21,696 INFO org.apache.hadoop.streaming.StreamJob (main):  map 0%  reduce 0%
  
  2013-01-01 12:08:02,238 INFO org.apache.hadoop.streaming.StreamJob (main):  map 100%  reduce 100%
  
  2013-01-01 12:08:02,239 INFO org.apache.hadoop.streaming.StreamJob (main): To kill this job, run:
  
  2013-01-01 12:08:02,240 INFO org.apache.hadoop.streaming.StreamJob (main): /home/hadoop/bin/hadoop job  -Dmapred.job.tracker=10.255.131.225:9001 -kill job_201301011204_0001
  
  2013-01-01 12:08:02,240 INFO org.apache.hadoop.streaming.StreamJob (main): Tracking URL: http://domU-12-31-39-01-7C-13.compute-1.internal:9100/jobdetails.jsp?jobid=job_201301011204_0001
  
   2013-01-01 12:08:02,240 ERROR org.apache.hadoop.streaming.StreamJob (main): Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201301011204_0001_m_000002
  
   2013-01-01 12:08:02,240 INFO org.apache.hadoop.streaming.StreamJob (main): killJob...
  </code></pre>
  
  <p>What am I doing wrong?</p>