title: How to read file from hdfs in python script when using Hadoop Streaming?
body: <p>My python script for mapper is like this:</p>
  
  <pre><code>import sys,re,string
  from os import listdir
  from os.path import isfile, join
  dir_path = "user/lexi/data/"
  sys.path.append('.')
  for filename in sys.stdin:
      file_path = dir_path + filename.strip()
      rfile = open(file_path,'r')
      #cat = subprocess.Popen(["hadoop","fs","-cat",file_path],stdout=subprocess.PIPE)
      for line in rfile.readlines():
             do something
  </code></pre>
  
  <p>for the sys.stdin, it is a <strong>filenames.txt</strong> file which containing a list of filenames like this:</p>
  
  <pre><code>123.html
  124.html
  125.html
  ...
  </code></pre>
  
  <p>all these html files are under "user/lexi/data/" on hdfs.</p>
  
  <p>Then I ran it in hadoop streaming:</p>
  
  <pre><code>hadoop jar /hadoop/contrib/streaming/hadoop-streaming-1.2.1.jar\
   -input 'input/filenames.txt'\
   -output 'test-output'\
   -file Mapper.py -file Reducer.py\
   -mapper 'Mapper.py' -reducer 'Reducer.py'
  </code></pre>
  
  <p>But I got this Error:</p>
  
  <pre><code>14/02/14 20:47:54 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  14/02/14 20:47:54 WARN snappy.LoadSnappy: Snappy native library not loaded
  14/02/14 20:47:54 INFO mapred.FileInputFormat: Total input paths to process : 1
  14/02/14 20:47:54 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop/mapred]
  14/02/14 20:47:54 INFO streaming.StreamJob: Running job: job_201401161224_3630
  14/02/14 20:47:54 INFO streaming.StreamJob: To kill this job, run:
  14/02/14 20:47:54 INFO streaming.StreamJob: /usr/libexec/../bin/hadoop job   -Dmapred.job.tracker=head:9000 -kill job_201401161224_3630
  14/02/14 20:47:54 INFO streaming.StreamJob: Tracking URL: http://mri-head.mri.cs.gmu.edu:50030/jobdetails.jsp?jobid=job_201401161224_3630
  14/02/14 20:47:55 INFO streaming.StreamJob:  map 0%  reduce 0%
  14/02/14 20:48:26 INFO streaming.StreamJob:  map 100%  reduce 100%
  14/02/14 20:48:26 INFO streaming.StreamJob: To kill this job, run:
  14/02/14 20:48:26 INFO streaming.StreamJob: /usr/libexec/../bin/hadoop job  -Dmapred.job.tracker=head:9000 -kill job_201401161224_3630
  14/02/14 20:48:26 INFO streaming.StreamJob: Tracking URL: http://mri-head.mri.cs.gmu.edu:50030/jobdetails.jsp?jobid=job_201401161224_3630
  14/02/14 20:48:26 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201401161224_3630_m_000001
  14/02/14 20:48:26 INFO streaming.StreamJob: killJob...
  Streaming Command Failed!
  </code></pre>
  
  <p>Someone can help me out of this???
  Really NEED HELP!!!</p>