title: Hadoop job suddenly fails
body: <p>When running my Hadoop program, it suddenly fails without any obvious reason. I have attached the output message. It seems to be an IO problem. It tells something must go wrong in line 525 which is the line in which I run the job (line 525: <code>RunningJob runjob = JobClient.runJob(conf);</code>). As can be seen in the error message, it does not say any thing more about the error line.</p>
  
  <p>I ran it a couple of times and it always gives me this error when it is 86% complete. Does anyone have some idea about the reason of failure?</p>
  
  <pre><code>13/10/14 10:25:11 INFO mapred.JobClient:  map 79% reduce 6%
  13/10/14 10:25:16 INFO mapred.JobClient:  map 80% reduce 6%
  13/10/14 10:25:43 INFO mapred.JobClient:  map 81% reduce 6%
  13/10/14 10:26:13 INFO mapred.JobClient:  map 82% reduce 6%
  13/10/14 10:26:41 INFO mapred.JobClient:  map 83% reduce 6%
  13/10/14 10:26:56 INFO mapred.JobClient:  map 83% reduce 8%
  13/10/14 10:27:09 INFO mapred.JobClient:  map 84% reduce 8%
  13/10/14 10:27:44 INFO mapred.JobClient:  map 85% reduce 8%
  13/10/14 10:28:29 INFO mapred.JobClient:  map 86% reduce 8%
  13/10/14 10:29:13 INFO mapred.JobClient: Job complete: job_201310101346_0102
  13/10/14 10:29:13 INFO mapred.JobClient: Counters: 31
  13/10/14 10:29:13 INFO mapred.JobClient:   File System Counters
  13/10/14 10:29:13 INFO mapred.JobClient:     FILE: Number of bytes read=316172729
  13/10/14 10:29:13 INFO mapred.JobClient:     FILE: Number of bytes written=501348856
  13/10/14 10:29:13 INFO mapred.JobClient:     FILE: Number of read operations=0
  13/10/14 10:29:13 INFO mapred.JobClient:     FILE: Number of large read operations=0
  13/10/14 10:29:13 INFO mapred.JobClient:     FILE: Number of write operations=0
  13/10/14 10:29:13 INFO mapred.JobClient:     HDFS: Number of bytes read=2144535743
  13/10/14 10:29:13 INFO mapred.JobClient:     HDFS: Number of bytes written=0
  13/10/14 10:29:13 INFO mapred.JobClient:     HDFS: Number of read operations=128
  13/10/14 10:29:13 INFO mapred.JobClient:     HDFS: Number of large read operations=0
  13/10/14 10:29:13 INFO mapred.JobClient:     HDFS: Number of write operations=0
  13/10/14 10:29:13 INFO mapred.JobClient:   Job Counters 
  13/10/14 10:29:13 INFO mapred.JobClient:     Failed map tasks=1
  13/10/14 10:29:13 INFO mapred.JobClient:     Launched map tasks=188
  13/10/14 10:29:13 INFO mapred.JobClient:     Launched reduce tasks=1
  13/10/14 10:29:13 INFO mapred.JobClient:     Data-local map tasks=125
  13/10/14 10:29:13 INFO mapred.JobClient:     Rack-local map tasks=63
  13/10/14 10:29:13 INFO mapred.JobClient:     Total time spent by all maps in occupied   slots (ms)=211239444
  13/10/14 10:29:13 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=406123
  13/10/14 10:29:13 INFO mapred.JobClient:     Total time spent by all maps waiting after     reserving slots (ms)=0
  13/10/14 10:29:13 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
  13/10/14 10:29:13 INFO mapred.JobClient:   Map-Reduce Framework
  13/10/14 10:29:13 INFO mapred.JobClient:     Map input records=45927922
  13/10/14 10:29:13 INFO mapred.JobClient:     Map output records=45927922
  13/10/14 10:29:13 INFO mapred.JobClient:     Map output bytes=11573153427
  13/10/14 10:29:13 INFO mapred.JobClient:     Input split bytes=8000
  13/10/14 10:29:13 INFO mapred.JobClient:     Combine input records=48701007
  13/10/14 10:29:13 INFO mapred.JobClient:     Combine output records=3833295
  13/10/14 10:29:13 INFO mapred.JobClient:     Spilled Records=3833295
  13/10/14 10:29:13 INFO mapred.JobClient:     CPU time spent (ms)=86130400
  13/10/14 10:29:13 INFO mapred.JobClient:     Physical memory (bytes) snapshot=13896142848
  13/10/14 10:29:13 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=25477984256
  13/10/14 10:29:13 INFO mapred.JobClient:     Total committed heap usage  (bytes)=9854910464
  13/10/14 10:29:13 INFO mapred.JobClient:    org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
  13/10/14 10:29:13 INFO mapred.JobClient:     BYTES_READ=2144450681
  13/10/14 10:29:13 INFO mapred.JobClient: Job Failed: NA
  Exception in thread "main" java.io.IOException: Job failed!
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1373)
      at Mondrian3.main(Mondrian3.java:525)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
  </code></pre>