title: TestDFSIO Benchmark - Bad Datanodes
body: <p>I am attempting to run the <strong>TestDFSIO - Write</strong> test to benchmark my hadoop yarn cluster. </p>
  
  <p>The result for 1 MB file size is as follows:</p>
  
  <pre><code>17/02/13 16:29:56 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
  17/02/13 16:29:56 INFO fs.TestDFSIO:            Date &amp; time: Mon Feb 13 16:29:56 IST 2017
  17/02/13 16:29:56 INFO fs.TestDFSIO:        Number of files: 10
  17/02/13 16:29:56 INFO fs.TestDFSIO: Total MBytes processed: 10.0
  17/02/13 16:29:56 INFO fs.TestDFSIO:      Throughput mb/sec: 0.08520645524104906
  17/02/13 16:29:56 INFO fs.TestDFSIO: Average IO rate mb/sec: 0.11449315398931503
  17/02/13 16:29:56 INFO fs.TestDFSIO:  IO rate std deviation: 0.051399135678609
  17/02/13 16:29:56 INFO fs.TestDFSIO:     Test exec time sec: 59.722
  </code></pre>
  
  <p>and for 10 MB file size, the result is</p>
  
  <pre><code>17/02/13 17:24:23 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
  17/02/13 17:24:23 INFO fs.TestDFSIO:            Date &amp; time: Mon Feb 13 17:24:23 IST 2017
  17/02/13 17:24:23 INFO fs.TestDFSIO:        Number of files: 10
  17/02/13 17:24:23 INFO fs.TestDFSIO: Total MBytes processed: 100.0
  17/02/13 17:24:23 INFO fs.TestDFSIO:      Throughput mb/sec: 0.13296139598828877
  17/02/13 17:24:23 INFO fs.TestDFSIO: Average IO rate mb/sec: 0.1448635458946228
  17/02/13 17:24:23 INFO fs.TestDFSIO:  IO rate std deviation: 0.044048737962458215
  17/02/13 17:24:23 INFO fs.TestDFSIO:     Test exec time sec: 135.863
  </code></pre>
  
  <p><strong>But when I try increasing the file size to 100 MB, I keep getting the following error:</strong></p>
  
  <pre><code>....
  17/02/13 17:28:33 INFO mapreduce.Job:  map 67% reduce 0%
  17/02/13 17:28:43 INFO mapreduce.Job:  map 73% reduce 0%
  17/02/13 17:29:45 INFO mapreduce.Job:  map 67% reduce 0%
  17/02/13 17:29:46 INFO mapreduce.Job: Task Id : attempt_1486983847790_0003_m_000003_0, Status : FAILED
  Error: java.io.IOException: All datanodes DatanodeInfoWithStorage[172.16.33.70:50010,DS-f6a1b4fe-66f7-4b6d-9164-f15f371471e0,DISK] are bad. Aborting...
          at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1109)
          at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:871)
          at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:401)
  
  Container killed by the ApplicationMaster.
  Container killed on request. Exit code is 143
  Container exited with a non-zero exit code 143
  ....
  </code></pre>
  
  <p><strong>The MR jobs gets stuck. Is my 3 machine cluster (Centos7 OS, 4 cores &amp; 32 GB RAM - Each) ONLY capable of writing 10 MB of file?</strong></p>
  
  <p>Best Regards</p>