title: Hadoop tasktracker issue
body: <p>This is a weird problem. I was running a MR job (single node) when my lappy lost power and shutdown itself. Thanks to the lack of "no battery" warning in ubuntu 10.04</p>
  
  <p>When I started up again and tried to start hadoop (single node), everything goes well and when I issue "jps" from the command line, all the master and slave processes are listed. I also tried issuing "netstat -plten | grep java".</p>
  
  <p>Output from command line:</p>
  
  <pre>
  hduser@fox:~$ jps
  10476 DataNode
  1819 
  10696 JobTracker
  10813 TaskTracker
  11741 Jps
  10605 SecondaryNameNode
  10360 NameNode
  
  hduser@fox:~$ sudo netstat -plten | grep java
  tcp        0      0 0.0.0.0:55376           0.0.0.0:*               LISTEN      1000       141925      10476/java      
  tcp        0      0 0.0.0.0:42352           0.0.0.0:*               LISTEN      1000       141790      10360/java      
  tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      1000       142727      10360/java      
  tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      1000       142818      10476/java      
  tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      1000       143308      10476/java      
  tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      1000       146358      10476/java      
  tcp        0      0 127.0.0.1:38533         0.0.0.0:*               LISTEN      1000       143217      10813/java      
  tcp        0      0 102.173.156.27:54310   0.0.0.0:*               LISTEN      1000       141913      10360/java      
  tcp        0      0 102.173.156.27:54311   0.0.0.0:*               LISTEN      1000       142359      10696/java      
  tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      1000       142827      10605/java      
  tcp        0      0 0.0.0.0:56779           0.0.0.0:*               LISTEN      1000       142191      10605/java      
  tcp        0      0 0.0.0.0:55373           0.0.0.0:*               LISTEN      1000       142251      10696/java      
  tcp        0      0 0.0.0.0:50030           0.0.0.0:*               LISTEN      1000     142921      10696/java
  
  </pre>
  
  <p>I checked the job tracker log file:</p>
  
  <pre>
  2013-01-09 16:40:52,286 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
  2013-01-09 16:40:52,286 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/home/hduser/hadoop/tmpnew/mapred/system/jobtracker.info" - Aborting...
  2013-01-09 16:40:52,286 WARN org.apache.hadoop.mapred.JobTracker: Writing to file hdfs://master:54310/home/hduser/hadoop/tmpnew/mapred/system/jobtracker.info failed!
  2013-01-09 16:40:52,287 WARN org.apache.hadoop.mapred.JobTracker: FileSystem is not ready yet!
  2013-01-09 16:40:52,290 WARN org.apache.hadoop.mapred.JobTracker: Failed to initialize recovery manager. 
  org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hduser/hadoop/tmpnew/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at 
  org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
      at org.apache.hadoop.ipc.Client.call(Client.java:1070)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
      at $Proxy5.addBlock(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
      at $Proxy5.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3510)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3373)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829)
  2013-01-09 16:41:02,291 WARN org.apache.hadoop.mapred.JobTracker: Retrying...
  2013-01-09 16:41:02,331 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hduser/hadoop/tmpnew/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
  
      at org.apache.hadoop.ipc.Client.call(Client.java:1070)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
      at $Proxy5.addBlock(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
      at $Proxy5.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3510)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3373)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829) </pre>
  
  <p>I am not able to see the tasktracker using the URL - master:50060</p>
  
  <p>Therefore I have a feeling that the tasktracker is not started up correctly. </p>
  
  <p>Help appreciated.</p>