title: Hadoop HDFS - Cannot connect to port on master
body: <p>I've set up a small Hadoop cluster for testing. Setup went fairly well with the NameNode (1 machine), SecondaryNameNode (1) and all DataNodes (3). The machines are named "master", "secondary" and "data01", "data02" and "data03". All DNS are properly set up, and passwordless SSH was configured from master/secondary to all machines and back. </p>
  
  <p>I formatted the cluster with <code>bin/hadoop namenode -format</code>, and then started all services using <code>bin/start-all.sh</code>. All processes on all nodes were checked to be up and running with <code>jps</code>. My basic configuration files look something like this:</p>
  
  <pre><code>&lt;!-- conf/core-site.xml --&gt;
  &lt;configuration&gt;
    &lt;property&gt;
      &lt;name&gt;fs.default.name&lt;/name&gt;
      &lt;!-- 
        on the master it's localhost
        on the others it's the master's DNS
        (ping works from everywhere)
      --&gt;
      &lt;value&gt;hdfs://localhost:9000&lt;/value&gt;
    &lt;/property&gt;
    &lt;property&gt;
      &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;
      &lt;!-- I picked /hdfs for the root FS --&gt;
      &lt;value&gt;/hdfs/tmp&lt;/value&gt;
    &lt;/property&gt;
  &lt;/configuration&gt;
  
  &lt;!-- conf/hdfs-site.xml --&gt;
  &lt;configuration&gt;
    &lt;property&gt;
      &lt;name&gt;dfs.name.dir&lt;/name&gt;
      &lt;value&gt;/hdfs/name&lt;/value&gt;
    &lt;/property&gt;
    &lt;property&gt;
      &lt;name&gt;dfs.data.dir&lt;/name&gt;
      &lt;value&gt;/hdfs/data&lt;/value&gt;
    &lt;/property&gt;
    &lt;property&gt;
      &lt;name&gt;dfs.replication&lt;/name&gt;
      &lt;value&gt;3&lt;/value&gt;
    &lt;/property&gt;
  &lt;/configuration&gt;
  
  # conf/masters
  secondary
  
  # conf/slaves
  data01
  data02
  data03
  </code></pre>
  
  <p>I'm just trying to get HDFS running properly now.</p>
  
  <p>I've created a dir for testing <code>hadoop fs -mkdir testing</code>, then tried to copy some files into it with <code>hadoop fs -copyFromLocal /tmp/*.txt testing</code>. This is when hadoop crashes, giving me more or less this:</p>
  
  <pre><code>WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hd/testing/wordcount1.txt could only be replicated to 0 nodes, instead of 1
    at ... (such and such)
  
  WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
    at ...
  
  WARN hdfs.DFSClient: Could not get block locations. Source file "/user/hd/testing/wordcount1.txt" - Aborting...
    at ...
  
  ERROR hdfs.DFSClient: Exception closing file /user/hd/testing/wordcount1.txt: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hd/testing/wordcount1.txt could only be replicated to 0 nodes, instead of 1
    at ...
  </code></pre>
  
  <p>And so on. A similar issue occurs when I try to run <code>hadoop fs -lsr .</code> from a DataNode machine, only to get the following:</p>
  
  <pre><code>12/01/02 10:02:11 INFO ipc.Client: Retrying connt to server master/192.162.10.10:9000. Already tried 0 time(s).
  12/01/02 10:02:12 INFO ipc.Client: Retrying connt to server master/192.162.10.10:9000. Already tried 1 time(s).
  12/01/02 10:02:13 INFO ipc.Client: Retrying connt to server master/192.162.10.10:9000. Already tried 2 time(s).
  ...
  </code></pre>
  
  <p>I'm saying it's similar, because I suspect this is a port availability issue. Running <code>telnet master 9000</code> reveals that the port is closed. I've read somewhere that this might be an IPv6 clash issue, and thus defined the following in conf/hadoop-env.sh:</p>
  
  <pre><code>export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
  </code></pre>
  
  <p>But that didn't do the trick. Running <code>netstat</code> on the master reveals something like this:</p>
  
  <pre><code>Proto Recv-Q Send-Q  Local Address       Foreign Address      State
  tcp        0      0  localhost:9000      localhost:56387      ESTABLISHED
  tcp        0      0  localhost:56386     localhost:9000       TIME_WAIT
  tcp        0      0  localhost:56387     localhost:9000       ESTABLISHED
  tcp        0      0  localhost:56384     localhost:9000       TIME_WAIT
  tcp        0      0  localhost:56385     localhost:9000       TIME_WAIT
  tcp        0      0  localhost:56383     localhost:9000       TIME_WAIT
  </code></pre>
  
  <p>At this point I'm pretty sure the problem is with the port (9000), but I'm not sure what I missed as far as configuration goes. Any ideas? Thanks.</p>
  
  <h2>update</h2>
  
  <p>I found that hard coding DNS names into <code>/etc/hosts</code> not only help resolve this, but also speeds up the connections. The downside is that you have to do this on all the machines in the cluster, and again when you add new nodes. Or you can just set up a DNS server, which I didn't. </p>
  
  <p>Here's a sample of my one node in my cluster (nodes are named <code>hadoop01</code>, <code>hadoop02</code>, etc, with the master and secondary being 01 and 02). Node that most of it are generated by the OS:</p>
  
  <pre><code># this is a sample for a machine with dns hadoop01
  ::1 localhost ip6-localhost ip6-loopback
  fe00::0 ip6-localnet
  ff00::0 ip6-mcastrprefix
  ff02::1 ip6-allnodes
  ff02::2 ip6-allroutes
  
  # --- Start list of nodes
  192.168.10.101 hadoop01
  192.168.10.102 hadoop02
  192.168.10.103 hadoop03
  192.168.10.104 hadoop04
  192.168.10.105 hadoop05
  192.168.10.106 hadoop06
  192.168.10.107 hadoop07
  192.168.10.108 hadoop08
  192.168.10.109 hadoop09
  192.168.10.110 hadoop10
  # ... and so on
  
  # --- End list of nodes
  
  # Auto-generated hostname. Please do not remove this comment.
  127.0.0.1 hadoop01 localhost localhost.localdomain
  </code></pre>
  
  <p>Hope this helps.</p>