title: hadoop "ipc.Client: Retrying connect to server" error
body: <p>There are a lot of ideas on how to resolve this hadoop error</p>
  
  <pre><code>15/04/17 10:59:57 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 0 time(s).
  </code></pre>
  
  <p>However, I tried all and still see that error! Here are my configurations</p>
  
  <p>1) core-site.xml</p>
  
  <pre><code>$ cat ../../apache/hadoop-1.0.2/conf/core-site.xml
  &lt;?xml version="1.0"?&gt;
  &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
  &lt;!-- Put site-specific property overrides in this file. --&gt;
  &lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;fs.default.name&lt;/name&gt;
    &lt;value&gt;hdfs://localhost:54310&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p>2) mapred-site.xml</p>
  
  <pre><code>$ cat ../../apache/hadoop-1.0.2/conf/mapred-site.xml
  &lt;?xml version="1.0"?&gt;
  &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
  &lt;!-- Put site-specific property overrides in this file. --&gt;
  &lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.job.tracker&lt;/name&gt;
    &lt;value&gt;localhost:54311&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.child.java.opts&lt;/name&gt;
    &lt;value&gt;-Xmx512m&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p>3) iptables for the port</p>
  
  <pre><code># cat /etc/sysconfig/iptables
  *filter
  :INP UT ACCEPT [0:0]
  :FORWARD ACCEPT [0:0]
  :OUTPUT ACCEPT [0:0]
  -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
  -A INPUT -p icmp -j ACCEPT
  -A INPUT -i lo -j ACCEPT
  -A INPUT -m state --state NEW -m tcp -p tcp --dport 5901 -j ACCEPT
  -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
  -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
  -A INPUT -m state --state NEW -m tcp -p tcp --dport 2049 -j ACCEPT
  -A INPUT -m state --state NEW -m tcp -p tcp --dport 54310 -j ACCEPT
  -A INPUT -m state --state NEW -m tcp -p tcp --dport 54311 -j ACCEPT
  -A INPUT -j REJECT --reject-with icmp-host-prohibited
  -A FORWARD -j REJECT --reject-with icmp-host-prohibited
  COMMIT
  
  # /etc/init.d/iptables restart
  iptables: Flushing firewall rules:                         [  OK  ]
  iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
  iptables: Unloading modules:                               [  OK  ]
  iptables: Applying firewall rules:                         [  OK  ]
  
  $ netstat -an | grep 54310
  $ netstat -an | grep 54311
  tcp        0      0 ::ffff:127.0.0.1:54311     :::*                        LISTEN
  tcp      238      0 ::ffff:127.0.0.1:54311      ::ffff:127.0.0.1:44216      ESTABLISHED
  tcp        0      0 ::ffff:127.0.0.1:44216      ::ffff:127.0.0.1:54311      ESTABLISHED
  </code></pre>
  
  <p>4) starting hadoop</p>
  
  <pre><code>$ $HADOOP_HOME/bin/start-all.sh
  Warning: $HADOOP_HOME is deprecated.
  
  starting namenode, logging to /home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-namenode-tiger.out
  mahmood@localhost's password:
  localhost: Warning: $HADOOP_HOME is deprecated.
  localhost:
  localhost: starting datanode, logging to /home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-datanode-tiger.out
  mahmood@localhost's password:
  localhost: Warning: $HADOOP_HOME is deprecated.
  localhost:
  localhost: secondarynamenode running as process 7583. Stop it first.
  jobtracker running as process 7792. Stop it first.
  mahmood@localhost's password:
  localhost: Warning: $HADOOP_HOME is deprecated.
  localhost:
  localhost: tasktracker running as process 8019. Stop it first.
  </code></pre>
  
  <p>5) check java processes</p>
  
  <pre><code>$ jps
  10292 Jps
  8019 TaskTracker
  7792 JobTracker
  7583 SecondaryNameNode
  </code></pre>
  
  <p>6) Still I get that error</p>
  
  <pre><code>$ hadoop fs -ls hdfs://localhost:54310/
  Warning: $HADOOP_HOME is deprecated.
  
  15/04/17 10:59:57 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 0 time(s).
  15/04/17 10:59:58 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 1 time(s).
  15/04/17 10:59:59 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 2 time(s).
  15/04/17 11:00:00 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 3 time(s).
  15/04/17 11:00:01 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 4 time(s).
  15/04/17 11:00:02 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 5 time(s).
  15/04/17 11:00:03 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 6 time(s).
  15/04/17 11:00:04 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 7 time(s).
  15/04/17 11:00:05 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 8 time(s).
  15/04/17 11:00:06 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 9 time(s).
  Bad connection to FS. command aborted. exception: Call to localhost/127.0.0.1:54310 failed on connection exception: java.net.ConnectException: Connection refused
  </code></pre>
  
  <p>UPDATE:</p>
  
  <p>While I thought that I had formatted the filesystem, it turns out that the hdfs format command aborted and I didn't noticed. The reason for that was I answered this question <code>Re-format filesystem in /home/mahmood/bigdatabench/apache/hadoop-1.0.2/folders/name ? (Y or N)</code> with <strong>y</strong>. However the correct answer is to press <strong>Y</strong> (capital letter!!).</p>
  
  <p>So the correct steps are</p>
  
  <p>1- stop-all.sh</p>
  
  <p>2- hadoop namenode -format</p>
  
  <p>3- start-all.sh</p>
  
  <p>The ipc error has been gone :)</p>
  
  <p>How can I fix that?</p>