title: Spark Cluster starting issue
body: <p>I'm new to spark, and trying to setup spark cluster. I did following things to set-up and check status of spark cluster, but not sure about status. </p>
  
  <p>I tried to check  master-ip:8081 (8080, 4040, 4041) in the browser, but didn't see any results. To start with, I set-up  and started hadoop cluster. </p>
  
  <pre><code> JPS gives:
  
   2436 SecondaryNameNode
   2708 NodeManager
   2151 NameNode
   5495 Master
   2252 DataNode
   2606 ResourceManager
   5710 Jps
  </code></pre>
  
  <p>Question (Was it necessary to start hadoop?)</p>
  
  <p>In Master /usr/local/spark/conf/slaves</p>
  
  <pre><code> localhost
   slave-node-1
   slave-node-2
  </code></pre>
  
  <p>Now, to start Spark; Master Started with  </p>
  
  <pre><code>  $SPARK_HOME/sbin/start-master.sh 
  </code></pre>
  
  <p>And tested with </p>
  
  <pre><code>  ps -ef|grep spark
    hduser    5495     1  0 18:12 pts/0    00:00:04 /usr/local/java/bin/java -cp /usr/local/spark/conf/:/usr/local/spark/jars/*:/usr/local/hadoop/etc/hadoop/ -Xmx1g org.apache.spark.deploy.master.Master --host master-hostname --port 7077 --webui-port 8080
  </code></pre>
  
  <p>On slave node 1</p>
  
  <pre><code> $SPARK_HOME/sbin/start-slave.sh spark://205.147.102.19:7077
  </code></pre>
  
  <p>Tested with </p>
  
  <pre><code> ps -ef|grep spark
   hduser    1847     1 20 18:24 pts/0    00:00:04 /usr/local/java/bin/java -cp /usr/local/spark/conf/:/usr/local/spark/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master-ip:7077
  </code></pre>
  
  <p>Same on slave-node 2</p>
  
  <pre><code>  $SPARK_HOME/sbin/start-slave.sh spark://master-ip:7077
    ps -ef|grep spark
    hduser    1948     1  3 18:18 pts/0    00:00:03 /usr/local/java/bin/java -cp /usr/local/spark/conf/:/usr/local/spark/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master-ip:7077
  </code></pre>
  
  <p>I was not able to see anything on the web console of spark.. so I thought problem may be with firewall. Here is my iptables..</p>
  
  <pre><code>  iptables -L -nv
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination         
    6136  587K fail2ban-ssh  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22
    151K   25M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    6   280 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    579 34740 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    34860 2856K ACCEPT     all  --  eth1   *       0.0.0.0/0            0.0.0.0/0           
    145  7608 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
    56156 5994K REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8080
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8081
  
    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination         
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
  
   Chain OUTPUT (policy ACCEPT 3531 packets, 464K bytes)
   pkts bytes target     prot opt in     out     source               destination         
  
   Chain fail2ban-ssh (1 references)
   pkts bytes target     prot opt in     out     source               destination         
   2   120 REJECT     all  --  *      *       218.87.109.153       0.0.0.0/0            reject-with icmp-port-unreachable
   5794  554K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  </code></pre>
  
  <p>I'm trying all I can to see if spark-cluster is set-up and how to check it properly. And if cluster is set-up, why am I not able to check that on web-console? What could be wrong? Any pointers would be helpful...</p>
  
  <p>EDIT - ADDING LOGS after spark-shell --master local command (in the master)</p>
  
  <pre><code> 17/01/11 18:12:46 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
   17/01/11 18:12:47 INFO master.Master: Starting Spark master at spark://master:7077
   17/01/11 18:12:47 INFO master.Master: Running Spark version 2.1.0
   17/01/11 18:12:47 INFO util.log: Logging initialized @3326ms
   17/01/11 18:12:47 INFO server.Server: jetty-9.2.z-SNAPSHOT
   17/01/11 18:12:47 INFO handler.ContextHandler: Started   o.s.j.s.ServletContextHandler@20f0b5ff{/app,null,AVAILABLE}
   17/01/11 18:12:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@734e74b2{/app/json,null,AVAILABLE}
   17/01/11 18:12:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1bc45d76{/,null,AVAILABLE}
   17/01/11 18:12:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6a274a23{/json,null,AVAILABLE}
   17/01/11 18:12:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4f5d45d5{/static,null,AVAILABLE}
   17/01/11 18:12:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4fb65368{/app/kill,null,AVAILABLE}
   17/01/11 18:12:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@76208805{/driver/kill,null,AVAILABLE}
   17/01/11 18:12:47 INFO server.ServerConnector: Started ServerConnector@258dbadd{HTTP/1.1}{0.0.0.0:8080}
   17/01/11 18:12:47 INFO server.Server: Started @3580ms
   17/01/11 18:12:47 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.
   17/01/11 18:12:47 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://master:8080
   17/01/11 18:12:47 INFO server.Server: jetty-9.2.z-SNAPSHOT
   17/01/11 18:12:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1cfbb7e9{/,null,AVAILABLE}
   17/01/11 18:12:47 INFO server.ServerConnector: Started ServerConnector@2f7af4e{HTTP/1.1}{master:6066}
   17/01/11 18:12:47 INFO server.Server: Started @3628ms
   17/01/11 18:12:47 INFO util.Utils: Successfully started service on port 6066.
   17/01/11 18:12:47 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
   17/01/11 18:12:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@799d5f4f{/metrics/master/json,null,AVAILABLE}
   17/01/11 18:12:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@647c46e3{/metrics/applications/json,null,AVAILABLE}
   17/01/11 18:12:47 INFO master.Master: I have been elected leader! New state: ALIVE
  </code></pre>
  
  <p>In  slave nodes-</p>
  
  <pre><code> 17/01/11 18:22:46 INFO Worker: Connecting to master master:7077...
   17/01/11 18:22:46 WARN Worker: Failed to connect to master master:7077
  </code></pre>
  
  <p>Tonnes of java errors..</p>
  
  <pre><code> 17/01/11 18:31:18 ERROR Worker: All masters are unresponsive! Giving up.
  </code></pre>