{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/4634","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/4634/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/4634/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/4634/events","html_url":"https://github.com/rancher/rancher/issues/4634","id":152045344,"number":4634,"title":"Rancher HA Stack Will Not Complete Initialization","user":{"login":"ThatsNinja","id":11461476,"avatar_url":"https://avatars.githubusercontent.com/u/11461476?v=3","gravatar_id":"","url":"https://api.github.com/users/ThatsNinja","html_url":"https://github.com/ThatsNinja","followers_url":"https://api.github.com/users/ThatsNinja/followers","following_url":"https://api.github.com/users/ThatsNinja/following{/other_user}","gists_url":"https://api.github.com/users/ThatsNinja/gists{/gist_id}","starred_url":"https://api.github.com/users/ThatsNinja/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThatsNinja/subscriptions","organizations_url":"https://api.github.com/users/ThatsNinja/orgs","repos_url":"https://api.github.com/users/ThatsNinja/repos","events_url":"https://api.github.com/users/ThatsNinja/events{/privacy}","received_events_url":"https://api.github.com/users/ThatsNinja/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancher/rancher/labels/area/ha","name":"area/ha","color":"FBCA04"}],"state":"open","locked":false,"assignee":{"login":"LLParse","id":6145659,"avatar_url":"https://avatars.githubusercontent.com/u/6145659?v=3","gravatar_id":"","url":"https://api.github.com/users/LLParse","html_url":"https://github.com/LLParse","followers_url":"https://api.github.com/users/LLParse/followers","following_url":"https://api.github.com/users/LLParse/following{/other_user}","gists_url":"https://api.github.com/users/LLParse/gists{/gist_id}","starred_url":"https://api.github.com/users/LLParse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LLParse/subscriptions","organizations_url":"https://api.github.com/users/LLParse/orgs","repos_url":"https://api.github.com/users/LLParse/repos","events_url":"https://api.github.com/users/LLParse/events{/privacy}","received_events_url":"https://api.github.com/users/LLParse/received_events","type":"User","site_admin":false},"milestone":null,"comments":7,"created_at":"2016-04-30T18:34:04Z","updated_at":"2016-05-02T22:50:01Z","closed_at":null,"body":"**Rancher Version:**\r\nv1.0.1\r\n\r\n**Docker Version:**\r\n1.10.3\r\n\r\n**OS:** \r\nRancherOS v0.4.4\r\n\r\n**Steps to Reproduce:** \r\n1.) Create RDS Maria DB Instance type `db.t2.medium`;\r\n2.) Create three new `t2.medium`-type instances based on Rancher-distributed AMI (https://aws.amazon.com/marketplace/pp/B01AB05EEA), with Security Group configured as follows:\r\n```\r\nInbound:\r\nHTTP / TCP / 80 / 0.0.0.0/0\r\nCustom TCP Rule / TCP / 2888 / 10.0.0.0/16\r\nCustom TCP Rule / TCP / 8080 / 0.0.0.0/0\r\nSSH / TCP / 22 / 0.0.0.0/0\r\nCustom TCP Rule / TCP / 6379 / 10.0.0.0/16\r\nCustom TCP Rule / TCP / 2376 / 10.0.0.0/16\r\nCustom TCP Rule / TCP / 3888 / 10.0.0.0/16\r\nCustom TCP Rule / TCP / 2181 / 10.0.0.0/16\r\nCustom UDP Rule / UDP / 500 / 10.0.0.0/16\r\nCustom UDP Rule / UDP / 4500 / 10.0.0.0/16\r\nCustom TCP Rule / TCP / 18080 / 0.0.0.0/0\r\nHTTPS / TCP / 443 / 0.0.0.0/0\r\n```\r\n3.) Create Elastic Load Balancer, configured as follows:\r\n*Listeners*\r\n```\r\nHTTP / 80 / HTTP / 80 / N/A / N/A\r\nSSL / 443 / TCP / 443 / (Change) / (Self-signed cert with ELB FQDN CN)\r\nHTTP / 18080 / HTTP / 18080 / N/A / N/A\r\n```\r\n*Security Group*\r\n```\r\nInbound:\r\nHTTP / TCP / 80 / 0.0.0.0/0\r\nSSH / TCP / 22 / 0.0.0.0/0\r\nCustom TCP Rule / TCP / 18080 / 0.0.0.0/0\r\nHTTPS / TCP / 443 / 0.0.0.0/0\r\n```\r\n4.) Add `ProxyProtocol` configuration to ELB:\r\n```\r\naws elb create-load-balancer-policy --load-balancer-name my-elb --policy-name my-ProxyProtocol-policy --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true\r\naws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-elb --instance-port 80 --policy-names my-ProxyProtocol-policy\r\naws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-elb --instance-port 443 --policy-names my-ProxyProtocol-policy\r\n```\r\n5.) Start standalone Rancher server on one of the newly configured RancherOS host instances (created at step 2, above):\r\n```\r\nsudo docker run -d -p 8080:8080 \\\r\n-e CATTLE_DB_CATTLE_MYSQL_HOST=my-rds-mariadb-instance.rds.amazonaws.com \\\r\n-e CATTLE_DB_CATTLE_MYSQL_PORT=3306 \\\r\n-e CATTLE_DB_CATTLE_MYSQL_NAME=cattle \\\r\n-e CATTLE_DB_CATTLE_USERNAME=cattle \\\r\n-e CATTLE_DB_CATTLE_PASSWORD=cattle \\\r\n-v /var/run/docker.sock:/var/run/docker.sock \\\r\nrancher/server:v1.0.1\r\n```\r\n6.) Navigate to `http://<step-5-standalone-rancher-host>:8080`. Via **Admin > HA**, verify that steps 1-2 of HA config are completed - that the standalone Rancher server has connectivity to the RDS DB instance.\r\n7.) Select the option to create a self-signed certificate.\r\n8.) Generate the `rancher-ha.sh` startup script, and download it.\r\n9.) Create a copy of the `rancher-ha.sh` script downloaded at 8, above, on each of the instances created at step 2, above.\r\n10.) Stop the standalone Rancher server created at step 5, above.\r\n11.) Execute the startup script on each of the hosts:\r\n```\r\nsudo bash ./rancher-ha.sh rancher/server:v1.0.1\r\n```\r\n**Expected:**\r\nA three-node HA Rancher Server cluster.\r\n\r\n**Results:** \r\nTwelve hours of hair-pulling frustration. Here's the most recent taste of what I'm seeing:\r\n*Node 1*\r\n```bash\r\n$ docker logs --tail=\"5\" rancher-ha\r\ntime=\"2016-04-30T17:07:46Z\" level=info msg=\"[10/10] [load-balancer-swarm]: Started \"\r\ntime=\"2016-04-30T17:07:46Z\" level=info msg=\"Done launching management stack\" component=service\r\ntime=\"2016-04-30T17:07:46Z\" level=info msg=\"You can access the site at https://ssg-tools-elb-1143025530.us-east-1.elb.amazonaws.com\" component=service\r\ntime=\"2016-04-30T17:07:52Z\" level=info msg=\"Container agent is not running in state &types.ContainerState{Status:\\\"exited\\\", Running:false, Paused:false, Restarting:false, OOMKilled:false, Dead:false, Pid:0, ExitCode:0, Error:\\\"\\\", StartedAt:\\\"2016-04-30T17:06:33.932685014Z\\\", FinishedAt:\\\"2016-04-30T17:06:42.479756753Z\\\"}\" component=docker\r\ntime=\"2016-04-30T17:07:52Z\" level=info msg=\"Deleting container ef28cc7bbb2343296df4e4d0e2729a56deb8484e132a430de5ca18d9dc62e676\" component=docker\r\n$ docker logs --tail=\"5\" rancher-ha-zk\r\n++ JAVA_OPTS=\r\n+ exec java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=WARN,CONSOLE -cp /etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg\r\n2016-04-30 17:05:46,617 - ERROR [main:QuorumPeerConfig@283] - Invalid configuration, only one server specified (ignoring)\r\n2016-04-30 17:05:46,624 - WARN  [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running  in standalone mode\r\n2016-04-30 17:05:46,708 - ERROR [main:QuorumPeerConfig@283] - Invalid configuration, only one server specified (ignoring)\r\n$ netstat -lnt\r\nActive Internet connections (only servers)\r\nProto Recv-Q Send-Q Local Address           Foreign Address         State\r\ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN\r\ntcp        0      0 :::22                   :::*                    LISTEN\r\ntcp        0      0 :::443                  :::*                    LISTEN\r\ntcp        0      0 :::444                  :::*                    LISTEN\r\ntcp        0      0 :::2376                 :::*                    LISTEN\r\ntcp        0      0 :::80                   :::*                    LISTEN\r\ntcp        0      0 :::81                   :::*                    LISTEN\r\n```\r\n*Local Machine*\r\n```bash\r\n$ curl https://<node-1-ip>/ping --insecure\r\n<html><body><h1>503 Service Unavailable</h1>\r\nNo server is available to handle this request.\r\n</body></html>\r\n\r\n$ curl http://<node-1-ip>/ping\r\n<html><body><h1>503 Service Unavailable</h1>\r\nNo server is available to handle this request.\r\n</body></html>\r\n```\r\n*Node 2*\r\n```bash\r\n$ docker logs --tail=\"5\" rancher-ha\r\ntime=\"2016-04-30T17:15:39Z\" level=info msg=\"Can not launch agent right now: Server not available at http://172.17.0.1:18080/ping:\" component=service\r\ntime=\"2016-04-30T17:15:44Z\" level=info msg=\"Waiting for server to be available\" component=cert\r\ntime=\"2016-04-30T17:15:44Z\" level=info msg=\"Can not launch agent right now: Server not available at http://172.17.0.1:18080/ping:\" component=service\r\ntime=\"2016-04-30T17:15:50Z\" level=info msg=\"Waiting for server to be available\" component=cert\r\ntime=\"2016-04-30T17:15:50Z\" level=info msg=\"Can not launch agent right now: Server not available at http://172.17.0.1:18080/ping:\" component=service\r\n$ docker logs --tail=\"5\" rancher-ha-zk\r\n2016-04-30 17:15:25,786 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception\r\nEndOfStreamException: Unable to read additional data from client sessionid 0x154682ac0380000, likely client has closed socket\r\n\tat org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)\r\n\tat org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n$ netstat -lnt\r\nActive Internet connections (only servers)\r\nProto Recv-Q Send-Q Local Address           Foreign Address         State\r\ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN\r\ntcp        0      0 :::22                   :::*                    LISTEN\r\ntcp        0      0 :::443                  :::*                    LISTEN\r\ntcp        0      0 :::444                  :::*                    LISTEN\r\ntcp        0      0 :::2376                 :::*                    LISTEN\r\ntcp        0      0 :::80                   :::*                    LISTEN\r\ntcp        0      0 :::81                   :::*                    LISTEN\r\n```\r\n*Node 3*\r\n```bash\r\n$ docker logs --tail=\"5\" rancher-ha\r\nError: No such container: rancher-ha\r\n$ docker logs --tail=\"5\" rancher-ha-zk\r\n2016-04-30 17:17:55,364 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception\r\nEndOfStreamException: Unable to read additional data from client sessionid 0x154682c2b620000, likely client has closed socket\r\n\tat org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)\r\n\tat org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n$ netstat -lnt\r\nActive Internet connections (only servers)\r\nProto Recv-Q Send-Q Local Address           Foreign Address         State\r\ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN\r\ntcp        0      0 :::22                   :::*                    LISTEN\r\ntcp        0      0 :::443                  :::*                    LISTEN\r\ntcp        0      0 :::444                  :::*                    LISTEN\r\ntcp        0      0 :::80                   :::*                    LISTEN\r\ntcp        0      0 :::81                   :::*                    LISTEN\r\n```\r\nThe above represents current state. I've been through three iterations of configuration, and decided to scorch the earth and start over, employing lessons learned. I'll be happy to provide any requested information via whatever means desired. Any thoughts, suggestions, criticisms, and/or outright insults to my intelligence would be hugely appreciated at this point."},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/216388182","html_url":"https://github.com/rancher/rancher/issues/4634#issuecomment-216388182","issue_url":"https://api.github.com/repos/rancher/rancher/issues/4634","id":216388182,"user":{"login":"ThatsNinja","id":11461476,"avatar_url":"https://avatars.githubusercontent.com/u/11461476?v=3","gravatar_id":"","url":"https://api.github.com/users/ThatsNinja","html_url":"https://github.com/ThatsNinja","followers_url":"https://api.github.com/users/ThatsNinja/followers","following_url":"https://api.github.com/users/ThatsNinja/following{/other_user}","gists_url":"https://api.github.com/users/ThatsNinja/gists{/gist_id}","starred_url":"https://api.github.com/users/ThatsNinja/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThatsNinja/subscriptions","organizations_url":"https://api.github.com/users/ThatsNinja/orgs","repos_url":"https://api.github.com/users/ThatsNinja/repos","events_url":"https://api.github.com/users/ThatsNinja/events{/privacy}","received_events_url":"https://api.github.com/users/ThatsNinja/received_events","type":"User","site_admin":false},"created_at":"2016-05-02T22:50:01Z","updated_at":"2016-05-02T22:50:01Z","body":"Two of the nodes respond with:\r\n```\r\n$ docker exec -it rancher-ha-zk bash\r\nError response from daemon: No such container: rancher-ha-zk\r\n```\r\nFrom the node that does have the container running:\r\n```\r\n$ docker exec -it rancher-ha-zk bash\r\nroot@da31c531a227:/# echo mntr | nc localhost 2181\r\nThis ZooKeeper instance is not currently serving requests\r\nroot@da31c531a227:/# echo mntr | nc localhost 2182\r\nroot@da31c531a227:/# echo mntr | nc localhost 2183\r\nroot@da31c531a227:/#\r\n```"}}