{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/4271","labels_url":"https://api.github.com/repos/coreos/etcd/issues/4271/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/4271/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/4271/events","html_url":"https://github.com/coreos/etcd/issues/4271","id":128370389,"number":4271,"title":"After rebooting, etcd2 not working on EBS-Optimized EC2 machines","user":{"login":"zhouqianli","id":1570319,"avatar_url":"https://avatars.githubusercontent.com/u/1570319?v=3","gravatar_id":"","url":"https://api.github.com/users/zhouqianli","html_url":"https://github.com/zhouqianli","followers_url":"https://api.github.com/users/zhouqianli/followers","following_url":"https://api.github.com/users/zhouqianli/following{/other_user}","gists_url":"https://api.github.com/users/zhouqianli/gists{/gist_id}","starred_url":"https://api.github.com/users/zhouqianli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhouqianli/subscriptions","organizations_url":"https://api.github.com/users/zhouqianli/orgs","repos_url":"https://api.github.com/users/zhouqianli/repos","events_url":"https://api.github.com/users/zhouqianli/events{/privacy}","received_events_url":"https://api.github.com/users/zhouqianli/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-01-24T03:52:30Z","updated_at":"2016-02-01T02:08:03Z","closed_at":null,"body":"We enable **EBS-Optimized** setting for machines.  After rebooting etcd cannot working again.\r\nError message from `fleetctl list-units`:\r\n```\r\nError retrieving list of units from repository: googleapi: Error 503: fleet server unable to communicate with etcd\r\n```\r\n### CoreOS version:\r\n```\r\nNAME=CoreOS\r\nID=coreos\r\nVERSION=835.9.0\r\nVERSION_ID=835.9.0\r\nBUILD_ID=\r\nPRETTY_NAME=\"CoreOS 835.9.0\"\r\nANSI_COLOR=\"1;32\"\r\nHOME_URL=\"https://coreos.com/\"\r\nBUG_REPORT_URL=\"https://github.com/coreos/bugs/issues\"\r\n```\r\n\r\n### sudo systemctl cat etcd2:\r\n```\r\n# /usr/lib64/systemd/system/etcd2.service\r\n[Unit]\r\nDescription=etcd2\r\nConflicts=etcd.service\r\n\r\n[Service]\r\nUser=etcd\r\nType=notify\r\nEnvironment=ETCD_DATA_DIR=/var/lib/etcd2\r\nEnvironment=ETCD_NAME=%m\r\nExecStart=/usr/bin/etcd2\r\nRestart=always\r\nRestartSec=10s\r\nLimitNOFILE=40000\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n\r\n# /run/systemd/system/etcd2.service.d/10-oem.conf\r\n[Service]\r\nEnvironment=ETCD_ELECTION_TIMEOUT=1200\r\n# /run/systemd/system/etcd2.service.d/20-cloudinit.conf\r\n[Service]\r\nEnvironment=\"ETCD_ADVERTISE_CLIENT_URLS=http://10.2.8.197:2379,http://10.2.8.197:4001\"\r\nEnvironment=\"ETCD_DISCOVERY=https://discovery.etcd.io/<token>\"\r\nEnvironment=\"ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.2.8.197:2380\"\r\nEnvironment=\"ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001\"\r\nEnvironment=\"ETCD_LISTEN_PEER_URLS=http://10.2.8.197:2380,http://10.2.8.197:7001\"\r\n\r\n```\r\n\r\n### sudo journalctl -u etcd2\r\n```\r\n-- Logs begin at Tue 2016-01-19 04:09:00 UTC, end at Tue 2016-01-26 08:02:50 UTC. --\r\nJan 19 04:09:24 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: Starting etcd2...\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://10.2.8.197:2379,http://10.2.8.197:4001\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/873f71bc011a204dd1731c9a456062ab\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=1200\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.2.8.197:2380\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://10.2.8.197:2380,http://10.2.8.197:7001\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: recognized and used environment variable ETCD_NAME=3beb5dfb4f6646eabd8a6bd61aa9eb53\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: etcd Version: 2.2.0\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: Git SHA: e4561dd\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: Go Version: go1.4.2\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: Go OS/Arch: linux/amd64\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: setting maximum number of CPUs to 1, total number of available CPUs is 8\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: listening for peers on http://10.2.8.197:2380\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: listening for peers on http://10.2.8.197:7001\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: listening for client requests on http://0.0.0.0:2379\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: listening for client requests on http://0.0.0.0:4001\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: stopping listening for client requests on http://0.0.0.0:4001\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: stopping listening for client requests on http://0.0.0.0:2379\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: stopping listening for peers on http://10.2.8.197:7001\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: stopping listening for peers on http://10.2.8.197:2380\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: discovery cluster full, falling back to proxy\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: proxy: using peer urls [http://10.2.2.150:2380 http://10.2.2.151:2380 http://10.2.2.243:2380]\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: proxy: listening for client requests on http://0.0.0.0:2379\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: Started etcd2.\r\nJan 19 04:09:27 ip-10-2-8-197.us-west-2.compute.internal etcd2[856]: proxy: listening for client requests on http://0.0.0.0:4001\r\nJan 19 13:56:51 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: Stopping etcd2...\r\nJan 19 13:56:51 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: Stopped etcd2.\r\n-- Reboot --\r\nJan 19 14:03:18 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: Starting etcd2...\r\nJan 19 14:03:18 ip-10-2-8-197.us-west-2.compute.internal etcd2[762]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://10.2.8.197:2379,http://10.2.8.197:4001\r\nJan 19 14:03:18 ip-10-2-8-197.us-west-2.compute.internal etcd2[762]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2\r\nJan 19 14:03:18 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: etcd2.service: Main process exited, code=exited, status=1/FAILURE\r\nJan 19 14:03:18 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: Failed to start etcd2.\r\nJan 19 14:03:18 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: etcd2.service: Unit entered failed state.\r\nJan 19 14:03:18 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: etcd2.service: Failed with result 'exit-code'.\r\nJan 19 14:03:28 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: etcd2.service: Service hold-off time over, scheduling restart.\r\nJan 19 14:03:28 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: Stopped etcd2.\r\nJan 19 14:03:28 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: Starting etcd2...\r\nJan 19 14:03:28 ip-10-2-8-197.us-west-2.compute.internal etcd2[786]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://10.2.8.197:2379,http://10.2.8.197:4001\r\nJan 19 14:03:28 ip-10-2-8-197.us-west-2.compute.internal etcd2[786]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2\r\nJan 19 14:03:28 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: etcd2.service: Main process exited, code=exited, status=1/FAILURE\r\nJan 19 14:03:28 ip-10-2-8-197.us-west-2.compute.internal systemd[1]: Failed to start etcd2.\r\n\r\n```\r\nThanks for any help."},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/177706962","html_url":"https://github.com/coreos/etcd/issues/4271#issuecomment-177706962","issue_url":"https://api.github.com/repos/coreos/etcd/issues/4271","id":177706962,"user":{"login":"zhouqianli","id":1570319,"avatar_url":"https://avatars.githubusercontent.com/u/1570319?v=3","gravatar_id":"","url":"https://api.github.com/users/zhouqianli","html_url":"https://github.com/zhouqianli","followers_url":"https://api.github.com/users/zhouqianli/followers","following_url":"https://api.github.com/users/zhouqianli/following{/other_user}","gists_url":"https://api.github.com/users/zhouqianli/gists{/gist_id}","starred_url":"https://api.github.com/users/zhouqianli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhouqianli/subscriptions","organizations_url":"https://api.github.com/users/zhouqianli/orgs","repos_url":"https://api.github.com/users/zhouqianli/repos","events_url":"https://api.github.com/users/zhouqianli/events{/privacy}","received_events_url":"https://api.github.com/users/zhouqianli/received_events","type":"User","site_admin":false},"created_at":"2016-02-01T02:08:03Z","updated_at":"2016-02-01T02:08:03Z","body":"Still not working.  I will request new machine to repo this issue.\r\n\r\n```\r\ncore@ip-10-2-8-230 ~ $ sudo journalctl -f -t etcd2\r\n-- Logs begin at Tue 2016-01-19 04:23:29 UTC. --\r\n```"}}