{"action":"created","issue":{"url":"https://api.github.com/repos/metral/corekube/issues/23","repository_url":"https://api.github.com/repos/metral/corekube","labels_url":"https://api.github.com/repos/metral/corekube/issues/23/labels{/name}","comments_url":"https://api.github.com/repos/metral/corekube/issues/23/comments","events_url":"https://api.github.com/repos/metral/corekube/issues/23/events","html_url":"https://github.com/metral/corekube/issues/23","id":131384917,"number":23,"title":"Failed to start flannel.service","user":{"login":"glenselle","id":1056932,"avatar_url":"https://avatars.githubusercontent.com/u/1056932?v=3","gravatar_id":"","url":"https://api.github.com/users/glenselle","html_url":"https://github.com/glenselle","followers_url":"https://api.github.com/users/glenselle/followers","following_url":"https://api.github.com/users/glenselle/following{/other_user}","gists_url":"https://api.github.com/users/glenselle/gists{/gist_id}","starred_url":"https://api.github.com/users/glenselle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glenselle/subscriptions","organizations_url":"https://api.github.com/users/glenselle/orgs","repos_url":"https://api.github.com/users/glenselle/repos","events_url":"https://api.github.com/users/glenselle/events{/privacy}","received_events_url":"https://api.github.com/users/glenselle/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-02-04T15:36:38Z","updated_at":"2016-02-04T17:22:45Z","closed_at":null,"body":"I'm trying to use the public cloud heat template without any changes with the Rackspace orchestration service but the etcd cluster never comes up. Doing `etcdctl cluster-health` on the discovery machine says all is well. But doing the same on kubernetes master yeilds the following:\r\n\r\n    cluster may be unhealthy: failed to list members\r\n    Error:  unexpected status code 404\r\n\r\nDoing `journalctl -f` outputs this:\r\n\r\n    Feb 04 15:33:06 kubernetes-master etcdctl[2224]: Error:  unexpected status code 404\r\n    Feb 04 15:33:06 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4\r\n    Feb 04 15:33:06 kubernetes-master systemd[1]: Failed to start flannel.service.\r\n    Feb 04 15:33:06 kubernetes-master systemd[1]: flannel.service: Unit entered failed state.\r\n    Feb 04 15:33:06 kubernetes-master systemd[1]: flannel.service: Failed with result 'exit-code'.\r\n    Feb 04 15:33:11 kubernetes-master systemd[1]: flannel.service: Service hold-off time over, scheduling restart.\r\n    Feb 04 15:33:11 kubernetes-master systemd[1]: Stopped flannel.service.\r\n    Feb 04 15:33:11 kubernetes-master systemd[1]: Starting flannel.service...\r\n    Feb 04 15:33:11 kubernetes-master etcdctl[2233]: Error:  unexpected status code 404\r\n    Feb 04 15:33:11 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4\r\n    Feb 04 15:33:11 kubernetes-master systemd[1]: Failed to start flannel.service.\r\n    Feb 04 15:33:11 kubernetes-master systemd[1]: flannel.service: Unit entered failed state.\r\n    Feb 04 15:33:11 kubernetes-master systemd[1]: flannel.service: Failed with result 'exit-code'.\r\n    Feb 04 15:33:16 kubernetes-master systemd[1]: flannel.service: Service hold-off time over, scheduling restart.\r\n    Feb 04 15:33:16 kubernetes-master systemd[1]: Stopped flannel.service.\r\n    Feb 04 15:33:16 kubernetes-master systemd[1]: Starting flannel.service...\r\n    Feb 04 15:33:16 kubernetes-master etcdctl[2241]: Error:  unexpected status code 404\r\n    Feb 04 15:33:16 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4\r\n    Feb 04 15:33:16 kubernetes-master systemd[1]: Failed to start flannel.service.\r\n    Feb 04 15:33:16 kubernetes-master systemd[1]: flannel.service: Unit entered failed state.\r\n    Feb 04 15:33:16 kubernetes-master systemd[1]: flannel.service: Failed with result 'exit-code'.\r\n    Feb 04 15:33:17 kubernetes-master kube-scheduler[1709]: I0204 15:33:17.309879    1709 event.go:206] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-dns-v8-fprbi\", UID:\"f7a1fc1d-cb53-11e5-8199-bc764e11c1a6\", APIVersion:\"v1\", ResourceVersion:\"19\", FieldPath:\"\"}): reason: 'FailedScheduling' no nodes available to schedule pods\r\n    Feb 04 15:33:18 kubernetes-master systemd[1]: Starting Generate /run/coreos/motd...\r\n    Feb 04 15:33:18 kubernetes-master systemd[1]: Started Generate /run/coreos/motd.\r\n    Feb 04 15:33:22 kubernetes-master systemd[1]: flannel.service: Service hold-off time over, scheduling restart.\r\n    Feb 04 15:33:22 kubernetes-master systemd[1]: Stopped flannel.service.\r\n    Feb 04 15:33:22 kubernetes-master systemd[1]: Starting flannel.service...\r\n    Feb 04 15:33:22 kubernetes-master etcdctl[2257]: Error:  unexpected status code 404\r\n    Feb 04 15:33:22 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4\r\n    Feb 04 15:33:22 kubernetes-master systemd[1]: Failed to start flannel.service.\r\n    Feb 04 15:33:22 kubernetes-master systemd[1]: flannel.service: Unit entered failed state.\r\n    Feb 04 15:33:22 kubernetes-master systemd[1]: flannel.service: Failed with result 'exit-code'.\r\n    Feb 04 15:33:27 kubernetes-master systemd[1]: flannel.service: Service hold-off time over, scheduling restart.\r\n    Feb 04 15:33:27 kubernetes-master systemd[1]: Stopped flannel.service.\r\n    Feb 04 15:33:27 kubernetes-master systemd[1]: Starting flannel.service...\r\n    Feb 04 15:33:27 kubernetes-master etcdctl[2265]: Error:  unexpected status code 404\r\n    Feb 04 15:33:27 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4\r\n    Feb 04 15:33:27 kubernetes-master systemd[1]: Failed to start flannel.service.\r\n\r\nInspecting just the etcd service looks just fine (again on the kubernetes master machine):\r\n\r\n    $ journalctl -u etcd.service\r\n    -- Logs begin at Thu 2016-02-04 15:25:00 UTC, end at Thu 2016-02-04 15:53:25 UTC. --\r\n    Feb 04 15:25:15 kubernetes-master systemd[1]: Started etcd.\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.306 INFO      | Discovery via http://10.210.193.89:2379 using prefix discovery/1mn3MOFVbKnC02nbnhi2BZ7nGAXjPkSR.\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.313 INFO      | Discovery found peers [http://10.209.104.221:7001]\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.313 INFO      | Discovery fetched back peer list: [10.209.104.221:7001]\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.320 INFO      | Send Join Request to http://10.209.104.221:7001/join\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.330 INFO      | kubernetes_master joined the cluster via peer 10.209.104.221:7001\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.335 INFO      | etcd server [name kubernetes_master, listen on :4001, advertised url http://10.209.102.21:4001]\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.335 INFO      | peer server [name kubernetes_master, listen on :7001, advertised url http://10.209.102.21:7001]\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.336 INFO      | kubernetes_master starting in peer mode\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.336 INFO      | kubernetes_master: state changed from 'initialized' to 'follower'.\r\n    Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:15.387 INFO      | kubernetes_master: peer added: 'overlord'\r\n    Feb 04 15:25:16 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:16.785 INFO      | kubernetes_master: peer added: 'kubernetes_minion_0'\r\n    Feb 04 15:25:18 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:18.402 INFO      | kubernetes_master: peer added: 'kubernetes_minion_1'\r\n    Feb 04 15:25:19 kubernetes-master etcd[1256]: [etcd] Feb  4 15:25:19.179 INFO      | kubernetes_master: peer added: 'kubernetes_minion_2'\r\n\r\nFor days I've been trying to diagnose the issue with no luck. Is this a template issue? I've also noticed any Docker commands just hang, but only on machines where the etcd cluster is not working. \r\n\r\nCoreOS 835.9.0"},"comment":{"url":"https://api.github.com/repos/metral/corekube/issues/comments/179955174","html_url":"https://github.com/metral/corekube/issues/23#issuecomment-179955174","issue_url":"https://api.github.com/repos/metral/corekube/issues/23","id":179955174,"user":{"login":"glenselle","id":1056932,"avatar_url":"https://avatars.githubusercontent.com/u/1056932?v=3","gravatar_id":"","url":"https://api.github.com/users/glenselle","html_url":"https://github.com/glenselle","followers_url":"https://api.github.com/users/glenselle/followers","following_url":"https://api.github.com/users/glenselle/following{/other_user}","gists_url":"https://api.github.com/users/glenselle/gists{/gist_id}","starred_url":"https://api.github.com/users/glenselle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glenselle/subscriptions","organizations_url":"https://api.github.com/users/glenselle/orgs","repos_url":"https://api.github.com/users/glenselle/repos","events_url":"https://api.github.com/users/glenselle/events{/privacy}","received_events_url":"https://api.github.com/users/glenselle/received_events","type":"User","site_admin":false},"created_at":"2016-02-04T17:22:45Z","updated_at":"2016-02-04T17:22:45Z","body":"Thanks for the response. First, this is not production cluster. I'm literally copy-pasting the template into the orchestration service and creating the stack--with no success. Can you verify it works for you?\r\n\r\nHere are some more logs from kubernetes-master:\r\n\r\n    kubernetes-master ~ # systemctl status flannel -l\r\n    ● flannel.service\r\n       Loaded: loaded (/etc/systemd/system/flannel.service; static; vendor preset: disabled)\r\n       Active: active (running) since Thu 2016-02-04 17:03:25 UTC; 11min ago\r\n      Process: 1314 ExecStartPre=/usr/bin/etcdctl mk /coreos.com/network/config {\"Network\":\"10.244.0.0/15\", \"Backend\": {\"Type\": \"vxlan\"}} (code=exited, status=0/SUCCESS)\r\n     Main PID: 1321 (flanneld)\r\n       CGroup: /system.slice/flannel.service\r\n               └─1321 /opt/bin/flanneld -iface=eth2\r\n\r\n    Feb 04 17:03:25 kubernetes-master flanneld[1321]: I0204 17:03:25.871428 01321 main.go:275] Installing signal handlers\r\n    Feb 04 17:03:25 kubernetes-master flanneld[1321]: I0204 17:03:25.880686 01321 main.go:188] Using 192.168.3.4 as external interface\r\n    Feb 04 17:03:25 kubernetes-master flanneld[1321]: I0204 17:03:25.885072 01321 main.go:189] Using 192.168.3.4 as external endpoint\r\n    Feb 04 17:03:25 kubernetes-master flanneld[1321]: I0204 17:03:25.906731 01321 etcd.go:204] Picking subnet in range 10.244.1.0 ... 10.245.255.0\r\n    Feb 04 17:03:25 kubernetes-master flanneld[1321]: I0204 17:03:25.913410 01321 etcd.go:84] Subnet lease acquired: 10.244.5.0/24\r\n    Feb 04 17:03:25 kubernetes-master flanneld[1321]: I0204 17:03:25.917238 01321 vxlan.go:153] Watching for L3 misses\r\n    Feb 04 17:03:25 kubernetes-master flanneld[1321]: I0204 17:03:25.918510 01321 vxlan.go:159] Watching for new subnet leases\r\n    Feb 04 17:03:25 kubernetes-master flanneld[1321]: I0204 17:03:25.919716 01321 vxlan.go:273] Handling initial subnet events\r\n    Feb 04 17:03:25 kubernetes-master flanneld[1321]: I0204 17:03:25.919731 01321 device.go:159] calling GetL2List() dev.link.Index: 5\r\n    Feb 04 17:13:26 kubernetes-master flanneld[1321]: E0204 17:13:26.596561 01321 watch.go:41] Watch subnets: client: etcd cluster is unavailable or misconfigured\r\n    kubernetes-master ~ # systemctl status etcd -l\r\n    ● etcd.service - etcd\r\n       Loaded: loaded (/usr/lib64/systemd/system/etcd.service; static; vendor preset: disabled)\r\n      Drop-In: /run/systemd/system/etcd.service.d\r\n               └─10-oem.conf\r\n       Active: active (running) since Thu 2016-02-04 17:03:25 UTC; 11min ago\r\n     Main PID: 1291 (etcd)\r\n       CGroup: /system.slice/etcd.service\r\n               └─1291 /usr/bin/etcd\r\n\r\n    Feb 04 17:03:25 kubernetes-master systemd[1]: Started etcd.\r\n    Feb 04 17:03:25 kubernetes-master etcd[1291]: [etcd] Feb  4 17:03:25.269 INFO      | b45fd7970b9d4961847d5de0d36224be is starting a new cluster\r\n    Feb 04 17:03:25 kubernetes-master etcd[1291]: [etcd] Feb  4 17:03:25.274 INFO      | etcd server [name b45fd7970b9d4961847d5de0d36224be, listen on :4001, advertised url http://127.0.0.1:4001]\r\n    Feb 04 17:03:25 kubernetes-master etcd[1291]: [etcd] Feb  4 17:03:25.277 INFO      | peer server [name b45fd7970b9d4961847d5de0d36224be, listen on :7001, advertised url http://127.0.0.1:7001]\r\n    Feb 04 17:03:25 kubernetes-master etcd[1291]: [etcd] Feb  4 17:03:25.279 INFO      | b45fd7970b9d4961847d5de0d36224be starting in peer mode\r\n    Feb 04 17:03:25 kubernetes-master etcd[1291]: [etcd] Feb  4 17:03:25.281 INFO      | b45fd7970b9d4961847d5de0d36224be: state changed from 'initialized' to 'follower'.\r\n    Feb 04 17:03:25 kubernetes-master etcd[1291]: [etcd] Feb  4 17:03:25.282 INFO      | b45fd7970b9d4961847d5de0d36224be: state changed from 'follower' to 'leader'.\r\n    Feb 04 17:03:25 kubernetes-master etcd[1291]: [etcd] Feb  4 17:03:25.282 INFO      | b45fd7970b9d4961847d5de0d36224be: leader changed from '' to 'b45fd7970b9d4961847d5de0d36224be'.\r\n\r\nAnd when I try to dump the logs from the discovery machine I get this:\r\n\r\n     discovery ~ # docker logs discovery \r\n    \"logs\" command is supported only for \"json-file\" logging driver (got: journald)\r\n\r\nI swapped out CoreOS for a newer version, but that throws all the other dependencies out of whack. Is there nothing special I have to do? And can you verify that using the Rackspace Control panel orchestration should work fine? I'm not using any of the OpenStack CLI tools."}}