{"action":"created","issue":{"url":"https://api.github.com/repos/deis/workflow/issues/422","repository_url":"https://api.github.com/repos/deis/workflow","labels_url":"https://api.github.com/repos/deis/workflow/issues/422/labels{/name}","comments_url":"https://api.github.com/repos/deis/workflow/issues/422/comments","events_url":"https://api.github.com/repos/deis/workflow/issues/422/events","html_url":"https://github.com/deis/workflow/issues/422","id":168882046,"number":422,"title":"Workflow is on but No Pods ","user":{"login":"DavidSie","id":8830246,"avatar_url":"https://avatars.githubusercontent.com/u/8830246?v=3","gravatar_id":"","url":"https://api.github.com/users/DavidSie","html_url":"https://github.com/DavidSie","followers_url":"https://api.github.com/users/DavidSie/followers","following_url":"https://api.github.com/users/DavidSie/following{/other_user}","gists_url":"https://api.github.com/users/DavidSie/gists{/gist_id}","starred_url":"https://api.github.com/users/DavidSie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DavidSie/subscriptions","organizations_url":"https://api.github.com/users/DavidSie/orgs","repos_url":"https://api.github.com/users/DavidSie/repos","events_url":"https://api.github.com/users/DavidSie/events{/privacy}","received_events_url":"https://api.github.com/users/DavidSie/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/workflow/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2016-08-02T13:15:17Z","updated_at":"2016-08-03T07:45:38Z","closed_at":null,"body":"Hello I was following guide: https://deis.com/docs/workflow/installing-workflow/\r\nI'am using:\r\n\r\nCoreOS 1010.5.0 Stable\r\n\r\nkubernetes:    \r\n          Client Version: version.Info{Major:\"1\", Minor:\"2\", GitVersion:\"v1.2.4\",       GitCommit:\"3eed1e3be6848b877ff80a93da3785d9034d0a4f\", GitTreeState:\"clean\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"2\", GitVersion:\"v1.2.4+coreos.1\", GitCommit:\"7f80f816ee1a23c26647aee8aecd32f0b21df754\", GitTreeState:\"clean\"} \r\n\r\n`helmc: helmc version 0.8.1+63c6d05`\r\n\r\n`helmc target\r\nKubernetes master is running at https://some_ip`\r\n\r\nAnd when I perform : \r\n\r\n ` helmc repo add deis https://github.com/deis/charts`\r\n ` helmc fetch deis/workflow-v2.2.0`\r\n ` helmc generate -x manifests workflow-v2.2.0`\r\n ` helmc install workflow-v2.2.0`\r\n\r\n\r\nhelm creates services,service account,  secrets  **but  no pods**"},"comment":{"url":"https://api.github.com/repos/deis/workflow/issues/comments/237166967","html_url":"https://github.com/deis/workflow/issues/422#issuecomment-237166967","issue_url":"https://api.github.com/repos/deis/workflow/issues/422","id":237166967,"user":{"login":"DavidSie","id":8830246,"avatar_url":"https://avatars.githubusercontent.com/u/8830246?v=3","gravatar_id":"","url":"https://api.github.com/users/DavidSie","html_url":"https://github.com/DavidSie","followers_url":"https://api.github.com/users/DavidSie/followers","following_url":"https://api.github.com/users/DavidSie/following{/other_user}","gists_url":"https://api.github.com/users/DavidSie/gists{/gist_id}","starred_url":"https://api.github.com/users/DavidSie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DavidSie/subscriptions","organizations_url":"https://api.github.com/users/DavidSie/orgs","repos_url":"https://api.github.com/users/DavidSie/repos","events_url":"https://api.github.com/users/DavidSie/events{/privacy}","received_events_url":"https://api.github.com/users/DavidSie/received_events","type":"User","site_admin":false},"created_at":"2016-08-03T07:45:38Z","updated_at":"2016-08-03T07:45:38Z","body":"```\r\n   kubectl get events\r\n```\r\nis not returning anything\r\n\r\nlogs from master\r\n```\r\n sudo systemctl status kubelet -l\r\n● kubelet.service\r\n   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)\r\n   Active: active (running) since Tue 2016-08-02 18:31:44 UTC; 13h ago\r\n  Process: 1506 ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)\r\n Main PID: 1513 (kubelet)\r\n    Tasks: 12\r\n   Memory: 234.3M\r\n      CPU: 13min 16.068s\r\n   CGroup: /system.slice/kubelet.service\r\n           ├─1513 /kubelet --api-servers=http://127.0.0.1:8080 --network-plugin-dir=/etc/kubernetes/cni/net.d --network-plugin=cni --register-schedulable=false --allow-privileged=true --config=/etc/kubernetes/ma\r\n           └─1634 journalctl -k -f\r\n\r\nAug 02 19:16:16 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0802 19:16:16.423823    1513 kubelet.go:2745] Error updating node status, will retry: nodes \"10.63.10.62\" cannot \r\nAug 02 19:18:52 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0802 19:18:52.954497    1513 kubelet.go:2745] Error updating node status, will retry: client: etcd cluster is una\r\nAug 02 19:19:17 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0802 19:19:17.986552    1513 kubelet.go:2745] Error updating node status, will retry: nodes \"10.63.10.62\" cannot \r\nAug 02 19:28:17 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0802 19:28:17.386025    1513 kubelet.go:2745] Error updating node status, will retry: nodes \"10.63.10.62\" cannot \r\nAug 02 19:29:32 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0802 19:29:32.843566    1513 kubelet.go:2745] Error updating node status, will retry: nodes \"10.63.10.62\" cannot \r\nAug 02 19:31:04 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0802 19:31:04.472829    1513 kubelet.go:2745] Error updating node status, will retry: nodes \"10.63.10.62\" cannot \r\nAug 02 19:32:08 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0802 19:32:08.092741    1513 kubelet.go:2745] Error updating node status, will retry: nodes \"10.63.10.62\" cannot \r\nAug 02 19:38:42 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0802 19:38:42.820935    1513 kubelet.go:2745] Error updating node status, will retry: nodes \"10.63.10.62\" cannot \r\nAug 02 21:46:47 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0802 21:46:47.489363    1513 kubelet.go:2745] Error updating node status, will retry: nodes \"10.63.10.62\" cannot \r\nAug 03 04:17:21 test-deis-stack2-kubernetes-master-cfhhg7dgeb3z.novalocal kubelet-wrapper[1513]: E0803 04:17:21.225077    1513 kubelet.go:2745] Error updating node status, will retry: nodes \"10.63.10.62\" cannot \r\nlines 1-22/22 (END)\r\n```\r\n\r\netcd viewed on master\r\n\r\n```\r\netcdctl  cluster-health\r\nmember some_id is healthy: got healthy result from http://10.63.10.60:2379\r\nmember some_id is healthy: got healthy result from http://10.63.10.62:2379\r\nmember some_id is healthy: got healthy result from http://10.63.10.61:2379\r\nmember some_id  is healthy: got healthy result from http://10.63.10.59:2379\r\ncluster is healthy\r\n```\r\n\r\nminion 1 was dead \r\n\r\nafter start:\r\n\r\n```\r\n systemctl status kubelet\r\n● kubelet.service\r\n   Loaded: loaded (/etc/systemd/system/kubelet.service; disabled; vendor preset: disabled)\r\n   Active: active (running) since Wed 2016-08-03 07:37:20 UTC; 43s ago\r\n  Process: 1487 ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)\r\n Main PID: 1493 (kubelet)\r\n    Tasks: 14\r\n   Memory: 471.9M\r\n      CPU: 6.177s\r\n   CGroup: /system.slice/kubelet.service\r\n           ├─1493 /kubelet --api-servers=https://10.63.10.62 --network-plugin-dir=/etc/kubernetes/cni/net.d --network-plugin=cni --register-node=true --allow-privileged=true --config=/etc/kubernetes/manifests --\r\n           └─1626 journalctl -k -f\r\n\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.461620    1493 manager.go:123] Starting to sync pod status with apiserver\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.462660    1493 kubelet.go:2356] Starting kubelet main sync loop.\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.464344    1493 kubelet.go:2365] skipping pod synchronization - [container runtime is down]\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.597840    1493 factory.go:233] Registering Docker factory\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.598982    1493 factory.go:97] Registering Raw factory\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.623964    1493 kubelet.go:1150] Node 10.63.10.61 was previously registered\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.728240    1493 manager.go:1003] Started watching for new ooms in manager\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.729692    1493 oomparser.go:182] oomparser using systemd\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.730878    1493 manager.go:256] Starting recovery of all containers\r\nAug 03 07:37:51 test-deis-stack2-kubernetes-minion1-awjht3il5pw6.novalocal kubelet-wrapper[1493]: I0803 07:37:51.852285    1493 manager.go:261] Recovery completed\r\n```\r\n\r\nminion 2(was also dead):\r\n\r\n```\r\nsystemctl status kubelet\r\n● kubelet.service\r\n   Loaded: loaded (/etc/systemd/system/kubelet.service; disabled; vendor preset: disabled)\r\n   Active: active (running) since Wed 2016-08-03 07:38:59 UTC; 46s ago\r\n  Process: 1498 ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)\r\n Main PID: 1505 (kubelet)\r\n    Tasks: 13\r\n   Memory: 475.8M\r\n      CPU: 6.621s\r\n   CGroup: /system.slice/kubelet.service\r\n           ├─1505 /kubelet --api-servers=https://10.63.10.62 --network-plugin-dir=/etc/kubernetes/cni/net.d --network-plugin=cni --register-node=true --allow-privileged=true --config=/etc/kubernetes/manifests --\r\n           └─1560 journalctl -k -f\r\n\r\nAug 03 07:39:31 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: I0803 07:39:31.134446    1505 kubelet.go:2356] Starting kubelet main sync loop.\r\nAug 03 07:39:31 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: I0803 07:39:31.134469    1505 kubelet.go:2365] skipping pod synchronization - [container runtime is down]\r\nAug 03 07:39:31 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: I0803 07:39:31.290318    1505 factory.go:233] Registering Docker factory\r\nAug 03 07:39:31 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: I0803 07:39:31.291037    1505 factory.go:97] Registering Raw factory\r\nAug 03 07:39:31 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: I0803 07:39:31.300563    1505 kubelet.go:1150] Node 10.63.10.60 was previously registered\r\nAug 03 07:39:31 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: I0803 07:39:31.432016    1505 manager.go:1003] Started watching for new ooms in manager\r\nAug 03 07:39:31 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: I0803 07:39:31.434208    1505 oomparser.go:182] oomparser using systemd\r\nAug 03 07:39:31 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: I0803 07:39:31.435531    1505 manager.go:256] Starting recovery of all containers\r\nAug 03 07:39:31 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: I0803 07:39:31.529716    1505 manager.go:261] Recovery completed\r\nAug 03 07:39:36 test-deis-stack2-kubernetes-minion2-lapgigoetbus.novalocal kubelet-wrapper[1505]: E0803 07:39:36.208578    1505 kubelet.go:1764] Failed creating a mirror pod for \"kube-proxy-10.63.10.60_kube-syst\r\n```\r\n\r\n\r\nminion 3 (also was dead)\r\n\r\n```\r\n\r\nsystemctl status kubelet\r\n● kubelet.service\r\n   Loaded: loaded (/etc/systemd/system/kubelet.service; disabled; vendor preset: disabled)\r\n   Active: active (running) since Wed 2016-08-03 07:41:16 UTC; 14s ago\r\n  Process: 1442 ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)\r\n Main PID: 1447 (rkt)\r\n    Tasks: 7\r\n   Memory: 318.6M\r\n      CPU: 2.845s\r\n   CGroup: /system.slice/kubelet.service\r\n           └─1447 /usr/bin/rkt run --volume etc-kubernetes,kind=host,source=/etc/kubernetes --volume etc-ssl-certs,kind=host,source=/usr/share/ca-certificates --volume var-lib-docker,kind=host,source=/var/lib/do\r\n\r\nAug 03 07:41:16 test-deis-stack2-kubernetes-minion3-vejtj5xchorz.novalocal systemd[1]: Starting kubelet.service...\r\nAug 03 07:41:16 test-deis-stack2-kubernetes-minion3-vejtj5xchorz.novalocal systemd[1]: Started kubelet.service.\r\nAug 03 07:41:16 test-deis-stack2-kubernetes-minion3-vejtj5xchorz.novalocal kubelet-wrapper[1447]: image: using image from file /usr/lib64/rkt/stage1-images/stage1-fly.aci\r\nAug 03 07:41:19 test-deis-stack2-kubernetes-minion3-vejtj5xchorz.novalocal kubelet-wrapper[1447]: image: using image from local store for image name quay.io/coreos/hyperkube:v1.2.4_coreos.cni.1\r\n```\r\n\r\nbut ` kubectl get nodes` all the time showed status READY :/\r\n\r\n\r\nafter those restarts \r\n```\r\nkubectl get events\r\nFIRSTSEEN   LASTSEEN   COUNT     NAME          KIND      SUBOBJECT   TYPE      REASON                  SOURCE                  MESSAGE\r\n1m          1m         1         10.63.10.59   Node                  Normal    Starting                {kubelet 10.63.10.59}   Starting kubelet.\r\n1m          1m         1         10.63.10.59   Node                  Normal    NodeHasSufficientDisk   {kubelet 10.63.10.59}   Node 10.63.10.59 status is now: NodeHasSufficientDisk\r\n1m          1m         1         10.63.10.59   Node                  Warning   Rebooted                {kubelet 10.63.10.59}   Node 10.63.10.59 has been rebooted, boot id: a3aa32de-1bef-46f6-9d73-1cc49e4d36cd\r\n3m          3m         1         10.63.10.60   Node                  Normal    Starting                {kubelet 10.63.10.60}   Starting kubelet.\r\n3m          3m         1         10.63.10.60   Node                  Normal    NodeHasSufficientDisk   {kubelet 10.63.10.60}   Node 10.63.10.60 status is now: NodeHasSufficientDisk\r\n3m          3m         1         10.63.10.60   Node                  Warning   Rebooted                {kubelet 10.63.10.60}   Node 10.63.10.60 has been rebooted, boot id: 55755b8f-9cc1-4865-90b7-eeb770c68a24\r\n3m          3m         1         10.63.10.60   Node                  Normal    NodeNotReady            {kubelet 10.63.10.60}   Node 10.63.10.60 status is now: NodeNotReady\r\n3m          3m         1         10.63.10.60   Node                  Normal    NodeReady               {kubelet 10.63.10.60}   Node 10.63.10.60 status is now: NodeReady\r\n5m          5m         1         10.63.10.61   Node                  Normal    Starting                {kubelet 10.63.10.61}   Starting kubelet.\r\n5m          5m         1         10.63.10.61   Node                  Normal    NodeHasSufficientDisk   {kubelet 10.63.10.61}   Node 10.63.10.61 status is now: NodeHasSufficientDisk\r\n5m          5m         1         10.63.10.61   Node                  Warning   Rebooted                {kubelet 10.63.10.61}   Node 10.63.10.61 has been rebooted, boot id: d9610c97-ce57-48df-b1d7-992211fd6514\r\n5m          5m         1         10.63.10.61   Node                  Normal    NodeNotReady            {kubelet 10.63.10.61}   Node 10.63.10.61 status is now: NodeNotReady\r\n5m          5m         1         10.63.10.61   Node                  Normal    NodeReady               {kubelet 10.63.10.61}   Node 10.63.10.61 status is now: NodeReady\r\n\r\n\r\n```\r\n\r\nbut status of deis is still unchanged I can see services but no pods"}}
{"actor":{"display_login":"DavidSie"}}