{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675","repository_url":"https://api.github.com/repos/coreos/coreos-kubernetes","labels_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675/labels{/name}","comments_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675/comments","events_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675/events","html_url":"https://github.com/coreos/coreos-kubernetes/issues/675","id":177461377,"number":675,"title":"kube-aws: decrypt-tls-assets.service failed in a controller node","user":{"login":"mumoshu","id":22009,"avatar_url":"https://avatars.githubusercontent.com/u/22009?v=3","gravatar_id":"","url":"https://api.github.com/users/mumoshu","html_url":"https://github.com/mumoshu","followers_url":"https://api.github.com/users/mumoshu/followers","following_url":"https://api.github.com/users/mumoshu/following{/other_user}","gists_url":"https://api.github.com/users/mumoshu/gists{/gist_id}","starred_url":"https://api.github.com/users/mumoshu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumoshu/subscriptions","organizations_url":"https://api.github.com/users/mumoshu/orgs","repos_url":"https://api.github.com/users/mumoshu/repos","events_url":"https://api.github.com/users/mumoshu/events{/privacy}","received_events_url":"https://api.github.com/users/mumoshu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-09-16T16:00:05Z","updated_at":"2016-11-08T03:36:45Z","closed_at":null,"body":"With the latest master(066d888caa85c00112d891f3c69a8416a59aaee4), it failed and kubelet.service didn't come up.\n\n`systemctl status decrypt-tls-assets` showed:\n\n```\n● decrypt-tls-assets.service - decrypt kubelet tls assets using amazon kms\n   Loaded: loaded (/etc/systemd/system/decrypt-tls-assets.service; enabled; vendor preset: disabled)\n   Active: failed (Result: exit-code) since Fri 2016-09-16 13:02:33 UTC; 3min 34s ago\n  Process: 884 ExecStart=/opt/bin/decrypt-tls-assets (code=exited, status=1/FAILURE)\n Main PID: 884 (code=exited, status=1/FAILURE)\n```\n\n`journalctl -u decrypt-tls-assets` showed:\n\n```\nSep 16 13:02:03 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: Starting decrypt kubelet tls assets using amazon kms...\nSep 16 13:02:04 ip-10-0-0-50.ap-northeast-1.compute.internal sudo[890]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/rkt run --volume=ssl,kind=host,source=/etc/kubernetes/ssl,readOnly=fal\nSep 16 13:02:04 ip-10-0-0-50.ap-northeast-1.compute.internal sudo[890]: pam_unix(sudo:session): session opened for user root by (uid=0)\nSep 16 13:02:04 ip-10-0-0-50.ap-northeast-1.compute.internal decrypt-tls-assets[884]: image: using image from file /usr/lib64/rkt/stage1-images/stage1-coreos.aci\nSep 16 13:02:18 ip-10-0-0-50.ap-northeast-1.compute.internal decrypt-tls-assets[884]: image: searching for app image quay.io/coreos/awscli\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal decrypt-tls-assets[884]: run: discovery failed\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: decrypt-tls-assets.service: Main process exited, code=exited, status=1/FAILURE\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: Failed to start decrypt kubelet tls assets using amazon kms.\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: decrypt-tls-assets.service: Unit entered failed state.\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: decrypt-tls-assets.service: Failed with result 'exit-code'.\n```\n\nA work-around for me was to run\n\n```\nsudo systemctl restart decrypt-tls-assets\nsudo systemctl restart kubelet\n```\n\nand then waiting several minutes until `curl localhost:8080` succeeds.\nI'm not really sure but the fix for #666 caused another issue?\n"},"comment":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/comments/259036777","html_url":"https://github.com/coreos/coreos-kubernetes/issues/675#issuecomment-259036777","issue_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675","id":259036777,"user":{"login":"mumoshu","id":22009,"avatar_url":"https://avatars.githubusercontent.com/u/22009?v=3","gravatar_id":"","url":"https://api.github.com/users/mumoshu","html_url":"https://github.com/mumoshu","followers_url":"https://api.github.com/users/mumoshu/followers","following_url":"https://api.github.com/users/mumoshu/following{/other_user}","gists_url":"https://api.github.com/users/mumoshu/gists{/gist_id}","starred_url":"https://api.github.com/users/mumoshu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumoshu/subscriptions","organizations_url":"https://api.github.com/users/mumoshu/orgs","repos_url":"https://api.github.com/users/mumoshu/repos","events_url":"https://api.github.com/users/mumoshu/events{/privacy}","received_events_url":"https://api.github.com/users/mumoshu/received_events","type":"User","site_admin":false},"created_at":"2016-11-08T03:36:45Z","updated_at":"2016-11-08T03:36:45Z","body":"@cmcconnell1 hi, thanks for your feedback here!\r\n\r\nDoes restarting `flanneld.service` and then `kubelet.service` work for you?\r\n\r\nIf you'd been already using the external etcd, `flanneld.service` might have failed due to e.g. ACL forbidding communication between etcd and workers/controllers, slow startup of etcd, etc.\r\n\r\nIf restarting `flanneld.service` and then `kubelet.service` does work for you, it may be the latter, which is already addressed in https://github.com/coreos/kube-aws/commit/75769392020c6a7278fb1e2dac98165da7b28015 (btw, we've moved our repo!)\r\n\r\nIt restarting doesn't work for you, it may be related to ACL or something I'm not aware of for now."}}
{"actor":{"display_login":"mumoshu"}}