{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675","repository_url":"https://api.github.com/repos/coreos/coreos-kubernetes","labels_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675/labels{/name}","comments_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675/comments","events_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675/events","html_url":"https://github.com/coreos/coreos-kubernetes/issues/675","id":177461377,"number":675,"title":"kube-aws: decrypt-tls-assets.service failed in a controller node","user":{"login":"mumoshu","id":22009,"avatar_url":"https://avatars.githubusercontent.com/u/22009?v=3","gravatar_id":"","url":"https://api.github.com/users/mumoshu","html_url":"https://github.com/mumoshu","followers_url":"https://api.github.com/users/mumoshu/followers","following_url":"https://api.github.com/users/mumoshu/following{/other_user}","gists_url":"https://api.github.com/users/mumoshu/gists{/gist_id}","starred_url":"https://api.github.com/users/mumoshu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumoshu/subscriptions","organizations_url":"https://api.github.com/users/mumoshu/orgs","repos_url":"https://api.github.com/users/mumoshu/repos","events_url":"https://api.github.com/users/mumoshu/events{/privacy}","received_events_url":"https://api.github.com/users/mumoshu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-09-16T16:00:05Z","updated_at":"2016-09-21T01:40:36Z","closed_at":null,"body":"With the latest master(066d888caa85c00112d891f3c69a8416a59aaee4), it failed and kubelet.service didn't come up.\r\n\r\n`systemctl status decrypt-tls-assets` showed:\r\n\r\n```\r\n● decrypt-tls-assets.service - decrypt kubelet tls assets using amazon kms\r\n   Loaded: loaded (/etc/systemd/system/decrypt-tls-assets.service; enabled; vendor preset: disabled)\r\n   Active: failed (Result: exit-code) since Fri 2016-09-16 13:02:33 UTC; 3min 34s ago\r\n  Process: 884 ExecStart=/opt/bin/decrypt-tls-assets (code=exited, status=1/FAILURE)\r\n Main PID: 884 (code=exited, status=1/FAILURE)\r\n```\r\n\r\n`journalctl -u decrypt-tls-assets` showed:\r\n\r\n```\r\nSep 16 13:02:03 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: Starting decrypt kubelet tls assets using amazon kms...\r\nSep 16 13:02:04 ip-10-0-0-50.ap-northeast-1.compute.internal sudo[890]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/rkt run --volume=ssl,kind=host,source=/etc/kubernetes/ssl,readOnly=fal\r\nSep 16 13:02:04 ip-10-0-0-50.ap-northeast-1.compute.internal sudo[890]: pam_unix(sudo:session): session opened for user root by (uid=0)\r\nSep 16 13:02:04 ip-10-0-0-50.ap-northeast-1.compute.internal decrypt-tls-assets[884]: image: using image from file /usr/lib64/rkt/stage1-images/stage1-coreos.aci\r\nSep 16 13:02:18 ip-10-0-0-50.ap-northeast-1.compute.internal decrypt-tls-assets[884]: image: searching for app image quay.io/coreos/awscli\r\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal decrypt-tls-assets[884]: run: discovery failed\r\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: decrypt-tls-assets.service: Main process exited, code=exited, status=1/FAILURE\r\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: Failed to start decrypt kubelet tls assets using amazon kms.\r\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: decrypt-tls-assets.service: Unit entered failed state.\r\nSep 16 13:02:33 ip-10-0-0-50.ap-northeast-1.compute.internal systemd[1]: decrypt-tls-assets.service: Failed with result 'exit-code'.\r\n```\r\n\r\nA work-around for me was to run\r\n\r\n```\r\nsudo systemctl restart decrypt-tls-assets\r\nsudo systemctl restart kubelet\r\n```\r\n\r\nand then waiting several minutes until `curl localhost:8080` succeeds.\r\nI'm not really sure but the fix for #666 caused another issue?"},"comment":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/comments/248486925","html_url":"https://github.com/coreos/coreos-kubernetes/issues/675#issuecomment-248486925","issue_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/675","id":248486925,"user":{"login":"mumoshu","id":22009,"avatar_url":"https://avatars.githubusercontent.com/u/22009?v=3","gravatar_id":"","url":"https://api.github.com/users/mumoshu","html_url":"https://github.com/mumoshu","followers_url":"https://api.github.com/users/mumoshu/followers","following_url":"https://api.github.com/users/mumoshu/following{/other_user}","gists_url":"https://api.github.com/users/mumoshu/gists{/gist_id}","starred_url":"https://api.github.com/users/mumoshu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumoshu/subscriptions","organizations_url":"https://api.github.com/users/mumoshu/orgs","repos_url":"https://api.github.com/users/mumoshu/repos","events_url":"https://api.github.com/users/mumoshu/events{/privacy}","received_events_url":"https://api.github.com/users/mumoshu/received_events","type":"User","site_admin":false},"created_at":"2016-09-21T01:40:36Z","updated_at":"2016-09-21T01:40:36Z","body":"@cgag Thanks for your detailed explanation. It was very helpful to me.\r\n\r\n> Say service B depends on service A. If you attempt to start B, it will then try to start A. If A fails, B will be marked as \"failed due to dependency\", and then systemd will never attempt to restart it. Things marked as failing due to dependency are dead for good.\r\n\r\nAck. That's also why I've proposed (1) in https://github.com/coreos/coreos-kubernetes/issues/675#issuecomment-248476062\r\n\r\n> If A has restart logic, systemd will try to restart it, but it won't go back and start B once A has come up.\r\n\r\nThanks. That's what I wasn't sure in the time of writing (1) above.\r\n\r\n> The idea behind the is-active workaround is to force B to fail due to exit-code rather than dependency. Systemd will attempt to restart things that fail due to bad exit codes.\r\n\r\nSounds nice. I've seen [someone in the internet also came up with that](http://unix.stackexchange.com/a/216254).\r\n\r\n> I thought that these restarts would cause it to re-run it's dependencies if they were inactive, but my initial tests seem to show I'm wrong. I need to validate further.\r\n\r\nThanks for sharing the great findings.\r\nSorry for saying something without real experiments on my side, but my assumption on that behavior is that `Once a dependency is failed, it will be permanent. I.e. starting dependant(e.g. kubelet) does trigger initial start of the dependency(e.g. decrypt-tls-assets) but won't try to restart failed dependency`\r\n\r\nSo:\r\n\r\n> We could do what you said and add the retry logic to the shell script, but if possible I'd prefer to keep any restart logic in the service files, which pushes me towards using the ExecStartPre option to directly run decrypt-tls-assets\r\n\r\nI'd also like the ExecStartPre option in that regard if it works.\r\nSorry for the back and forth but, now, your initial comment totally made sense to me. Thanks a lot 👍 \r\n"}}
{"actor":{"display_login":"mumoshu"}}