{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/pai/issues/1166","repository_url":"https://api.github.com/repos/Microsoft/pai","labels_url":"https://api.github.com/repos/Microsoft/pai/issues/1166/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/pai/issues/1166/comments","events_url":"https://api.github.com/repos/Microsoft/pai/issues/1166/events","html_url":"https://github.com/Microsoft/pai/issues/1166","id":354311782,"node_id":"MDU6SXNzdWUzNTQzMTE3ODI=","number":1166,"title":"`paictl.py service delele` hangs on `Unable to retrieve pull secret`","user":{"login":"mzmssg","id":11887940,"node_id":"MDQ6VXNlcjExODg3OTQw","avatar_url":"https://avatars3.githubusercontent.com/u/11887940?v=4","gravatar_id":"","url":"https://api.github.com/users/mzmssg","html_url":"https://github.com/mzmssg","followers_url":"https://api.github.com/users/mzmssg/followers","following_url":"https://api.github.com/users/mzmssg/following{/other_user}","gists_url":"https://api.github.com/users/mzmssg/gists{/gist_id}","starred_url":"https://api.github.com/users/mzmssg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mzmssg/subscriptions","organizations_url":"https://api.github.com/users/mzmssg/orgs","repos_url":"https://api.github.com/users/mzmssg/repos","events_url":"https://api.github.com/users/mzmssg/events{/privacy}","received_events_url":"https://api.github.com/users/mzmssg/received_events","type":"User","site_admin":false},"labels":[{"id":702147111,"node_id":"MDU6TGFiZWw3MDIxNDcxMTE=","url":"https://api.github.com/repos/Microsoft/pai/labels/won't%20fix","name":"won't fix","color":"ffffff","default":false}],"state":"open","locked":false,"assignee":{"login":"hao1939","id":1547343,"node_id":"MDQ6VXNlcjE1NDczNDM=","avatar_url":"https://avatars1.githubusercontent.com/u/1547343?v=4","gravatar_id":"","url":"https://api.github.com/users/hao1939","html_url":"https://github.com/hao1939","followers_url":"https://api.github.com/users/hao1939/followers","following_url":"https://api.github.com/users/hao1939/following{/other_user}","gists_url":"https://api.github.com/users/hao1939/gists{/gist_id}","starred_url":"https://api.github.com/users/hao1939/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hao1939/subscriptions","organizations_url":"https://api.github.com/users/hao1939/orgs","repos_url":"https://api.github.com/users/hao1939/repos","events_url":"https://api.github.com/users/hao1939/events{/privacy}","received_events_url":"https://api.github.com/users/hao1939/received_events","type":"User","site_admin":false},"assignees":[{"login":"hao1939","id":1547343,"node_id":"MDQ6VXNlcjE1NDczNDM=","avatar_url":"https://avatars1.githubusercontent.com/u/1547343?v=4","gravatar_id":"","url":"https://api.github.com/users/hao1939","html_url":"https://github.com/hao1939","followers_url":"https://api.github.com/users/hao1939/followers","following_url":"https://api.github.com/users/hao1939/following{/other_user}","gists_url":"https://api.github.com/users/hao1939/gists{/gist_id}","starred_url":"https://api.github.com/users/hao1939/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hao1939/subscriptions","organizations_url":"https://api.github.com/users/hao1939/orgs","repos_url":"https://api.github.com/users/hao1939/repos","events_url":"https://api.github.com/users/hao1939/events{/privacy}","received_events_url":"https://api.github.com/users/hao1939/received_events","type":"User","site_admin":false},{"login":"ydye","id":8675883,"node_id":"MDQ6VXNlcjg2NzU4ODM=","avatar_url":"https://avatars2.githubusercontent.com/u/8675883?v=4","gravatar_id":"","url":"https://api.github.com/users/ydye","html_url":"https://github.com/ydye","followers_url":"https://api.github.com/users/ydye/followers","following_url":"https://api.github.com/users/ydye/following{/other_user}","gists_url":"https://api.github.com/users/ydye/gists{/gist_id}","starred_url":"https://api.github.com/users/ydye/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ydye/subscriptions","organizations_url":"https://api.github.com/users/ydye/orgs","repos_url":"https://api.github.com/users/ydye/repos","events_url":"https://api.github.com/users/ydye/events{/privacy}","received_events_url":"https://api.github.com/users/ydye/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2018-08-27T13:08:42Z","updated_at":"2018-08-29T09:13:09Z","closed_at":null,"author_association":"MEMBER","body":"`service delete` hangs because service hasn't started actually, but we call `service delete` without checking it.\r\n\r\n**Details:**\r\nNow our deployment have some occasional failure, an typical error is\r\n```\r\n2018-08-27 10:59:16,887 [INFO] - k8sPaiLibrary.maintainlib.kubectl_install : Successfully install kubectl and configure it!\r\n\r\n2018-08-27 10:59:16,888 [INFO] - k8sPaiLibrary.maintainlib.deploy : Create kube-proxy daemon for kuberentes cluster.\r\n\r\nerror: unable to recognize \"kube-proxy.yaml\": no matches for kind \"DaemonSet\" in version \"apps/v1\"\r\n\r\n2018-08-27 10:59:17,194 [ERROR] - k8sPaiLibrary.maintainlib.common : Failed to create kube-proxy\r\n```\r\n\r\nThen deployment failed and PAI service haven't been created. \r\n\r\nJenkins will do clean job after that, in which `paictl.py service delele` invoked.\r\nBut `paictl.py service delele` have a assumption that `imagePullSecrets` create successfully, otherwise it will hang due to `image pull` failed.\r\n```\r\nE0827 11:45:27.404029   60205 pod_workers.go:186] Error syncing pod 5d56b9ae-a9e9-11e8-8499-000d3a10c86f (\"delete-batch-job-zookeeper-q5ct7_default(5d56b9ae-a9e9-11e8-8499-000d3a10c86f)\"), skipping: failed to \"StartContainer\" for \"cleaning-one-shot\" with ImagePullBackOff: \"Back-off pulling image \\\"openpai.azurecr.io/paiclusterint/cleaning-image:zimiao-jenkins_fix-8a13ebe-10\\\"\"\r\nW0827 11:45:40.402114   60205 kubelet_pods.go:875] Unable to retrieve pull secret default/pai-secret for default/delete-batch-job-zookeeper-q5ct7 due to secrets \"pai-secret\" not found.  The image pull may not succeed.\r\n\r\n```\r\n\r\n\r\n\r\n\r\n \r\n"},"comment":{"url":"https://api.github.com/repos/Microsoft/pai/issues/comments/416883283","html_url":"https://github.com/Microsoft/pai/issues/1166#issuecomment-416883283","issue_url":"https://api.github.com/repos/Microsoft/pai/issues/1166","id":416883283,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjg4MzI4Mw==","user":{"login":"ydye","id":8675883,"node_id":"MDQ6VXNlcjg2NzU4ODM=","avatar_url":"https://avatars2.githubusercontent.com/u/8675883?v=4","gravatar_id":"","url":"https://api.github.com/users/ydye","html_url":"https://github.com/ydye","followers_url":"https://api.github.com/users/ydye/followers","following_url":"https://api.github.com/users/ydye/following{/other_user}","gists_url":"https://api.github.com/users/ydye/gists{/gist_id}","starred_url":"https://api.github.com/users/ydye/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ydye/subscriptions","organizations_url":"https://api.github.com/users/ydye/orgs","repos_url":"https://api.github.com/users/ydye/repos","events_url":"https://api.github.com/users/ydye/events{/privacy}","received_events_url":"https://api.github.com/users/ydye/received_events","type":"User","site_admin":false},"created_at":"2018-08-29T09:13:09Z","updated_at":"2018-08-29T09:13:09Z","author_association":"CONTRIBUTOR","body":"@fanyangCS Yes, I will close it, if no more comments after 24 hours. "}}
{"actor":{"display_login":"ydye"}}