{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69141","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69141/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69141/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69141/events","html_url":"https://github.com/kubernetes/kubernetes/pull/69141","id":364374282,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE4NTc2MTEy","number":69141,"title":"feat: refresh ecr token every 6 hours","user":{"login":"knight42","id":4237254,"node_id":"MDQ6VXNlcjQyMzcyNTQ=","avatar_url":"https://avatars2.githubusercontent.com/u/4237254?v=4","gravatar_id":"","url":"https://api.github.com/users/knight42","html_url":"https://github.com/knight42","followers_url":"https://api.github.com/users/knight42/followers","following_url":"https://api.github.com/users/knight42/following{/other_user}","gists_url":"https://api.github.com/users/knight42/gists{/gist_id}","starred_url":"https://api.github.com/users/knight42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/knight42/subscriptions","organizations_url":"https://api.github.com/users/knight42/orgs","repos_url":"https://api.github.com/users/knight42/repos","events_url":"https://api.github.com/users/knight42/events{/privacy}","received_events_url":"https://api.github.com/users/knight42/received_events","type":"User","site_admin":false},"labels":[{"id":477397086,"node_id":"MDU6TGFiZWw0NzczOTcwODY=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/cncf-cla:%20yes","name":"cncf-cla: yes","color":"bfe5bf","default":false},{"id":1047217136,"node_id":"MDU6TGFiZWwxMDQ3MjE3MTM2","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/needs-kind","name":"needs-kind","color":"ededed","default":false},{"id":573014062,"node_id":"MDU6TGFiZWw1NzMwMTQwNjI=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/needs-ok-to-test","name":"needs-ok-to-test","color":"b60205","default":false},{"id":617149945,"node_id":"MDU6TGFiZWw2MTcxNDk5NDU=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/needs-sig","name":"needs-sig","color":"ededed","default":false},{"id":200149833,"node_id":"MDU6TGFiZWwyMDAxNDk4MzM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/release-note","name":"release-note","color":"c2e0c6","default":false},{"id":253450793,"node_id":"MDU6TGFiZWwyNTM0NTA3OTM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/size/XS","name":"size/XS","color":"009900","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-09-27T09:05:52Z","updated_at":"2018-09-27T09:18:34Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/69141","html_url":"https://github.com/kubernetes/kubernetes/pull/69141","diff_url":"https://github.com/kubernetes/kubernetes/pull/69141.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/69141.patch"},"body":"<!--  Thanks for sending a pull request!  Here are some tips for you:\r\n1. If this is your first time, read our contributor guidelines https://git.k8s.io/community/contributors/guide#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide\r\n2. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews\r\n3. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md\r\n4. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests\r\n-->\r\n\r\n**What this PR does / why we need it**:\r\n\r\nWe pull images from AWS ECR in our k8s cluster, but we occasionally encounter a problem that the auth token got expired earlier than 12 hours as the following log shows.\r\n\r\nkubelet log:\r\n```\r\nAug 22 15:31:12 ip-172-31-72-199 kubelet[1018]: I0822 15:31:12.600432    1018 provider.go:119] Refreshing cache for provider: *credentials.ecrProvider\r\n\r\nAug 22 15:31:12 ip-172-31-72-199 kubelet[1018]: I0822 15:31:12.895399    1018 kube_docker_client.go:348] Stop pulling image \"<id>.dkr.ecr.cn-north-1.amazonaws.com.cn/topic-service:master-a6cd66e4\": \"Status: Image is up to date for <id>.dkr.ecr.cn-north-1.amazonaws.com.cn/topic-service:master-a6cd66e4\"\r\n\r\nAug 23 03:09:05 ip-172-31-72-199 kubelet[1018]: E0823 03:09:05.926097    1018 remote_image.go:108] PullImage \"<id>.dkr.ecr.cn-north-1.amazonaws.com.cn/topic-service:master-a6cd66e4\" from image service failed: rpc error: code = Unknown desc = Error response from daemon: repository <id>.dkr.ecr.cn-north-1.amazonaws.com.cn/topic-service not found: does not exist or no pull access\r\n```\r\n\r\ndockerd log:\r\n```\r\nAug 23 03:09:05 ip-172-31-72-199 dockerd[1251]: time=\"2018-08-23T03:09:05.925621539Z\" level=error msg=\"Not continuing with pull after error: denied: The security token included in the request is invalid\"\r\n\r\nAug 23 03:09:05 ip-172-31-72-199 dockerd[1251]: time=\"2018-08-23T03:09:05.925694770Z\" level=error msg=\"Handler for POST /v1.27/images/create returned error: repository <id>.dkr.ecr.cn-north-1.amazonaws.com.cn/topic-service not found: does not exist or no pull access\"\r\n```\r\n\r\nAs reflected in kubelet logs, the very image `<id>.dkr.ecr.cn-north-1.amazonaws.com.cn/topic-service:master-a6cd66e4` does exist and can be pulled successfully after the ecr token got refreshed at `Aug 22 15:31:12`. \r\n\r\nIdeally, that token would only be expired after `Aug 23 03:31:12`, but it seems that the token became invalid at `Aug 23 03:09:05`, about 20 minutes earlier than expected.\r\n\r\nTherefore, I think we could refresh the ECR token much more earlier to workaround this problem. In addition, the AWS official [amazon-ecr-credential-helper](https://github.com/awslabs/amazon-ecr-credential-helper/blob/master/ecr-login/cache/credentials.go#L32) also refreshes the credentials every 6 hours.\r\n\r\n**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:\r\nFixes #\r\n\r\n**Special notes for your reviewer**:\r\n\r\n**Release note**:\r\n<!--  Write your release note:\r\n1. Enter your extended release note in the below block. If the PR requires additional action from users switching to the new release, include the string \"action required\".\r\n2. If no release note is required, just write \"NONE\".\r\n-->\r\n```release-note\r\nRefresh ECR token every 6 hours.\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/425020443","html_url":"https://github.com/kubernetes/kubernetes/pull/69141#issuecomment-425020443","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69141","id":425020443,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTAyMDQ0Mw==","user":{"login":"knight42","id":4237254,"node_id":"MDQ6VXNlcjQyMzcyNTQ=","avatar_url":"https://avatars2.githubusercontent.com/u/4237254?v=4","gravatar_id":"","url":"https://api.github.com/users/knight42","html_url":"https://github.com/knight42","followers_url":"https://api.github.com/users/knight42/followers","following_url":"https://api.github.com/users/knight42/following{/other_user}","gists_url":"https://api.github.com/users/knight42/gists{/gist_id}","starred_url":"https://api.github.com/users/knight42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/knight42/subscriptions","organizations_url":"https://api.github.com/users/knight42/orgs","repos_url":"https://api.github.com/users/knight42/repos","events_url":"https://api.github.com/users/knight42/events{/privacy}","received_events_url":"https://api.github.com/users/knight42/received_events","type":"User","site_admin":false},"created_at":"2018-09-27T09:18:34Z","updated_at":"2018-09-27T09:18:34Z","author_association":"NONE","body":"/sig aws"}}
{"actor":{"display_login":"knight42"}}