{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/50102","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/50102/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/50102/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/50102/events","html_url":"https://github.com/kubernetes/kubernetes/pull/50102","id":247794821,"node_id":"MDExOlB1bGxSZXF1ZXN0MTM0MDEwOTI1","number":50102,"title":"Fix broken watches","user":{"login":"tnozicka","id":11869554,"node_id":"MDQ6VXNlcjExODY5NTU0","avatar_url":"https://avatars2.githubusercontent.com/u/11869554?v=4","gravatar_id":"","url":"https://api.github.com/users/tnozicka","html_url":"https://github.com/tnozicka","followers_url":"https://api.github.com/users/tnozicka/followers","following_url":"https://api.github.com/users/tnozicka/following{/other_user}","gists_url":"https://api.github.com/users/tnozicka/gists{/gist_id}","starred_url":"https://api.github.com/users/tnozicka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tnozicka/subscriptions","organizations_url":"https://api.github.com/users/tnozicka/orgs","repos_url":"https://api.github.com/users/tnozicka/repos","events_url":"https://api.github.com/users/tnozicka/events{/privacy}","received_events_url":"https://api.github.com/users/tnozicka/received_events","type":"User","site_admin":false},"labels":[{"id":477397086,"node_id":"MDU6TGFiZWw0NzczOTcwODY=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/cncf-cla:%20yes","name":"cncf-cla: yes","color":"bfe5bf","default":false},{"id":687565804,"node_id":"MDU6TGFiZWw2ODc1NjU4MDQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/do-not-merge/hold","name":"do-not-merge/hold","color":"e11d21","default":false},{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":711852621,"node_id":"MDU6TGFiZWw3MTE4NTI2MjE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/milestone/needs-approval","name":"milestone/needs-approval","color":"ededed","default":false},{"id":114528223,"node_id":"MDU6TGFiZWwxMTQ1MjgyMjM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/important-soon","name":"priority/important-soon","color":"eb6420","default":false},{"id":200149833,"node_id":"MDU6TGFiZWwyMDAxNDk4MzM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/release-note","name":"release-note","color":"c2e0c6","default":false},{"id":173493835,"node_id":"MDU6TGFiZWwxNzM0OTM4MzU=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/api-machinery","name":"sig/api-machinery","color":"d2b48c","default":false},{"id":404091735,"node_id":"MDU6TGFiZWw0MDQwOTE3MzU=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/apps","name":"sig/apps","color":"d2b48c","default":false},{"id":450823910,"node_id":"MDU6TGFiZWw0NTA4MjM5MTA=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/cli","name":"sig/cli","color":"d2b48c","default":false},{"id":173494222,"node_id":"MDU6TGFiZWwxNzM0OTQyMjI=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/cluster-lifecycle","name":"sig/cluster-lifecycle","color":"d2b48c","default":false},{"id":125010198,"node_id":"MDU6TGFiZWwxMjUwMTAxOTg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/scalability","name":"sig/scalability","color":"d2b48c","default":false},{"id":125550211,"node_id":"MDU6TGFiZWwxMjU1NTAyMTE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/scheduling","name":"sig/scheduling","color":"d2b48c","default":false},{"id":253451093,"node_id":"MDU6TGFiZWwyNTM0NTEwOTM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/size/XXL","name":"size/XXL","color":"ee0000","default":false}],"state":"open","locked":false,"assignee":{"login":"mfojtik","id":44136,"node_id":"MDQ6VXNlcjQ0MTM2","avatar_url":"https://avatars2.githubusercontent.com/u/44136?v=4","gravatar_id":"","url":"https://api.github.com/users/mfojtik","html_url":"https://github.com/mfojtik","followers_url":"https://api.github.com/users/mfojtik/followers","following_url":"https://api.github.com/users/mfojtik/following{/other_user}","gists_url":"https://api.github.com/users/mfojtik/gists{/gist_id}","starred_url":"https://api.github.com/users/mfojtik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mfojtik/subscriptions","organizations_url":"https://api.github.com/users/mfojtik/orgs","repos_url":"https://api.github.com/users/mfojtik/repos","events_url":"https://api.github.com/users/mfojtik/events{/privacy}","received_events_url":"https://api.github.com/users/mfojtik/received_events","type":"User","site_admin":false},"assignees":[{"login":"mfojtik","id":44136,"node_id":"MDQ6VXNlcjQ0MTM2","avatar_url":"https://avatars2.githubusercontent.com/u/44136?v=4","gravatar_id":"","url":"https://api.github.com/users/mfojtik","html_url":"https://github.com/mfojtik","followers_url":"https://api.github.com/users/mfojtik/followers","following_url":"https://api.github.com/users/mfojtik/following{/other_user}","gists_url":"https://api.github.com/users/mfojtik/gists{/gist_id}","starred_url":"https://api.github.com/users/mfojtik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mfojtik/subscriptions","organizations_url":"https://api.github.com/users/mfojtik/orgs","repos_url":"https://api.github.com/users/mfojtik/repos","events_url":"https://api.github.com/users/mfojtik/events{/privacy}","received_events_url":"https://api.github.com/users/mfojtik/received_events","type":"User","site_admin":false},{"login":"liggitt","id":980082,"node_id":"MDQ6VXNlcjk4MDA4Mg==","avatar_url":"https://avatars0.githubusercontent.com/u/980082?v=4","gravatar_id":"","url":"https://api.github.com/users/liggitt","html_url":"https://github.com/liggitt","followers_url":"https://api.github.com/users/liggitt/followers","following_url":"https://api.github.com/users/liggitt/following{/other_user}","gists_url":"https://api.github.com/users/liggitt/gists{/gist_id}","starred_url":"https://api.github.com/users/liggitt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liggitt/subscriptions","organizations_url":"https://api.github.com/users/liggitt/orgs","repos_url":"https://api.github.com/users/liggitt/repos","events_url":"https://api.github.com/users/liggitt/events{/privacy}","received_events_url":"https://api.github.com/users/liggitt/received_events","type":"User","site_admin":false},{"login":"wojtek-t","id":10743879,"node_id":"MDQ6VXNlcjEwNzQzODc5","avatar_url":"https://avatars3.githubusercontent.com/u/10743879?v=4","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/39","html_url":"https://github.com/kubernetes/kubernetes/milestone/39","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/39/labels","id":2913712,"node_id":"MDk6TWlsZXN0b25lMjkxMzcxMg==","number":39,"title":"v1.12","description":"","creator":{"login":"thockin","id":5595220,"node_id":"MDQ6VXNlcjU1OTUyMjA=","avatar_url":"https://avatars0.githubusercontent.com/u/5595220?v=4","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"open_issues":39,"closed_issues":27,"state":"open","created_at":"2017-11-15T06:32:51Z","updated_at":"2018-08-02T02:25:26Z","due_on":null,"closed_at":null},"comments":80,"created_at":"2017-08-03T18:27:05Z","updated_at":"2018-08-02T08:04:34Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/50102","html_url":"https://github.com/kubernetes/kubernetes/pull/50102","diff_url":"https://github.com/kubernetes/kubernetes/pull/50102.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/50102.patch"},"body":"The main reason for having this is that `watch.Until` can exit before the timeout is reached. There are several issues about it in Kubernetes repo already (linked below). It is caused by the watcher being closed prematurely. The usual causes are API timeout or ETCD timeout for watch. This is destined to cause a lot of test flakes! And `watch.Until` already had to be removed from some tests. Also I think this was an issue with `kubectl rollout status --timeout=...` so not just tests.\r\n\r\nThe idea behind this PR is to inspect all the events being processed and keeping track of last resourceVersion of its objects. If the watch fails it will restart the watch from the last resourceVersion.\r\nThis is all wrapped in object RetryWatcher implementing watch.Interface so it looks like a regular watcher and the reader won't even notice those restarts on background.\r\n\r\nThis PR is not fixing `watch.Until` itself but only providing you with a watcher wrapper to make it behave properly. The reason why I haven't plumbed it into `watch.Until` itself is that it would require changing it's arguments which is a fairly big change I guess - but still, it's broken now. It would need to accept WatchFunc instead of a watcher. Otherwise the change would be just in using RetryWatcher internally in `watch.until` and fixing the code that calls it. Please let me know your opinion here about fixing that as well! Doing this in `watch.Until` would be probably best; it would fix all the related issues... like https://github.com/kubernetes/kubernetes/issues/40224\r\n\r\nPlease note that this fixes most of the problems but not all. There can be a situation when (between the background restarts) the lastResourceVersion is not in ETCD cache anymore causing it to fail. I guess the only other path forward would be to use Informers but that's kind of heavy and would make the user experience worse.\r\n\r\n### Example Usage\r\n```go\r\ninitialResourceVersion := \"\"\r\nwatch.Until(10*time.Minute, watch.WithRetry(initialResourceVersion, func(rv string) watch.Interface {\r\n\twatcher, err := c.CoreV1().Secrets(secret.Namespace).Watch(metav1.SingleObject(metav1.ObjectMeta{Name: secret.Name, ResourceVersion: rv}))\r\n\tif err != nil {\r\n\t\tt.Fatalf(\"Failed to create watcher on Secret: %v\", err)\r\n\t}\r\n\treturn watcher\r\n}))\r\n```\r\n\r\nRelated issues and PRs about broken watch.Until:\r\n- https://github.com/kubernetes/kubernetes/pull/41112\r\n- https://github.com/kubernetes/kubernetes/pull/47756\r\n- https://github.com/kubernetes/kubernetes/issues/40224\r\n- https://github.com/kubernetes/kubernetes/pull/33942#issuecomment-305817825\r\n- https://github.com/kubernetes/kubernetes/issues/31345\r\n- https://github.com/kubernetes/kubernetes/issues/39879#issuecomment-278059099\r\n- https://github.com/openshift/origin/pull/14936#issuecomment-311957325\r\n\r\n(Feel free to link more.)\r\n\r\nccing few people from previous issues that might be interested:\r\n@smarterclayton @mfojtik @kargakis @soltysh @janetkuo @liggitt @wojtek-t \r\n\r\n**Release note**:\r\n```release-note\r\nFix premature timeouts/failures in commands that wait using WATCH\r\n```\r\n\r\nTODO:\r\n - [x] - go through all the usages of context for Until* in CLI and make sure 0 timeout creates context without timeout"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/409842221","html_url":"https://github.com/kubernetes/kubernetes/pull/50102#issuecomment-409842221","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/50102","id":409842221,"node_id":"MDEyOklzc3VlQ29tbWVudDQwOTg0MjIyMQ==","user":{"login":"k8s-merge-robot","id":13653959,"node_id":"MDQ6VXNlcjEzNjUzOTU5","avatar_url":"https://avatars0.githubusercontent.com/u/13653959?v=4","gravatar_id":"","url":"https://api.github.com/users/k8s-merge-robot","html_url":"https://github.com/k8s-merge-robot","followers_url":"https://api.github.com/users/k8s-merge-robot/followers","following_url":"https://api.github.com/users/k8s-merge-robot/following{/other_user}","gists_url":"https://api.github.com/users/k8s-merge-robot/gists{/gist_id}","starred_url":"https://api.github.com/users/k8s-merge-robot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k8s-merge-robot/subscriptions","organizations_url":"https://api.github.com/users/k8s-merge-robot/orgs","repos_url":"https://api.github.com/users/k8s-merge-robot/repos","events_url":"https://api.github.com/users/k8s-merge-robot/events{/privacy}","received_events_url":"https://api.github.com/users/k8s-merge-robot/received_events","type":"User","site_admin":false},"created_at":"2018-08-02T08:04:34Z","updated_at":"2018-08-02T08:04:34Z","author_association":"COLLABORATOR","body":"[MILESTONENOTIFIER] Milestone Pull Request **Needs Approval**\n\n@liggitt @mfojtik @tnozicka @wojtek-t @kubernetes/sig-api-machinery-misc @kubernetes/sig-apps-misc @kubernetes/sig-cli-misc @kubernetes/sig-cluster-lifecycle-misc @kubernetes/sig-scalability-misc @kubernetes/sig-scheduling-misc\n\n\n**Action required**: This pull request must have the `status/approved-for-milestone` label applied by a SIG maintainer. If the label is not applied within 2 days, the pull request will be moved out of the v1.12 milestone.\n<details>\n<summary>Pull Request Labels</summary>\n\n- `sig/api-machinery` `sig/apps` `sig/cli` `sig/cluster-lifecycle` `sig/scalability` `sig/scheduling`: Pull Request will be escalated to these SIGs if needed.\n- `priority/important-soon`: Escalate to the pull request owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.\n- `kind/bug`: Fixes a bug discovered during the current release.\n</details>\n<details>\n<summary>Help</summary>\n<ul>\n <li><a href=\"https://git.k8s.io/sig-release/ephemera/issues.md\">Additional instructions</a></li>\n <li><a href=\"https://go.k8s.io/bot-commands\">Commands for setting labels</a></li>\n</ul>\n</details>"}}
{"actor":{"display_login":"k8s-merge-robot"}}