{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-cpp/issues/1454","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-cpp","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-cpp/issues/1454/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-cpp/issues/1454/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-cpp/issues/1454/events","html_url":"https://github.com/GoogleCloudPlatform/google-cloud-cpp/issues/1454","id":381225118,"node_id":"MDU6SXNzdWUzODEyMjUxMTg=","number":1454,"title":"Design how to handle multipage responses.","user":{"login":"dopiera","id":406969,"node_id":"MDQ6VXNlcjQwNjk2OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/406969?v=4","gravatar_id":"","url":"https://api.github.com/users/dopiera","html_url":"https://github.com/dopiera","followers_url":"https://api.github.com/users/dopiera/followers","following_url":"https://api.github.com/users/dopiera/following{/other_user}","gists_url":"https://api.github.com/users/dopiera/gists{/gist_id}","starred_url":"https://api.github.com/users/dopiera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dopiera/subscriptions","organizations_url":"https://api.github.com/users/dopiera/orgs","repos_url":"https://api.github.com/users/dopiera/repos","events_url":"https://api.github.com/users/dopiera/events{/privacy}","received_events_url":"https://api.github.com/users/dopiera/received_events","type":"User","site_admin":false},"labels":[{"id":770782350,"node_id":"MDU6TGFiZWw3NzA3ODIzNTA=","url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-cpp/labels/api:%20bigtable","name":"api: bigtable","color":"d4c5f9","default":false},{"id":952110209,"node_id":"MDU6TGFiZWw5NTIxMTAyMDk=","url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-cpp/labels/type:%20process","name":"type: process","color":"c5def5","default":false}],"state":"open","locked":false,"assignee":{"login":"dopiera","id":406969,"node_id":"MDQ6VXNlcjQwNjk2OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/406969?v=4","gravatar_id":"","url":"https://api.github.com/users/dopiera","html_url":"https://github.com/dopiera","followers_url":"https://api.github.com/users/dopiera/followers","following_url":"https://api.github.com/users/dopiera/following{/other_user}","gists_url":"https://api.github.com/users/dopiera/gists{/gist_id}","starred_url":"https://api.github.com/users/dopiera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dopiera/subscriptions","organizations_url":"https://api.github.com/users/dopiera/orgs","repos_url":"https://api.github.com/users/dopiera/repos","events_url":"https://api.github.com/users/dopiera/events{/privacy}","received_events_url":"https://api.github.com/users/dopiera/received_events","type":"User","site_admin":false},"assignees":[{"login":"dopiera","id":406969,"node_id":"MDQ6VXNlcjQwNjk2OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/406969?v=4","gravatar_id":"","url":"https://api.github.com/users/dopiera","html_url":"https://github.com/dopiera","followers_url":"https://api.github.com/users/dopiera/followers","following_url":"https://api.github.com/users/dopiera/following{/other_user}","gists_url":"https://api.github.com/users/dopiera/gists{/gist_id}","starred_url":"https://api.github.com/users/dopiera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dopiera/subscriptions","organizations_url":"https://api.github.com/users/dopiera/orgs","repos_url":"https://api.github.com/users/dopiera/repos","events_url":"https://api.github.com/users/dopiera/events{/privacy}","received_events_url":"https://api.github.com/users/dopiera/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-11-15T15:55:37Z","updated_at":"2018-11-16T15:12:42Z","closed_at":null,"author_association":"COLLABORATOR","body":"Requests like `ListInstances` stream responses page by page, utilizing `next_page_token` to indicate if it's an end of stream or not.\r\n\r\nIt is not immediately obvious how to handle retries in such API calls.\r\n\r\nMy idea is to:\r\n * go over pages without any sleeps between them\r\n * accumulate result (we should make it future proof to allow for streaming)\r\n * if getting a page fails, keep retrying until it succeeds; if all retries on a page fail, fail the whole request\r\n * even if getting a page had to be retried (but eventually succeeded), move on to the next page without any delay\r\n\r\nThis is how I'd implement it:\r\n * Create a templated `GetPage` operation, which would meet `MeetsAsyncOperationRequirements` \r\n   requirements\r\n * Create a templated `PageIterator` class, which would keep going over pages without any delay, \r\n   accumulating the results and not doing any retries (an instantiation of `AsyncPollOp` with a \r\n  `PollingPolicy` which fails on first error and doesn't sleep between attempts)\r\n * The operation passed to `PageIterator` would be an `AsyncRetryOp<GetPage>`\r\n * Each operation which `PageIterator` starts, shares the same `RetryPolicy`, but gets a new `RPCBackoffPolicy`\r\n\r\nThis is going to allow for easily handling the operations like `ListInstances` while ignoring `failed_locations`. Handling `failed_locations` can be built on top of this."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-cpp/issues/comments/439423598","html_url":"https://github.com/GoogleCloudPlatform/google-cloud-cpp/issues/1454#issuecomment-439423598","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-cpp/issues/1454","id":439423598,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTQyMzU5OA==","user":{"login":"coryan","id":6241635,"node_id":"MDQ6VXNlcjYyNDE2MzU=","avatar_url":"https://avatars3.githubusercontent.com/u/6241635?v=4","gravatar_id":"","url":"https://api.github.com/users/coryan","html_url":"https://github.com/coryan","followers_url":"https://api.github.com/users/coryan/followers","following_url":"https://api.github.com/users/coryan/following{/other_user}","gists_url":"https://api.github.com/users/coryan/gists{/gist_id}","starred_url":"https://api.github.com/users/coryan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coryan/subscriptions","organizations_url":"https://api.github.com/users/coryan/orgs","repos_url":"https://api.github.com/users/coryan/repos","events_url":"https://api.github.com/users/coryan/events{/privacy}","received_events_url":"https://api.github.com/users/coryan/received_events","type":"User","site_admin":false},"created_at":"2018-11-16T15:12:42Z","updated_at":"2018-11-16T15:12:42Z","author_association":"COLLABORATOR","body":"> I don't understand why you're saying it's an alternative.\r\n\r\nI was responding to this comment:\r\n\r\n> >  Handling `failed_locations` can be built on top of this.\r\n\r\nIf we just reset the backoff policy we may not create the necessary infrastructure, but we can deal with the `failed_locations` by building on top of an API that does not.\r\n\r\n> > Do not worry about iteration for these things. The number of instances for a project is expected to fit in  memory.\r\n\r\n> I'm not sure I understand. Maybe I picked a confusing name (PageIterator). Perhaps AllPagesFetcher would be more descriptive. I wanted it to be responsible for kicking off the retried fetching of operations and I wanted to implement it by abusing AsyncPollOp.\r\n\r\nMy bad, you are right.\r\n"}}
{"actor":{"display_login":"coryan"}}