{"action":"created","issue":{"url":"https://api.github.com/repos/broadinstitute/clio/issues/212","repository_url":"https://api.github.com/repos/broadinstitute/clio","labels_url":"https://api.github.com/repos/broadinstitute/clio/issues/212/labels{/name}","comments_url":"https://api.github.com/repos/broadinstitute/clio/issues/212/comments","events_url":"https://api.github.com/repos/broadinstitute/clio/issues/212/events","html_url":"https://github.com/broadinstitute/clio/pull/212","id":296548278,"number":212,"title":"DSDEGP-2068 Collect ES updates into batches to relieve recovery backpressure.","user":{"login":"danxmoran","id":2755881,"avatar_url":"https://avatars2.githubusercontent.com/u/2755881?v=4","gravatar_id":"","url":"https://api.github.com/users/danxmoran","html_url":"https://github.com/danxmoran","followers_url":"https://api.github.com/users/danxmoran/followers","following_url":"https://api.github.com/users/danxmoran/following{/other_user}","gists_url":"https://api.github.com/users/danxmoran/gists{/gist_id}","starred_url":"https://api.github.com/users/danxmoran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danxmoran/subscriptions","organizations_url":"https://api.github.com/users/danxmoran/orgs","repos_url":"https://api.github.com/users/danxmoran/repos","events_url":"https://api.github.com/users/danxmoran/events{/privacy}","received_events_url":"https://api.github.com/users/danxmoran/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-02-12T22:32:02Z","updated_at":"2018-02-13T14:22:25Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/broadinstitute/clio/pulls/212","html_url":"https://github.com/broadinstitute/clio/pull/212","diff_url":"https://github.com/broadinstitute/clio/pull/212.diff","patch_url":"https://github.com/broadinstitute/clio/pull/212.patch"},"body":"### Description\r\n\r\n@jacarey noticed from repeated profiling that the `recovery-parallelism` setting had steeply diminishing returns, because the whole stream was getting backpressured waiting on the sequential\r\nupdates to Elasticsearch. This PR tweaks the end stage of the stream to collect updates into batches, which are sent as bulk updates to ES.\r\n\r\nThe change halves the runtime of the `RecoveryIntegrationSpec` when I run it locally 🎉\r\n\r\n----\r\n\r\n### Checklist (never delete this)\r\n\r\nNever delete this, it is our record that procedure was followed. If you find that for whatever reason one of the checklist points doesn't apply to your PR, you can leave it unchecked but please add an explanation below.\r\n\r\n- [ ] **Submitter**: Include the JIRA issue number in the PR description\r\n- [ ] **Submitter**: Check documentation and code comments. Add explanatory PR comments if helpful.\r\n- [ ] **Submitter**: Ensure that you have added or updated tests\r\n- [ ] **Submitter**: If you're adding/switching libraries or otherwise changing the design, update the [design documentation](https://broadinstitute.atlassian.net/wiki/pages/viewpage.action?pageId=114531509).\r\n- [ ] **Submitter**: JIRA ticket checks:\r\n  * Acceptance criteria exists and is met\r\n  * Note any changes to implementation from the description\r\n  * Add notes on what you've tested\r\n* Review cycle:\r\n  * Team may comment on PR at will\r\n  * **Reviewer assigns to submitter** for feedback fixes\r\n  * Submitter rebases to develop again if necessary\r\n  * Submitter makes further commits. DO NOT SQUASH\r\n  * Submitter updates documentation as needed\r\n  * Submitter **reassigns to reviewer** for further feedback\r\n- [ ] **Submitter**: Verify all tests go green, including CI tests\r\n- [ ] **Submitter**: Squash commits and merge to develop\r\n- [ ] **Submitter**: Delete branch after merge\r\n- [ ] **Submitter**: **Test this change works on dev environment after deployment**. YOU own getting it fixed if dev isn't working for ANY reason!\r\n- [ ] **Submitter**: Move the JIRA issue to QA once this checklist is completed\r\n"},"comment":{"url":"https://api.github.com/repos/broadinstitute/clio/issues/comments/365280755","html_url":"https://github.com/broadinstitute/clio/pull/212#issuecomment-365280755","issue_url":"https://api.github.com/repos/broadinstitute/clio/issues/212","id":365280755,"user":{"login":"danxmoran","id":2755881,"avatar_url":"https://avatars2.githubusercontent.com/u/2755881?v=4","gravatar_id":"","url":"https://api.github.com/users/danxmoran","html_url":"https://github.com/danxmoran","followers_url":"https://api.github.com/users/danxmoran/followers","following_url":"https://api.github.com/users/danxmoran/following{/other_user}","gists_url":"https://api.github.com/users/danxmoran/gists{/gist_id}","starred_url":"https://api.github.com/users/danxmoran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danxmoran/subscriptions","organizations_url":"https://api.github.com/users/danxmoran/orgs","repos_url":"https://api.github.com/users/danxmoran/repos","events_url":"https://api.github.com/users/danxmoran/events{/privacy}","received_events_url":"https://api.github.com/users/danxmoran/received_events","type":"User","site_admin":false},"created_at":"2018-02-13T14:22:25Z","updated_at":"2018-02-13T14:22:25Z","author_association":"CONTRIBUTOR","body":"I tried to beef up the recovery integration test (right now it only loads & recovers 50 of each document...), and hit weirdness where some records fail to be updated. Looking into it now."}}
{"actor":{"display_login":"danxmoran"}}