{"action":"created","issue":{"url":"https://api.github.com/repos/palantir/atlasdb/issues/982","repository_url":"https://api.github.com/repos/palantir/atlasdb","labels_url":"https://api.github.com/repos/palantir/atlasdb/issues/982/labels{/name}","comments_url":"https://api.github.com/repos/palantir/atlasdb/issues/982/comments","events_url":"https://api.github.com/repos/palantir/atlasdb/issues/982/events","html_url":"https://github.com/palantir/atlasdb/issues/982","id":179180744,"number":982,"title":"Sweep shouldn't OOM when iterating through RowResults","user":{"login":"gsheasby","id":15610492,"avatar_url":"https://avatars.githubusercontent.com/u/15610492?v=3","gravatar_id":"","url":"https://api.github.com/users/gsheasby","html_url":"https://github.com/gsheasby","followers_url":"https://api.github.com/users/gsheasby/followers","following_url":"https://api.github.com/users/gsheasby/following{/other_user}","gists_url":"https://api.github.com/users/gsheasby/gists{/gist_id}","starred_url":"https://api.github.com/users/gsheasby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gsheasby/subscriptions","organizations_url":"https://api.github.com/users/gsheasby/orgs","repos_url":"https://api.github.com/users/gsheasby/repos","events_url":"https://api.github.com/users/gsheasby/events{/privacy}","received_events_url":"https://api.github.com/users/gsheasby/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/palantir/atlasdb/labels/priority:%20P1","name":"priority: P1","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-09-26T09:34:24Z","updated_at":"2016-10-13T17:30:05Z","closed_at":null,"body":"From internal QA ticket 96341.\r\n\r\n@berler said:\r\nIn other words, this OOM has nothing to do with paging on the KVS layer. It is a problem in the Sweep layer which takes an iterator of RowResults and turns that into an ImmutableList of RowResults, and then for each row loads all the Cells for that row. The problem code is this:\r\n\r\nhttps://github.com/palantir/atlasdb/blob/0.15.1/atlasdb-impl-shared/src/main/java/com/palantir/atlasdb/sweep/SweepTaskRunnerImpl.java#L144-L148\r\n\r\nBatching at the Cell level needs to be implemented in Sweep (since even if the KVS batched loading the Cells, it stuffs all the Cells into a map before actually doing the sweep), currently it only batches at the RowResult level."},"comment":{"url":"https://api.github.com/repos/palantir/atlasdb/issues/comments/253581555","html_url":"https://github.com/palantir/atlasdb/issues/982#issuecomment-253581555","issue_url":"https://api.github.com/repos/palantir/atlasdb/issues/982","id":253581555,"user":{"login":"tkleingarn","id":19295396,"avatar_url":"https://avatars.githubusercontent.com/u/19295396?v=3","gravatar_id":"","url":"https://api.github.com/users/tkleingarn","html_url":"https://github.com/tkleingarn","followers_url":"https://api.github.com/users/tkleingarn/followers","following_url":"https://api.github.com/users/tkleingarn/following{/other_user}","gists_url":"https://api.github.com/users/tkleingarn/gists{/gist_id}","starred_url":"https://api.github.com/users/tkleingarn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tkleingarn/subscriptions","organizations_url":"https://api.github.com/users/tkleingarn/orgs","repos_url":"https://api.github.com/users/tkleingarn/repos","events_url":"https://api.github.com/users/tkleingarn/events{/privacy}","received_events_url":"https://api.github.com/users/tkleingarn/received_events","type":"User","site_admin":false},"created_at":"2016-10-13T17:30:05Z","updated_at":"2016-10-13T17:30:05Z","body":"Hey @gbrova Robert mentioned that you were starting work on this.  Let us know if you'd like any advice, tagging @berler for SA.  This is high priority on our end, as it could result in an OOM for anyone with a wide row."}}
{"actor":{"display_login":"tkleingarn"}}