{"action":"created","comment":{"url":"https://api.github.com/repos/pravega/pravega/pulls/comments/234733365","pull_request_review_id":176438665,"id":234733365,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzNDczMzM2NQ==","diff_hunk":"@@ -693,17 +814,22 @@ private ByteArraySegment generateMinKey() {\n         Preconditions.checkArgument(pageCollection.getIndexLength() == offset, \"IndexLength mismatch.\");\n         pages.add(new AbstractMap.SimpleImmutableEntry<>(offset, getFooter(lastPage.getOffset(), lastPage.getPage().getLength())));\n \n-        // Also collect the old footer's offset, as it will be replaced by a more recent value.\n+        // Collect the old footer's offset, as it will be replaced by a more recent value.\n         long oldFooterOffset = getFooterOffset(state.length);\n         if (oldFooterOffset >= 0) {\n             oldOffsets.add(oldFooterOffset);\n         }\n \n+        // Collect the offsets of removed pages. These are no longer needed.\n+        pageCollection.collectRemovedPageOffsets(oldOffsets);","path":"common/src/main/java/io/pravega/common/util/btree/BTreeIndex.java","position":324,"original_position":324,"commit_id":"38fcc648491125f38ce8ed348d3b5a5e7d22ab28","original_commit_id":"749d64bed10de405452203de566134faae8ab9c8","user":{"login":"fpj","id":572920,"node_id":"MDQ6VXNlcjU3MjkyMA==","avatar_url":"https://avatars3.githubusercontent.com/u/572920?v=4","gravatar_id":"","url":"https://api.github.com/users/fpj","html_url":"https://github.com/fpj","followers_url":"https://api.github.com/users/fpj/followers","following_url":"https://api.github.com/users/fpj/following{/other_user}","gists_url":"https://api.github.com/users/fpj/gists{/gist_id}","starred_url":"https://api.github.com/users/fpj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fpj/subscriptions","organizations_url":"https://api.github.com/users/fpj/orgs","repos_url":"https://api.github.com/users/fpj/repos","events_url":"https://api.github.com/users/fpj/events{/privacy}","received_events_url":"https://api.github.com/users/fpj/received_events","type":"User","site_admin":false},"body":"Ok.","created_at":"2018-11-19T18:35:55Z","updated_at":"2018-11-19T18:35:55Z","html_url":"https://github.com/pravega/pravega/pull/3090#discussion_r234733365","pull_request_url":"https://api.github.com/repos/pravega/pravega/pulls/3090","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/pravega/pravega/pulls/comments/234733365"},"html":{"href":"https://github.com/pravega/pravega/pull/3090#discussion_r234733365"},"pull_request":{"href":"https://api.github.com/repos/pravega/pravega/pulls/3090"}},"in_reply_to_id":232843095},"pull_request":{"url":"https://api.github.com/repos/pravega/pravega/pulls/3090","id":229835143,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI5ODM1MTQz","html_url":"https://github.com/pravega/pravega/pull/3090","diff_url":"https://github.com/pravega/pravega/pull/3090.diff","patch_url":"https://github.com/pravega/pravega/pull/3090.patch","issue_url":"https://api.github.com/repos/pravega/pravega/issues/3090","number":3090,"state":"open","locked":false,"title":"Issue 2834: (Segment Store) B+Tree Incremental Compaction","user":{"login":"andreipaduroiu","id":14354377,"node_id":"MDQ6VXNlcjE0MzU0Mzc3","avatar_url":"https://avatars2.githubusercontent.com/u/14354377?v=4","gravatar_id":"","url":"https://api.github.com/users/andreipaduroiu","html_url":"https://github.com/andreipaduroiu","followers_url":"https://api.github.com/users/andreipaduroiu/followers","following_url":"https://api.github.com/users/andreipaduroiu/following{/other_user}","gists_url":"https://api.github.com/users/andreipaduroiu/gists{/gist_id}","starred_url":"https://api.github.com/users/andreipaduroiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreipaduroiu/subscriptions","organizations_url":"https://api.github.com/users/andreipaduroiu/orgs","repos_url":"https://api.github.com/users/andreipaduroiu/repos","events_url":"https://api.github.com/users/andreipaduroiu/events{/privacy}","received_events_url":"https://api.github.com/users/andreipaduroiu/received_events","type":"User","site_admin":false},"body":"**Change log description**  \r\n- Enhancing the B+Tree implementation with incremental compaction. \r\n    - Upon every update, it will fetch the Page with smallest offset and move it to the end of the index file.\r\n- SegmentAttributeBTreeIndex will auto-truncate the Attribute Segment as instructed by the B+Tree write callback.\r\n\r\n**Purpose of the change**  \r\nFixes #2834.\r\n\r\n**What the code does**  \r\n- B+Trees on append-only storage suffer from copy-on-write problems. Every update to a leaf page results in having to rewrite that page, as well as all of its ancestors.\r\n- This leaves the B+Tree file representation with a lot of gaps in the middle of the file (where there used to be valid pages, but which have since become obsolete).\r\n- The index file cannot be truncated since we may be deleting valid pages from the beginning.\r\n- The incremental compaction picks the page with smallest offset and moves it to the end. Over time, this helps remove the obstacles that prevented truncation in the first place.\r\n- The BTreeIndex also holds extra metadata in the index nodes so that it knows what is the safest Truncation offset to cut data from.\r\n    - This value is picked up by the SegmentAttributeBTreeIndex which applies truncation as needed.\r\n\r\n**How to verify it**  \r\nExisting unit tests must pass. New/updated unit tests must pass as well.\r\n","created_at":"2018-11-09T22:19:34Z","updated_at":"2018-11-19T18:35:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"75e6bf7b426f6b3ed4b8d87564254d68c30bb176","assignee":{"login":"fpj","id":572920,"node_id":"MDQ6VXNlcjU3MjkyMA==","avatar_url":"https://avatars3.githubusercontent.com/u/572920?v=4","gravatar_id":"","url":"https://api.github.com/users/fpj","html_url":"https://github.com/fpj","followers_url":"https://api.github.com/users/fpj/followers","following_url":"https://api.github.com/users/fpj/following{/other_user}","gists_url":"https://api.github.com/users/fpj/gists{/gist_id}","starred_url":"https://api.github.com/users/fpj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fpj/subscriptions","organizations_url":"https://api.github.com/users/fpj/orgs","repos_url":"https://api.github.com/users/fpj/repos","events_url":"https://api.github.com/users/fpj/events{/privacy}","received_events_url":"https://api.github.com/users/fpj/received_events","type":"User","site_admin":false},"assignees":[{"login":"fpj","id":572920,"node_id":"MDQ6VXNlcjU3MjkyMA==","avatar_url":"https://avatars3.githubusercontent.com/u/572920?v=4","gravatar_id":"","url":"https://api.github.com/users/fpj","html_url":"https://github.com/fpj","followers_url":"https://api.github.com/users/fpj/followers","following_url":"https://api.github.com/users/fpj/following{/other_user}","gists_url":"https://api.github.com/users/fpj/gists{/gist_id}","starred_url":"https://api.github.com/users/fpj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fpj/subscriptions","organizations_url":"https://api.github.com/users/fpj/orgs","repos_url":"https://api.github.com/users/fpj/repos","events_url":"https://api.github.com/users/fpj/events{/privacy}","received_events_url":"https://api.github.com/users/fpj/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/pravega/pravega/pulls/3090/commits","review_comments_url":"https://api.github.com/repos/pravega/pravega/pulls/3090/comments","review_comment_url":"https://api.github.com/repos/pravega/pravega/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pravega/pravega/issues/3090/comments","statuses_url":"https://api.github.com/repos/pravega/pravega/statuses/38fcc648491125f38ce8ed348d3b5a5e7d22ab28","head":{"label":"andreipaduroiu:issue-2834-btree-compaction","ref":"issue-2834-btree-compaction","sha":"38fcc648491125f38ce8ed348d3b5a5e7d22ab28","user":{"login":"andreipaduroiu","id":14354377,"node_id":"MDQ6VXNlcjE0MzU0Mzc3","avatar_url":"https://avatars2.githubusercontent.com/u/14354377?v=4","gravatar_id":"","url":"https://api.github.com/users/andreipaduroiu","html_url":"https://github.com/andreipaduroiu","followers_url":"https://api.github.com/users/andreipaduroiu/followers","following_url":"https://api.github.com/users/andreipaduroiu/following{/other_user}","gists_url":"https://api.github.com/users/andreipaduroiu/gists{/gist_id}","starred_url":"https://api.github.com/users/andreipaduroiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreipaduroiu/subscriptions","organizations_url":"https://api.github.com/users/andreipaduroiu/orgs","repos_url":"https://api.github.com/users/andreipaduroiu/repos","events_url":"https://api.github.com/users/andreipaduroiu/events{/privacy}","received_events_url":"https://api.github.com/users/andreipaduroiu/received_events","type":"User","site_admin":false},"repo":{"id":136227276,"node_id":"MDEwOlJlcG9zaXRvcnkxMzYyMjcyNzY=","name":"pravega","full_name":"andreipaduroiu/pravega","private":false,"owner":{"login":"andreipaduroiu","id":14354377,"node_id":"MDQ6VXNlcjE0MzU0Mzc3","avatar_url":"https://avatars2.githubusercontent.com/u/14354377?v=4","gravatar_id":"","url":"https://api.github.com/users/andreipaduroiu","html_url":"https://github.com/andreipaduroiu","followers_url":"https://api.github.com/users/andreipaduroiu/followers","following_url":"https://api.github.com/users/andreipaduroiu/following{/other_user}","gists_url":"https://api.github.com/users/andreipaduroiu/gists{/gist_id}","starred_url":"https://api.github.com/users/andreipaduroiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreipaduroiu/subscriptions","organizations_url":"https://api.github.com/users/andreipaduroiu/orgs","repos_url":"https://api.github.com/users/andreipaduroiu/repos","events_url":"https://api.github.com/users/andreipaduroiu/events{/privacy}","received_events_url":"https://api.github.com/users/andreipaduroiu/received_events","type":"User","site_admin":false},"html_url":"https://github.com/andreipaduroiu/pravega","description":"Pravega - Streaming as a new software defined storage primitive","fork":true,"url":"https://api.github.com/repos/andreipaduroiu/pravega","forks_url":"https://api.github.com/repos/andreipaduroiu/pravega/forks","keys_url":"https://api.github.com/repos/andreipaduroiu/pravega/keys{/key_id}","collaborators_url":"https://api.github.com/repos/andreipaduroiu/pravega/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/andreipaduroiu/pravega/teams","hooks_url":"https://api.github.com/repos/andreipaduroiu/pravega/hooks","issue_events_url":"https://api.github.com/repos/andreipaduroiu/pravega/issues/events{/number}","events_url":"https://api.github.com/repos/andreipaduroiu/pravega/events","assignees_url":"https://api.github.com/repos/andreipaduroiu/pravega/assignees{/user}","branches_url":"https://api.github.com/repos/andreipaduroiu/pravega/branches{/branch}","tags_url":"https://api.github.com/repos/andreipaduroiu/pravega/tags","blobs_url":"https://api.github.com/repos/andreipaduroiu/pravega/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/andreipaduroiu/pravega/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/andreipaduroiu/pravega/git/refs{/sha}","trees_url":"https://api.github.com/repos/andreipaduroiu/pravega/git/trees{/sha}","statuses_url":"https://api.github.com/repos/andreipaduroiu/pravega/statuses/{sha}","languages_url":"https://api.github.com/repos/andreipaduroiu/pravega/languages","stargazers_url":"https://api.github.com/repos/andreipaduroiu/pravega/stargazers","contributors_url":"https://api.github.com/repos/andreipaduroiu/pravega/contributors","subscribers_url":"https://api.github.com/repos/andreipaduroiu/pravega/subscribers","subscription_url":"https://api.github.com/repos/andreipaduroiu/pravega/subscription","commits_url":"https://api.github.com/repos/andreipaduroiu/pravega/commits{/sha}","git_commits_url":"https://api.github.com/repos/andreipaduroiu/pravega/git/commits{/sha}","comments_url":"https://api.github.com/repos/andreipaduroiu/pravega/comments{/number}","issue_comment_url":"https://api.github.com/repos/andreipaduroiu/pravega/issues/comments{/number}","contents_url":"https://api.github.com/repos/andreipaduroiu/pravega/contents/{+path}","compare_url":"https://api.github.com/repos/andreipaduroiu/pravega/compare/{base}...{head}","merges_url":"https://api.github.com/repos/andreipaduroiu/pravega/merges","archive_url":"https://api.github.com/repos/andreipaduroiu/pravega/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/andreipaduroiu/pravega/downloads","issues_url":"https://api.github.com/repos/andreipaduroiu/pravega/issues{/number}","pulls_url":"https://api.github.com/repos/andreipaduroiu/pravega/pulls{/number}","milestones_url":"https://api.github.com/repos/andreipaduroiu/pravega/milestones{/number}","notifications_url":"https://api.github.com/repos/andreipaduroiu/pravega/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/andreipaduroiu/pravega/labels{/name}","releases_url":"https://api.github.com/repos/andreipaduroiu/pravega/releases{/id}","deployments_url":"https://api.github.com/repos/andreipaduroiu/pravega/deployments","created_at":"2018-06-05T19:49:10Z","updated_at":"2018-06-05T19:55:27Z","pushed_at":"2018-11-19T17:55:57Z","git_url":"git://github.com/andreipaduroiu/pravega.git","ssh_url":"git@github.com:andreipaduroiu/pravega.git","clone_url":"https://github.com/andreipaduroiu/pravega.git","svn_url":"https://github.com/andreipaduroiu/pravega","homepage":"http://pravega.io","size":26859,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"pravega:master","ref":"master","sha":"592ea91c0b8067a752f07e2a57080f0d3b18f66d","user":{"login":"pravega","id":25698199,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI1Njk4MTk5","avatar_url":"https://avatars1.githubusercontent.com/u/25698199?v=4","gravatar_id":"","url":"https://api.github.com/users/pravega","html_url":"https://github.com/pravega","followers_url":"https://api.github.com/users/pravega/followers","following_url":"https://api.github.com/users/pravega/following{/other_user}","gists_url":"https://api.github.com/users/pravega/gists{/gist_id}","starred_url":"https://api.github.com/users/pravega/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pravega/subscriptions","organizations_url":"https://api.github.com/users/pravega/orgs","repos_url":"https://api.github.com/users/pravega/repos","events_url":"https://api.github.com/users/pravega/events{/privacy}","received_events_url":"https://api.github.com/users/pravega/received_events","type":"Organization","site_admin":false},"repo":{"id":63096822,"node_id":"MDEwOlJlcG9zaXRvcnk2MzA5NjgyMg==","name":"pravega","full_name":"pravega/pravega","private":false,"owner":{"login":"pravega","id":25698199,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI1Njk4MTk5","avatar_url":"https://avatars1.githubusercontent.com/u/25698199?v=4","gravatar_id":"","url":"https://api.github.com/users/pravega","html_url":"https://github.com/pravega","followers_url":"https://api.github.com/users/pravega/followers","following_url":"https://api.github.com/users/pravega/following{/other_user}","gists_url":"https://api.github.com/users/pravega/gists{/gist_id}","starred_url":"https://api.github.com/users/pravega/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pravega/subscriptions","organizations_url":"https://api.github.com/users/pravega/orgs","repos_url":"https://api.github.com/users/pravega/repos","events_url":"https://api.github.com/users/pravega/events{/privacy}","received_events_url":"https://api.github.com/users/pravega/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/pravega/pravega","description":"Pravega - Streaming as a new software defined storage primitive","fork":false,"url":"https://api.github.com/repos/pravega/pravega","forks_url":"https://api.github.com/repos/pravega/pravega/forks","keys_url":"https://api.github.com/repos/pravega/pravega/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pravega/pravega/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pravega/pravega/teams","hooks_url":"https://api.github.com/repos/pravega/pravega/hooks","issue_events_url":"https://api.github.com/repos/pravega/pravega/issues/events{/number}","events_url":"https://api.github.com/repos/pravega/pravega/events","assignees_url":"https://api.github.com/repos/pravega/pravega/assignees{/user}","branches_url":"https://api.github.com/repos/pravega/pravega/branches{/branch}","tags_url":"https://api.github.com/repos/pravega/pravega/tags","blobs_url":"https://api.github.com/repos/pravega/pravega/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pravega/pravega/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pravega/pravega/git/refs{/sha}","trees_url":"https://api.github.com/repos/pravega/pravega/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pravega/pravega/statuses/{sha}","languages_url":"https://api.github.com/repos/pravega/pravega/languages","stargazers_url":"https://api.github.com/repos/pravega/pravega/stargazers","contributors_url":"https://api.github.com/repos/pravega/pravega/contributors","subscribers_url":"https://api.github.com/repos/pravega/pravega/subscribers","subscription_url":"https://api.github.com/repos/pravega/pravega/subscription","commits_url":"https://api.github.com/repos/pravega/pravega/commits{/sha}","git_commits_url":"https://api.github.com/repos/pravega/pravega/git/commits{/sha}","comments_url":"https://api.github.com/repos/pravega/pravega/comments{/number}","issue_comment_url":"https://api.github.com/repos/pravega/pravega/issues/comments{/number}","contents_url":"https://api.github.com/repos/pravega/pravega/contents/{+path}","compare_url":"https://api.github.com/repos/pravega/pravega/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pravega/pravega/merges","archive_url":"https://api.github.com/repos/pravega/pravega/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pravega/pravega/downloads","issues_url":"https://api.github.com/repos/pravega/pravega/issues{/number}","pulls_url":"https://api.github.com/repos/pravega/pravega/pulls{/number}","milestones_url":"https://api.github.com/repos/pravega/pravega/milestones{/number}","notifications_url":"https://api.github.com/repos/pravega/pravega/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pravega/pravega/labels{/name}","releases_url":"https://api.github.com/repos/pravega/pravega/releases{/id}","deployments_url":"https://api.github.com/repos/pravega/pravega/deployments","created_at":"2016-07-11T19:41:03Z","updated_at":"2018-11-19T16:17:03Z","pushed_at":"2018-11-19T17:55:58Z","git_url":"git://github.com/pravega/pravega.git","ssh_url":"git@github.com:pravega/pravega.git","clone_url":"https://github.com/pravega/pravega.git","svn_url":"https://github.com/pravega/pravega","homepage":"http://pravega.io","size":27436,"stargazers_count":283,"watchers_count":283,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":93,"mirror_url":null,"archived":false,"open_issues_count":390,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":93,"open_issues":390,"watchers":283,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pravega/pravega/pulls/3090"},"html":{"href":"https://github.com/pravega/pravega/pull/3090"},"issue":{"href":"https://api.github.com/repos/pravega/pravega/issues/3090"},"comments":{"href":"https://api.github.com/repos/pravega/pravega/issues/3090/comments"},"review_comments":{"href":"https://api.github.com/repos/pravega/pravega/pulls/3090/comments"},"review_comment":{"href":"https://api.github.com/repos/pravega/pravega/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pravega/pravega/pulls/3090/commits"},"statuses":{"href":"https://api.github.com/repos/pravega/pravega/statuses/38fcc648491125f38ce8ed348d3b5a5e7d22ab28"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"fpj"}}