{"action":"created","comment":{"url":"https://api.github.com/repos/facebook/rocksdb/pulls/comments/208774618","pull_request_review_id":144565911,"id":208774618,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwODc3NDYxOA==","diff_hunk":"@@ -377,27 +430,69 @@ bool IndexBlockIter::ParseNextIndexKey() {\n     key_pinned_ = false;\n   }\n   value_ = Slice(p + non_shared, value_length);\n-  while (restart_index_ + 1 < num_restarts_ &&\n-         GetRestartPoint(restart_index_ + 1) < current_) {\n-    ++restart_index_;\n+  if (shared == 0) {\n+    while (restart_index_ + 1 < num_restarts_ &&\n+           GetRestartPoint(restart_index_ + 1) < current_) {\n+      ++restart_index_;\n+    }\n+  }\n+  // else we are in the middle of a restart interval and the restart_index_\n+  // thus has not changed\n+  if (value_delta_encoded_) {\n+    assert(value_length == 0);\n+    DecodeCurrentValue(shared);\n   }\n   return true;\n }\n \n+// The format:\n+// restart_point   0: k, v (off,sz), k, v (delta-sz), ..., k, v (delta-sz)\n+// restart_point   1: k, v (off,sz), k, v (delta-sz), ..., k, v (delta-sz)\n+// ...\n+// restart_point n-1: k, v (off,sz), k, v (delta-sz), ..., k, v (delta-sz)\n+// where, k is key, v is value, and its encoding is in parenthesis.\n+// The format of each key is (shared_size, non_shared_size, shared, non_shared)\n+// The format of each value, i.e., block hanlde, is (offset,size) whenever the\n+// shared_size is 0, which included the first entry in each restart point.\n+// Otherwise the format is delta-size = block handle size - size of last block\n+// handle.\n+void IndexBlockIter::DecodeCurrentValue(uint32_t shared) {\n+  assert(value_delta_encoded_);\n+  const char* l = value_.data() + 20;  // limit of two var-size uint64_t\n+  if (shared == 0) {\n+    uint64_t o, s;\n+    auto newp = GetVarint64Ptr(value_.data(), l, &o);\n+    newp = GetVarint64Ptr(newp, l, &s);\n+    decoded_value_ = BlockHandle(o, s);\n+    value_.set_size(newp - value_.data());\n+  } else {\n+    uint64_t next_value_base =\n+        decoded_value_.offset() + decoded_value_.size() + kBlockTrailerSize;\n+    int64_t delta;\n+    auto newp = GetVarsignedint64Ptr(value_.data(), l, &delta);","path":"table/block.cc","position":217,"original_position":217,"commit_id":"6606844312cd9d8356f822eee93b02d2bfd1b7b6","original_commit_id":"6606844312cd9d8356f822eee93b02d2bfd1b7b6","user":{"login":"siying","id":5334567,"node_id":"MDQ6VXNlcjUzMzQ1Njc=","avatar_url":"https://avatars3.githubusercontent.com/u/5334567?v=4","gravatar_id":"","url":"https://api.github.com/users/siying","html_url":"https://github.com/siying","followers_url":"https://api.github.com/users/siying/followers","following_url":"https://api.github.com/users/siying/following{/other_user}","gists_url":"https://api.github.com/users/siying/gists{/gist_id}","starred_url":"https://api.github.com/users/siying/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/siying/subscriptions","organizations_url":"https://api.github.com/users/siying/orgs","repos_url":"https://api.github.com/users/siying/repos","events_url":"https://api.github.com/users/siying/events{/privacy}","received_events_url":"https://api.github.com/users/siying/received_events","type":"User","site_admin":false},"body":"Same here. Can you name `char*`?  This gives me headache.","created_at":"2018-08-09T00:19:34Z","updated_at":"2018-08-09T00:25:26Z","html_url":"https://github.com/facebook/rocksdb/pull/3983#discussion_r208774618","pull_request_url":"https://api.github.com/repos/facebook/rocksdb/pulls/3983","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/comments/208774618"},"html":{"href":"https://github.com/facebook/rocksdb/pull/3983#discussion_r208774618"},"pull_request":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3983"}}},"pull_request":{"url":"https://api.github.com/repos/facebook/rocksdb/pulls/3983","id":194069703,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk0MDY5NzAz","html_url":"https://github.com/facebook/rocksdb/pull/3983","diff_url":"https://github.com/facebook/rocksdb/pull/3983.diff","patch_url":"https://github.com/facebook/rocksdb/pull/3983.patch","issue_url":"https://api.github.com/repos/facebook/rocksdb/issues/3983","number":3983,"state":"open","locked":false,"title":"Index value delta encoding","user":{"login":"maysamyabandeh","id":1167741,"node_id":"MDQ6VXNlcjExNjc3NDE=","avatar_url":"https://avatars2.githubusercontent.com/u/1167741?v=4","gravatar_id":"","url":"https://api.github.com/users/maysamyabandeh","html_url":"https://github.com/maysamyabandeh","followers_url":"https://api.github.com/users/maysamyabandeh/followers","following_url":"https://api.github.com/users/maysamyabandeh/following{/other_user}","gists_url":"https://api.github.com/users/maysamyabandeh/gists{/gist_id}","starred_url":"https://api.github.com/users/maysamyabandeh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maysamyabandeh/subscriptions","organizations_url":"https://api.github.com/users/maysamyabandeh/orgs","repos_url":"https://api.github.com/users/maysamyabandeh/repos","events_url":"https://api.github.com/users/maysamyabandeh/events{/privacy}","received_events_url":"https://api.github.com/users/maysamyabandeh/received_events","type":"User","site_admin":false},"body":"Given that index value is a BlockHandle, which is basically an <offset, size> pair we can apply delta encoding on the values. The first value at each index restart interval encoded the full BlockHandle but the rest encode only the size. This reduces the index size.\r\nCurrently the value is encoded again when retrieved by ::value(). We can later optimize this to skip this extra step and read the BlockHandle directly from the index iterator.","created_at":"2018-06-11T18:48:24Z","updated_at":"2018-08-09T00:25:26Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e7a42056e7ff53859b47d56377482e7a60251060","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":196859108,"node_id":"MDU6TGFiZWwxOTY4NTkxMDg=","url":"https://api.github.com/repos/facebook/rocksdb/labels/CLA%20Signed","name":"CLA Signed","color":"009900","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/facebook/rocksdb/pulls/3983/commits","review_comments_url":"https://api.github.com/repos/facebook/rocksdb/pulls/3983/comments","review_comment_url":"https://api.github.com/repos/facebook/rocksdb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/3983/comments","statuses_url":"https://api.github.com/repos/facebook/rocksdb/statuses/6606844312cd9d8356f822eee93b02d2bfd1b7b6","head":{"label":"maysamyabandeh:format4","ref":"format4","sha":"6606844312cd9d8356f822eee93b02d2bfd1b7b6","user":{"login":"maysamyabandeh","id":1167741,"node_id":"MDQ6VXNlcjExNjc3NDE=","avatar_url":"https://avatars2.githubusercontent.com/u/1167741?v=4","gravatar_id":"","url":"https://api.github.com/users/maysamyabandeh","html_url":"https://github.com/maysamyabandeh","followers_url":"https://api.github.com/users/maysamyabandeh/followers","following_url":"https://api.github.com/users/maysamyabandeh/following{/other_user}","gists_url":"https://api.github.com/users/maysamyabandeh/gists{/gist_id}","starred_url":"https://api.github.com/users/maysamyabandeh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maysamyabandeh/subscriptions","organizations_url":"https://api.github.com/users/maysamyabandeh/orgs","repos_url":"https://api.github.com/users/maysamyabandeh/repos","events_url":"https://api.github.com/users/maysamyabandeh/events{/privacy}","received_events_url":"https://api.github.com/users/maysamyabandeh/received_events","type":"User","site_admin":false},"repo":{"id":73221890,"node_id":"MDEwOlJlcG9zaXRvcnk3MzIyMTg5MA==","name":"rocksdb","full_name":"maysamyabandeh/rocksdb","owner":{"login":"maysamyabandeh","id":1167741,"node_id":"MDQ6VXNlcjExNjc3NDE=","avatar_url":"https://avatars2.githubusercontent.com/u/1167741?v=4","gravatar_id":"","url":"https://api.github.com/users/maysamyabandeh","html_url":"https://github.com/maysamyabandeh","followers_url":"https://api.github.com/users/maysamyabandeh/followers","following_url":"https://api.github.com/users/maysamyabandeh/following{/other_user}","gists_url":"https://api.github.com/users/maysamyabandeh/gists{/gist_id}","starred_url":"https://api.github.com/users/maysamyabandeh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maysamyabandeh/subscriptions","organizations_url":"https://api.github.com/users/maysamyabandeh/orgs","repos_url":"https://api.github.com/users/maysamyabandeh/repos","events_url":"https://api.github.com/users/maysamyabandeh/events{/privacy}","received_events_url":"https://api.github.com/users/maysamyabandeh/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/maysamyabandeh/rocksdb","description":"A library that provides an embeddable, persistent key-value store for fast storage.","fork":true,"url":"https://api.github.com/repos/maysamyabandeh/rocksdb","forks_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/forks","keys_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/teams","hooks_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/hooks","issue_events_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/issues/events{/number}","events_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/events","assignees_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/assignees{/user}","branches_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/branches{/branch}","tags_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/tags","blobs_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/statuses/{sha}","languages_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/languages","stargazers_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/stargazers","contributors_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/contributors","subscribers_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/subscribers","subscription_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/subscription","commits_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/contents/{+path}","compare_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/merges","archive_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/downloads","issues_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/issues{/number}","pulls_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/pulls{/number}","milestones_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/milestones{/number}","notifications_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/labels{/name}","releases_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/releases{/id}","deployments_url":"https://api.github.com/repos/maysamyabandeh/rocksdb/deployments","created_at":"2016-11-08T19:56:13Z","updated_at":"2016-11-08T19:56:19Z","pushed_at":"2018-08-08T23:34:45Z","git_url":"git://github.com/maysamyabandeh/rocksdb.git","ssh_url":"git@github.com:maysamyabandeh/rocksdb.git","clone_url":"https://github.com/maysamyabandeh/rocksdb.git","svn_url":"https://github.com/maysamyabandeh/rocksdb","homepage":"http://rocksdb.org","size":115433,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"facebook:master","ref":"master","sha":"bbf30330b4cf459de3a7de0d3af77006e91d1f55","user":{"login":"facebook","id":69631,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY5NjMx","avatar_url":"https://avatars3.githubusercontent.com/u/69631?v=4","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"repo":{"id":6934395,"node_id":"MDEwOlJlcG9zaXRvcnk2OTM0Mzk1","name":"rocksdb","full_name":"facebook/rocksdb","owner":{"login":"facebook","id":69631,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY5NjMx","avatar_url":"https://avatars3.githubusercontent.com/u/69631?v=4","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/facebook/rocksdb","description":"A library that provides an embeddable, persistent key-value store for fast storage.","fork":false,"url":"https://api.github.com/repos/facebook/rocksdb","forks_url":"https://api.github.com/repos/facebook/rocksdb/forks","keys_url":"https://api.github.com/repos/facebook/rocksdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/facebook/rocksdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/facebook/rocksdb/teams","hooks_url":"https://api.github.com/repos/facebook/rocksdb/hooks","issue_events_url":"https://api.github.com/repos/facebook/rocksdb/issues/events{/number}","events_url":"https://api.github.com/repos/facebook/rocksdb/events","assignees_url":"https://api.github.com/repos/facebook/rocksdb/assignees{/user}","branches_url":"https://api.github.com/repos/facebook/rocksdb/branches{/branch}","tags_url":"https://api.github.com/repos/facebook/rocksdb/tags","blobs_url":"https://api.github.com/repos/facebook/rocksdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/facebook/rocksdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/facebook/rocksdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/facebook/rocksdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/facebook/rocksdb/statuses/{sha}","languages_url":"https://api.github.com/repos/facebook/rocksdb/languages","stargazers_url":"https://api.github.com/repos/facebook/rocksdb/stargazers","contributors_url":"https://api.github.com/repos/facebook/rocksdb/contributors","subscribers_url":"https://api.github.com/repos/facebook/rocksdb/subscribers","subscription_url":"https://api.github.com/repos/facebook/rocksdb/subscription","commits_url":"https://api.github.com/repos/facebook/rocksdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/facebook/rocksdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/facebook/rocksdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/facebook/rocksdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/facebook/rocksdb/contents/{+path}","compare_url":"https://api.github.com/repos/facebook/rocksdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/facebook/rocksdb/merges","archive_url":"https://api.github.com/repos/facebook/rocksdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/facebook/rocksdb/downloads","issues_url":"https://api.github.com/repos/facebook/rocksdb/issues{/number}","pulls_url":"https://api.github.com/repos/facebook/rocksdb/pulls{/number}","milestones_url":"https://api.github.com/repos/facebook/rocksdb/milestones{/number}","notifications_url":"https://api.github.com/repos/facebook/rocksdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/facebook/rocksdb/labels{/name}","releases_url":"https://api.github.com/repos/facebook/rocksdb/releases{/id}","deployments_url":"https://api.github.com/repos/facebook/rocksdb/deployments","created_at":"2012-11-30T06:16:18Z","updated_at":"2018-08-08T22:57:43Z","pushed_at":"2018-08-08T23:45:56Z","git_url":"git://github.com/facebook/rocksdb.git","ssh_url":"git@github.com:facebook/rocksdb.git","clone_url":"https://github.com/facebook/rocksdb.git","svn_url":"https://github.com/facebook/rocksdb","homepage":"http://rocksdb.org","size":116761,"stargazers_count":11220,"watchers_count":11220,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2468,"mirror_url":null,"archived":false,"open_issues_count":212,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":2468,"open_issues":212,"watchers":11220,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3983"},"html":{"href":"https://github.com/facebook/rocksdb/pull/3983"},"issue":{"href":"https://api.github.com/repos/facebook/rocksdb/issues/3983"},"comments":{"href":"https://api.github.com/repos/facebook/rocksdb/issues/3983/comments"},"review_comments":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3983/comments"},"review_comment":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3983/commits"},"statuses":{"href":"https://api.github.com/repos/facebook/rocksdb/statuses/6606844312cd9d8356f822eee93b02d2bfd1b7b6"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"siying"}}