{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/4174","repository_url":"https://api.github.com/repos/facebook/rocksdb","labels_url":"https://api.github.com/repos/facebook/rocksdb/issues/4174/labels{/name}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/4174/comments","events_url":"https://api.github.com/repos/facebook/rocksdb/issues/4174/events","html_url":"https://github.com/facebook/rocksdb/pull/4174","id":343914691,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAzNDMyMTg0","number":4174,"title":"DataBlockHashIndex: Adding HashIndex feature to Block and BlockBuilder","user":{"login":"fgwu","id":2732324,"node_id":"MDQ6VXNlcjI3MzIzMjQ=","avatar_url":"https://avatars1.githubusercontent.com/u/2732324?v=4","gravatar_id":"","url":"https://api.github.com/users/fgwu","html_url":"https://github.com/fgwu","followers_url":"https://api.github.com/users/fgwu/followers","following_url":"https://api.github.com/users/fgwu/following{/other_user}","gists_url":"https://api.github.com/users/fgwu/gists{/gist_id}","starred_url":"https://api.github.com/users/fgwu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fgwu/subscriptions","organizations_url":"https://api.github.com/users/fgwu/orgs","repos_url":"https://api.github.com/users/fgwu/repos","events_url":"https://api.github.com/users/fgwu/events{/privacy}","received_events_url":"https://api.github.com/users/fgwu/received_events","type":"User","site_admin":false},"labels":[{"id":196859108,"node_id":"MDU6TGFiZWwxOTY4NTkxMDg=","url":"https://api.github.com/repos/facebook/rocksdb/labels/CLA%20Signed","name":"CLA Signed","color":"009900","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":86,"created_at":"2018-07-24T07:21:56Z","updated_at":"2018-08-15T04:03:43Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/facebook/rocksdb/pulls/4174","html_url":"https://github.com/facebook/rocksdb/pull/4174","diff_url":"https://github.com/facebook/rocksdb/pull/4174.diff","patch_url":"https://github.com/facebook/rocksdb/pull/4174.patch"},"body":"Summary:\r\n\r\nAdding the DataBlockHashIndex feature to Block and BlockBuilder.\r\n\r\nTest Plan:\r\n\r\nadded unit test `DataBlockHashIndexTest` in `block_test`\r\n`make -j32 check` and make sure all test pass\r\n\r\nSome performance numbers.\r\n`CPU Util` is the perf number of the `DataBlockIter::Seek()`, including its subroutines. \r\n```\r\n# large cache 20GB\r\nindex(util_ratio) | TP (MB/s) | CPU Util(%) | fall_back_ratio | cache_miss_ratio | DB Space (GB)\r\n----------------- | ----------| ----------- | --------------- | ---------------- | ------------\r\n           binary |       116 |       27.17 |           1.000 |         0.000494 |         5.41\r\n            hash1 |       123 |       22.21 |           0.524 |         0.000502 |         5.59\r\n          hash0.9 |       126 |       22.89 |           0.559 |         0.000502 |         5.61\r\n          hash0.8 |       129 |       21.65 |           0.487 |         0.000504 |         5.63\r\n          hash0.7 |       127 |       21.12 |           0.463 |         0.000504 |         5.65\r\n          hash0.6 |       130 |       20.62 |           0.423 |         0.000506 |         5.69\r\n          hash0.5 |       132 |       19.34 |           0.311 |         0.000510 |         5.75\r\n\r\n\r\n# small cache 1GB\r\nindex(util_ratio) | TP (MB/s) | CPU Util(%) | fall_back_ratio | cache_miss_ratio | DB Space (GB)\r\n----------------- | ----------| ----------- | --------------- | ---------------- | ------------\r\n           binary |      26.8 |        2.02 |           1.000 |         0.923345 |         5.41\r\n            hash1 |      25.9 |        1.49 |           0.524 |         0.924571 |         5.59\r\n          hash0.9 |      27.5 |        1.59 |           0.559 |         0.924561 |         5.61\r\n          hash0.8 |      27.4 |        1.52 |           0.487 |         0.924868 |         5.63\r\n          hash0.7 |      27.7 |        1.44 |           0.463 |         0.924858 |         5.65\r\n          hash0.6 |      26.8 |        1.36 |           0.423 |         0.925160 |         5.69\r\n          hash0.5 |      28.0 |        1.22 |           0.311 |         0.925779 |         5.75\r\n\r\n```\r\n\r\nAlso we compare the `master` branch the `feature` PR based on to make sure there is no performance regression on the default binary seek case.\r\n\r\n```\r\n branch | Throughput (MB/s) | CPU Util(%) | cache miss ratio | DB Space (GB)\r\n------- | ----------------- | ----------- | ---------------- | -------------\r\n master |             123.2 |        26.6 |         0.000494 |          5.41\r\nfeature |             124.6 |       26.02 |         0.000494 |          5.41\r\n\r\n```\r\n\r\n```\r\n# benchmarking command\r\n# setting: num=200 million, reads=100 million, key_size=8B, value_size=40B, thread=16\r\n$DB_BENCH  --data_block_index_type=${block_index} \\\r\n           --db=${db} \\\r\n           --block_size=16000 --level_compaction_dynamic_level_bytes=1 \\\r\n           --num=$num \\\r\n           --key_size=$ks \\\r\n           --value_size=$vs \\\r\n           --benchmarks=fillseq --compression_type=snappy \\\r\n           --statistics=false --block_restart_interval=1 \\\r\n           --compression_ratio=0.4 \\\r\n            --data_block_hash_table_util_ratio=${util_ratio} \\\r\n           --statistics=true \\\r\n           --perf_level=2 \\\r\n           >${write_log}\r\n\r\ntime perf record -g -F 99 \\\r\n$DB_BENCH  --data_block_index_type=${block_index} \\\r\n           --db=${db} \\\r\n           --block_size=16000 --level_compaction_dynamic_level_bytes=1 \\\r\n           --use_existing_db=true \\\r\n           --num=${num} \\\r\n           --reads=${reads} \\\r\n           --key_size=$ks \\\r\n           --value_size=$vs \\\r\n           --benchmarks=readtocache,readrandom \\\r\n           --compression_type=snappy \\\r\n           --block_restart_interval=16 \\\r\n           --compression_ratio=0.4 \\\r\n           --cache_size=${cache_size} \\\r\n            --data_block_hash_table_util_ratio=${util_ratio} \\\r\n           --use_direct_reads \\\r\n           --disable_auto_compactions \\\r\n           --threads=${threads} \\\r\n           --statistics=true \\\r\n           --perf_level=2 \\\r\n           > ${read_log}\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/comments/413088136","html_url":"https://github.com/facebook/rocksdb/pull/4174#issuecomment-413088136","issue_url":"https://api.github.com/repos/facebook/rocksdb/issues/4174","id":413088136,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzA4ODEzNg==","user":{"login":"facebook-github-bot","id":6422482,"node_id":"MDQ6VXNlcjY0MjI0ODI=","avatar_url":"https://avatars3.githubusercontent.com/u/6422482?v=4","gravatar_id":"","url":"https://api.github.com/users/facebook-github-bot","html_url":"https://github.com/facebook-github-bot","followers_url":"https://api.github.com/users/facebook-github-bot/followers","following_url":"https://api.github.com/users/facebook-github-bot/following{/other_user}","gists_url":"https://api.github.com/users/facebook-github-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook-github-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook-github-bot/subscriptions","organizations_url":"https://api.github.com/users/facebook-github-bot/orgs","repos_url":"https://api.github.com/users/facebook-github-bot/repos","events_url":"https://api.github.com/users/facebook-github-bot/events{/privacy}","received_events_url":"https://api.github.com/users/facebook-github-bot/received_events","type":"User","site_admin":false},"created_at":"2018-08-15T04:03:43Z","updated_at":"2018-08-15T04:03:43Z","author_association":"NONE","body":"[@fgwu](https://github.com/fgwu) has updated the pull request. [Re-import the pull request](https://our.intern.facebook.com/intern/opensource/github/pull_request/confirm/import/1864064067235640/)"}}
{"actor":{"display_login":"facebook-github-bot"}}