{"action":"created","comment":{"url":"https://api.github.com/repos/facebook/rocksdb/pulls/comments/180619270","pull_request_review_id":111069998,"id":180619270,"diff_hunk":"@@ -919,27 +986,57 @@ class SharedState {\n     }\n   }\n \n+  std::atomic<uint32_t>& Value(int cf, int64_t key) const {\n+    return values_[cf * max_key_ + key];\n+  }\n+\n   void ClearColumnFamily(int cf) {\n-    std::fill(values_[cf].begin(), values_[cf].end(), SENTINEL);\n+    std::fill(&Value(cf, 0 /* key */), &Value(cf + 1, 0 /* key */),\n+              DELETION_SENTINEL);\n   }\n \n-  void Put(int cf, int64_t key, uint32_t value_base) {\n-    values_[cf][key] = value_base;\n+  // @param pending True if the update may have started but is not yet\n+  //    guaranteed finished. This is useful for crash-recovery testing when the\n+  //    process may crash before updating the expected values array.\n+  void Put(int cf, int64_t key, uint32_t value_base, bool pending) {\n+    if (!pending) {\n+      // prevent expected-value update from reordering before Write\n+      std::atomic_thread_fence(std::memory_order_release);","path":"tools/db_stress.cc","position":135,"original_position":135,"commit_id":"05f1419c19ad367441f185aa2875ce6df99b7b26","original_commit_id":"05f1419c19ad367441f185aa2875ce6df99b7b26","user":{"login":"ajkr","id":4780362,"avatar_url":"https://avatars0.githubusercontent.com/u/4780362?v=4","gravatar_id":"","url":"https://api.github.com/users/ajkr","html_url":"https://github.com/ajkr","followers_url":"https://api.github.com/users/ajkr/followers","following_url":"https://api.github.com/users/ajkr/following{/other_user}","gists_url":"https://api.github.com/users/ajkr/gists{/gist_id}","starred_url":"https://api.github.com/users/ajkr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajkr/subscriptions","organizations_url":"https://api.github.com/users/ajkr/orgs","repos_url":"https://api.github.com/users/ajkr/repos","events_url":"https://api.github.com/users/ajkr/events{/privacy}","received_events_url":"https://api.github.com/users/ajkr/received_events","type":"User","site_admin":false},"body":"So this isn't needed for hardware reordering on x86 as it promises not to do store-store reordering. I am still not sure about compiler reordering, though... To me it looks impossible as the DB ops are in different compilation units","created_at":"2018-04-11T02:26:28Z","updated_at":"2018-04-11T02:26:28Z","html_url":"https://github.com/facebook/rocksdb/pull/3629#discussion_r180619270","pull_request_url":"https://api.github.com/repos/facebook/rocksdb/pulls/3629","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/comments/180619270"},"html":{"href":"https://github.com/facebook/rocksdb/pull/3629#discussion_r180619270"},"pull_request":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3629"}}},"pull_request":{"url":"https://api.github.com/repos/facebook/rocksdb/pulls/3629","id":176310456,"html_url":"https://github.com/facebook/rocksdb/pull/3629","diff_url":"https://github.com/facebook/rocksdb/pull/3629.diff","patch_url":"https://github.com/facebook/rocksdb/pull/3629.patch","issue_url":"https://api.github.com/repos/facebook/rocksdb/issues/3629","number":3629,"state":"open","locked":false,"title":"Add crash-recovery correctness check to db_stress","user":{"login":"ajkr","id":4780362,"avatar_url":"https://avatars0.githubusercontent.com/u/4780362?v=4","gravatar_id":"","url":"https://api.github.com/users/ajkr","html_url":"https://github.com/ajkr","followers_url":"https://api.github.com/users/ajkr/followers","following_url":"https://api.github.com/users/ajkr/following{/other_user}","gists_url":"https://api.github.com/users/ajkr/gists{/gist_id}","starred_url":"https://api.github.com/users/ajkr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajkr/subscriptions","organizations_url":"https://api.github.com/users/ajkr/orgs","repos_url":"https://api.github.com/users/ajkr/repos","events_url":"https://api.github.com/users/ajkr/events{/privacy}","received_events_url":"https://api.github.com/users/ajkr/received_events","type":"User","site_admin":false},"body":"Previously, our `db_stress` tool held the expected state of the DB in-memory, so after crash-recovery, there was no way to verify data correctness. This PR adds an option, `--expected_values_file`, which specifies a file holding the expected values.\r\n\r\nIn black-box testing, the `db_stress` process can be killed arbitrarily, so updates to the `--expected_values_file` must be atomic. We achieve this by `mmap`ing the file and relying on `std::atomic<uint32_t>` for atomicity. Actually this doesn't provide a total guarantee on what we want as `std::atomic<uint32_t>` could, in theory, be translated into multiple stores surrounded by a mutex. We can verify our assumption by looking at `std::atomic::is_always_lock_free`.\r\n\r\nFor the `mmap`'d file, we didn't have an existing way to expose its contents as a raw memory buffer. This PR adds it in the `Env::NewMemoryMappedFileBuffer` function, and `MemoryMappedFileBuffer` class.\r\n\r\n`db_crashtest.py` is updated to use an expected values file for black-box testing. On the first iteration (when the DB is created), an empty file is provided as `db_stress` will populate it when it runs. On subsequent iterations, that same filename is provided so `db_stress` can check the data is as expected on startup.\r\n\r\nTest Plan:\r\n\r\n- new unit test\r\n- `python ./tools/db_crashtest.py --simple blackbox --nooverwritepercent=1 -duration=600`\r\n- manually induced corruption to `--expected_values_file` and verified crash-recovery check fails","created_at":"2018-03-20T20:18:15Z","updated_at":"2018-04-11T02:26:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7f77b19a1d808675deadb3a61916897b1da0b71e","assignee":null,"assignees":[],"requested_reviewers":[{"login":"anand1976","id":33647610,"avatar_url":"https://avatars0.githubusercontent.com/u/33647610?v=4","gravatar_id":"","url":"https://api.github.com/users/anand1976","html_url":"https://github.com/anand1976","followers_url":"https://api.github.com/users/anand1976/followers","following_url":"https://api.github.com/users/anand1976/following{/other_user}","gists_url":"https://api.github.com/users/anand1976/gists{/gist_id}","starred_url":"https://api.github.com/users/anand1976/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anand1976/subscriptions","organizations_url":"https://api.github.com/users/anand1976/orgs","repos_url":"https://api.github.com/users/anand1976/repos","events_url":"https://api.github.com/users/anand1976/events{/privacy}","received_events_url":"https://api.github.com/users/anand1976/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[{"id":196859108,"url":"https://api.github.com/repos/facebook/rocksdb/labels/CLA%20Signed","name":"CLA Signed","color":"009900","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/facebook/rocksdb/pulls/3629/commits","review_comments_url":"https://api.github.com/repos/facebook/rocksdb/pulls/3629/comments","review_comment_url":"https://api.github.com/repos/facebook/rocksdb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/3629/comments","statuses_url":"https://api.github.com/repos/facebook/rocksdb/statuses/05f1419c19ad367441f185aa2875ce6df99b7b26","head":{"label":"ajkr:db-stress-crash-recovery","ref":"db-stress-crash-recovery","sha":"05f1419c19ad367441f185aa2875ce6df99b7b26","user":{"login":"ajkr","id":4780362,"avatar_url":"https://avatars0.githubusercontent.com/u/4780362?v=4","gravatar_id":"","url":"https://api.github.com/users/ajkr","html_url":"https://github.com/ajkr","followers_url":"https://api.github.com/users/ajkr/followers","following_url":"https://api.github.com/users/ajkr/following{/other_user}","gists_url":"https://api.github.com/users/ajkr/gists{/gist_id}","starred_url":"https://api.github.com/users/ajkr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajkr/subscriptions","organizations_url":"https://api.github.com/users/ajkr/orgs","repos_url":"https://api.github.com/users/ajkr/repos","events_url":"https://api.github.com/users/ajkr/events{/privacy}","received_events_url":"https://api.github.com/users/ajkr/received_events","type":"User","site_admin":false},"repo":{"id":69681666,"name":"rocksdb","full_name":"ajkr/rocksdb","owner":{"login":"ajkr","id":4780362,"avatar_url":"https://avatars0.githubusercontent.com/u/4780362?v=4","gravatar_id":"","url":"https://api.github.com/users/ajkr","html_url":"https://github.com/ajkr","followers_url":"https://api.github.com/users/ajkr/followers","following_url":"https://api.github.com/users/ajkr/following{/other_user}","gists_url":"https://api.github.com/users/ajkr/gists{/gist_id}","starred_url":"https://api.github.com/users/ajkr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajkr/subscriptions","organizations_url":"https://api.github.com/users/ajkr/orgs","repos_url":"https://api.github.com/users/ajkr/repos","events_url":"https://api.github.com/users/ajkr/events{/privacy}","received_events_url":"https://api.github.com/users/ajkr/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ajkr/rocksdb","description":"A library that provides an embeddable, persistent key-value store for fast storage.","fork":true,"url":"https://api.github.com/repos/ajkr/rocksdb","forks_url":"https://api.github.com/repos/ajkr/rocksdb/forks","keys_url":"https://api.github.com/repos/ajkr/rocksdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ajkr/rocksdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ajkr/rocksdb/teams","hooks_url":"https://api.github.com/repos/ajkr/rocksdb/hooks","issue_events_url":"https://api.github.com/repos/ajkr/rocksdb/issues/events{/number}","events_url":"https://api.github.com/repos/ajkr/rocksdb/events","assignees_url":"https://api.github.com/repos/ajkr/rocksdb/assignees{/user}","branches_url":"https://api.github.com/repos/ajkr/rocksdb/branches{/branch}","tags_url":"https://api.github.com/repos/ajkr/rocksdb/tags","blobs_url":"https://api.github.com/repos/ajkr/rocksdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ajkr/rocksdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ajkr/rocksdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/ajkr/rocksdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ajkr/rocksdb/statuses/{sha}","languages_url":"https://api.github.com/repos/ajkr/rocksdb/languages","stargazers_url":"https://api.github.com/repos/ajkr/rocksdb/stargazers","contributors_url":"https://api.github.com/repos/ajkr/rocksdb/contributors","subscribers_url":"https://api.github.com/repos/ajkr/rocksdb/subscribers","subscription_url":"https://api.github.com/repos/ajkr/rocksdb/subscription","commits_url":"https://api.github.com/repos/ajkr/rocksdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/ajkr/rocksdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/ajkr/rocksdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/ajkr/rocksdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/ajkr/rocksdb/contents/{+path}","compare_url":"https://api.github.com/repos/ajkr/rocksdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ajkr/rocksdb/merges","archive_url":"https://api.github.com/repos/ajkr/rocksdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ajkr/rocksdb/downloads","issues_url":"https://api.github.com/repos/ajkr/rocksdb/issues{/number}","pulls_url":"https://api.github.com/repos/ajkr/rocksdb/pulls{/number}","milestones_url":"https://api.github.com/repos/ajkr/rocksdb/milestones{/number}","notifications_url":"https://api.github.com/repos/ajkr/rocksdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ajkr/rocksdb/labels{/name}","releases_url":"https://api.github.com/repos/ajkr/rocksdb/releases{/id}","deployments_url":"https://api.github.com/repos/ajkr/rocksdb/deployments","created_at":"2016-09-30T16:09:32Z","updated_at":"2016-09-30T16:09:37Z","pushed_at":"2018-04-11T00:58:17Z","git_url":"git://github.com/ajkr/rocksdb.git","ssh_url":"git@github.com:ajkr/rocksdb.git","clone_url":"https://github.com/ajkr/rocksdb.git","svn_url":"https://github.com/ajkr/rocksdb","homepage":"http://rocksdb.org","size":136618,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"facebook:master","ref":"master","sha":"d12112d05e20b13fb030fcc3ae043b58e8d74286","user":{"login":"facebook","id":69631,"avatar_url":"https://avatars3.githubusercontent.com/u/69631?v=4","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"repo":{"id":6934395,"name":"rocksdb","full_name":"facebook/rocksdb","owner":{"login":"facebook","id":69631,"avatar_url":"https://avatars3.githubusercontent.com/u/69631?v=4","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/facebook/rocksdb","description":"A library that provides an embeddable, persistent key-value store for fast storage.","fork":false,"url":"https://api.github.com/repos/facebook/rocksdb","forks_url":"https://api.github.com/repos/facebook/rocksdb/forks","keys_url":"https://api.github.com/repos/facebook/rocksdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/facebook/rocksdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/facebook/rocksdb/teams","hooks_url":"https://api.github.com/repos/facebook/rocksdb/hooks","issue_events_url":"https://api.github.com/repos/facebook/rocksdb/issues/events{/number}","events_url":"https://api.github.com/repos/facebook/rocksdb/events","assignees_url":"https://api.github.com/repos/facebook/rocksdb/assignees{/user}","branches_url":"https://api.github.com/repos/facebook/rocksdb/branches{/branch}","tags_url":"https://api.github.com/repos/facebook/rocksdb/tags","blobs_url":"https://api.github.com/repos/facebook/rocksdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/facebook/rocksdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/facebook/rocksdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/facebook/rocksdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/facebook/rocksdb/statuses/{sha}","languages_url":"https://api.github.com/repos/facebook/rocksdb/languages","stargazers_url":"https://api.github.com/repos/facebook/rocksdb/stargazers","contributors_url":"https://api.github.com/repos/facebook/rocksdb/contributors","subscribers_url":"https://api.github.com/repos/facebook/rocksdb/subscribers","subscription_url":"https://api.github.com/repos/facebook/rocksdb/subscription","commits_url":"https://api.github.com/repos/facebook/rocksdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/facebook/rocksdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/facebook/rocksdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/facebook/rocksdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/facebook/rocksdb/contents/{+path}","compare_url":"https://api.github.com/repos/facebook/rocksdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/facebook/rocksdb/merges","archive_url":"https://api.github.com/repos/facebook/rocksdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/facebook/rocksdb/downloads","issues_url":"https://api.github.com/repos/facebook/rocksdb/issues{/number}","pulls_url":"https://api.github.com/repos/facebook/rocksdb/pulls{/number}","milestones_url":"https://api.github.com/repos/facebook/rocksdb/milestones{/number}","notifications_url":"https://api.github.com/repos/facebook/rocksdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/facebook/rocksdb/labels{/name}","releases_url":"https://api.github.com/repos/facebook/rocksdb/releases{/id}","deployments_url":"https://api.github.com/repos/facebook/rocksdb/deployments","created_at":"2012-11-30T06:16:18Z","updated_at":"2018-04-11T02:02:34Z","pushed_at":"2018-04-11T02:02:26Z","git_url":"git://github.com/facebook/rocksdb.git","ssh_url":"git@github.com:facebook/rocksdb.git","clone_url":"https://github.com/facebook/rocksdb.git","svn_url":"https://github.com/facebook/rocksdb","homepage":"http://rocksdb.org","size":112524,"stargazers_count":10275,"watchers_count":10275,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2258,"mirror_url":null,"archived":false,"open_issues_count":157,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":2258,"open_issues":157,"watchers":10275,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3629"},"html":{"href":"https://github.com/facebook/rocksdb/pull/3629"},"issue":{"href":"https://api.github.com/repos/facebook/rocksdb/issues/3629"},"comments":{"href":"https://api.github.com/repos/facebook/rocksdb/issues/3629/comments"},"review_comments":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3629/comments"},"review_comment":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3629/commits"},"statuses":{"href":"https://api.github.com/repos/facebook/rocksdb/statuses/05f1419c19ad367441f185aa2875ce6df99b7b26"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"ajkr"}}