{"action":"created","comment":{"url":"https://api.github.com/repos/facebook/rocksdb/pulls/comments/212103689","pull_request_review_id":148666009,"id":212103689,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxMjEwMzY4OQ==","diff_hunk":"@@ -1367,40 +1415,53 @@ Status DBImpl::BackgroundFlush(bool* made_progress, JobContext* job_context,\n     return status;\n   }\n \n-  ColumnFamilyData* cfd = nullptr;\n+  autovector<ColumnFamilyData*> cfds;\n+  autovector<MutableCFOptions> mutable_cf_options_list;\n+  autovector<uint64_t> memtable_ids;\n   while (!flush_queue_.empty()) {\n     // This cfd is already referenced\n-    auto first_cfd = PopFirstFromFlushQueue();\n-\n-    if (first_cfd->IsDropped() || !first_cfd->imm()->IsFlushPending()) {\n-      // can't flush this CF, try next one\n-      if (first_cfd->Unref()) {\n-        delete first_cfd;\n+    const auto& flush_req = PopFirstFromFlushQueue();\n+\n+    for (auto& iter : flush_req) {\n+      auto cfd = iter.first;\n+      if (cfd->IsDropped() || !cfd->imm()->IsFlushPending()) {\n+        // can't flush this CF, try next one\n+        if (cfd->Unref()) {\n+          delete cfd;\n+        }\n+        continue;\n       }\n-      continue;\n+      cfds.push_back(cfd);\n+      mutable_cf_options_list.emplace_back(*cfd->GetLatestMutableCFOptions());\n+      memtable_ids.push_back(iter.second);\n+      job_context->superversion_contexts.emplace_back(","path":"db/db_impl_compaction_flush.cc","position":null,"original_position":243,"commit_id":"79ee80f8a9585cafe7c6d1d52a5afebfa731d854","original_commit_id":"a48ee308acfb42224e45f71fdd15786e19d70a42","user":{"login":"ajkr","id":4780362,"node_id":"MDQ6VXNlcjQ3ODAzNjI=","avatar_url":"https://avatars0.githubusercontent.com/u/4780362?v=4","gravatar_id":"","url":"https://api.github.com/users/ajkr","html_url":"https://github.com/ajkr","followers_url":"https://api.github.com/users/ajkr/followers","following_url":"https://api.github.com/users/ajkr/following{/other_user}","gists_url":"https://api.github.com/users/ajkr/gists{/gist_id}","starred_url":"https://api.github.com/users/ajkr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajkr/subscriptions","organizations_url":"https://api.github.com/users/ajkr/orgs","repos_url":"https://api.github.com/users/ajkr/repos","events_url":"https://api.github.com/users/ajkr/events{/privacy}","received_events_url":"https://api.github.com/users/ajkr/received_events","type":"User","site_admin":false},"body":"Got it thanks. If column family is available already through that chain, can you access it in that way?","created_at":"2018-08-22T20:51:27Z","updated_at":"2018-08-22T20:51:27Z","html_url":"https://github.com/facebook/rocksdb/pull/3952#discussion_r212103689","pull_request_url":"https://api.github.com/repos/facebook/rocksdb/pulls/3952","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/comments/212103689"},"html":{"href":"https://github.com/facebook/rocksdb/pull/3952#discussion_r212103689"},"pull_request":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3952"}},"in_reply_to_id":211757645},"pull_request":{"url":"https://api.github.com/repos/facebook/rocksdb/pulls/3952","id":192554815,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkyNTU0ODE1","html_url":"https://github.com/facebook/rocksdb/pull/3952","diff_url":"https://github.com/facebook/rocksdb/pull/3952.diff","patch_url":"https://github.com/facebook/rocksdb/pull/3952.patch","issue_url":"https://api.github.com/repos/facebook/rocksdb/issues/3952","number":3952,"state":"open","locked":false,"title":"Refactor flush request queueing and processing","user":{"login":"riversand963","id":20094703,"node_id":"MDQ6VXNlcjIwMDk0NzAz","avatar_url":"https://avatars0.githubusercontent.com/u/20094703?v=4","gravatar_id":"","url":"https://api.github.com/users/riversand963","html_url":"https://github.com/riversand963","followers_url":"https://api.github.com/users/riversand963/followers","following_url":"https://api.github.com/users/riversand963/following{/other_user}","gists_url":"https://api.github.com/users/riversand963/gists{/gist_id}","starred_url":"https://api.github.com/users/riversand963/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riversand963/subscriptions","organizations_url":"https://api.github.com/users/riversand963/orgs","repos_url":"https://api.github.com/users/riversand963/repos","events_url":"https://api.github.com/users/riversand963/events{/privacy}","received_events_url":"https://api.github.com/users/riversand963/received_events","type":"User","site_admin":false},"body":"RocksDB used to queue individual column family for flushing. This is not sufficient to support the needs of some applications that want to enforce order/dependency between column families, given that multiple foreground and background activities can trigger flushing in RocksDB.\r\n\r\nThis PR aims to address this limitation. Each flush request is described as a `FlushRequest` that can contain multiple column families. A background flushing thread pops one flush request from the queue at a time and processes it.\r\n\r\nThis PR does not enable atomic_flush yet, but is a subset of [PR 3752](https://github.com/facebook/rocksdb/pull/3752).\r\n\r\nTest plan:\r\n```\r\n$make clean && make -j20 all check\r\n```","created_at":"2018-06-04T23:32:35Z","updated_at":"2018-08-22T20:51:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"cfbe1983d688d80bc564c8e985bb97b9e19624ba","assignee":{"login":"riversand963","id":20094703,"node_id":"MDQ6VXNlcjIwMDk0NzAz","avatar_url":"https://avatars0.githubusercontent.com/u/20094703?v=4","gravatar_id":"","url":"https://api.github.com/users/riversand963","html_url":"https://github.com/riversand963","followers_url":"https://api.github.com/users/riversand963/followers","following_url":"https://api.github.com/users/riversand963/following{/other_user}","gists_url":"https://api.github.com/users/riversand963/gists{/gist_id}","starred_url":"https://api.github.com/users/riversand963/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riversand963/subscriptions","organizations_url":"https://api.github.com/users/riversand963/orgs","repos_url":"https://api.github.com/users/riversand963/repos","events_url":"https://api.github.com/users/riversand963/events{/privacy}","received_events_url":"https://api.github.com/users/riversand963/received_events","type":"User","site_admin":false},"assignees":[{"login":"riversand963","id":20094703,"node_id":"MDQ6VXNlcjIwMDk0NzAz","avatar_url":"https://avatars0.githubusercontent.com/u/20094703?v=4","gravatar_id":"","url":"https://api.github.com/users/riversand963","html_url":"https://github.com/riversand963","followers_url":"https://api.github.com/users/riversand963/followers","following_url":"https://api.github.com/users/riversand963/following{/other_user}","gists_url":"https://api.github.com/users/riversand963/gists{/gist_id}","starred_url":"https://api.github.com/users/riversand963/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riversand963/subscriptions","organizations_url":"https://api.github.com/users/riversand963/orgs","repos_url":"https://api.github.com/users/riversand963/repos","events_url":"https://api.github.com/users/riversand963/events{/privacy}","received_events_url":"https://api.github.com/users/riversand963/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":196859108,"node_id":"MDU6TGFiZWwxOTY4NTkxMDg=","url":"https://api.github.com/repos/facebook/rocksdb/labels/CLA%20Signed","name":"CLA Signed","color":"009900","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/facebook/rocksdb/pulls/3952/commits","review_comments_url":"https://api.github.com/repos/facebook/rocksdb/pulls/3952/comments","review_comment_url":"https://api.github.com/repos/facebook/rocksdb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/3952/comments","statuses_url":"https://api.github.com/repos/facebook/rocksdb/statuses/79ee80f8a9585cafe7c6d1d52a5afebfa731d854","head":{"label":"riversand963:atomic_flush_foreground","ref":"atomic_flush_foreground","sha":"79ee80f8a9585cafe7c6d1d52a5afebfa731d854","user":{"login":"riversand963","id":20094703,"node_id":"MDQ6VXNlcjIwMDk0NzAz","avatar_url":"https://avatars0.githubusercontent.com/u/20094703?v=4","gravatar_id":"","url":"https://api.github.com/users/riversand963","html_url":"https://github.com/riversand963","followers_url":"https://api.github.com/users/riversand963/followers","following_url":"https://api.github.com/users/riversand963/following{/other_user}","gists_url":"https://api.github.com/users/riversand963/gists{/gist_id}","starred_url":"https://api.github.com/users/riversand963/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riversand963/subscriptions","organizations_url":"https://api.github.com/users/riversand963/orgs","repos_url":"https://api.github.com/users/riversand963/repos","events_url":"https://api.github.com/users/riversand963/events{/privacy}","received_events_url":"https://api.github.com/users/riversand963/received_events","type":"User","site_admin":false},"repo":{"id":124924567,"node_id":"MDEwOlJlcG9zaXRvcnkxMjQ5MjQ1Njc=","name":"rocksdb","full_name":"riversand963/rocksdb","owner":{"login":"riversand963","id":20094703,"node_id":"MDQ6VXNlcjIwMDk0NzAz","avatar_url":"https://avatars0.githubusercontent.com/u/20094703?v=4","gravatar_id":"","url":"https://api.github.com/users/riversand963","html_url":"https://github.com/riversand963","followers_url":"https://api.github.com/users/riversand963/followers","following_url":"https://api.github.com/users/riversand963/following{/other_user}","gists_url":"https://api.github.com/users/riversand963/gists{/gist_id}","starred_url":"https://api.github.com/users/riversand963/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riversand963/subscriptions","organizations_url":"https://api.github.com/users/riversand963/orgs","repos_url":"https://api.github.com/users/riversand963/repos","events_url":"https://api.github.com/users/riversand963/events{/privacy}","received_events_url":"https://api.github.com/users/riversand963/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/riversand963/rocksdb","description":"A library that provides an embeddable, persistent key-value store for fast storage.","fork":true,"url":"https://api.github.com/repos/riversand963/rocksdb","forks_url":"https://api.github.com/repos/riversand963/rocksdb/forks","keys_url":"https://api.github.com/repos/riversand963/rocksdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/riversand963/rocksdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/riversand963/rocksdb/teams","hooks_url":"https://api.github.com/repos/riversand963/rocksdb/hooks","issue_events_url":"https://api.github.com/repos/riversand963/rocksdb/issues/events{/number}","events_url":"https://api.github.com/repos/riversand963/rocksdb/events","assignees_url":"https://api.github.com/repos/riversand963/rocksdb/assignees{/user}","branches_url":"https://api.github.com/repos/riversand963/rocksdb/branches{/branch}","tags_url":"https://api.github.com/repos/riversand963/rocksdb/tags","blobs_url":"https://api.github.com/repos/riversand963/rocksdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/riversand963/rocksdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/riversand963/rocksdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/riversand963/rocksdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/riversand963/rocksdb/statuses/{sha}","languages_url":"https://api.github.com/repos/riversand963/rocksdb/languages","stargazers_url":"https://api.github.com/repos/riversand963/rocksdb/stargazers","contributors_url":"https://api.github.com/repos/riversand963/rocksdb/contributors","subscribers_url":"https://api.github.com/repos/riversand963/rocksdb/subscribers","subscription_url":"https://api.github.com/repos/riversand963/rocksdb/subscription","commits_url":"https://api.github.com/repos/riversand963/rocksdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/riversand963/rocksdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/riversand963/rocksdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/riversand963/rocksdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/riversand963/rocksdb/contents/{+path}","compare_url":"https://api.github.com/repos/riversand963/rocksdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/riversand963/rocksdb/merges","archive_url":"https://api.github.com/repos/riversand963/rocksdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/riversand963/rocksdb/downloads","issues_url":"https://api.github.com/repos/riversand963/rocksdb/issues{/number}","pulls_url":"https://api.github.com/repos/riversand963/rocksdb/pulls{/number}","milestones_url":"https://api.github.com/repos/riversand963/rocksdb/milestones{/number}","notifications_url":"https://api.github.com/repos/riversand963/rocksdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/riversand963/rocksdb/labels{/name}","releases_url":"https://api.github.com/repos/riversand963/rocksdb/releases{/id}","deployments_url":"https://api.github.com/repos/riversand963/rocksdb/deployments","created_at":"2018-03-12T17:16:33Z","updated_at":"2018-08-22T04:09:09Z","pushed_at":"2018-08-22T16:59:33Z","git_url":"git://github.com/riversand963/rocksdb.git","ssh_url":"git@github.com:riversand963/rocksdb.git","clone_url":"https://github.com/riversand963/rocksdb.git","svn_url":"https://github.com/riversand963/rocksdb","homepage":"http://rocksdb.org","size":98329,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"facebook:master","ref":"master","sha":"eb8885a08a688e08131c2a7acb94bca746de6789","user":{"login":"facebook","id":69631,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY5NjMx","avatar_url":"https://avatars3.githubusercontent.com/u/69631?v=4","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"repo":{"id":6934395,"node_id":"MDEwOlJlcG9zaXRvcnk2OTM0Mzk1","name":"rocksdb","full_name":"facebook/rocksdb","owner":{"login":"facebook","id":69631,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY5NjMx","avatar_url":"https://avatars3.githubusercontent.com/u/69631?v=4","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/facebook/rocksdb","description":"A library that provides an embeddable, persistent key-value store for fast storage.","fork":false,"url":"https://api.github.com/repos/facebook/rocksdb","forks_url":"https://api.github.com/repos/facebook/rocksdb/forks","keys_url":"https://api.github.com/repos/facebook/rocksdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/facebook/rocksdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/facebook/rocksdb/teams","hooks_url":"https://api.github.com/repos/facebook/rocksdb/hooks","issue_events_url":"https://api.github.com/repos/facebook/rocksdb/issues/events{/number}","events_url":"https://api.github.com/repos/facebook/rocksdb/events","assignees_url":"https://api.github.com/repos/facebook/rocksdb/assignees{/user}","branches_url":"https://api.github.com/repos/facebook/rocksdb/branches{/branch}","tags_url":"https://api.github.com/repos/facebook/rocksdb/tags","blobs_url":"https://api.github.com/repos/facebook/rocksdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/facebook/rocksdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/facebook/rocksdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/facebook/rocksdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/facebook/rocksdb/statuses/{sha}","languages_url":"https://api.github.com/repos/facebook/rocksdb/languages","stargazers_url":"https://api.github.com/repos/facebook/rocksdb/stargazers","contributors_url":"https://api.github.com/repos/facebook/rocksdb/contributors","subscribers_url":"https://api.github.com/repos/facebook/rocksdb/subscribers","subscription_url":"https://api.github.com/repos/facebook/rocksdb/subscription","commits_url":"https://api.github.com/repos/facebook/rocksdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/facebook/rocksdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/facebook/rocksdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/facebook/rocksdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/facebook/rocksdb/contents/{+path}","compare_url":"https://api.github.com/repos/facebook/rocksdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/facebook/rocksdb/merges","archive_url":"https://api.github.com/repos/facebook/rocksdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/facebook/rocksdb/downloads","issues_url":"https://api.github.com/repos/facebook/rocksdb/issues{/number}","pulls_url":"https://api.github.com/repos/facebook/rocksdb/pulls{/number}","milestones_url":"https://api.github.com/repos/facebook/rocksdb/milestones{/number}","notifications_url":"https://api.github.com/repos/facebook/rocksdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/facebook/rocksdb/labels{/name}","releases_url":"https://api.github.com/repos/facebook/rocksdb/releases{/id}","deployments_url":"https://api.github.com/repos/facebook/rocksdb/deployments","created_at":"2012-11-30T06:16:18Z","updated_at":"2018-08-22T20:47:52Z","pushed_at":"2018-08-22T18:33:55Z","git_url":"git://github.com/facebook/rocksdb.git","ssh_url":"git@github.com:facebook/rocksdb.git","clone_url":"https://github.com/facebook/rocksdb.git","svn_url":"https://github.com/facebook/rocksdb","homepage":"http://rocksdb.org","size":117547,"stargazers_count":11345,"watchers_count":11345,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2488,"mirror_url":null,"archived":false,"open_issues_count":207,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":2488,"open_issues":207,"watchers":11345,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3952"},"html":{"href":"https://github.com/facebook/rocksdb/pull/3952"},"issue":{"href":"https://api.github.com/repos/facebook/rocksdb/issues/3952"},"comments":{"href":"https://api.github.com/repos/facebook/rocksdb/issues/3952/comments"},"review_comments":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3952/comments"},"review_comment":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/facebook/rocksdb/pulls/3952/commits"},"statuses":{"href":"https://api.github.com/repos/facebook/rocksdb/statuses/79ee80f8a9585cafe7c6d1d52a5afebfa731d854"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"ajkr"}}