{"action":"created","comment":{"url":"https://api.github.com/repos/ceph/ceph/pulls/comments/70664935","id":70664935,"diff_hunk":"@@ -0,0 +1,117 @@\n+==============================\n+Multi-Object Write Transaction\n+==============================\n+\n+Multi-object write transaction enables a group of write operations \n+on multi-objects complete atomically.\n+\n+Description of the Process\n+--------------------------\n+\n+The high level process is,\n+\n+  1. Define the following class,\n+\n+     class *CEPH_RADOS_API MultiObjectWriteOperation*\n+\n+     {\n+\n+       std::map<std::string, ObjectWriteOperation*> slaves;\n+\n+       ObjectWriteOperation master;\n+\n+     };\n+\n+     based on this, client could send a group of ops on multi-objects \n+     through *MOSDOp* to the primary osd in charge of the *master* object \n+     which is explicitly specified by client, call this osd *MASTER*\n+\n+  #. *MASTER* receives the multi-object write operation request, performs \n+     dead lock check (detailed below), extracts *ObjectWriteOperations* \n+     by objects from *slaves*, and sends them to corresponding primary \n+     osds individually, calls those osds *SLAVE*, the transaction \n+     enters the *PREPARE* state\n+\n+  #. *SLAVE* receives *MOSDOp* from *MASTER*, note each op received by *SLAVE*\n+     is a group of write operations on a single slave object. *SLAVE* \n+     performs dead lock check, does the normal *prepare_transaction()* stuff, \n+     if everything is ok, *SLAVE* will lock the object, and delay further \n+     read/write operations on this object until the transaction finishs. \n+     *SLAVE* sends a *QUERY* message to *MASTER*, and returns *MASTER* \n+     *EINPROGRESS*. The *QUERY* message is used for *MASTER* to notify *SLAVE* \n+     when to release transaction context, *MASTER* will not ack *QUERY* until \n+     itself has released transaction context\n+\n+  #. *MASTER* collects response from *SLAVE*, if all *EINPROGRESS*, then sends\n+     *SLAVE* *LOG* message, the transaction enters the *LOG* state\n+     \n+  #. *SLAVE* receives *LOG*, constructs a write op which persists the \n+     multi-object operation information in the object attribute named \n+     *MOWO_ATTR*, and also synchronizes the op within its replica osds, \n+     utilizing the present single object write transaction mechanism, then \n+     returns *MASTER* zero\n+     \n+  #. *MASTER* collects response from *SLAVE*, if all zero, persists the \n+     multi-object operation information itself, then performs the write op on \n+     the master object, and synchronizes the op within its replica osds, return \n+     client *ACK*\n+     \n+  #. *MASTER* sends *SLAVE* *COMMIT & UNLOCK* message, the transaction enters \n+     the *COMMIT* state\n+     \n+  #. *SLAVE* receives *COMMIT & UNLOCK* message, first processes *COMMIT*, \n+     performs the write op on slave object, and synchronizes the op within its\n+     replica osds, returns *MASTER* zero, *SLAVE* processes *UNLOCK*, cleans up \n+     the transaction and deletes the persisted object attribute\n+     \n+  #. *MASTER* collects response from *SLAVE*, if all zero, clean up the \n+     transaction and deletes the persisted object attribute, acks *SLAVE* \n+     *QUERY* message\n+     \n+     \n+Fault Tolerance\n+---------------\n+\n+- If *SLAVE* down at any time before persisting transaction to object \n+  attribute, when *MASTER* resends *PREPARE* or (re)sends *LOG*, since *SLAVE* \n+  has lost the in-memory transaction information, it will return *EPERM*. If \n+  *MASTER* receives *EPERM*, it recognizes the situation and redo *PREPARE* for \n+  this *SLAVE* to resend the operation to it to proceed\n+\n+- If *MASTER* down at any time before persisting transaction to object \n+  attribute, when client resends the request, it will reconstruct the \n+  transaction, and resends *PREPARE* and *LOG* to *SLAVE*, if *SLAVE* is alive, \n+  it will recognize these resent messages, does nothing and replies with \n+  correct response to proceed; If *SLAVE* also restarts (for example, share the \n+  same osd with *MASTER*), it will return *EPERM* and *MASTER* will resends \n+  *PREPARE* which falls back to the above situation\n+\n+- If *SLAVE* down at any time after persisting transaction to object attribute, \n+  when *MASTER* resends *LOG*/sends *COMMIT*, *SLAVE* will reconstruct the \n+  transaction from the object attribute and proceed. If *SLAVE* down during \n+  *UNLOCK*, when *MASTER* finishes transaction, it will ack *QUERY* to the \n+  *SLAVE* to release the transaction to proceed\n+\n+- If *MASTER* down at any time after persisting transaction to object \n+  attribute, when client resends request/*SLAVE* resends *QUERY*, *MASTER* will \n+  reconstruct the transaction from the object attribute and send *COMMIT* to \n+  *SLAVE* to proceed\n+","path":"doc/dev/osd_internals/multi_object_write_transaction.rst","position":99,"original_position":99,"commit_id":"7dfbeb0a8adfac06e141501797aca019ea5964d4","original_commit_id":"7dfbeb0a8adfac06e141501797aca019ea5964d4","user":{"login":"athanatos","id":544782,"avatar_url":"https://avatars.githubusercontent.com/u/544782?v=3","gravatar_id":"","url":"https://api.github.com/users/athanatos","html_url":"https://github.com/athanatos","followers_url":"https://api.github.com/users/athanatos/followers","following_url":"https://api.github.com/users/athanatos/following{/other_user}","gists_url":"https://api.github.com/users/athanatos/gists{/gist_id}","starred_url":"https://api.github.com/users/athanatos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/athanatos/subscriptions","organizations_url":"https://api.github.com/users/athanatos/orgs","repos_url":"https://api.github.com/users/athanatos/repos","events_url":"https://api.github.com/users/athanatos/events{/privacy}","received_events_url":"https://api.github.com/users/athanatos/received_events","type":"User","site_admin":false},"body":"Clients don't undergo interval changes, they just die.  The slave can't be continuously sending queries to the master, something would have to trigger it.  It also can't merely do it when it gets an IO on the object -- it would either be redundant or inadequate (what if the master dies after the blocked IO arrives but before it completes the transaction?)  Does the slave monitor osdmap changes to detect an interval change on the primary PG for all operations for which it is the slave?  If so, that seems like it duplicates a great deal of the Objecter.  Does it poll?  If so, at what interval?","created_at":"2016-07-13T16:58:27Z","updated_at":"2016-07-13T16:58:27Z","html_url":"https://github.com/ceph/ceph/pull/9398#discussion_r70664935","pull_request_url":"https://api.github.com/repos/ceph/ceph/pulls/9398","_links":{"self":{"href":"https://api.github.com/repos/ceph/ceph/pulls/comments/70664935"},"html":{"href":"https://github.com/ceph/ceph/pull/9398#discussion_r70664935"},"pull_request":{"href":"https://api.github.com/repos/ceph/ceph/pulls/9398"}}},"pull_request":{"url":"https://api.github.com/repos/ceph/ceph/pulls/9398","id":71905360,"html_url":"https://github.com/ceph/ceph/pull/9398","diff_url":"https://github.com/ceph/ceph/pull/9398.diff","patch_url":"https://github.com/ceph/ceph/pull/9398.patch","issue_url":"https://api.github.com/repos/ceph/ceph/issues/9398","number":9398,"state":"open","locked":false,"title":"Wip multi object write transaction support","user":{"login":"dragonylffly","id":4452507,"avatar_url":"https://avatars.githubusercontent.com/u/4452507?v=3","gravatar_id":"","url":"https://api.github.com/users/dragonylffly","html_url":"https://github.com/dragonylffly","followers_url":"https://api.github.com/users/dragonylffly/followers","following_url":"https://api.github.com/users/dragonylffly/following{/other_user}","gists_url":"https://api.github.com/users/dragonylffly/gists{/gist_id}","starred_url":"https://api.github.com/users/dragonylffly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dragonylffly/subscriptions","organizations_url":"https://api.github.com/users/dragonylffly/orgs","repos_url":"https://api.github.com/users/dragonylffly/repos","events_url":"https://api.github.com/users/dragonylffly/events{/privacy}","received_events_url":"https://api.github.com/users/dragonylffly/received_events","type":"User","site_admin":false},"body":"This patch implements multi-object write transaction. It is originally proposed by Sage at CDS for \r\nInfernalis (http://tracker.ceph.com/projects/ceph/wiki/Osd_-_Transactions), with more discussion at  CDS for Jewel (http://tracker.ceph.com/projects/ceph/wiki/Rados_-_multi-object_transaction_support). The support for multi-object write transaction at RADOS could potentially greatly simplify the client design, such as CephFS, which can expect many file system calls finish atomically, eliminating the need of journals. Similar idea is presented at  https://www.usenix.org/conference/fast16/technical-sessions/presentation/shin. \r\n\r\nIt is theoretically fully on top of the current RADOS mechanism, and in terms of implementation, it keeps the minimal intrusion to the ReplicatedPG code.\r\n\r\nsrc/test/librados/multiobjectwriteoperation.cc gives the usage example, and ceph_test_rados_api_multiobjectwriteoperation runs the test.\r\n","created_at":"2016-05-30T13:59:17Z","updated_at":"2016-07-13T16:58:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"dab020ed73ebdce1d03344d1a4704b0970fe65fe","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/ceph/ceph/pulls/9398/commits","review_comments_url":"https://api.github.com/repos/ceph/ceph/pulls/9398/comments","review_comment_url":"https://api.github.com/repos/ceph/ceph/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ceph/ceph/issues/9398/comments","statuses_url":"https://api.github.com/repos/ceph/ceph/statuses/7dfbeb0a8adfac06e141501797aca019ea5964d4","head":{"label":"kylinstorage:wip-multi-object-write-transaction","ref":"wip-multi-object-write-transaction","sha":"7dfbeb0a8adfac06e141501797aca019ea5964d4","user":{"login":"kylinstorage","id":15884581,"avatar_url":"https://avatars.githubusercontent.com/u/15884581?v=3","gravatar_id":"","url":"https://api.github.com/users/kylinstorage","html_url":"https://github.com/kylinstorage","followers_url":"https://api.github.com/users/kylinstorage/followers","following_url":"https://api.github.com/users/kylinstorage/following{/other_user}","gists_url":"https://api.github.com/users/kylinstorage/gists{/gist_id}","starred_url":"https://api.github.com/users/kylinstorage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylinstorage/subscriptions","organizations_url":"https://api.github.com/users/kylinstorage/orgs","repos_url":"https://api.github.com/users/kylinstorage/repos","events_url":"https://api.github.com/users/kylinstorage/events{/privacy}","received_events_url":"https://api.github.com/users/kylinstorage/received_events","type":"Organization","site_admin":false},"repo":{"id":46329840,"name":"ceph","full_name":"kylinstorage/ceph","owner":{"login":"kylinstorage","id":15884581,"avatar_url":"https://avatars.githubusercontent.com/u/15884581?v=3","gravatar_id":"","url":"https://api.github.com/users/kylinstorage","html_url":"https://github.com/kylinstorage","followers_url":"https://api.github.com/users/kylinstorage/followers","following_url":"https://api.github.com/users/kylinstorage/following{/other_user}","gists_url":"https://api.github.com/users/kylinstorage/gists{/gist_id}","starred_url":"https://api.github.com/users/kylinstorage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylinstorage/subscriptions","organizations_url":"https://api.github.com/users/kylinstorage/orgs","repos_url":"https://api.github.com/users/kylinstorage/repos","events_url":"https://api.github.com/users/kylinstorage/events{/privacy}","received_events_url":"https://api.github.com/users/kylinstorage/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kylinstorage/ceph","description":"Ceph is a distributed object, block, and file storage platform ","fork":true,"url":"https://api.github.com/repos/kylinstorage/ceph","forks_url":"https://api.github.com/repos/kylinstorage/ceph/forks","keys_url":"https://api.github.com/repos/kylinstorage/ceph/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kylinstorage/ceph/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kylinstorage/ceph/teams","hooks_url":"https://api.github.com/repos/kylinstorage/ceph/hooks","issue_events_url":"https://api.github.com/repos/kylinstorage/ceph/issues/events{/number}","events_url":"https://api.github.com/repos/kylinstorage/ceph/events","assignees_url":"https://api.github.com/repos/kylinstorage/ceph/assignees{/user}","branches_url":"https://api.github.com/repos/kylinstorage/ceph/branches{/branch}","tags_url":"https://api.github.com/repos/kylinstorage/ceph/tags","blobs_url":"https://api.github.com/repos/kylinstorage/ceph/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kylinstorage/ceph/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kylinstorage/ceph/git/refs{/sha}","trees_url":"https://api.github.com/repos/kylinstorage/ceph/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kylinstorage/ceph/statuses/{sha}","languages_url":"https://api.github.com/repos/kylinstorage/ceph/languages","stargazers_url":"https://api.github.com/repos/kylinstorage/ceph/stargazers","contributors_url":"https://api.github.com/repos/kylinstorage/ceph/contributors","subscribers_url":"https://api.github.com/repos/kylinstorage/ceph/subscribers","subscription_url":"https://api.github.com/repos/kylinstorage/ceph/subscription","commits_url":"https://api.github.com/repos/kylinstorage/ceph/commits{/sha}","git_commits_url":"https://api.github.com/repos/kylinstorage/ceph/git/commits{/sha}","comments_url":"https://api.github.com/repos/kylinstorage/ceph/comments{/number}","issue_comment_url":"https://api.github.com/repos/kylinstorage/ceph/issues/comments{/number}","contents_url":"https://api.github.com/repos/kylinstorage/ceph/contents/{+path}","compare_url":"https://api.github.com/repos/kylinstorage/ceph/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kylinstorage/ceph/merges","archive_url":"https://api.github.com/repos/kylinstorage/ceph/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kylinstorage/ceph/downloads","issues_url":"https://api.github.com/repos/kylinstorage/ceph/issues{/number}","pulls_url":"https://api.github.com/repos/kylinstorage/ceph/pulls{/number}","milestones_url":"https://api.github.com/repos/kylinstorage/ceph/milestones{/number}","notifications_url":"https://api.github.com/repos/kylinstorage/ceph/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kylinstorage/ceph/labels{/name}","releases_url":"https://api.github.com/repos/kylinstorage/ceph/releases{/id}","deployments_url":"https://api.github.com/repos/kylinstorage/ceph/deployments","created_at":"2015-11-17T07:15:48Z","updated_at":"2015-11-17T07:16:27Z","pushed_at":"2016-06-30T14:04:23Z","git_url":"git://github.com/kylinstorage/ceph.git","ssh_url":"git@github.com:kylinstorage/ceph.git","clone_url":"https://github.com/kylinstorage/ceph.git","svn_url":"https://github.com/kylinstorage/ceph","homepage":"http://ceph.com","size":136644,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"ceph:master","ref":"master","sha":"f2c2c29c840bae85c1089655a8882dab4b131d79","user":{"login":"ceph","id":1015767,"avatar_url":"https://avatars.githubusercontent.com/u/1015767?v=3","gravatar_id":"","url":"https://api.github.com/users/ceph","html_url":"https://github.com/ceph","followers_url":"https://api.github.com/users/ceph/followers","following_url":"https://api.github.com/users/ceph/following{/other_user}","gists_url":"https://api.github.com/users/ceph/gists{/gist_id}","starred_url":"https://api.github.com/users/ceph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ceph/subscriptions","organizations_url":"https://api.github.com/users/ceph/orgs","repos_url":"https://api.github.com/users/ceph/repos","events_url":"https://api.github.com/users/ceph/events{/privacy}","received_events_url":"https://api.github.com/users/ceph/received_events","type":"Organization","site_admin":false},"repo":{"id":2310495,"name":"ceph","full_name":"ceph/ceph","owner":{"login":"ceph","id":1015767,"avatar_url":"https://avatars.githubusercontent.com/u/1015767?v=3","gravatar_id":"","url":"https://api.github.com/users/ceph","html_url":"https://github.com/ceph","followers_url":"https://api.github.com/users/ceph/followers","following_url":"https://api.github.com/users/ceph/following{/other_user}","gists_url":"https://api.github.com/users/ceph/gists{/gist_id}","starred_url":"https://api.github.com/users/ceph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ceph/subscriptions","organizations_url":"https://api.github.com/users/ceph/orgs","repos_url":"https://api.github.com/users/ceph/repos","events_url":"https://api.github.com/users/ceph/events{/privacy}","received_events_url":"https://api.github.com/users/ceph/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ceph/ceph","description":"Ceph is a distributed object, block, and file storage platform ","fork":false,"url":"https://api.github.com/repos/ceph/ceph","forks_url":"https://api.github.com/repos/ceph/ceph/forks","keys_url":"https://api.github.com/repos/ceph/ceph/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ceph/ceph/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ceph/ceph/teams","hooks_url":"https://api.github.com/repos/ceph/ceph/hooks","issue_events_url":"https://api.github.com/repos/ceph/ceph/issues/events{/number}","events_url":"https://api.github.com/repos/ceph/ceph/events","assignees_url":"https://api.github.com/repos/ceph/ceph/assignees{/user}","branches_url":"https://api.github.com/repos/ceph/ceph/branches{/branch}","tags_url":"https://api.github.com/repos/ceph/ceph/tags","blobs_url":"https://api.github.com/repos/ceph/ceph/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ceph/ceph/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ceph/ceph/git/refs{/sha}","trees_url":"https://api.github.com/repos/ceph/ceph/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ceph/ceph/statuses/{sha}","languages_url":"https://api.github.com/repos/ceph/ceph/languages","stargazers_url":"https://api.github.com/repos/ceph/ceph/stargazers","contributors_url":"https://api.github.com/repos/ceph/ceph/contributors","subscribers_url":"https://api.github.com/repos/ceph/ceph/subscribers","subscription_url":"https://api.github.com/repos/ceph/ceph/subscription","commits_url":"https://api.github.com/repos/ceph/ceph/commits{/sha}","git_commits_url":"https://api.github.com/repos/ceph/ceph/git/commits{/sha}","comments_url":"https://api.github.com/repos/ceph/ceph/comments{/number}","issue_comment_url":"https://api.github.com/repos/ceph/ceph/issues/comments{/number}","contents_url":"https://api.github.com/repos/ceph/ceph/contents/{+path}","compare_url":"https://api.github.com/repos/ceph/ceph/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ceph/ceph/merges","archive_url":"https://api.github.com/repos/ceph/ceph/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ceph/ceph/downloads","issues_url":"https://api.github.com/repos/ceph/ceph/issues{/number}","pulls_url":"https://api.github.com/repos/ceph/ceph/pulls{/number}","milestones_url":"https://api.github.com/repos/ceph/ceph/milestones{/number}","notifications_url":"https://api.github.com/repos/ceph/ceph/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ceph/ceph/labels{/name}","releases_url":"https://api.github.com/repos/ceph/ceph/releases{/id}","deployments_url":"https://api.github.com/repos/ceph/ceph/deployments","created_at":"2011-09-01T21:41:26Z","updated_at":"2016-07-13T08:34:58Z","pushed_at":"2016-07-13T16:52:20Z","git_url":"git://github.com/ceph/ceph.git","ssh_url":"git@github.com:ceph/ceph.git","clone_url":"https://github.com/ceph/ceph.git","svn_url":"https://github.com/ceph/ceph","homepage":"http://ceph.com","size":226883,"stargazers_count":2301,"watchers_count":2301,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1365,"mirror_url":null,"open_issues_count":481,"forks":1365,"open_issues":481,"watchers":2301,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ceph/ceph/pulls/9398"},"html":{"href":"https://github.com/ceph/ceph/pull/9398"},"issue":{"href":"https://api.github.com/repos/ceph/ceph/issues/9398"},"comments":{"href":"https://api.github.com/repos/ceph/ceph/issues/9398/comments"},"review_comments":{"href":"https://api.github.com/repos/ceph/ceph/pulls/9398/comments"},"review_comment":{"href":"https://api.github.com/repos/ceph/ceph/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ceph/ceph/pulls/9398/commits"},"statuses":{"href":"https://api.github.com/repos/ceph/ceph/statuses/7dfbeb0a8adfac06e141501797aca019ea5964d4"}}}}
{"actor":{"display_login":"athanatos"}}