{"action":"created","comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments/93757469","pull_request_review_id":14289524,"id":93757469,"diff_hunk":"@@ -0,0 +1,136 @@\n+\n+[[docs-replication]]\n+== Reading and Writing documents\n+\n+=== Introduction\n+\n+Each index in Elasticsearch is <<getting-started-shards-and-replicas,divided into shards>>\n+and each shard can have multiple copies. These copies are known as `replication group` and must be kept in sync when documents\n+are added to and removed. If we fail to do so, reading from one copy will result in very different results than reading from another.\n+The process of keeping the shard copies in sync and serving reads from them is what we call the *data replication model*.\n+\n+Elasticsearch’s data replication model is based on the primary-backup model and is described very well in the https://www.microsoft.com/en-us/research/publication/pacifica-replication-in-log-based-distributed-storage-systems/[PacificA paper] of\n+Microsoft Research. That model is based on having a single copy from the replication group that is acting as the primary shard.\n+The other copies are called “replica shards”. The primary serves as the main entry point for all indexing operations. It is in charge of\n+validating them and making sure they are correct. Once an index operation has been accepted by the primary, the primary is also\n+responsible for replicating the operation to the other copies.\n+\n+This purpose of this section is to give a high level overview of the Elasticsearch replication model and discuss the implications\n+it has for various interactions between write and read operations.\n+\n+=== Basic write model\n+\n+Every indexing operation in Elasticsearch is first resolved to a replication group.\n+This is <<index-routing,typically>> done by looking at the document ID. Once the replication group is determined,\n+the operation is internally forwarded to the current primary shard of the group. The primary shard is responsible\n+for validating the operation and forwarding it to the other replicas. Since replicas can be off-line, the primary\n+is not required to replicate to all replicas. Instead, Elasticsearch maintains a list of shard copies that should\n+receive the operation. This list is called the “in-sync copies” and is maintained by the master node. As the name implies,\n+these are the set of \"good\" shard copies that are guaranteed to have processed all of the index and delete operations that\n+have been acknowledged to the user. The primary is responsible for maintaining this invariant and thus has to replicate all\n+operations to each copy in this set.\n+\n+The primary shard follows this basic flow:\n+\n+. Validate incoming operation and reject it if structurally invalid (Example: have an object field where a number is expected)\n+. Execute the operation locally - i.e., indexing or deleting the relevant document. This will also validate the content of fields\n+   and reject if needed (Example: a keyword value is too long for indexing in Lucene).\n+. Forward the operation to each replica in the current in-sync copies set. If there are multiple replicas, this is done concurrently.\n+. Once all replicas have successfully performed the operation and responded to the primary, the primary acknowledges the successful\n+   completion of the request to the client.\n+\n+==== Failure handling\n+\n+Many things can go wrong during indexing - disks can get corrupted, nodes can be disconnected from each other or some\n+configuration mistake could cause an operation to fail on a replica despite it being successful on the primary. These\n+are infrequent but the primary has to respond to them.\n+\n+In the case that the primary itself fails, a message will be sent to the master about it. The indexing operation will\n+wait (up to 1 minute, by [default](link to doc)) for the master to promote one of the replicas to be a new primary.\n+The operation will then be forwarded to the new primary for processing. Note that the master also monitors the health\n+of the nodes and may decide to pro-actively demote a primary. This typically happens when the node holding the primary\n+ is isolated from the cluster by a networking issue. More on this later.\n+\n+Once the operation has been  successfully performed on the primary, the primary has to deal with potential failures\n+when executing it on the replica shards. This may be caused by an actual failure on the replica or due to a network\n+issue preventing the operation from reaching the replica (or preventing the replica from responding). All of these\n+share the same end result - a replica which is part of the in-sync replica set misses an operation that is about to\n+be acknowledged. In order to avoid violating the invariant, the primary sends a message to  the master requesting\n+that the problematic shard be removed from the in-sync replica set. Only once this has successfully been done, does\n+the primary acknowledge the operation. Note that the master will also instruct another node to start building a new\n+shard copy in order to restore the system to a healthy state.\n+\n+While forwarding an operation to the replicas, the primary will use the replicas to validate that it is still the\n+active primary. If the primary has been isolated due to a network partition (or a long GC) it may continue to process\n+ incoming indexing operations before realising that it has been demoted. Operations that come from a stale primary\n+ will be rejected by the replicas. The primary, following its protocol, will then reach out to the master and will\n+ learn that it has been replaced. The operation is then routed to the new primary.\n+\n+What happens if there are no replicas? This is a valid scenario that can happen due to index configuration or simply\n+because the replicas have failed. In that case the primary is processing operations without any external validation,\n+which may seem problematic. On the other hand, the primary cannot fail other shards on its own. This means that the\n+master knows that the primary is the only single good copy. We are therefore guaranteed that the master will not promote\n+any other (out-of-date) shard copy to be a  new primary and that any operation indexed into the primary will not be lost.\n+Of course, since at that point we run with only single copy of the data, physical hardware issues can cause data loss\n+(see <<index-wait-for-active-shards>> for some mitigation options).\n+\n+=== Basic read model\n+\n+Reads in Elasticsearch can be very lightweight lookups by ID or a heavy search request with complex aggregations that\n+take non-trivial CPU power. One of the beauties of the primary-backup model is that it keeps all shard copies identical\n+(with the exception of in-flight operations).. As such, a single copy is sufficient to serve a read requests.\n+\n+When a read request is sent to a node, that node is responsible for forwarding it to the right places, collating the","path":"docs/reference/docs/data-replication.asciidoc","position":83,"original_position":83,"commit_id":"de678b1c04299a674e15ef441c10063679388276","original_commit_id":"de678b1c04299a674e15ef441c10063679388276","user":{"login":"clintongormley","id":56599,"avatar_url":"https://avatars.githubusercontent.com/u/56599?v=3","gravatar_id":"","url":"https://api.github.com/users/clintongormley","html_url":"https://github.com/clintongormley","followers_url":"https://api.github.com/users/clintongormley/followers","following_url":"https://api.github.com/users/clintongormley/following{/other_user}","gists_url":"https://api.github.com/users/clintongormley/gists{/gist_id}","starred_url":"https://api.github.com/users/clintongormley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clintongormley/subscriptions","organizations_url":"https://api.github.com/users/clintongormley/orgs","repos_url":"https://api.github.com/users/clintongormley/repos","events_url":"https://api.github.com/users/clintongormley/events{/privacy}","received_events_url":"https://api.github.com/users/clintongormley/received_events","type":"User","site_admin":false},"body":"When a read request from a client is received by a node\r\nright places -> nodes that hold the relevant shards","created_at":"2016-12-23T11:50:50Z","updated_at":"2016-12-23T12:01:19Z","html_url":"https://github.com/elastic/elasticsearch/pull/22327#discussion_r93757469","pull_request_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/22327","_links":{"self":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments/93757469"},"html":{"href":"https://github.com/elastic/elasticsearch/pull/22327#discussion_r93757469"},"pull_request":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/22327"}}},"pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/22327","id":99101904,"html_url":"https://github.com/elastic/elasticsearch/pull/22327","diff_url":"https://github.com/elastic/elasticsearch/pull/22327.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/22327.patch","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/22327","number":22327,"state":"open","locked":false,"title":"Add a high level description of ES's data replication model","user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"body":"This commit adds a document describing our data replication model in high level terms. The goal is give people basic insight into how things work in order to better understand how read and writes interact, both during normal operations and under failures.\r\n","created_at":"2016-12-22T13:11:01Z","updated_at":"2016-12-23T12:01:20Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9b68245d7e1c7795ea45c89c513fae715989eb2b","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/22327/commits","review_comments_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/22327/comments","review_comment_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/22327/comments","statuses_url":"https://api.github.com/repos/elastic/elasticsearch/statuses/de678b1c04299a674e15ef441c10063679388276","head":{"label":"bleskes:replication_docs","ref":"replication_docs","sha":"de678b1c04299a674e15ef441c10063679388276","user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"repo":{"id":9219509,"name":"elasticsearch","full_name":"bleskes/elasticsearch","owner":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/bleskes/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine","fork":true,"url":"https://api.github.com/repos/bleskes/elasticsearch","forks_url":"https://api.github.com/repos/bleskes/elasticsearch/forks","keys_url":"https://api.github.com/repos/bleskes/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bleskes/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bleskes/elasticsearch/teams","hooks_url":"https://api.github.com/repos/bleskes/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/bleskes/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/bleskes/elasticsearch/events","assignees_url":"https://api.github.com/repos/bleskes/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/bleskes/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/bleskes/elasticsearch/tags","blobs_url":"https://api.github.com/repos/bleskes/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bleskes/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bleskes/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/bleskes/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bleskes/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/bleskes/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/bleskes/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/bleskes/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/bleskes/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/bleskes/elasticsearch/subscription","commits_url":"https://api.github.com/repos/bleskes/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/bleskes/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/bleskes/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/bleskes/elasticsearch/issues/comments{/number}","contents_url":"https://api.github.com/repos/bleskes/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/bleskes/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bleskes/elasticsearch/merges","archive_url":"https://api.github.com/repos/bleskes/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bleskes/elasticsearch/downloads","issues_url":"https://api.github.com/repos/bleskes/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/bleskes/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/bleskes/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/bleskes/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bleskes/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/bleskes/elasticsearch/releases{/id}","deployments_url":"https://api.github.com/repos/bleskes/elasticsearch/deployments","created_at":"2013-04-04T14:06:17Z","updated_at":"2015-04-24T10:42:15Z","pushed_at":"2016-12-22T13:09:21Z","git_url":"git://github.com/bleskes/elasticsearch.git","ssh_url":"git@github.com:bleskes/elasticsearch.git","clone_url":"https://github.com/bleskes/elasticsearch.git","svn_url":"https://github.com/bleskes/elasticsearch","homepage":"http://elasticsearch.org","size":257999,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"elastic:master","ref":"master","sha":"06576ed13b038da71a23d0cd01b49fc1fd240625","user":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"repo":{"id":507775,"name":"elasticsearch","full_name":"elastic/elasticsearch","owner":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elastic/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine","fork":false,"url":"https://api.github.com/repos/elastic/elasticsearch","forks_url":"https://api.github.com/repos/elastic/elasticsearch/forks","keys_url":"https://api.github.com/repos/elastic/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elastic/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elastic/elasticsearch/teams","hooks_url":"https://api.github.com/repos/elastic/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/elastic/elasticsearch/events","assignees_url":"https://api.github.com/repos/elastic/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/elastic/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/elastic/elasticsearch/tags","blobs_url":"https://api.github.com/repos/elastic/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elastic/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elastic/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/elastic/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elastic/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/elastic/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/elastic/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/elastic/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/elastic/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/elastic/elasticsearch/subscription","commits_url":"https://api.github.com/repos/elastic/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/elastic/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments{/number}","contents_url":"https://api.github.com/repos/elastic/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/elastic/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elastic/elasticsearch/merges","archive_url":"https://api.github.com/repos/elastic/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elastic/elasticsearch/downloads","issues_url":"https://api.github.com/repos/elastic/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/elastic/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/elastic/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/elastic/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elastic/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/elastic/elasticsearch/releases{/id}","deployments_url":"https://api.github.com/repos/elastic/elasticsearch/deployments","created_at":"2010-02-08T13:20:56Z","updated_at":"2016-12-23T11:23:41Z","pushed_at":"2016-12-23T11:23:53Z","git_url":"git://github.com/elastic/elasticsearch.git","ssh_url":"git@github.com:elastic/elasticsearch.git","clone_url":"https://github.com/elastic/elasticsearch.git","svn_url":"https://github.com/elastic/elasticsearch","homepage":"https://www.elastic.co/products/elasticsearch","size":299878,"stargazers_count":19949,"watchers_count":19949,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":6852,"mirror_url":null,"open_issues_count":1170,"forks":6852,"open_issues":1170,"watchers":19949,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/22327"},"html":{"href":"https://github.com/elastic/elasticsearch/pull/22327"},"issue":{"href":"https://api.github.com/repos/elastic/elasticsearch/issues/22327"},"comments":{"href":"https://api.github.com/repos/elastic/elasticsearch/issues/22327/comments"},"review_comments":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/22327/comments"},"review_comment":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/22327/commits"},"statuses":{"href":"https://api.github.com/repos/elastic/elasticsearch/statuses/de678b1c04299a674e15ef441c10063679388276"}}}}
{"actor":{"display_login":"clintongormley"}}