{"action":"created","comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments/48887878","id":48887878,"diff_hunk":"@@ -0,0 +1,608 @@\n+[[plugins-reindex]]\n+=== Reindex Plugin\n+\n+The reindex plugin adds two APIs:\n+\n+* `_update_by_query` updates all documents matching a query in place.\n+* `_reindex` copies documents from one index to another.\n+\n+These APIs are brothers so they live in the same plugin. Both use\n+{ref}/search-request-scroll.html[Scroll] and {ref}/docs-bulk.html[Bulk] APIs\n+to send an index request per document though they may not work that way in the\n+future. Regardless of how they work internally they will always work **as**\n+**though** they send an index request per document that matches a scroll.\n+\n+[float]\n+==== Installation\n+\n+This plugin can be installed using the plugin manager:\n+\n+[source,sh]\n+----------------------------------------------------------------\n+sudo bin/plugin install reindex\n+----------------------------------------------------------------\n+\n+The plugin must be installed on every node in the cluster, and each node must\n+be restarted after installation.\n+\n+[float]\n+==== Removal\n+\n+The plugin can be removed with the following command:\n+\n+[source,sh]\n+----------------------------------------------------------------\n+sudo bin/plugin remove reindex\n+----------------------------------------------------------------\n+\n+The node must be stopped before removing the plugin.\n+\n+[[update-by-query-usage]]\n+==== Using `_update_by_query`\n+\n+The simplest usage of `_update_by_query` just `touch`es every document in an\n+index:\n+\n+[source,js]\n+--------------------------------------------------\n+POST /twitter/_update_by_query\n+--------------------------------------------------\n+// AUTOSENSE\n+\n+That will return something like this:\n+\n+[source,js]\n+--------------------------------------------------\n+{\n+  \"took\" : 639,\n+  \"updated\": 1235,\n+  \"batches\": 13,\n+  \"version_conflicts\": 2,\n+  \"failures\" : [ ]\n+}\n+--------------------------------------------------\n+\n+`_update_by_query` gets a snapshot of the index when it starts and indexes what\n+it finds using `internal` versioning. That means that if the document changes\n+between the time when the snapshot was taken and when the index request is\n+processed then you'll get a version conflict. When the versions match the\n+document is changed and the version number is incremented.\n+\n+Version conflicts are just counted and not returned by default. To count them\n+you must set `conflicts=abort` on the url or `\"conflicts\": \"abort\"` in the\n+request body. If you do then the `failures` element will contain version\n+conflict information. There is no `conflicts=record` because we number of\n+conflicts could grow without bounds so its either `abort` or nothing.\n+\n+Speaking of the `failures` element, it will contain a list of failures if\n+there are any. The only \"optional\" sort of failures are version conflicts. All\n+other failures always cause the operation to abort. While the first failure\n+causes the abort, it's possible to encounter lots of subsequent failures due to\n+batching so all failures are reported. Just in case you want them.\n+\n+It's important to remember that \"abort\" here doesn't mean \"rollback\". This\n+operation isn't atomic. If it fails part way through then its changes have been\n+made.\n+\n+Back to the API format, you can limit the copy to a single type. This will only\n+update `tweet`s from the `twitter` index:\n+\n+[source,js]\n+--------------------------------------------------\n+POST /twitter/tweet/_update_by_query\n+--------------------------------------------------\n+// AUTOSENSE\n+\n+You can limit using the {ref}/query-dsl.html[Query DSL]. This will update","path":"docs/plugins/reindex.asciidoc","position":null,"original_position":96,"commit_id":"c0bc003d4efbcf45e591de3438fc7ccd61b33ea7","original_commit_id":"088a978d212e95b597838f7f0abcb65184503614","user":{"login":"nik9000","id":215970,"avatar_url":"https://avatars.githubusercontent.com/u/215970?v=3","gravatar_id":"","url":"https://api.github.com/users/nik9000","html_url":"https://github.com/nik9000","followers_url":"https://api.github.com/users/nik9000/followers","following_url":"https://api.github.com/users/nik9000/following{/other_user}","gists_url":"https://api.github.com/users/nik9000/gists{/gist_id}","starred_url":"https://api.github.com/users/nik9000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nik9000/subscriptions","organizations_url":"https://api.github.com/users/nik9000/orgs","repos_url":"https://api.github.com/users/nik9000/repos","events_url":"https://api.github.com/users/nik9000/events{/privacy}","received_events_url":"https://api.github.com/users/nik9000/received_events","type":"User","site_admin":false},"body":"Fixing.","created_at":"2016-01-05T19:48:55Z","updated_at":"2016-01-05T19:48:55Z","html_url":"https://github.com/elastic/elasticsearch/pull/15125#discussion_r48887878","pull_request_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/15125","_links":{"self":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments/48887878"},"html":{"href":"https://github.com/elastic/elasticsearch/pull/15125#discussion_r48887878"},"pull_request":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/15125"}}},"pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/15125","id":52176503,"html_url":"https://github.com/elastic/elasticsearch/pull/15125","diff_url":"https://github.com/elastic/elasticsearch/pull/15125.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/15125.patch","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15125","number":15125,"state":"open","locked":false,"title":"Basic reindex implementation","user":{"login":"nik9000","id":215970,"avatar_url":"https://avatars.githubusercontent.com/u/215970?v=3","gravatar_id":"","url":"https://api.github.com/users/nik9000","html_url":"https://github.com/nik9000","followers_url":"https://api.github.com/users/nik9000/followers","following_url":"https://api.github.com/users/nik9000/following{/other_user}","gists_url":"https://api.github.com/users/nik9000/gists{/gist_id}","starred_url":"https://api.github.com/users/nik9000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nik9000/subscriptions","organizations_url":"https://api.github.com/users/nik9000/orgs","repos_url":"https://api.github.com/users/nik9000/repos","events_url":"https://api.github.com/users/nik9000/events{/privacy}","received_events_url":"https://api.github.com/users/nik9000/received_events","type":"User","site_admin":false},"body":"Note: this has been edited from my first comment.\r\n\r\nThis creates basic reindex and update-by-query implementations that work like delete-by-query. They scroll documents and then turn around and issue bulk requests for each.\r\n\r\nThis isn't the last word on reindex. Before we merge the feature branch down to master we'll need to fix it to retry on bulk rejection and integrate it with the task management api.\r\n\r\nFor those of you reading the issues, realize that all the discussion about APIs is discussion. The API we're going to merge is the result of those discussions. You can read about it by reading the asciidoc files in the PR.","created_at":"2015-11-30T20:43:51Z","updated_at":"2016-01-05T19:48:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"4a802f706ed7ab66cfbf360a203e4c293d464f08","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/15125/commits","review_comments_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/15125/comments","review_comment_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15125/comments","statuses_url":"https://api.github.com/repos/elastic/elasticsearch/statuses/c0bc003d4efbcf45e591de3438fc7ccd61b33ea7","head":{"label":"nik9000:first_reindex","ref":"first_reindex","sha":"c0bc003d4efbcf45e591de3438fc7ccd61b33ea7","user":{"login":"nik9000","id":215970,"avatar_url":"https://avatars.githubusercontent.com/u/215970?v=3","gravatar_id":"","url":"https://api.github.com/users/nik9000","html_url":"https://github.com/nik9000","followers_url":"https://api.github.com/users/nik9000/followers","following_url":"https://api.github.com/users/nik9000/following{/other_user}","gists_url":"https://api.github.com/users/nik9000/gists{/gist_id}","starred_url":"https://api.github.com/users/nik9000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nik9000/subscriptions","organizations_url":"https://api.github.com/users/nik9000/orgs","repos_url":"https://api.github.com/users/nik9000/repos","events_url":"https://api.github.com/users/nik9000/events{/privacy}","received_events_url":"https://api.github.com/users/nik9000/received_events","type":"User","site_admin":false},"repo":{"id":11487195,"name":"elasticsearch","full_name":"nik9000/elasticsearch","owner":{"login":"nik9000","id":215970,"avatar_url":"https://avatars.githubusercontent.com/u/215970?v=3","gravatar_id":"","url":"https://api.github.com/users/nik9000","html_url":"https://github.com/nik9000","followers_url":"https://api.github.com/users/nik9000/followers","following_url":"https://api.github.com/users/nik9000/following{/other_user}","gists_url":"https://api.github.com/users/nik9000/gists{/gist_id}","starred_url":"https://api.github.com/users/nik9000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nik9000/subscriptions","organizations_url":"https://api.github.com/users/nik9000/orgs","repos_url":"https://api.github.com/users/nik9000/repos","events_url":"https://api.github.com/users/nik9000/events{/privacy}","received_events_url":"https://api.github.com/users/nik9000/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/nik9000/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine","fork":true,"url":"https://api.github.com/repos/nik9000/elasticsearch","forks_url":"https://api.github.com/repos/nik9000/elasticsearch/forks","keys_url":"https://api.github.com/repos/nik9000/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/nik9000/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/nik9000/elasticsearch/teams","hooks_url":"https://api.github.com/repos/nik9000/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/nik9000/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/nik9000/elasticsearch/events","assignees_url":"https://api.github.com/repos/nik9000/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/nik9000/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/nik9000/elasticsearch/tags","blobs_url":"https://api.github.com/repos/nik9000/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/nik9000/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/nik9000/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/nik9000/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/nik9000/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/nik9000/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/nik9000/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/nik9000/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/nik9000/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/nik9000/elasticsearch/subscription","commits_url":"https://api.github.com/repos/nik9000/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/nik9000/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/nik9000/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/nik9000/elasticsearch/issues/comments{/number}","contents_url":"https://api.github.com/repos/nik9000/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/nik9000/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/nik9000/elasticsearch/merges","archive_url":"https://api.github.com/repos/nik9000/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/nik9000/elasticsearch/downloads","issues_url":"https://api.github.com/repos/nik9000/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/nik9000/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/nik9000/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/nik9000/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/nik9000/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/nik9000/elasticsearch/releases{/id}","created_at":"2013-07-17T20:48:13Z","updated_at":"2014-12-15T18:37:06Z","pushed_at":"2016-01-05T19:47:54Z","git_url":"git://github.com/nik9000/elasticsearch.git","ssh_url":"git@github.com:nik9000/elasticsearch.git","clone_url":"https://github.com/nik9000/elasticsearch.git","svn_url":"https://github.com/nik9000/elasticsearch","homepage":"http://elasticsearch.org","size":185279,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"elastic:feature/reindex","ref":"feature/reindex","sha":"f8026ed8fd80ece83f33dd7a5d7e66b371dc12a4","user":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"repo":{"id":507775,"name":"elasticsearch","full_name":"elastic/elasticsearch","owner":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elastic/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine","fork":false,"url":"https://api.github.com/repos/elastic/elasticsearch","forks_url":"https://api.github.com/repos/elastic/elasticsearch/forks","keys_url":"https://api.github.com/repos/elastic/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elastic/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elastic/elasticsearch/teams","hooks_url":"https://api.github.com/repos/elastic/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/elastic/elasticsearch/events","assignees_url":"https://api.github.com/repos/elastic/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/elastic/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/elastic/elasticsearch/tags","blobs_url":"https://api.github.com/repos/elastic/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elastic/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elastic/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/elastic/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elastic/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/elastic/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/elastic/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/elastic/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/elastic/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/elastic/elasticsearch/subscription","commits_url":"https://api.github.com/repos/elastic/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/elastic/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments{/number}","contents_url":"https://api.github.com/repos/elastic/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/elastic/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elastic/elasticsearch/merges","archive_url":"https://api.github.com/repos/elastic/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elastic/elasticsearch/downloads","issues_url":"https://api.github.com/repos/elastic/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/elastic/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/elastic/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/elastic/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elastic/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/elastic/elasticsearch/releases{/id}","created_at":"2010-02-08T13:20:56Z","updated_at":"2016-01-05T19:14:25Z","pushed_at":"2016-01-05T19:47:56Z","git_url":"git://github.com/elastic/elasticsearch.git","ssh_url":"git@github.com:elastic/elasticsearch.git","clone_url":"https://github.com/elastic/elasticsearch.git","svn_url":"https://github.com/elastic/elasticsearch","homepage":"https://www.elastic.co/products/elasticsearch","size":234253,"stargazers_count":14236,"watchers_count":14236,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":4745,"mirror_url":null,"open_issues_count":1260,"forks":4745,"open_issues":1260,"watchers":14236,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/15125"},"html":{"href":"https://github.com/elastic/elasticsearch/pull/15125"},"issue":{"href":"https://api.github.com/repos/elastic/elasticsearch/issues/15125"},"comments":{"href":"https://api.github.com/repos/elastic/elasticsearch/issues/15125/comments"},"review_comments":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/15125/comments"},"review_comment":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/15125/commits"},"statuses":{"href":"https://api.github.com/repos/elastic/elasticsearch/statuses/c0bc003d4efbcf45e591de3438fc7ccd61b33ea7"}}}}