{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/12573","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12573/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12573/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12573/events","html_url":"https://github.com/elastic/elasticsearch/issues/12573","id":98374056,"number":12573,"title":"Cluster state delay can cause endless index request loop","user":{"login":"brwe","id":4320215,"avatar_url":"https://avatars.githubusercontent.com/u/4320215?v=3","gravatar_id":"","url":"https://api.github.com/users/brwe","html_url":"https://github.com/brwe","followers_url":"https://api.github.com/users/brwe/followers","following_url":"https://api.github.com/users/brwe/following{/other_user}","gists_url":"https://api.github.com/users/brwe/gists{/gist_id}","starred_url":"https://api.github.com/users/brwe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brwe/subscriptions","organizations_url":"https://api.github.com/users/brwe/orgs","repos_url":"https://api.github.com/users/brwe/repos","events_url":"https://api.github.com/users/brwe/events{/privacy}","received_events_url":"https://api.github.com/users/brwe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Recovery","name":":Recovery","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/bug","name":"bug","color":"9e2c2c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-31T11:41:34Z","updated_at":"2016-01-27T18:26:35Z","closed_at":null,"body":"When a primary is relocating from `node_1` to `node_2`, there can be a short time where the old primary is removed from the node already (closed, not deleted) but the new primary is still in `POST_RECOVERY`. In this state indexing requests might be sent back and forth between `node_1` and `node_2` endlessly.\r\n\r\nCourse of events: \r\n\r\n1. primary (`[index][0]`) relocates from `node_1` to `node_2`\r\n1. `node_2` is done recovering, moves its shard to `IndexShardState.POST_RECOVERY` and sends a message to master that the shard is `ShardRoutingState.STARTED` \r\n\r\n  ```\r\n   Cluster state 1: \r\n   node_1: [index][0] RELOCATING (ShardRoutingState), (STARTED from IndexShardState perspective on node_1) \r\n   node_2: [index][0] INITIALIZING (ShardRoutingState), (at this point already POST_RECOVERY from IndexShardState perspective on node_2) \r\n   ```\r\n2. master receives shard started and updates cluster state to: \r\n\r\n   ```   \r\n   Cluster state 2: \r\n   node_1: [index][0] no shard \r\n   node_2: [index][0] STARTED (ShardRoutingState), (at this point still in POST_RECOVERY from IndexShardState perspective on node_2) \r\n   ```\r\nmaster sends this to `node_1` and `node_2 `\r\n3. `node_1` receives the new cluster state and removes its shard because it is not allocated on `node_1` anymore \r\n4. index a document \r\n\r\nAt this point `node_1` is already on cluster state 2 and does not have the shard anymore so it forwards the request to `node_2`. But `node_2` is behind with cluster state processing, is still on cluster state 1 and therefore has the shard in `IndexShardState.POST_RECOVERY` and thinks `node_1` has the primary. So it will send the request back to `node_1`. This goes on until either `node_2` finally catches up and processes cluster state 2 or both nodes OOM.\r\n\r\nI will make a pull request with a test shortly"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/175782401","html_url":"https://github.com/elastic/elasticsearch/issues/12573#issuecomment-175782401","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12573","id":175782401,"user":{"login":"ywelsch","id":3718355,"avatar_url":"https://avatars.githubusercontent.com/u/3718355?v=3","gravatar_id":"","url":"https://api.github.com/users/ywelsch","html_url":"https://github.com/ywelsch","followers_url":"https://api.github.com/users/ywelsch/followers","following_url":"https://api.github.com/users/ywelsch/following{/other_user}","gists_url":"https://api.github.com/users/ywelsch/gists{/gist_id}","starred_url":"https://api.github.com/users/ywelsch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ywelsch/subscriptions","organizations_url":"https://api.github.com/users/ywelsch/orgs","repos_url":"https://api.github.com/users/ywelsch/repos","events_url":"https://api.github.com/users/ywelsch/events{/privacy}","received_events_url":"https://api.github.com/users/ywelsch/received_events","type":"User","site_admin":false},"created_at":"2016-01-27T18:26:35Z","updated_at":"2016-01-27T18:26:35Z","body":"I've opened #16274 to address this issue."}}