{"action":"opened","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/31976","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/31976/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/31976/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/31976/events","html_url":"https://github.com/elastic/elasticsearch/issues/31976","id":340377998,"node_id":"MDU6SXNzdWUzNDAzNzc5OTg=","number":31976,"title":"Possible to index duplicate documents with same id and routing id.","user":{"login":"kylelyk","id":3401359,"node_id":"MDQ6VXNlcjM0MDEzNTk=","avatar_url":"https://avatars1.githubusercontent.com/u/3401359?v=4","gravatar_id":"","url":"https://api.github.com/users/kylelyk","html_url":"https://github.com/kylelyk","followers_url":"https://api.github.com/users/kylelyk/followers","following_url":"https://api.github.com/users/kylelyk/following{/other_user}","gists_url":"https://api.github.com/users/kylelyk/gists{/gist_id}","starred_url":"https://api.github.com/users/kylelyk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylelyk/subscriptions","organizations_url":"https://api.github.com/users/kylelyk/orgs","repos_url":"https://api.github.com/users/kylelyk/repos","events_url":"https://api.github.com/users/kylelyk/events{/privacy}","received_events_url":"https://api.github.com/users/kylelyk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-11T19:37:34Z","updated_at":"2018-07-11T19:37:34Z","closed_at":null,"author_association":"NONE","body":"**Elasticsearch version**: 6.2.4\r\n\r\n**Plugins installed**: []\r\n\r\n**JVM version**: 1.8.0_172\r\n\r\n**OS version**: MacOS (Darwin Kernel Version 15.6.0)\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\nOver the past few months, we've been seeing completely identical documents pop up which have the same id, type and routing id. We're using custom routing to get parent-child joins working correctly and we make sure to delete the existing documents when re-indexing them to avoid two copies of the same document on the same shard. We use Bulk Index API calls to delete and index the documents. The indexTime field below is set by the service that indexes the document into ES and as you can see, the documents were indexed about 1 second apart from each other. This problem only seems to happen on our production server which has more traffic and 1 read replica, and it's only ever 2 documents that are duplicated on what I believe to be a single shard.\r\n\r\nThe problem can be fixed by deleting the existing documents with that id and re-indexing it again which is weird since that is what the indexing service is doing in the first place.\r\n\r\nQueries:\r\nGET /my-index/_search\r\n```\r\n{\r\n  \"size\": 0,\r\n  \"aggs\": {\r\n    \"duplicateCount\": {\r\n      \"terms\": {\r\n      \"field\": \"id\",\r\n        \"min_doc_count\": 2\r\n      },\r\n      \"aggs\": {\r\n        \"duplicateDocuments\": {\r\n          \"top_hits\": {}\r\n        }\r\n      }\r\n    }\r\n  }\r\n}\r\n```\r\n```\r\n{\r\n  \"took\": 2588,\r\n  \"timed_out\": false,\r\n  \"_shards\": {\r\n    \"total\": 4,\r\n    \"successful\": 4,\r\n    \"skipped\": 0,\r\n    \"failed\": 0\r\n  },\r\n  \"hits\": {\r\n    \"total\": 15430904,\r\n    \"max_score\": 0,\r\n    \"hits\": []\r\n  },\r\n  \"aggregations\": {\r\n    \"duplicateCount\": {\r\n      \"doc_count_error_upper_bound\": 4,\r\n      \"sum_other_doc_count\": 15430801,\r\n      \"buckets\": [\r\n        {\r\n          \"key\": \"746004ff8168bbe5672605fad34704a5\",\r\n          \"doc_count\": 2,\r\n          \"duplicateDocuments\": {\r\n            \"hits\": {\r\n              \"total\": 2,\r\n              \"max_score\": 1,\r\n              \"hits\": [\r\n                {\r\n                  \"_index\": \"my-index\",\r\n                  \"_type\": \"ce\",\r\n                  \"_id\": \"746004ff8168bbe5672605fad34704a5\",\r\n                  \"_score\": 1,\r\n                  \"_routing\": \"746004ff8168bbe5672605fad34704a5\",\r\n                  \"_source\": {\r\n                    \"indexTime\": 1531249623788\r\n                  }\r\n                },\r\n                {\r\n                  \"_index\": \"my-index\",\r\n                  \"_type\": \"ce\",\r\n                  \"_id\": \"746004ff8168bbe5672605fad34704a5\",\r\n                  \"_score\": 1,\r\n                  \"_routing\": \"746004ff8168bbe5672605fad34704a5\",\r\n                  \"_source\": {\r\n                    \"indexTime\": 1531249622605\r\n                  }\r\n                }\r\n              ]\r\n            }\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}\r\n```"}}
{"actor":{"display_login":"kylelyk"}}