{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/32453","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/32453/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/32453/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/32453/events","html_url":"https://github.com/elastic/elasticsearch/issues/32453","id":345455533,"node_id":"MDU6SXNzdWUzNDU0NTU1MzM=","number":32453,"title":"ShardFollowNodeTask fetch operations twice","user":{"login":"dnhatn","id":13474362,"node_id":"MDQ6VXNlcjEzNDc0MzYy","avatar_url":"https://avatars3.githubusercontent.com/u/13474362?v=4","gravatar_id":"","url":"https://api.github.com/users/dnhatn","html_url":"https://github.com/dnhatn","followers_url":"https://api.github.com/users/dnhatn/followers","following_url":"https://api.github.com/users/dnhatn/following{/other_user}","gists_url":"https://api.github.com/users/dnhatn/gists{/gist_id}","starred_url":"https://api.github.com/users/dnhatn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dnhatn/subscriptions","organizations_url":"https://api.github.com/users/dnhatn/orgs","repos_url":"https://api.github.com/users/dnhatn/repos","events_url":"https://api.github.com/users/dnhatn/events{/privacy}","received_events_url":"https://api.github.com/users/dnhatn/received_events","type":"User","site_admin":false},"labels":[{"id":912824565,"node_id":"MDU6TGFiZWw5MTI4MjQ1NjU=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Distributed/CCR","name":":Distributed/CCR","color":"0e8a16","default":false},{"id":23173,"node_id":"MDU6TGFiZWwyMzE3Mw==","url":"https://api.github.com/repos/elastic/elasticsearch/labels/%3Ebug","name":">bug","color":"b60205","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-07-28T15:10:49Z","updated_at":"2018-07-30T17:17:14Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Since #31581, ShardFollowNodeTask may fetch some range twice. The following log indicates that we fetched the range [1680 to 2024] twice.\r\n\r\n```\r\n[ShardFollowNodeTask] fetch from=1620, to=1679, receive [1620 2024]\r\n[ShardFollowNodeTask] fetch from=1680, to=1984, receive [1680 2084]\r\n```\r\n\r\nThis bug and if the follower is not using the FollowingEngine (PR #32448) can explain why we have many deletes on the replicas of the follower (but not on the primaries of the follower).\r\n\r\n```\r\n[ShardFollowNodeTask] send shard request from=0, batch=405, max=-1, retry=0, gcp=-1, num_reads=1\r\n[ShardFollowNodeTask] fetch from=0, to=-1, receive [0 404]\r\n[ShardFollowNodeTask] send shard request from=405, batch=405, max=809, retry=0, gcp=1679, num_reads=1\r\n[ShardFollowNodeTask] send shard request from=810, batch=405, max=1214, retry=0, gcp=1679, num_reads=2\r\n[ShardFollowNodeTask] send shard request from=1215, batch=405, max=1619, retry=0, gcp=1679, num_reads=3\r\n[ShardFollowNodeTask] send shard request from=1620, batch=405, max=1679, retry=0, gcp=1679, num_reads=4\r\n\r\n[ShardFollowNodeTask] fetch from=810, to=1214, receive [810 1214]\r\n[ShardFollowNodeTask] send shard request from=1680, batch=405, max=1984, retry=0, gcp=1984, num_reads=4\r\n[ShardFollowNodeTask] fetch from=405, to=809, receive [405 809]\r\n[ShardFollowNodeTask] fetch from=1215, to=1619, receive [1215 1619]\r\n[ShardFollowNodeTask] fetch from=1620, to=1679, receive [1620 2024] (*)\r\n[ShardFollowNodeTask] send shard request from=2025, batch=405, max=2322, retry=0, gcp=2322, num_reads=2\r\n[ShardFollowNodeTask] fetch from=1680, to=1984, receive [1680 2084] (**)\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/408941019","html_url":"https://github.com/elastic/elasticsearch/issues/32453#issuecomment-408941019","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/32453","id":408941019,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODk0MTAxOQ==","user":{"login":"bleskes","id":1006375,"node_id":"MDQ6VXNlcjEwMDYzNzU=","avatar_url":"https://avatars1.githubusercontent.com/u/1006375?v=4","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"created_at":"2018-07-30T17:16:31Z","updated_at":"2018-07-30T17:16:31Z","author_association":"MEMBER","body":"Thx @dnhatn I forgot that we don't use the bulk shard request and that exceptions on the primary don't map to lack of replications. It does. Thanks!"}}
{"actor":{"display_login":"bleskes"}}