{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/30247","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30247/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30247/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30247/events","html_url":"https://github.com/elastic/elasticsearch/issues/30247","id":318809188,"node_id":"MDU6SXNzdWUzMTg4MDkxODg=","number":30247,"title":"Remote cross cluster search stops working...","user":{"login":"asatsi","id":10648621,"node_id":"MDQ6VXNlcjEwNjQ4NjIx","avatar_url":"https://avatars0.githubusercontent.com/u/10648621?v=4","gravatar_id":"","url":"https://api.github.com/users/asatsi","html_url":"https://github.com/asatsi","followers_url":"https://api.github.com/users/asatsi/followers","following_url":"https://api.github.com/users/asatsi/following{/other_user}","gists_url":"https://api.github.com/users/asatsi/gists{/gist_id}","starred_url":"https://api.github.com/users/asatsi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asatsi/subscriptions","organizations_url":"https://api.github.com/users/asatsi/orgs","repos_url":"https://api.github.com/users/asatsi/repos","events_url":"https://api.github.com/users/asatsi/events{/privacy}","received_events_url":"https://api.github.com/users/asatsi/received_events","type":"User","site_admin":false},"labels":[{"id":146832564,"node_id":"MDU6TGFiZWwxNDY4MzI1NjQ=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Search/Search","name":":Search/Search","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-04-30T07:47:06Z","updated_at":"2018-10-12T17:06:31Z","closed_at":null,"author_association":"NONE","body":"**Elasticsearch version** (`bin/elasticsearch --version`):\r\n$ bin/elasticsearch --version\r\nVersion: 6.2.2, Build: 10b1edd/2018-02-16T19:01:30.685723Z, JVM: 1.8.0_161\r\n\r\n**Plugins installed**: \r\nNo external plugins installed.\r\n\r\n**JVM version** (`java -version`): \r\n\"1.8.0_161\"\r\n\r\n**OS version** (`uname -a` if on a Unix-like system):\r\nLinux myhostname 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Dec 28 14:23:39 EST 2017 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\nWe have two elasticsearch clusters(each 16 nodes, 4 nodes per machine, in all 4 machines per cluster) configured across 8 machines. They are joined together by a third cluster configured as a cross elasticsearch cluster.\r\n\r\n$ curl -s -XGET http://192.168.97.15:9400/_cluster/settings?pretty\r\n{\r\n  \"persistent\" : {\r\n    \"search\" : {\r\n      \"remote\" : {\r\n        \"prod1_cluster\" : {\r\n          \"seeds\" : [\r\n            \"192.168.97.15:9301\",\r\n            \"192.168.97.16:9301\",\r\n            \"192.168.97.26:9301\",\r\n            \"192.168.97.27:9301\"\r\n          ]\r\n        },\r\n        \"prod2_cluster\" : {\r\n          \"seeds\" : [\r\n            \"192.168.181.15:9300\",\r\n            \"192.168.181.16:9300\",\r\n            \"192.168.181.26:9300\",\r\n            \"192.168.181.27:9300\"\r\n          ]\r\n        }\r\n      }\r\n    }\r\n  },\r\n  \"transient\" : { }\r\n}\r\n\r\nI can create the indices in kibana for two of these clusters initially. However after sometime the search for prod2_cluster (prod2_cluster:myindex_on_prod2) times out. Search for prod1_cluster always work. Also noticed that even though the _cluster/settings give a nice response. _remote/info errors out as below:\r\n\r\n$ curl -s -XGET \"http://192.168.97.15:9400/_remote/info?pretty\"\r\n{\r\n  \"error\" : {\r\n    \"root_cause\" : [\r\n      {\r\n        \"type\" : \"node_disconnected_exception\",\r\n        \"reason\" : \"[node-es4-1][192.168.181.27:9300][cluster:monitor/nodes/info] disconnected\"\r\n      }\r\n    ],\r\n    \"type\" : \"node_disconnected_exception\",\r\n    \"reason\" : \"[node-es4-1][192.168.181.27:9300][cluster:monitor/nodes/info] disconnected\"\r\n  },\r\n  \"status\" : 500\r\n}\r\n\r\nAll ports are opened and there is no connectivity issue.\r\n\r\n**Steps to reproduce**:\r\nSetup total three clusters. Two clusters are the actual clusters holding data and a third cluster which is used to \"join\" these two as part of a cross cluster search.\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/429394617","html_url":"https://github.com/elastic/elasticsearch/issues/30247#issuecomment-429394617","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30247","id":429394617,"node_id":"MDEyOklzc3VlQ29tbWVudDQyOTM5NDYxNw==","user":{"login":"javanna","id":832460,"node_id":"MDQ6VXNlcjgzMjQ2MA==","avatar_url":"https://avatars1.githubusercontent.com/u/832460?v=4","gravatar_id":"","url":"https://api.github.com/users/javanna","html_url":"https://github.com/javanna","followers_url":"https://api.github.com/users/javanna/followers","following_url":"https://api.github.com/users/javanna/following{/other_user}","gists_url":"https://api.github.com/users/javanna/gists{/gist_id}","starred_url":"https://api.github.com/users/javanna/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javanna/subscriptions","organizations_url":"https://api.github.com/users/javanna/orgs","repos_url":"https://api.github.com/users/javanna/repos","events_url":"https://api.github.com/users/javanna/events{/privacy}","received_events_url":"https://api.github.com/users/javanna/received_events","type":"User","site_admin":false},"created_at":"2018-10-12T17:06:31Z","updated_at":"2018-10-12T17:06:31Z","author_association":"MEMBER","body":"See #34405 , I believe the outlined solutions will help with this issue. Stay tuned for a fix. I am not sure that skip_unavailable helps as in some cases we are not quick enough detecting network disconnections and due to firewalls dropping connections in-between, we may have CCS searches hanging which is a problem regardless of the `skip_unavailable` setting value."}}
{"actor":{"display_login":"javanna"}}