title: Add node to running cluster elasticsearch causes master not discovered exception
body: <h1>Problem</h1>
  
  <p>I have a running cluster and I would like to add a data node into it. The running cluster is </p>
  
  <pre><code>x.x.x.246
  </code></pre>
  
  <p>and the data node is </p>
  
  <pre><code>x.x.x.99
  </code></pre>
  
  <p>each server can see each other by ping.
  Machine OS: CentOS7
  Elasticsearch: 7.61</p>
  
  <h2>configs:</h2>
  
  <p>here is elasticsearch.yml of x.x.x.246:</p>
  
  <pre><code>cluster.name: elasticsearch
  node.master: true
  node.name: Node_master
  node.data: true
  path.data: /var/lib/elasticsearch
  path.logs: /var/log/elasticsearch
  network.host: x.x.x.246
  http.port: 9200
  discovery.seed_hosts: ["x.x.x.99:9300"]
  cluster.initial_master_nodes: ["x.x.x.246:9300"]
  </code></pre>
  
  <p>here is elasticsearch.yml of x.x.x.99</p>
  
  <pre><code>cluster.name: elasticsearch
  node.name: Node_master
  node.data: true
  path.data: /var/lib/elasticsearch
  path.logs: /var/log/elasticsearch
  network.host: x.x.x.99
  http.port: 9200
  discovery.seed_hosts: ["x.x.x.245:9300"]
  cluster.initial_master_nodes: ["x.x.x.246:9300"]
  </code></pre>
  
  <h1>Testing running elasticsearch on machine</h1>
  
  <p>When I run <code>systemctl start elasticsearch</code> on each machine, it works well. </p>
  
  <h2>test run on x.x.x.246</h2>
  
  <pre><code>curl -X GET "X.X.X.246:9200/_cluster/health?pretty"
  </code></pre>
  
  <p>show:number of the node not changing</p>
  
  <pre><code>curl -X GET "X.X.X.99:9200/_cluster/health?pretty
  </code></pre>
  
  <p>show:</p>
  
  <pre><code>{
    "error" : {
      "root_cause" : [
        {
          "type" : "master_not_discovered_exception",
          "reason" : null
        }
      ],
      "type" : "master_not_discovered_exception",
      "reason" : null
    },
    "status" : 503
  }
  </code></pre>
  
  <h1>edited</h1>
  
  <p>here is elasticsearch.yml of x.x.x.246:</p>
  
  <pre><code>cluster.name: elasticsearch
  node.name: master
  node.master: true
  node.data: true
  path.data: /var/lib/elasticsearch
  path.logs: /var/log/elasticsearch
  network.host: 0.0.0.0
  http.port: 9200
  discovery.seed_hosts: ["x.x.x.99","x.x.x.246]
  cluster.initial_master_nodes: ["x.x.x.246"]
  logger.org.elasticsearch.discovery: TRACE
  
  </code></pre>
  
  <p>here is elasticsearch.yml of x.x.x.99</p>
  
  <pre><code>cluster.name: elasticsearch
  node.name: node
  node.data: true
  path.data: /var/lib/elasticsearch
  path.logs: /var/log/elasticsearch
  network.host: 0.0.0.0
  http.port: 9200
  discovery.seed_hosts: ["x.x.x.246","x.x.x.99"]
  cluster.initial_master_nodes: ["x.x.x.246"]
  logger.org.elasticsearch.discovery: TRACE
  
  </code></pre>
  
  <p>log on x.x.x.99:</p>
  
  <pre><code>[root@dev ~]# tail -30 /var/log/elasticsearch/elasticsearch.log
          at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
          at java.lang.Thread.run(Thread.java:830) ~[?:?]
  [2020-03-19T12:12:04,462][INFO ][o.e.c.c.JoinHelper       ] [node-1] failed to join {master}{0UHYehfNQ2-WCadTC_VVkA}{1FNy5AJrTpKOCAejBLKR2w}{10.64.2.246}{10.64.2.246:9300}{dilm}{ml.machine_memory=1907810304, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{a4TYDhG7SWqL3CSG4tusEg}{10.64.2.99}{10.64.2.99:9300}{d}{xpack.installed=true}, optionalJoin=Optional[Join{term=178, lastAcceptedTerm=8, lastAcceptedVersion=100, sourceNode={node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{a4TYDhG7SWqL3CSG4tusEg}{10.64.2.99}{10.64.2.99:9300}{d}{xpack.installed=true}, targetNode={master}{0UHYehfNQ2-WCadTC_VVkA}{1FNy5AJrTpKOCAejBLKR2w}{10.64.2.246}{10.64.2.246:9300}{dilm}{ml.machine_memory=1907810304, ml.max_open_jobs=20, xpack.installed=true}}]}
  org.elasticsearch.transport.RemoteTransportException: [master][10.64.2.246:9300][internal:cluster/coordination/join]
  Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
          at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:514) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.6.1.jar:7.6.1]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
          at java.lang.Thread.run(Thread.java:830) [?:?]
  Caused by: org.elasticsearch.transport.RemoteTransportException: [node-1][10.64.2.99:9300][internal:cluster/coordination/join/validate]
  Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid P4QlwvuRRGSmlT77RroSjA than local cluster uuid oUoIe2-bSbS2UPg722ud9Q, rejecting
          at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
          at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
          at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1]
          at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
          at java.lang.Thread.run(Thread.java:830) ~[?:?]
  </code></pre>