title: Upgrading Elasticsearch 2.1.1 to 2.2.0 - missing authentication token?
body: <p>I decided to try upgrade the current cluster from ES2.1.1 to ES2.2.0.
  A mirror pair. The cluster is running within AWS, so I'm using the <code>cloud-aws</code> plugin for communication.</p>
  
  <p>I successfully upgraded the first node, and it has assumed master status, but I have encountered a strange communication/authentication issue when upgrading the second node.</p>
  
  <p>I paid attention to the guidelines <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.2/rolling-upgrades.html" rel="nofollow">here</a>, but I still seem to be experiencing a strange issue.</p>
  
  <p>From main cluster log on 2nd node:</p>
  
  <pre><code>[2016-02-03 12:29:41,241][INFO ][discovery.ec2            ] [Sharon Ventura] failed to send join request to master [{Space Phantom}{NzN7b7ZHT8uPu6oXJAORMg}{10.60.164.147}{10.60.164.147:9300}], reason [RemoteTransportException[[Space Phantom][10.60.164.147:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[Sharon Ventura][10.60.163.74:9300][internal:discovery/zen/join/validate]]; nested: ElasticsearchSecurityException[missing authentication token for action [internal:discovery/zen/join/validate]]; ]
  [2016-02-03 12:29:42,455][DEBUG][action.admin.cluster.health] [Sharon Ventura] no known master node, scheduling a retry
  [2016-02-03 12:29:44,255][INFO ][discovery.ec2            ] [Sharon Ventura] failed to send join request to master [{Space Phantom}{NzN7b7ZHT8uPu6oXJAORMg}{10.60.164.147}{10.60.164.147:9300}], reason [RemoteTransportException[[Space Phantom][10.60.164.147:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[Sharon Ventura][10.60.163.74:9300][internal:discovery/zen/join/validate]]; nested: ElasticsearchSecurityException[missing authentication token for action [internal:discovery/zen/join/validate]]; ]
  [2016-02-03 12:29:47,269][INFO ][discovery.ec2            ] [Sharon Ventura] failed to send join request to master [{Space Phantom}{NzN7b7ZHT8uPu6oXJAORMg}{10.60.164.147}{10.60.164.147:9300}], reason [RemoteTransportException[[Space Phantom][10.60.164.147:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[Sharon Ventura][10.60.163.74:9300][internal:discovery/zen/join/validate]]; nested: ElasticsearchSecurityException[missing authentication token for action [internal:discovery/zen/join/validate]]; ]
  [2016-02-03 12:29:49,472][DEBUG][action.admin.cluster.state] [Sharon Ventura] timed out while retrying [cluster:monitor/state] after failure (timeout [30s])
  [2016-02-03 12:29:49,473][INFO ][rest.suppressed          ] /_cluster/settings Params: {}
  MasterNotDiscoveredException[null]
          at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:205)
          at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239)
          at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:794)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
  [2016-02-03 12:29:50,283][INFO ][discovery.ec2            ] [Sharon Ventura] failed to send join request to master [{Space Phantom}{NzN7b7ZHT8uPu6oXJAORMg}{10.60.164.147}{10.60.164.147:9300}], reason [RemoteTransportException[[Space Phantom][10.60.164.147:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[Sharon Ventura][10.60.163.74:9300][internal:discovery/zen/join/validate]]; nested: ElasticsearchSecurityException[missing authentication token for action [internal:discovery/zen/join/validate]]; ]
  </code></pre>
  
  <p>My elasticsearch.yml file:</p>
  
  <pre><code>cluster.name: cluster01
  http.cors.enabled: true
  network.host: 0.0.0.0
  discovery.type: ec2
  discovery.ec2.tag.project_code_info: "cluster01"
  cloud.aws.region: eu-central-1
  </code></pre>
  
  <p>I can see in the logs that it has detected the 1st node: <code>[Space Phantom][10.60.164.147:9300]</code>
  It has detected it without any intervention, but it apparently cannot authenticate.</p>
  
  <p>I suspect this may be related to the <code>Shield</code> plugin, which is installed also, but the correct and identical permissions are setup the same as before. Nothing else has changed.</p>
  
  <p>I'm using a username and password in shield, no SSL configured.</p>
  
  <p>Can anyone assist?</p>