title: Two nodes in cluster showing DN to each other, UN to everyone else
body: <p>I have a nine node Cassandra cluster and everything seems to be working fine, except for two of my servers show each other as DN.  All other nodes in the cluster show all nodes as UN.  These two show all nodes UN except for each other, where they show each other as DN.  There are no errors in the system.log on either server that indicates a problem.  All nodes are listed as seed nodes across the cluster.  I am able to telnet between the servers on port 7001, so I don't think it is a network issue.  We are using Internode Communication Encryption so I wonder if it might be an issue with that?</p>
  
  <p>Related Nodetool Status Snippet on 64.6.220.249:</p>
  
  <pre><code>DN  64.6.220.251    106.19 GB  256     ?       e008bc26-5d12-48b5-a381-6a175b085496  Rack1
  </code></pre>
  
  <p>Related Nodetool Status Snippet on 64.6.220.251:</p>
  
  <pre><code>DN  64.6.220.249    105.31 GB  256     ?       59709c2a-6270-40be-a444-042bdf18873e  Rack1
  </code></pre>
  
  <p>Related Nodetol Status Snippet from another node in the cluster (all nodes show this, except for the two above):</p>
  
  <pre><code>UN  64.6.220.251    106.19 GB  256     ?       e008bc26-5d12-48b5-a381-6a175b085496  Rack1
  UN  64.6.220.249    105.31 GB  256     ?       59709c2a-6270-40be-a444-042bdf18873e  Rack1
  </code></pre>
  
  <p>GossipInfo ran from 64.6.220.249:</p>
  
  <pre><code>/64.6.220.251
  generation:1473238188
    heartbeat:12693992
    SCHEMA:a7b7f6f4-24ba-3153-90cc-dc8ad2754251
    RACK:Rack1
    SEVERITY:0.0
    RPC_ADDRESS:64.6.220.251
    HOST_ID:e008bc26-5d12-48b5-a381-6a175b085496
    INTERNAL_IP:64.6.220.251
    X_11_PADDING:{"workload":"Cassandra","active":"true"}
    LOAD:1.14019618013E11
    NET_VERSION:8
    DC:Cassandra-ALPHA
    RELEASE_VERSION:2.1.5.469
    STATUS:NORMAL,-1122920019547920198
  </code></pre>
  
  <p>GossipInfo ran from 64.6.220.251:</p>
  
  <pre><code>  /64.6.220.249
    generation:1473237564
    heartbeat:12696040
    RACK:Rack1
    DC:Cassandra-ALPHA
    RPC_ADDRESS:64.6.220.249
    SCHEMA:a7b7f6f4-24ba-3153-90cc-dc8ad2754251
    INTERNAL_IP:64.6.220.249
    SEVERITY:0.0
    X_11_PADDING:{"workload":"Cassandra","active":"true"}
    RELEASE_VERSION:2.1.5.469
    NET_VERSION:8
    LOAD:1.13072884091E11
    HOST_ID:59709c2a-6270-40be-a444-042bdf18873e
    STATUS:NORMAL,-1027844444513030305
  </code></pre>
  
  <p>Nodetool describecluster from 64.6.220.249:</p>
  
  <pre><code>Cluster Information:
      Name: Fusion Cluster
      Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
      Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
      Schema versions:
          a7b7f6f4-24ba-3153-90cc-dc8ad2754251: [64.6.220.254, 170.75.212.226, 170.75.212.225, 64.6.220.252, 170.75.212.224, 64.6.220.253, 64.6.220.250, 64.6.220.249]
  
          UNREACHABLE: [64.6.220.251]
  </code></pre>
  
  <p>Nodetool describecluster from 64.6.220.251:</p>
  
  <pre><code>Cluster Information:
      Name: Fusion Cluster
      Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
      Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
      Schema versions:
          a7b7f6f4-24ba-3153-90cc-dc8ad2754251: [64.6.220.254, 170.75.212.226, 170.75.212.225, 64.6.220.252, 170.75.212.224, 64.6.220.253, 64.6.220.250, 64.6.220.251]
  
          UNREACHABLE: [64.6.220.249]
  </code></pre>
  
  <p>Can anyone point me in the right direction as to why these two nodes show each other as "DN", even though all other nodes see them as "UN?</p>