title: HA cluster - Pacemaker - OFFLINE nodes status
body: <p>I'm using Pacemaker + Corosync in Centos7
  Create Cluster using these commands:</p>
  
  <pre><code>pcs cluster auth pcmk01-cr pcmk02-cr -u hacluster -p passwd
  pcs cluster setup --name my_cluster pcmk01-cr pcmk02-cr
  
  [pcmk01]# pcs cluster start --all
  [root@pcmk01 /]# pcs cluster cib clust_cfg
  [pcmk01]# pcs -f clust_cfg property set stonith-enabled=false
  [root@pcmk01 /]# pcs -f clust_cfg property set no-quorum-policy=ignore
  [root@pcmk01 /]# pcs -f clust_cfg resource defaults resource-stickiness=200 
  </code></pre>
  
  <p>When I check the status of cluster I see strange and diffrent behavior between nodes, it's looks like the nodes doesn't know each other.</p>
  
  <p>pcs status on <strong>NODE1</strong>:</p>
  
  <pre><code>[root@rvpcmk01 ~]# pcs status
  Cluster name: my_cluster
  Stack: corosync
  Current DC: rvpcmk01-cr (version 1.1.15-11.el7_3.2-e174ec8) - partition WITHOUT quorum
  Last updated: Mon Feb  6 15:18:18 2017          Last change: Mon Feb  6 15:03:03 2017 by root via cibadmin on rvpcmk01-cr
  
  2 nodes and 0 resources configured
  
  Online: [ rvpcmk01-cr ]
  OFFLINE: [ rvpcmk02-cr ]
  
  No resources
  
  
  PCSD Status:
    rvpcmk01-cr: Online
    rvpcmk02-cr: Online
  
  Daemon Status:
    corosync: active/enabled
    pacemaker: active/enabled
    pcsd: active/enabled
  </code></pre>
  
  <p>pcs status on <strong>NODE2</strong>:</p>
  
  <pre><code>[root@rvpcmk02 ~]# pcs status
  Cluster name: RV_cluster
  Stack: corosync
  Current DC: rvpcmk02-cr (version 1.1.15-11.el7_3.2-e174ec8) - partition WITHOUT quorum
  Last updated: Mon Feb  6 15:19:53 2017          Last change: Mon Feb  6 15:04:12 2017 by root via crm_attribute on rvpcmk02-cr
  
  2 nodes and 0 resources configured
  
  Online: [ rvpcmk02-cr ]
  OFFLINE: [ rvpcmk01-cr ]
  
  No resources
  
  
  PCSD Status:
    rvpcmk01-cr: Offline
    rvpcmk02-cr: Online
  
  Daemon Status:
    corosync: active/enabled
    pacemaker: active/enabled
    pcsd: active/enabled
  </code></pre>
  
  <p>I also run this command on both nodes:</p>
  
  <p>NODE1:</p>
  
  <pre><code>[root@rvpcmk01 ~]#  pcs status corosync
  
  Membership information
  ----------------------
      Nodeid      Votes Name
           1          1 rvpcmk01-cr (local)
  </code></pre>
  
  <p>NODE2:</p>
  
  <pre><code>[root@rvpcmk02 ~]# pcs status corosync
  
  Membership information
  ----------------------
      Nodeid      Votes Name
           2          1 rvpcmk02-cr (local)
  </code></pre>
  
  <p>As I know both nodes should be appear in this above status.</p>
  
  <p>Can you please help &amp; advise what i'm missing here?
  why nodes seems not know one on each other?</p>
  
  <p>here is also my /etc/hosts files on both servers:</p>
  
  <pre><code>[root@rvpcmk01 ~]# cat /etc/hosts
  127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
  ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
  172.17.235.109 rvpcmkvip
  172.17.235.43 rvpcmk01
  172.17.235.44 rvpcmk02
  172.17.235.75 rvpcmk01-cr
  172.17.235.106 rvpcmk02-cr
  172.17.235.119 rvpcmk01-drbd
  172.17.235.46 rvpcmk02-drbd
  
  [root@rvpcmk02 ~]# cat /etc/hosts
  127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
  ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
  172.17.235.109 rvpcmkvip
  172.17.235.43 rvpcmk01
  172.17.235.44 rvpcmk02
  172.17.235.75 rvpcmk01-cr
  172.17.235.106 rvpcmk02-cr
  172.17.235.119 rvpcmk01-drbd
  172.17.235.46 rvpcmk02-drbd
  </code></pre>
  
  <p>I check the authorization (that for sure was authorized when I start to configure my cluster and now I can see that there is a problem but I don't understand what it is and what is the root cause for it:</p>
  
  <pre><code>[root@rvpcmk02 drbd.d]# pcs cluster auth rvpcmk01-cr rvpcmk02-cr -u hacluster -p passwd --debug
  Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
  --Debug Input Start--
  {"username": "hacluster", "local": false, "nodes": ["rvpcmk01-cr", "rvpcmk02-cr"], "password": "passwd", "force": false}
  --Debug Input End--
  
  Return Value: 0
  --Debug Output Start--
  {
    "status": "ok",
    "data": {
      "auth_responses": {
        "rvpcmk01-cr": {
          "status": "noresponse"
        },
        "rvpcmk02-cr": {
          "status": "ok",
          "token": "e340f461-12ef-4701-a1a6-ef44439dda94"
        }
      },
      "sync_successful": true,
      "sync_nodes_err": [
        "rvpcmk01-cr"
      ],
      "sync_responses": {
        "rvpcmk01-cr": {
          "status": "error"
        },
        "rvpcmk02-cr": {
          "status": "ok",
          "result": {
            "tokens": "accepted"
          }
        }
      }
    },
    "log": [
      "I, [2017-02-06T17:58:16.146090 #27368]  INFO -- : PCSD Debugging enabled\n",
      "D, [2017-02-06T17:58:16.146225 #27368] DEBUG -- : Did not detect RHEL 6\n",
      "I, [2017-02-06T17:58:16.146304 #27368]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
      "I, [2017-02-06T17:58:16.146375 #27368]  INFO -- : CIB USER: hacluster, groups: \n",
      "D, [2017-02-06T17:58:16.156239 #27368] DEBUG -- : [\"totem.cluster_name (str) = RV-cluster\\n\"]\n",
      "D, [2017-02-06T17:58:16.156390 #27368] DEBUG -- : Duration: 0.009834697s\n",
      "I, [2017-02-06T17:58:16.156525 #27368]  INFO -- : Return Value: 0\n",
      "I, [2017-02-06T17:58:16.157212 #27368]  INFO -- : SRWT Node: rvpcmk02-cr Request: check_auth\n",
      "I, [2017-02-06T17:58:16.160119 #27368]  INFO -- : SRWT Node: rvpcmk01-cr Request: check_auth\n",
      "I, [2017-02-06T17:58:16.162104 #27368]  INFO -- : No response from: rvpcmk01-cr request: /check_auth, exception: Connection refused - connect(2)\n",
      "I, [2017-02-06T17:58:16.240477 #27368]  INFO -- : No response from: rvpcmk01-cr request: /auth, exception: Connection refused - connect(2)\n",
      "I, [2017-02-06T17:58:16.387075 #27368]  INFO -- : Running: /usr/sbin/pcs status nodes corosync\n",
      "I, [2017-02-06T17:58:16.387208 #27368]  INFO -- : CIB USER: hacluster, groups: \n",
      "D, [2017-02-06T17:58:16.670689 #27368] DEBUG -- : [\"Corosync Nodes:\\n\", \" Online: rvpcmk02-cr \\n\", \" Offline: rvpcmk01-cr \\n\"]\n",
      "D, [2017-02-06T17:58:16.670889 #27368] DEBUG -- : Duration: 0.28344494s\n",
      "I, [2017-02-06T17:58:16.671033 #27368]  INFO -- : Return Value: 0\n",
      "I, [2017-02-06T17:58:16.671978 #27368]  INFO -- : Sending config 'tokens' version 36 1f36b1c29146694381cd47b96c01d65876ba8db9 to nodes: rvpcmk02-cr, rvpcmk01-cr\n",
      "I, [2017-02-06T17:58:16.672428 #27368]  INFO -- : SRWT Node: rvpcmk01-cr Request: set_configs\n",
      "I, [2017-02-06T17:58:16.673819 #27368]  INFO -- : SRWT Node: rvpcmk02-cr Request: set_configs\n",
      "I, [2017-02-06T17:58:16.681322 #27368]  INFO -- : No response from: rvpcmk01-cr request: /set_configs, exception: Connection refused - connect(2)\n",
      "I, [2017-02-06T17:58:16.783061 #27368]  INFO -- : Sending config response from rvpcmk01-cr: {\"status\"=&gt;\"error\"}\n",
      "I, [2017-02-06T17:58:16.783169 #27368]  INFO -- : Sending config response from rvpcmk02-cr: {\"status\"=&gt;\"ok\", \"result\"=&gt;{\"tokens\"=&gt;\"accepted\"}}\n"
    ]
  }
  
  --Debug Output End--
  
  Error: Unable to communicate with rvpcmk01-cr
  rvpcmk02-cr: Authorized
  Error: Unable to synchronize and save tokens on nodes: rvpcmk01-cr. Are they authorized?
  </code></pre>
  
  <p>If you need some other info please tell me what and I will provide.</p>