<p>I'm following <a href="https://github.com/heketi/heketi/blob/master/docs/admin/install-kubernetes.md" rel="nofollow noreferrer">this instruction</a> to setup glusterfs on my kubernetes cluster. At <code>heketi-client/bin/heketi-cli setup-openshift-heketi-storage</code> part, heketi-cli tells me :</p>

<blockquote>
  <p>Error: volume create: heketidbstorage: failed: Host 192.168.99.25 not connected</p>
</blockquote>

<p>or sometimes: </p>

<blockquote>
  <p>Error: volume create: heketidbstorage: failed: Staging failed on 192.168.99.26. Error: Host 192.168.99.25 not connected</p>
</blockquote>

<p>heketi.json is</p>

<pre><code>{
  "_port_comment": "Heketi Server Port Number",
  "port": "8080",

  "_use_auth": "Enable JWT authorization. Please enable for deployment",
  "use_auth": false,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "7319"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "7319"
    }
  },

  "_glusterfs_comment": "GlusterFS Configuration",
  "glusterfs": {
    "_executor_comment": "Execute plugin. Possible choices: mock, kubernetes, ssh",
    "executor": "kubernetes",

    "_db_comment": "Database file name",
    "db": "/var/lib/heketi/heketi.db",

    "kubeexec": {
      "rebalance_on_expansion": true
    },

    "sshexec": {
      "rebalance_on_expansion": true,
      "keyfile": "/etc/heketi/private_key",
      "fstab": "/etc/fstab",
      "port": "22",
      "user": "root",
      "sudo": false
    }
  },

  "_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.",
  "backup_db_to_kube_secret": false
}
</code></pre>

<p>topology-sample.json is</p>

<pre><code>{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "redis-test25"
              ],
              "storage": [
                "192.168.99.25"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sda7",
              "destroydata": true
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "redis-test26"
              ],
              "storage": [
                "192.168.99.26"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sda7",
              "destroydata": true
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "redis-test01"
              ],
              "storage": [
                "192.168.99.113"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sda7",
              "destroydata": true
            }
          ]
        }
      ]
    }
  ]
}
</code></pre>

<p>The heketi-cli is v8.0.0 and kubernetes is v1.12.3</p>

<p>How do I fix this problem?</p>

<hr>

<p>Update: Just found that I missed the iptables part, but now the message becomes</p>

<blockquote>
  <p>Error: volume create: heketidbstorage: failed: Host 192.168.99.25 is not in 'Peer in Cluster' state</p>
</blockquote>

<p>seems that one of the glusterfs pod cannot connect to others, I tried <code>kubectl exec -i glusterfs-59ftx -- gluster peer status</code>:</p>

<pre><code>Number of Peers: 2

Hostname: 192.168.99.26
Uuid: 6950db9a-3d60-4625-b642-da5882396bee
State: Peer Rejected (Disconnected)

Hostname: 192.168.99.113
Uuid: 78983466-4499-48d2-8411-2c3e8c70f89f
State: Peer Rejected (Disconnected)
</code></pre>

<p>while the other one said:</p>

<pre><code>Number of Peers: 1

Hostname: 192.168.99.26
Uuid: 23a0114d-65b8-42d6-8067-7efa014af68d
State: Peer in Cluster (Connected)
</code></pre>

Leisen Chang
<p>I solved these problems by myself.</p>

<p>For first part, the reason is that I didn't setup iptables in every nodes according to <a href="https://github.com/heketi/heketi/blob/master/docs/admin/install-kubernetes.md#infrastructure-requirements" rel="nofollow noreferrer">Infrastructure Requirements</a>. </p>

<p>For second part according to <a href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Resolving%20Peer%20Rejected/" rel="nofollow noreferrer">this article</a>, delete all file in /var/lib/glusterd except glusterd.info and then start over from <a href="https://github.com/heketi/heketi/blob/master/docs/admin/install-kubernetes.md#kubernetes-deployment" rel="nofollow noreferrer">Kubernete Deploy</a>. </p>
