<p>I am trying to build an OrientDB cluster which consists of 3 nodes with Kubernetes (in Minikube). I am using StatefulSet with all the cluster configurations mounted as volumes.
The problem is when I check all the pods' logs it looks like each pod creates its own cluster instead of joining a single cluster.
What could be the cause of it?</p>

<p>Hazelcast.xml config looks like this:</p>

<pre><code>&lt;hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.3.xsd"
           xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"&gt;
  &lt;properties&gt;
    &lt;property name="hazelcast.discovery.enabled"&gt;true&lt;/property&gt;
  &lt;/properties&gt;
  &lt;network&gt;
    &lt;join&gt;
      &lt;!-- deactivate normal discovery --&gt;
      &lt;multicast enabled="false"/&gt;
      &lt;tcp-ip enabled="false"/&gt;
      &lt;!-- activate the Kubernetes plugin --&gt;
      &lt;discovery-strategies&gt;
        &lt;discovery-strategy enabled="true"
                            class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy"&gt;
          &lt;properties&gt;
            &lt;property name="service-dns"&gt;orientdbservice.default.svc.cluster.local&lt;/property&gt;
            &lt;property name="service-dns-timeout"&gt;10&lt;/property&gt;
          &lt;/properties&gt;
        &lt;/discovery-strategy&gt;
      &lt;/discovery-strategies&gt;
    &lt;/join&gt;
  &lt;/network&gt;
&lt;/hazelcast&gt; 
</code></pre>

<p>I am using service dns name as one of the approaches to use Hazelcast plugin with Kubernetes.</p>

<p>default-distributed-db-config.json file:</p>

<pre><code>{
    "autoDeploy": true,
    "hotAlignment": false, 
    "executionMode": "undefined",
    "readQuorum": 1,
    "writeQuorum": "majority",
    "readYourWrites": true,
    "newNodeStrategy": "static",
    "servers": {
        "*": "master"
    },
    "clusters":{
        "internal": {
        },
        "index": {
    },
        "*": { 
      "servers" : [ "&lt;NEW_NODE" ] 
    } 
    }
}   
</code></pre>

<p>Listeners part from orientdb-server-config.xml:</p>

<pre><code>&lt;listeners&gt;
    &lt;listener protocol="binary" socket="default" port-range="2424-2430" ip-address="0.0.0.0"/&gt;
    &lt;listener protocol="http" socket="default" port-range="2480-2490" ip-address="0.0.0.0"&gt;
        &lt;commands&gt;
            &lt;command implementation="com.orientechnologies.orient.server.network.protocol.http.command.get.OServerCommandGetStaticContent" pattern="GET|www GET|studio/ GET| GET|*.htm GET|*.html GET|*.xml GET|*.jpeg GET|*.jpg GET|*.png GET|*.gif GET|*.js GET|*.css GET|*.swf GET|*.ico GET|*.txt GET|*.otf GET|*.pjs GET|*.svg GET|*.json GET|*.woff GET|*.woff2 GET|*.ttf GET|*.svgz" stateful="false"&gt;
                &lt;parameters&gt;
                    &lt;entry value="Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\nPragma: no-cache" name="http.cache:*.htm *.html"/&gt;
                    &lt;entry value="Cache-Control: max-age=120" name="http.cache:default"/&gt;
                &lt;/parameters&gt;
            &lt;/command&gt;
        &lt;/commands&gt;
        &lt;parameters&gt;
            &lt;parameter value="utf-8" name="network.http.charset"/&gt;
            &lt;parameter value="true" name="network.http.jsonResponseError"/&gt;
            &lt;parameter value="Access-Control-Allow-Origin:*;Access-Control-Allow-Credentials: true" name="network.http.additionalResponseHeaders"/&gt;
        &lt;/parameters&gt;
    &lt;/listener&gt;
&lt;/listeners&gt;
</code></pre>

<p>StatefulSet I have defined:</p>

<pre><code>kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: orientdbservice
spec:
  serviceName: orientdbservice
  replicas: 3
  selector:
    matchLabels:
      service: orientdb
      type: container-deployment
  template:
    metadata:
      labels:
        service: orientdb
        type: container-deployment
    spec:
      containers:
      - name: orientdbservice
        image: orientdb:2.2.36
        command: ["/bin/sh","-c", "cp /configs/* /orientdb/config/ ; /orientdb/bin/server.sh -Ddistributed=true" ]
        env:
        - name: ORIENTDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: orientdb-password
              key: password.txt
        ports:
        - containerPort: 2424
          name: port-binary
        - containerPort: 2480
          name: port-http
        - containerPort: 5701
          name: hazelcast
        volumeMounts:
        - name: config
          mountPath: /orientdb/config
        - name: orientdb-config-backups
          mountPath: /configs/backups.json
          subPath: backups.json
        - name: orientdb-config-events
          mountPath: /configs/events.json
          subPath: events.json
        - name: orientdb-config-distributed
          mountPath: /configs/default-distributed-db-config.json
          subPath: default-distributed-db-config.json
        - name: orientdb-config-hazelcast
          mountPath: /configs/hazelcast.xml
          subPath: hazelcast.xml
        - name: orientdb-config-server
          mountPath: /configs/orientdb-server-config.xml
          subPath: orientdb-server-config.xml
        - name: orientdb-config-client-logs
          mountPath: /configs/orientdb-client-log.properties
          subPath: orientdb-client-log.properties
        - name: orientdb-config-server-logs
          mountPath: /configs/orientdb-server-log.properties
          subPath: orientdb-server-log.properties
        - name: orientdb-config-plugin
          mountPath: /configs/pom.xml
          subPath: pom.xml
        - name: orientdb-databases
          mountPath: /orientdb/databases
        - name: orientdb-backup
          mountPath: /orientdb/backup
        - name: orientdb-data
          mountPath: /orientdb/bin/data
      volumes:
      - name: config
        emptyDir: {}
      - name: orientdb-config-backups
        configMap:
          name: orientdb-configmap-backups
      - name: orientdb-config-events
        configMap:
          name: orientdb-configmap-events
      - name: orientdb-config-distributed
        configMap:
          name: orientdb-configmap-distributed
      - name: orientdb-config-hazelcast
        configMap:
          name: orientdb-configmap-hazelcast
      - name: orientdb-config-server
        configMap:
          name: orientdb-configmap-server
      - name: orientdb-config-client-logs
        configMap:
          name: orientdb-configmap-client-logs
      - name: orientdb-config-server-logs
        configMap:
          name: orientdb-configmap-server-logs
      - name: orientdb-config-plugin
        configMap:
          name: orientdb-configmap-plugin
      - name: orientdb-data
        hostPath:
          path: /import_data
          type: Directory
  volumeClaimTemplates:
  - metadata:
      name: orientdb-databases
      labels:
        service: orientdb
        type: pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 20Gi
  - metadata:
      name: orientdb-backup
      labels:
        service: orientdb
        type: pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi
</code></pre>

<p>Headkess service looks like this:</p>

<pre><code>kind: Service
apiVersion: v1
metadata:
  name: orientdbservice
  labels:
    service: orientdb
    type: headless-service
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    service: orientdb
    type: container-deployment
  ports:
  - port: 2424
    name: port-binary
  - port: 2480
    name: port-http
  - name: hazelcast
    port: 5701
</code></pre>

Cassie
<p>For Hazelcast members to discover themselves, you also need to:</p>

<ul>
<li>specify <code>containerPort</code> for Hazelcast (port <code>5701</code>)</li>
<li>create a Kubernetes Service for Hazelcast (port <code>5701</code>, in case of using DNS Lookup Discovery, the service needs to be <code>type: ClusterIP</code> and <code>clusterIP: None</code>)</li>
</ul>

<p>You can find more information about the configuration in the following sources:</p>

<ul>
<li><a href="https://github.com/hazelcast/hazelcast-kubernetes" rel="nofollow noreferrer">Official README</a> (<a href="https://github.com/hazelcast/hazelcast-kubernetes#dns-lookup" rel="nofollow noreferrer">DNS Lookup section</a>)</li>
<li><a href="https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/kubernetes/samples/springboot-k8s-hello-world" rel="nofollow noreferrer">Code Sample using DNS Lookup</a></li>
<li><a href="https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/kubernetes" rel="nofollow noreferrer">General Hazelcast Code Sample</a></li>
</ul>
