<p>Writing a spring based application written in kotlin with hazelcast I encounter issue when deploying to kubernetes.</p>
<p>For hazelcast on kubernetes I use <a href="https://docs.hazelcast.com/hazelcast/5.1/deploy/configuring-kubernetes#using-kubernetes-in-dns-lookup-mode" rel="nofollow noreferrer">DNS lookup mode</a> for discovery.
I have the following hazelcast configuration:</p>
<pre><code>hazelcast:
  network:
    join:
      multicast:
        enabled: false
      kubernetes:
        enabled: true
        service-dns: my-application-hs
</code></pre>
<p>And the following service.yaml for the kubernetes deployment:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: my-application

spec:
  type: ClusterIP
  selector:
    component: my-application
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
---
apiVersion: v1
kind: Service
metadata:
  name: my-application-hs

spec:
  type: ClusterIP
  clusterIP: None
  selector:
    component: my-application
  ports:
    - name: hazelcast
      port: 5701
</code></pre>
<p>A hazelcast map is used like this:</p>
<pre><code>@Component
class CacheClientImplHazelcast(){

    private val hz: HazelcastInstance

    init{
        val serializer = SerializerConfig()
            .setTypeClass(MyDto::class.java)
            .setImplementation(MyDtoSerializer())
        val config = Config()
        config.serializationConfig.addSerializerConfig(serializer)
        hz = Hazelcast.newHazelcastInstance(config)
    }

    fun getAllData(): List&lt;MyDto&gt; {
        val map: IMap&lt;String, MyDto&gt; = hz.getMap(&quot;my-map&quot;)
        return map.values.toList()
    }

    fun putData(key:String, myDto: MyDto) {
        val map: IMap&lt;String, MyDto&gt; = hz.getMap(&quot;my-map&quot;)
        map.put(key, myDto)
    }

    override fun clear() {
        val map: IMap&lt;String, MyDto&gt; = hz.getMap(&quot;my-map&quot;)
        map.clear()
    }
}
</code></pre>
<p>When running 3 instances on kubernetes the logs from hazelcast always show me 4 entries, something like this:</p>
<pre><code>Members {size:4, ver:53} [
  Member [10.4.2.32]:5701 - c1e70d6f-a62d-4924-9815-36bb1f98f141
  Member [10.4.3.25]:5702 - be96c292-8847-4f56-ae32-f27f380d7c5b
  Member [10.4.2.32]:5702 - 6ca96bfd-eb74-4149-8630-a2488e76d97d
  Member [10.4.11.41]:5702 - 7e8b0bc9-ad2b-41eb-afbf-b7af9ed497bd this
] 
</code></pre>
<p>(Side question 1: why do I see 4 here instead of 3?)</p>
<p>Now even though it seems the member are connected (at least the logs of the nodes show all the same member uuids) when I write data on one node it is not available on the other nodes. Calls to <code>getAllData</code> only show data that has been put into the hazelcast map on this node. When I send requests to the individual nodes (curl on shell) I only see a fraction of the data. When I send a request on the normal url of the pod then with round-robin I get data of the different nodes, which is not synchronized.</p>
<p>If I run the very same application locally with the following hazelcast.yaml:</p>
<pre><code>hazelcast:
  network:
    join:
      multicast:
        enabled: false
</code></pre>
<p>Then it all works correctly and the cache really seems to be &quot;synchronized&quot; across different server instances running locally. For this test I start the application on different ports.</p>
<p>However what is also strange, even if I run 2 instances locally, the logs I see from hazelcast indicate there are 4 members:</p>
<pre><code>Members {size:4, ver:4} [
    Member [192.168.68.106]:5701 - 01713c9f-7718-4ed4-b532-aaf62443c425
    Member [192.168.68.106]:5702 - 09921004-88ef-4fe5-9376-b88869fde2bc
    Member [192.168.68.106]:5703 - cea4b13f-d538-48f1-b0f2-6c53678c5823 this
    Member [192.168.68.106]:5704 - 44d84e70-5b68-4c69-a45b-fee39bd75554
]  
</code></pre>
<p>(Side questions 2: Why do I see 4 members even I only started 2 locally?)</p>
<p>The main question now is: Why does this setup not work on kubernetes? Why does each node have a separated map that is not in sync with the other nodes?</p>
<p>Here some log message that could be relevant but I was not able to identify an issue with:</p>
<pre><code>2022-04-19T12:29:31.414588357Z2022-04-19 12:29:31.414 INFO 1 --- [cached.thread-7] c.h.i.server.tcp.TcpServerConnector : [10.4.11.41]:5702 [dev] [5.1.1] Connecting to /10.4.3.25:5702, timeout: 10000, bind-any: true
2022-04-19T12:29:31.414806473Z2022-04-19 12:29:31.414 INFO 1 --- [.IO.thread-in-2] c.h.i.server.tcp.TcpServerConnection : [10.4.11.41]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.11.41:5702 and /10.4.3.25:46475
2022-04-19T12:29:31.414905573Z2022-04-19 12:29:31.414 INFO 1 --- [cached.thread-4] c.h.i.server.tcp.TcpServerConnector : [10.4.11.41]:5702 [dev] [5.1.1] Connecting to /10.4.2.32:5702, timeout: 10000, bind-any: true
2022-04-19T12:29:31.416520854Z2022-04-19 12:29:31.416 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.4.3.25]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.3.25:5702 and /10.4.11.41:40455
2022-04-19T12:29:31.416833551Z2022-04-19 12:29:31.416 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.4.2.32]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.2.32:54433 and /10.4.11.41:5702
2022-04-19T12:29:31.417377114Z2022-04-19 12:29:31.417 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.4.11.41]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.11.41:40455 and /10.4.3.25:5702
2022-04-19T12:29:31.417545174Z2022-04-19 12:29:31.417 INFO 1 --- [.IO.thread-in-2] c.h.i.server.tcp.TcpServerConnection : [10.4.2.32]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.2.32:5702 and /10.4.11.41:53547
2022-04-19T12:29:31.418541840Z2022-04-19 12:29:31.418 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.4.11.41]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.11.41:53547 and /10.4.2.32:5702
2022-04-19T12:29:31.419763311Z2022-04-19 12:29:31.419 INFO 1 --- [.IO.thread-in-2] c.h.i.server.tcp.TcpServerConnection : [10.4.3.25]:5702 [dev] [5.1.1] Initialized new cluster connection between /10.4.3.25:46475 and /10.4.11.41:5702
2022-04-19T12:29:31.676218042Z2022-04-19 12:29:31.675 INFO 1 --- [gulis.migration] c.h.i.partition.impl.MigrationManager : [10.4.2.32]:5701 [dev] [5.1.1] Repartitioning cluster data. Migration tasks count: 271
                                                                                                                                                   
</code></pre>

Sjoerd222888
<p>There are two issues here but only one problem.</p>
<ol>
<li>why the extra instances</li>
</ol>
<p>Spring (Boot) will create a Hazelcast instance for you if it finds a Hazelcast config file and no <code>HazelcastInstance</code> @Bean. You can fix by excluding the <code>HazelcastAutoConfiguration.class</code> or by returning the instance you create in your component class as a @Bean.</p>
<ol start="2">
<li>why the data sync issue on kubernetes</li>
</ol>
<p>Each pod accidentally has 2 Hazelcast nodes, one on 5701 and one on 5702. But your <code>ClusterIP</code> only lists 5701. Some instances in the pod can't be reached from outside. This will go away when you fix the first issue.</p>
