<p>Iam using embedded Hazelcast deployed in Kubernetes(GKE) to enable 2nd level cache Hibernate in Spring Boot application. Spingboot app connect to Spanner by using spanner-jdbc (<a href="https://github.com/olavloite/spanner-jdbc" rel="nofollow noreferrer">https://github.com/olavloite/spanner-jdbc</a>). 
I deploy 1 service (replicas=2) to k8s, however hazelcast instances located in 2 pods cannot connect stably (it works when server starts but then seem disconnected after, please the below log). This causes 2nd level cache cannot be shared between 2 pods. Anyone can help this situation? Thank you.</p>

<pre><code>&lt;groupId&gt;com.hazelcast.poc&lt;/groupId&gt;
&lt;artifactId&gt;hazelcast-poc&lt;/artifactId&gt;
&lt;version&gt;0.0.1-SNAPSHOT&lt;/version&gt;
&lt;name&gt;hazelcast-poc&lt;/name&gt;
&lt;description&gt;Demo project for Spring Boot&lt;/description&gt;

&lt;properties&gt;
    &lt;java.version&gt;1.8&lt;/java.version&gt;
&lt;/properties&gt;

&lt;dependencies&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;nl.topicus&lt;/groupId&gt;
        &lt;artifactId&gt;spanner-jdbc&lt;/artifactId&gt;
        &lt;version&gt;1.0.8&lt;/version&gt;
    &lt;/dependency&gt;

    &lt;dependency&gt;
        &lt;groupId&gt;nl.topicus&lt;/groupId&gt;
        &lt;artifactId&gt;spanner-hibernate&lt;/artifactId&gt;
        &lt;version&gt;0.7&lt;/version&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
        &lt;artifactId&gt;spring-boot-starter-data-jpa&lt;/artifactId&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
        &lt;artifactId&gt;spring-boot-starter-data-rest&lt;/artifactId&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;com.hazelcast&lt;/groupId&gt;
        &lt;artifactId&gt;hazelcast&lt;/artifactId&gt;
        &lt;version&gt;3.11&lt;/version&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;com.hazelcast&lt;/groupId&gt;
        &lt;artifactId&gt;hazelcast-hibernate5&lt;/artifactId&gt;
        &lt;version&gt;1.3.0&lt;/version&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;com.hazelcast&lt;/groupId&gt;
        &lt;artifactId&gt;hazelcast-kubernetes&lt;/artifactId&gt;
        &lt;version&gt;1.3.1&lt;/version&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
        &lt;artifactId&gt;spring-boot-starter-test&lt;/artifactId&gt;
        &lt;scope&gt;test&lt;/scope&gt;
    &lt;/dependency&gt;

    &lt;dependency&gt;
        &lt;groupId&gt;org.assertj&lt;/groupId&gt;
        &lt;artifactId&gt;assertj-core&lt;/artifactId&gt;
        &lt;version&gt;3.8.0&lt;/version&gt;
        &lt;scope&gt;test&lt;/scope&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;org.hamcrest&lt;/groupId&gt;
        &lt;artifactId&gt;hamcrest-all&lt;/artifactId&gt;
        &lt;version&gt;1.3&lt;/version&gt;
        &lt;scope&gt;test&lt;/scope&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;com.fasterxml.uuid&lt;/groupId&gt;
        &lt;artifactId&gt;java-uuid-generator&lt;/artifactId&gt;
        &lt;version&gt;3.1.5&lt;/version&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;org.hibernate&lt;/groupId&gt;
        &lt;artifactId&gt;hibernate-java8&lt;/artifactId&gt;
        &lt;version&gt;${hibernate.version}&lt;/version&gt;
    &lt;/dependency&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;junit&lt;/groupId&gt;
        &lt;artifactId&gt;junit&lt;/artifactId&gt;
        &lt;scope&gt;test&lt;/scope&gt;
    &lt;/dependency&gt;
&lt;/dependencies&gt;

&lt;build&gt;
    &lt;plugins&gt;
        &lt;plugin&gt;
            &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
            &lt;artifactId&gt;spring-boot-maven-plugin&lt;/artifactId&gt;
            &lt;version&gt;2.0.1.RELEASE&lt;/version&gt;
            &lt;executions&gt;
                &lt;execution&gt;
                    &lt;goals&gt;
                        &lt;goal&gt;repackage&lt;/goal&gt;
                    &lt;/goals&gt;
                &lt;/execution&gt;
            &lt;/executions&gt;
        &lt;/plugin&gt;
        &lt;plugin&gt;
            &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
            &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;
            &lt;version&gt;3.8.0&lt;/version&gt;
            &lt;configuration&gt;
                &lt;source&gt;1.8&lt;/source&gt;
                &lt;target&gt;1.8&lt;/target&gt;
                &lt;encoding&gt;UTF-8&lt;/encoding&gt;
            &lt;/configuration&gt;
        &lt;/plugin&gt;
    &lt;/plugins&gt;
&lt;/build&gt;
</code></pre>

<p></p>

<p>Springboot Application.java:</p>

<pre><code>public static void main(String[] args) throws IOException {
    CloudSpannerDriver.setLogLevel(CloudSpannerDriver.DEBUG);
    Writer writer = new FileWriter("jdbc.log", false);
    DriverManager.setLogWriter(new PrintWriter(writer));SpringApplication.run(Application.class, args);
}

@Bean
public Config hazelCastConfig(){
    Config config = new Config();
    config
            .addMapConfig(
                    new MapConfig()
                            .setName("employees")
                            .setMaxSizeConfig(new MaxSizeConfig(200, MaxSizeConfig.MaxSizePolicy.FREE_HEAP_SIZE))
                            .setEvictionPolicy(EvictionPolicy.LRU)
                            .setTimeToLiveSeconds(20000));
    JoinConfig joinConfig = config.getNetworkConfig().getJoin();
    joinConfig.getMulticastConfig().setEnabled(false);
    joinConfig.getKubernetesConfig().setEnabled(true).setProperty("namespace", "default");

    return config;
}
</code></pre>

<p>}</p>

<p>deployment.yaml:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
 name: hazelcast-embedded-0
spec:
  replicas: 2
  selector:
  matchLabels:
  app: hazelcast-embedded-0
  template:
   metadata:
    labels:
      app: hazelcast-embedded-0
    spec:
    volumes:
      - name: hazelcast-key
        secret:
        secretName: hazelcast-key
  containers:
    - name: hazelcast-embedded
      image: gcr.io/testproject-221915/hazelcast- poc@sha256:5a103d7e2e56c46e9e42617232e472891bda5d121139fe7e41057596bd71bf00
      imagePullPolicy: Always
      volumeMounts:
        - name: hazelcast-key
          mountPath: /var/secrets/google
      env:
      - name: GOOGLE_APPLICATION_CREDENTIALS
        value: /var/secrets/google/key.json
      ports:
       - containerPort: 5701
       - containerPort: 8080


  apiVersion: v1
  kind: Service
  metadata:
  name: hazelcast-embedded-0
  spec:
   type: LoadBalancer
   selector:
    app: hazelcast-embedded-0
   ports:
   - name: hazelcast
    port: 5701
   - name: app
     port: 8080
</code></pre>

<p>Spanner Datasource file:</p>

<pre><code>@Bean
public LocalContainerEntityManagerFactoryBean spannerEntityManager() {
    LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
    em.setDataSource(spannerDataSource());
    em.setPersistenceUnitName("SpannerPU");
    HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
    em.setJpaVendorAdapter(vendorAdapter);
    HashMap&lt;String, Object&gt; properties = new HashMap&lt;&gt;();
    properties.put("hibernate.dialect", "nl.topicus.hibernate.dialect.CloudSpannerDialect");
    properties.put("hibernate.temp.use_jdbc_metadata_defaults", false);
    properties.put("hibernate.show_sql", showSql);
    properties.put("hibernate.format_sql", formatSql);
    properties.put("hibernate.cache.use_query_cache", true);
    properties.put("hibernate.cache.use_second_level_cache", true);
    properties.put("hibernate.cache.region.factory_class", "com.hazelcast.hibernate.HazelcastCacheRegionFactory");
    if("development".equals(profileActive)) {
        properties.put("hibernate.generate_statistics", statistic);
        properties.put("org.hibernate.stat", hibernateStat);            
    }        
    em.setJpaPropertyMap(properties);

    return em;
}
</code></pre>

<p>Application.properties</p>

<pre><code>spanner.datasource.className=nl.topicus.jdbc.CloudSpannerDriver
spanner.datasource.url=jdbc:cloudspanner://;Project=testproject- 
221915;Instance=test-hazelcast;Database=hazelcast
spanner.hibernate.dialect=
 nl.topicus.hibernate.dialect.CloudSpannerDialect
spanner.hibernate.show-sql=true
spanner.hibernate.format_sql=true
spanner.datasource.hikari.maximum-pool-size=2 #depend on 
 microservice
spanner.datasource.hikari.minimum-idle=2
spanner.datasource.hikari.connection-timeout=60000

spring.profiles.active=dev
hibernate.generate_statistics=true
org.hibernate.stat=DEBUG
</code></pre>

<p><a href="https://i.stack.imgur.com/rVCC7.png" rel="nofollow noreferrer">Log_pod1</a>
<a href="https://i.stack.imgur.com/ctbqu.png" rel="nofollow noreferrer">Log_pod2</a></p>

<p>As you see, the second log has only 1 member whereas the first has 2 members.
I tested by issuing rest request to service which then distributed to each pods, the test result is sometimes cache are shared between two pods, sometimes not.</p>

<p>I expected 2nd Hibernate cache should be shared between 2 pods in the service</p>

Tom Ho
<p>You are creating two Hazelcast instances</p>

<p>At 9:36:44.990 a kubernetes discovery instance</p>

<p>At 9:36:53.706 a multicast discovery instance</p>

<p>Might be <a href="https://stackoverflow.com/questions/40325232/spring-boot-hazelcast-hibernate-5-l2-cache">this</a> again</p>

<p>Set <code>hibernate.cache.hazelcast.instance_name</code> system property and <code>config.setInstanceName("whatever")</code> pairing should match</p>
