<p>I have a spring boot 2.5 application with spring spring security 5 where I am using embedded hazelcast cache to back spring sessions. This application is deployed on openshift with two pods where same application is running, hence I have used hazelcast kubernetes plugin for service discovery. Everything is working as expected. However, I can see application logs are flooded with below log lines.
<strong>Any suggestion what is wrong with the hazelcast configuration ? Why so many log lines are generated ?</strong></p>
<p><strong>Generated logs</strong></p>
<p>10.1.8.58 is IP address of second pod which joined cluster later and logs are printed in this pod only.</p>
<pre><code>{&quot;timestamp&quot;:&quot;2021-06-03T19:00:43.732+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;hz.silly_gates.priority-generic-operation.thread-0&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] Removing null, since it thinks it's already split from this cluster and looking to merge.&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:00:43.733+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;ERROR&quot;,&quot;thread&quot;:&quot;hz.silly_gates.priority-generic-operation.thread-0&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] Target is this node! -&gt; [10.1.8.58]:5701&quot;,&quot;stack_trace&quot;:&quot;&lt;#d3566be0&gt; j.l.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701\n\tat c.h.s.i.o.i.OutboundResponseHandler.checkTarget(OutboundResponseHandler.java:226)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendNormalResponse(OutboundResponseHandler.java:125)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendResponse(OutboundResponseHandler.java:88)\n\tat c.h.s.i.o.Operation.sendResponse(Operation.java:475)\n\tat c.h.s.i.o.i.OperationRunnerImpl.call(OperationRunnerImpl.java:282)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:248)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:469)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:197)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:137)\n\tat c.h.s.i.o.i.OperationThread.executeRun(OperationThread.java:123)\n\tat c.h.i.u.e.HazelcastManagedThread.run(HazelcastManagedThread.java:102)\n&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:00:43.734+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;WARN&quot;,&quot;thread&quot;:&quot;hz.silly_gates.priority-generic-operation.thread-0&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] While sending op error... op: com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp{serviceName='hz:core:clusterService', identityHash=1017993990, partitionId=-1, replicaIndex=0, callId=324850, invocationTime=1622710843731 (2021-06-03 19:00:43.731), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, error: java.lang.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701&quot;,&quot;stack_trace&quot;:&quot;&lt;#dbcc9949&gt; j.l.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701, response: ErrorResponse{callId=324850, urgent=true, cause=java.lang.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701}\n\tat c.h.s.i.o.i.OutboundResponseHandler.send(OutboundResponseHandler.java:113)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendResponse(OutboundResponseHandler.java:96)\n\tat c.h.s.i.o.Operation.sendResponse(Operation.java:475)\n\tat c.h.s.i.o.i.OperationRunnerImpl.sendResponseAfterOperationError(OperationRunnerImpl.java:425)\n\tat c.h.s.i.o.i.OperationRunnerImpl.handleOperationError(OperationRunnerImpl.java:419)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:253)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:469)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:197)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:137)\n\tat c.h.s.i.o.i.OperationThread.executeRun(OperationThread.java:123)\n\tat c.h.i.u.e.HazelcastManagedThread.run(HazelcastManagedThread.java:102)\n&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:00:43.786+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;hz.silly_gates.InvocationMonitorThread&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] Invocations:2 timeouts:1 backup-timeouts:0&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:02:43.732+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;hz.silly_gates.priority-generic-operation.thread-0&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] Removing null, since it thinks it's already split from this cluster and looking to merge.&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:02:43.733+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;ERROR&quot;,&quot;thread&quot;:&quot;hz.silly_gates.priority-generic-operation.thread-0&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] Target is this node! -&gt; [10.1.8.58]:5701&quot;,&quot;stack_trace&quot;:&quot;&lt;#d3566be0&gt; j.l.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701\n\tat c.h.s.i.o.i.OutboundResponseHandler.checkTarget(OutboundResponseHandler.java:226)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendNormalResponse(OutboundResponseHandler.java:125)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendResponse(OutboundResponseHandler.java:88)\n\tat c.h.s.i.o.Operation.sendResponse(Operation.java:475)\n\tat c.h.s.i.o.i.OperationRunnerImpl.call(OperationRunnerImpl.java:282)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:248)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:469)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:197)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:137)\n\tat c.h.s.i.o.i.OperationThread.executeRun(OperationThread.java:123)\n\tat c.h.i.u.e.HazelcastManagedThread.run(HazelcastManagedThread.java:102)\n&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:02:43.734+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;WARN&quot;,&quot;thread&quot;:&quot;hz.silly_gates.priority-generic-operation.thread-0&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] While sending op error... op: com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp{serviceName='hz:core:clusterService', identityHash=1466719815, partitionId=-1, replicaIndex=0, callId=324859, invocationTime=1622710963731 (2021-06-03 19:02:43.731), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, error: java.lang.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701&quot;,&quot;stack_trace&quot;:&quot;&lt;#dbcc9949&gt; j.l.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701, response: ErrorResponse{callId=324859, urgent=true, cause=java.lang.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701}\n\tat c.h.s.i.o.i.OutboundResponseHandler.send(OutboundResponseHandler.java:113)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendResponse(OutboundResponseHandler.java:96)\n\tat c.h.s.i.o.Operation.sendResponse(Operation.java:475)\n\tat c.h.s.i.o.i.OperationRunnerImpl.sendResponseAfterOperationError(OperationRunnerImpl.java:425)\n\tat c.h.s.i.o.i.OperationRunnerImpl.handleOperationError(OperationRunnerImpl.java:419)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:253)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:469)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:197)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:137)\n\tat c.h.s.i.o.i.OperationThread.executeRun(OperationThread.java:123)\n\tat c.h.i.u.e.HazelcastManagedThread.run(HazelcastManagedThread.java:102)\n&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:02:43.786+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;hz.silly_gates.InvocationMonitorThread&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] Invocations:2 timeouts:1 backup-timeouts:0&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:04:43.732+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;hz.silly_gates.priority-generic-operation.thread-0&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] Removing null, since it thinks it's already split from this cluster and looking to merge.&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:04:43.733+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;ERROR&quot;,&quot;thread&quot;:&quot;hz.silly_gates.priority-generic-operation.thread-0&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] Target is this node! -&gt; [10.1.8.58]:5701&quot;,&quot;stack_trace&quot;:&quot;&lt;#d3566be0&gt; j.l.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701\n\tat c.h.s.i.o.i.OutboundResponseHandler.checkTarget(OutboundResponseHandler.java:226)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendNormalResponse(OutboundResponseHandler.java:125)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendResponse(OutboundResponseHandler.java:88)\n\tat c.h.s.i.o.Operation.sendResponse(Operation.java:475)\n\tat c.h.s.i.o.i.OperationRunnerImpl.call(OperationRunnerImpl.java:282)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:248)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:469)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:197)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:137)\n\tat c.h.s.i.o.i.OperationThread.executeRun(OperationThread.java:123)\n\tat c.h.i.u.e.HazelcastManagedThread.run(HazelcastManagedThread.java:102)\n&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:04:43.733+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;WARN&quot;,&quot;thread&quot;:&quot;hz.silly_gates.priority-generic-operation.thread-0&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] While sending op error... op: com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp{serviceName='hz:core:clusterService', identityHash=244619937, partitionId=-1, replicaIndex=0, callId=324868, invocationTime=1622711083731 (2021-06-03 19:04:43.731), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, error: java.lang.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701&quot;,&quot;stack_trace&quot;:&quot;&lt;#dbcc9949&gt; j.l.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701, response: ErrorResponse{callId=324868, urgent=true, cause=java.lang.IllegalArgumentException: Target is this node! -&gt; [10.1.8.58]:5701}\n\tat c.h.s.i.o.i.OutboundResponseHandler.send(OutboundResponseHandler.java:113)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendResponse(OutboundResponseHandler.java:96)\n\tat c.h.s.i.o.Operation.sendResponse(Operation.java:475)\n\tat c.h.s.i.o.i.OperationRunnerImpl.sendResponseAfterOperationError(OperationRunnerImpl.java:425)\n\tat c.h.s.i.o.i.OperationRunnerImpl.handleOperationError(OperationRunnerImpl.java:419)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:253)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:469)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:197)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:137)\n\tat c.h.s.i.o.i.OperationThread.executeRun(OperationThread.java:123)\n\tat c.h.i.u.e.HazelcastManagedThread.run(HazelcastManagedThread.java:102)\n&quot;}
{&quot;timestamp&quot;:&quot;2021-06-03T19:04:43.786+10:00&quot;,&quot;app&quot;:&quot;my-protected-application&quot;,&quot;logLevel&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;hz.silly_gates.InvocationMonitorThread&quot;,&quot;eventSource&quot;:&quot;com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor&quot;,&quot;message&quot;:&quot;[10.1.8.58]:5701 [spring-session-cluster] [4.2] Invocations:2 timeouts:1 backup-timeouts:0&quot;}
</code></pre>
<p>Maven dependencies included are</p>
<pre><code>&lt;dependency&gt;
    &lt;groupId&gt;com.hazelcast&lt;/groupId&gt;
    &lt;artifactId&gt;hazelcast&lt;/artifactId&gt;
    &lt;version&gt;4.2&lt;/version&gt;
&lt;/dependency&gt;
&lt;dependency&gt;
    &lt;groupId&gt;org.springframework.session&lt;/groupId&gt;
    &lt;artifactId&gt;spring-session-hazelcast&lt;/artifactId&gt;
&lt;/dependency&gt;
&lt;dependency&gt;
    &lt;groupId&gt;com.hazelcast&lt;/groupId&gt;
    &lt;artifactId&gt;hazelcast-kubernetes&lt;/artifactId&gt;
    &lt;version&gt;2.2.2&lt;/version&gt;
&lt;/dependency&gt;
</code></pre>
<p>Kubernetes service discovery is configured as below :</p>
<pre><code>var joinConfig = config.getNetworkConfig().getJoin();
joinConfig.getTcpIpConfig().setEnabled(false);
joinConfig.getMulticastConfig().setEnabled(false);
joinConfig.getKubernetesConfig().setEnabled(true)
                .setProperty(&quot;service-dns&quot;, &quot;dns-name&quot;);
</code></pre>

user2800089
<p>The exception you get <code>SplitBrainMergeValidationOp</code> means that the Hazelcast cluster might have been started in the split-brain and later tries to merge into one cluster. Could you check if you follow all the <a href="https://github.com/hazelcast/hazelcast-kubernetes#requirements-and-recommendations" rel="nofollow noreferrer">Hazelcast Kubernetes recommendations</a>?</p>
<p>Especially, check if you use StatefulSet (not Deployment). In the case of DNS Lookup discovery, using Deployment may cause Hazelcast to start in the split-brain mode.</p>
