<p>We're trying to use Red Hat Data Grid (RHDG)/Infinispan in our OCP (4.5.36) cluster. We have the latest official RHDG Operator installed and a Cache type cluster defined. (Which is apparently a k8s StatefulSet.)</p>
<p>I've then configured a WebSphere Liberty container/Deployment to try to use that Infinispan cluster for its sessions, as described in <a href="https://github.com/WASdev/ci.docker#session-caching" rel="nofollow noreferrer">https://github.com/WASdev/ci.docker#session-caching</a>.</p>
<p>Both the Infinispan cluster and the Liberty Deployment are in the same Project/namespace.</p>
<p>However, the Liberty container fails to connect, and the Infinispan containers are reporting several warnings of their own.</p>
<p>The Liberty container &quot;client&quot; log:</p>
<pre><code>INFINISPAN_SERVICE_NAME(original): session-infinispan
INFINISPAN_SERVICE_NAME(normalized): SESSION_INFINISPAN
INFINISPAN_HOST: 172.30.137.86
INFINISPAN_PORT: 11222
INFINISPAN_USER: developer
INFINISPAN_PASS: &lt;redacted&gt;

Launching defaultServer (WebSphere Application Server 21.0.0.3/wlp-1.0.50.cl210320210309-1101) on Eclipse OpenJ9 VM, version 1.8.0_282-b08 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKE0100I: This product is licensed for development, and limited production use. The full license terms can be viewed here: https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/license/base_ilan/ilan/21.0.0.3/lafiles/en.html
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ibm/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ibm/wlp/usr/servers/defaultServer/configDropins/overrides/infinispan-client-sessioncache.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://payment-engine-6dcc5b6d5-jclx2:9080/payment/
[ERROR   ] ISPN004007: Exception encountered. Retry 10 out of 10
org.infinispan.client.hotrod.exceptions.TransportException:: ISPN004071: Connection to 172.30.137.86/172.30.137.86:11222 was closed while waiting for response.
[ERROR   ] SESN0307E: An exception occurred when initializing the cache. The exception is: org.infinispan.client.hotrod.exceptions.TransportException:: org.infinispan.client.hotrod.exceptions.TransportException:: ISPN004071: Connection to 172.30.137.86/172.30.137.86:11222 was closed while waiting for response.
    at org.infinispan.client.hotrod.impl.transport.netty.ActivationHandler.exceptionCaught(ActivationHandler.java:53)
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300)
...
</code></pre>
<p>What looks like the relevant part of the Inifinispan container log:</p>
<pre><code>03:40:18,628 WARN  (SINGLE_PORT-ServerIO-4-2) [io.netty.handler.ssl.ApplicationProtocolNegotiationHandler] [id: 0xc39380c8, L:/10.254.0.248:11222 ! R:/10.254.2.65:32986] TLS handshake failed: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: a0061e21000003ffffffff0f0000
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1254)
</code></pre>
<p>(Actually, there are several Infinispan startup WARNs, mostly about deprecated capabilities. But this is the only one with a stack trace, so I'm jumping to the conclusion that it might be &quot;the culprit&quot;)</p>
<p>Also, this is the Infinispan Service, so you can see the IP and port match what the Liberty container is using:</p>
<p><a href="https://i.stack.imgur.com/P4imC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/P4imC.png" alt="Infinispan Service details" /></a></p>

dbreaux
<p>Working through this on <a href="https://infinispan.zulipchat.com/#narrow/stream/118645-infinispan/topic/RHDG.20OCP.20Liberty.20failure" rel="nofollow noreferrer">the Infinispan chat service</a>, it does appear that there's incorrect or incomplete setup of SSL/TLS.</p>
<p>I had attempted to remove encryption in the Infinispan cluster, but I either didn't sufficiently restart components or you can't change it after the fact. Removing the cluster and recreating with it disabled, though, enabled the Liberty communication to work.</p>
<p>The following CR YAML works:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: session-infinispan
spec:
  replicas: 1
  service:
    type: Cache 
  security:
    endpointEncryption:
      type: None
</code></pre>
<p>Now to pursue what's missing from the Liberty setup to make use of SSL correctly. The Infinispan chat conversation says that this Liberty XML setup from the official image:</p>
<pre class="lang-xml prettyprint-override"><code>&lt;server&gt;
  &lt;featureManager&gt;
    &lt;feature&gt;sessionCache-1.0&lt;/feature&gt;
  &lt;/featureManager&gt;
  &lt;httpSessionCache libraryRef=&quot;InfinispanLib&quot;&gt;
    &lt;properties infinispan.client.hotrod.server_list=&quot;${INFINISPAN_HOST}:${INFINISPAN_PORT}&quot;/&gt;
    &lt;properties infinispan.client.hotrod.marshaller=&quot;org.infinispan.commons.marshall.JavaSerializationMarshaller&quot;/&gt;
    &lt;properties infinispan.client.hotrod.java_serial_whitelist=&quot;.*&quot;/&gt;
    &lt;properties infinispan.client.hotrod.auth_username=&quot;${INFINISPAN_USER}&quot;/&gt;
    &lt;properties infinispan.client.hotrod.auth_password=&quot;${INFINISPAN_PASS}&quot;/&gt;
    &lt;properties infinispan.client.hotrod.auth_realm=&quot;default&quot;/&gt;
    &lt;properties infinispan.client.hotrod.sasl_mechanism=&quot;DIGEST-MD5&quot;/&gt;
    &lt;properties infinispan.client.hotrod.auth_server_name=&quot;infinispan&quot;/&gt;
  &lt;/httpSessionCache&gt;
  &lt;httpSessionCache enableBetaSupportForInfinispan=&quot;true&quot;/&gt; &lt;!-- TODO remove once no longer gated --&gt;
  &lt;library id=&quot;InfinispanLib&quot;&gt;
    &lt;fileset dir=&quot;${shared.resource.dir}/infinispan&quot; includes=&quot;*.jar&quot;/&gt;
  &lt;/library&gt;
&lt;/server&gt;
</code></pre>
<p><a href="https://infinispan.org/docs/infinispan-operator/master/operator.html#hotrod_properties-clients" rel="nofollow noreferrer">Needs the following properties added</a>:</p>
<pre><code># Encryption
infinispan.client.hotrod.sni_host_name=$SERVICE_HOSTNAME
# Path to the TLS certificate.
# Clients automatically generate trust stores from certificates.
infinispan.client.hotrod.trust_store_path=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
</code></pre>
