<p>We wanted to set up a high available Keycloak cluster on Kubernetes (with ldap as a user federation). We decided to use <a href="https://github.com/codecentric/helm-charts/tree/master/charts/keycloak" rel="nofollow noreferrer">codecentrics helm charts</a> since we were trying them out for a single Keycloak instance setup and that worked well. For the cluster we ran into a few issues while trying to set up everything correctly and didn't find the best sources in the wide internet. Therefore I decided to write a short summary what our main issues where and how we got through them.</p>
<p>Solutions to our problems where described on <a href="https://www.kainos.pl/blog/keycloak-ha-setup-in-multinode-kubernetes-cluster/" rel="nofollow noreferrer">this website</a> (amongst others), but things are described kind of very briefly and felt partly incomplete.</p>
<p>Issues we faced where:</p>
<ol>
<li>Choosing the correct <code>jgroups.discoveryProtocol</code></li>
<li>Adding the correct <code>discoveryProperties</code></li>
<li>Parts that need to be overridden in your own <code>values.yaml</code></li>
</ol>
<p>Bonus issues (we already faced with the single instance setup):</p>
<ol>
<li>Setting up an truststore to connect ldap as a user federation via ladps</li>
<li>Adding a custom theme for keycloak</li>
</ol>
<p>I will try and update this if things change due to codecentrics updating their helm charts.</p>
<p>Thanks to codecentrics for providing the helm charts by the way!</p>

<p><strong>Disclaimer:</strong><br />
This is the way we set it up - I hope this is helpful, but I do not take responsibility for configuration errors and resulting security flaws. Also we went through many different sources on the internet, I am sorry that I can't give credits to all of them, but it has been a few days since than an I can't get them together anymore...</p>
<h2>CODECENTRIC CHART VERSION &lt; 9.0.0</h2>
<p><strong>The main issues:</strong><br />
<strong>1. Choosing the correct jgroups.discoveryProtocol:</strong><br />
I will not explain things here but for us the correct protocol to use was <code>org.jgroups.protocols.JDBC_PING</code>. Find out more about the protocols (and general cluster setup) <a href="https://www.keycloak.org/2019/05/keycloak-cluster-setup.html" rel="nofollow noreferrer">here</a>.</p>
<pre><code>discoveryProtocol: org.jgroups.protocols.JDBC_PING
</code></pre>
<p>With JDBC_PING jgroups will manage instance discovery. Therefore and for caching user sessions  the database provided for keycloak will be enhanced with extra tables, e.g. <code>JGROUPSPING</code>.</p>
<p><strong>2. Setting up the discoveryProperties:</strong>
This needs to be set to</p>
<pre><code>discoveryProperties: &gt;
      &quot;datasource_jndi_name=java:jboss/datasources/KeycloakDS&quot;
</code></pre>
<p>to avoid an error like:</p>
<pre><code>java.lang.IllegalStateException: java.lang.IllegalArgumentException: 
Either the 4 configuration properties starting with 'connection_' or 
the datasource_jndi_name must be set
</code></pre>
<p><strong>3. Other parts that need to be set</strong> (as mostly described in the readme of codecentrics github and in the comments of the values.yaml in github as well):</p>
<ul>
<li>setting the <code>clusterDomain</code> according to your cluster</li>
<li>setting the number of <code>replicas</code> greater than 1 to enable clustering</li>
<li>setting the <code>service.type</code>: We went with <code>ClusterIP</code> but it also can work with other setups like <code>LoadBalancer</code> depending on your setup</li>
<li>optional but recommended: Setting either <code>maxUnavailable</code> or <code>minAvailable</code> to always have sufficient pods available according to your needs.</li>
<li>setting up our Ingress (which looks pretty much standard):</li>
</ul>
<pre><code> ingress:
    enabled: true
    path: /
    annotations: {
      kubernetes.io/ingress.class: nginx
    }
    hosts:
      - your.host.org
</code></pre>
<p><strong>Bonus issues:</strong></p>
<p><strong>1. The truststore:</strong><br />
To have Keycloak communicate with ldap via ldaps we had to set up a truststore with the certificate of our ldap in it:</p>
<p>Receive the certificate from ldap and save it somewhere:</p>
<pre class="lang-sh prettyprint-override"><code>openssl s_client -connect your.ldap.domain.org &lt; /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' &gt; /path/to/ldap.cert
</code></pre>
<p>Create a new keystore:</p>
<pre class="lang-sh prettyprint-override"><code>keytool -genkey -alias replserver \
-keyalg RSA -keystore /path/to/keystore.jks \
-dname &quot;CN=AddCommonName, OU=AddOrganizationalUnit, O=AddOrganisation, L=AddLocality, S=AddStateOrProvinceName, C=AddCountryName&quot; \
-storepass use_the_same_password \
-keypass use_the_same_password \
-deststoretype pkcs12
</code></pre>
<p>Add the downloaded certificate to the keystore:</p>
<pre class="lang-sh prettyprint-override"><code>keytool -import -alias ldaps -file /path/to/ldap.cert -storetype JKS -keystore path/to/keystore.jks   
</code></pre>
<p>Type in the required password: <code>use_the_same_password</code>.<br />
Trust the certificate by typing 'yes'.</p>
<p>Provide the keystore in a configmap:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl create configmap cert-keystore --from-file=path/to/keystore.jks
</code></pre>
<p>Enhance your values.yaml for the truststore:</p>
<p>Add and mount the config map:</p>
<pre><code>extraVolumes: |
    - name: cert-keystore
      configMap:
        name: cert-keystore

extraVolumeMounts: |
    - name: cert-keystore
      mountPath: &quot;/keystore/&quot;
      readOnly: true
</code></pre>
<p>Tell java tu use it:</p>
<pre><code>javaToolOptions: &gt;-
    -[maybe some other settings of yours]
    -Djavax.net.ssl.trustStore=/keystore/keystore.jks
    -Djavax.net.ssl.trustStorePassword=&lt;&lt;keystore_password&gt;&gt;
</code></pre>
<p>Since we didn't want to upload the keystore password to git we added a step to our pipeline where it gets <code>sed</code> into the <code>values.yaml</code>, replacing the <code>&lt;&lt;keystore_password&gt;&gt;</code>.</p>
<p><strong>2. Adding a custom theme:</strong><br />
Mainly we are providing a docker container with our custom theme in it:</p>
<pre><code>extraInitContainers: |
    - name: theme-provider
      image: docker_repo_url/themeContainer:version
      imagePullPolicy: IfNotPresent
      command:
        - sh
      args:
        - -c
        - |
          echo &quot;Copying theme...&quot;
          cp -R /custom-theme/* /theme
      volumeMounts:
        - name: theme
          mountPath: /theme
</code></pre>
<p>Add and mount the theme:</p>
<pre><code>extraVolumes: |
    - name: theme
      emptyDir: {}

extraVolumeMounts: |
    - name: theme
      mountPath: /opt/jboss/keycloak/themes/custom-theme
</code></pre>
<p>You now should be able to choose the custom theme in the Keycloak admin UI via <code>Realm Settings -&gt; Themes</code>.</p>
<h2>CODECENTRIC CHART VERSION 9.0.0 to 9.3.2 (and maybe higher)</h2>
<p><strong>1. Clustering</strong><br />
We are still going with JDBC_PING since we had problems with DNS_PING as described in the Codecentric Repo readme:</p>
<pre><code>extraEnv: |
  ## KEYCLOAK CONFIG
  - name: PROXY_ADDRESS_FORWARDING
    value: &quot;true&quot;
  ### CLUSTERING
  - name: JGROUPS_DISCOVERY_PROTOCOL
    value: org.jgroups.protocols.JDBC_PING
  - name: JGROUPS_DISCOVERY_PROPERTIES
    value: 'datasource_jndi_name=java:jboss/datasources/KeycloakDS'
  - name: CACHE_OWNERS_COUNT
    value: &quot;2&quot;
  - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
    value: &quot;2&quot;
</code></pre>
<p>With the service set up as ClusterIP:</p>
<pre><code>service:
  annotations: {}
  labels: {}
  type: ClusterIP
  loadBalancerIP: &quot;&quot;
  httpPort: 80
  httpNodePort: null
  httpsPort: 8443
  httpsNodePort: null
  httpManagementPort: 9990
  httpManagementNodePort: null
  extraPorts: []
</code></pre>
<p><strong>2. 502 Error Ingress Problem</strong><br />
We encountered a 502 error with Codecentrics chart 9.x.x for which fixing took a while to figure out. A solution for this is also described <a href="https://www.ibm.com/support/pages/502-error-ingress-keycloak-response" rel="nofollow noreferrer">here</a>, where we took our inspiration but for us the following ingress setup was enough:</p>
<pre><code>ingress:
  enabled: true
  servicePort: http
  # Ingress annotations
  annotations: {
    kubernetes.io/ingress.class: nginx,
    nginx.ingress.kubernetes.io/proxy-buffer-size: 128k,
  }
</code></pre>
<h2>CODECENTRIC CHART VERSION 9.5.0 (and maybe higher)</h2>
<p>Updating to 9.5.0 needs to be tested. Especially if desired to go with KUBE_PING and maybe even Autoscaling.
I will update after testing if something changed significantly.</p>
