<p>I'm running Keycloak in Kubernetes (Microk8s/Vagrant) and it's crashing on start up.</p>

<ul>
<li>If I don't add the MySQL environment variables then Keycloak starts up fine.</li>
<li>I can successfully connect to MySQL from other containers.</li>
<li>Logs can vary between start ups. Sometimes with no errors at all (but still exits)</li>
</ul>

<p>Keycloak also seems to be ignoring KEYCLOAK_LOGLEVEL and ROOT_LOGLEVEL.</p>

<h3>Keycloak Log</h3>

<pre><code>Added 'admin' to '/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json', restart server to load user
-b 0.0.0.0
=========================================================================

  Using MySQL database

=========================================================================

10:11:25,069 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.9.1.Final
10:11:25,210 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
10:11:25,239 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
10:11:25,445 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) starting
10:11:25,611 INFO  [org.jboss.vfs] (MSC service thread 1-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
10:11:27,152 INFO  [org.wildfly.security] (ServerService Thread Pool -- 20) ELY00001: WildFly Elytron version 1.10.4.Final
10:11:28,589 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:11:28,869 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:11:29,177 INFO  [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
10:11:29,239 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-1) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost                                                                                                                                                                                                          
10:11:29,424 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
10:11:29,431 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) started in 4333ms - Started 55 of 78 services (32 services are lazy, passive or on-demand)
The batch executed successfully
10:11:29,786 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) stopped in 35ms
10:11:32,602 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.9.1.Final
10:11:32,733 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
10:11:32,742 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
10:11:32,953 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) starting
10:11:33,122 INFO  [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
10:11:34,751 INFO  [org.wildfly.security] (ServerService Thread Pool -- 22) ELY00001: WildFly Elytron version 1.10.4.Final
10:11:36,595 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:11:36,815 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:11:37,215 INFO  [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
10:11:37,262 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost                                                                                                                                                                                                          
10:11:37,466 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
10:11:37,475 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) started in 4848ms - Started 55 of 85 services (39 services are lazy, passive or on-demand)
The batch executed successfully
10:11:37,847 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) stopped in 49ms
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss/keycloak

  JAVA: java

  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED

=========================================================================

10:11:39,080 INFO  [org.jboss.modules] (main) JBoss Modules version 1.9.1.Final
10:11:40,028 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.11.Final
10:11:40,053 INFO  [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
10:11:40,300 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) starting
10:11:40,466 INFO  [org.jboss.vfs] (MSC service thread 1-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
*** JBossAS process (325) received TERM signal ***
10:11:40,686 INFO  [org.jboss.as.server] (Thread-1) WFLYSRV0220: Server shutdown has been requested via an OS signal
10:11:41,114 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.as.server-controller: org.jboss.msc.service.StartException in service jboss.as.server-controller: Failed to start service
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Container is down
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:739)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2065)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractControllerService.start(AbstractControllerService.java:357)
        at org.jboss.as.server@10.0.3.Final//org.jboss.as.server.ServerService.start(ServerService.java:292)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
        ... 6 more

java.util.concurrent.ExecutionException: JBTHR00005: Operation failed
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:253)
        at org.jboss.as.server@10.0.3.Final//org.jboss.as.server.Main.main(Main.java:109)
        at org.jboss.modules.Module.run(Module.java:352)
        at org.jboss.modules.Module.run(Module.java:320)
        at org.jboss.modules.Main.main(Main.java:593)
Caused by: org.jboss.msc.service.StartException in service jboss.as.server-controller: Failed to start service
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Container is down
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:739)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2065)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractControllerService.start(AbstractControllerService.java:357)
        at org.jboss.as.server@10.0.3.Final//org.jboss.as.server.ServerService.start(ServerService.java:292)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
        ... 6 more
10:11:41,134 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) stopped in 442ms
*** JBossAS process (325) received TERM signal ***
</code></pre>

<h3>keycloak deployment</h3>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
   name: keycloak-deployment
   labels:
      app: keycloak
spec:
   replicas: 1
   selector:
      matchLabels:
         app: keycloak
   template:
      metadata:
         labels:
           app: keycloak
      spec:
         containers:
         - name: keyclock
           image: jboss/keycloak:latest
           env:
           - name: KEYCLOAK_LOGLEVEL
             value: ALL
           - name: ROOT_LOGLEVEL
             value: ALL
           - name: KEYCLOAK_USER
             value: admin
           - name: KEYCLOAK_PASSWORD
             value: admin
           - name: DB_VENDOR
             value: mysql
           - name: DB_ADDR
             value: mysql-service
           - name: DB_DATABASE
             value: keycloak
           - name: DB_USER
             value: keycloak
           - name: DB_PASSWORD
             value: keycloak
           ports:
           - containerPort: 8080
</code></pre>

<h3>MySQL Deployment</h3>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
   name: mysql-deployment
   labels:
      app: mysql
spec:
   replicas: 1
   selector:
      matchLabels:
         app: mysql
   template:
      metadata:
         labels:
           app: mysql
      spec:
         containers:
         - name: mysql
           image: mysql:5.7
           env:
           - name: MYSQL_ROOT_PASSWORD
             value: root
           ports:
           - name: mysql-port
             containerPort: 3306
</code></pre>

<h3>MySQL Service</h3>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: mysql-service
spec:
  selector:
    app: mysql
  strategy:
    type: Recreate
  ports:
  - port: 3306
    targetPort: mysql-port
    protocol: TCP
  type: NodePort
</code></pre>

Jroddev
<p>After rebuilding the cluster the DEBUG logs started to come through and so did more error messages.</p>

<p>The problem turned out to be an SSL issue
<code>Establishing SSL connection without server's identity verification is not recommended.</code></p>

<p>I have added this to the keycloak env</p>

<pre><code>- name: JDBC_PARAMS
  value: "useSSL=false"
</code></pre>

<p>It's not a proper solution, but it will get me by until I setup certificates in Kubernetes.</p>
