<p>I may be encountering the same issue described in Horizon: does not exit if database connection fails #898 (<a href="https://github.com/stellar/go/issues/898" rel="nofollow noreferrer">https://github.com/stellar/go/issues/898</a>) but with a different set up scenario.</p>

<p>I am in the process of migrating <a href="https://github.com/satoshipay/docker-stellar-horizon" rel="nofollow noreferrer">https://github.com/satoshipay/docker-stellar-horizon</a> Docker Compose definitions to Kubernetes.   I have been able to migrate most of the set up but hitting a problem with Horizon where the DB is not getting created during startup.    I believe I have stellar core with the dependency on Postgres working as designed and the DB created as part of startup but the set up is different for Horizon.</p>

<p>The current issue I am hitting is the following...</p>

<p><strong>Horizon Server Pod Logs</strong></p>

<pre><code>todkapmcbookpro:kubernetes todd$ kubectl get pods
NAME                                READY     STATUS             RESTARTS   AGE
postgres-horizon-564d479db4-2xvqd   1/1       Running            0          20m
postgres-sc-9f5f7fb4-prlpr          1/1       Running            0          22m
stellar-core-7ff77b4db8-tx4mt       1/1       Running            0          18m
stellar-horizon-6cff98554b-d7djn    0/1       CrashLoopBackOff   8          18m
todkapmcbookpro:kubernetes todd$ kubectl logs stellar-horizon-6cff98554b-d7djn
Initializing Horizon database...
2019/05/02 12:58:09 connect failed: pq: database "stellar-horizon" does not exist
Horizon database initialization failed (possibly because it has been done before)
2019/05/02 12:58:09 pq: database "stellar-horizon" does not exist
todkapmcbookpro:kubernetes todd$ 
</code></pre>

<p><strong>Horizon Postgres DB pod logs</strong></p>

<pre><code>todkapmcbookpro:kubernetes todd$ kubectl logs postgres-horizon-564d479db4-2xvqd
2019-05-02 12:40:06.424 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-05-02 12:40:06.424 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2019-05-02 12:40:06.437 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-05-02 12:40:06.444 UTC [23] LOG:  database system was interrupted; last known up at 2019-05-02 12:38:19 UTC
2019-05-02 12:40:06.453 UTC [23] LOG:  database system was not properly shut down; automatic recovery in progress
2019-05-02 12:40:06.454 UTC [23] LOG:  redo starts at 0/1636FB8
2019-05-02 12:40:06.454 UTC [23] LOG:  invalid record length at 0/1636FF0: wanted 24, got 0
2019-05-02 12:40:06.454 UTC [23] LOG:  redo done at 0/1636FB8
2019-05-02 12:40:06.459 UTC [1] LOG:  database system is ready to accept connections
2019-05-02 12:42:35.675 UTC [30] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:35.690 UTC [31] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:37.123 UTC [32] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:37.136 UTC [33] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:50.131 UTC [34] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:50.153 UTC [35] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:43:16.094 UTC [36] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:43:16.115 UTC [37] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:43:57.097 UTC [38] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:43:57.111 UTC [39] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:45:21.050 UTC [40] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:45:21.069 UTC [41] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:48:05.122 UTC [42] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:48:05.145 UTC [43] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:53:07.077 UTC [44] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:53:07.099 UTC [45] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:58:09.084 UTC [46] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:58:09.098 UTC [47] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:03:18.055 UTC [48] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:03:18.071 UTC [49] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:08:28.057 UTC [50] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:08:28.078 UTC [51] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:13:42.071 UTC [52] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:13:42.097 UTC [53] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:18:55.128 UTC [54] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:18:55.152 UTC [55] FATAL:  database "stellar-horizon" does not exist
</code></pre>

<p>It would be ideal if the setup for Horizon and Core were the same (especially as it relates to the DB configuration env properties).    I think I have the settings correct but may be missing something subtle.</p>

<p>I have a branch of this WIP where the failure occurs.   I have included a quick set up script as well as a minikube set up in this branch.
<a href="https://github.com/todkap/stellar-testnet/tree/k8-deploy/kubernetes" rel="nofollow noreferrer">https://github.com/todkap/stellar-testnet/tree/k8-deploy/kubernetes</a></p>

Todd Kaplinger
<p>We were able to resolve and published an article demonstrating the end to end flow. 
<a href="https://itnext.io/how-to-deploy-a-stellar-validator-on-kubernetes-with-helm-a111e5dfe437" rel="nofollow noreferrer">https://itnext.io/how-to-deploy-a-stellar-validator-on-kubernetes-with-helm-a111e5dfe437</a></p>
