<p>I'm pushing out my Phoenix app to a Kubernetes cluster for testing via GitLab. I'd like to be able to run <code>mix ecto.migrate</code> in my <code>gitlab-ci.yml</code> script once my app and the postgres service are ready. Here's a snippet from the <code>gitlab-ci.yml</code> file:</p>

<pre><code>review:
  stage: review
  image: dtzar/helm-kubectl

  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: https://$CI_PROJECT_NAME-${CI_ENVIRONMENT_SLUG}.$KUBE_DOMAIN
    on_stop: stop_review

  before_script:
    - command deploy/kinit.sh

  script:
    - helm upgrade --install db --wait --set postgresDatabase=app_db stable/postgresql
    - helm upgrade --install app ./deploy/app_chart --wait --set env.DATABASE_URL="${DATABASE_URL}"

    - export POD_NAME=`kubectl get pod -l "app=${CI_ENVIRONMENT_SLUG}" -o jsonpath='{.items[0].metadata.name}'`
    - kubectl exec $POD_NAME -- mix ecto.migrate
</code></pre>

<p>From what I understand, the <code>--wait</code> parameter means that each deployment will complete (in its entirety) before moving on. What I'm finding is that although the postgres deployment is complete, that does not mean that the postgres server is ready.</p>

<p>More often than not, when the <code>kubectl exec</code> command runs, I get the following error:</p>

<pre><code>** (exit) exited in: :gen_server.call(#PID&lt;0.183.0&gt;, {:checkout, #Reference&lt;0.0.1.2678&gt;, true, :infinity}, 5000)
    ** (EXIT) time out
    (db_connection) lib/db_connection/poolboy.ex:112: DBConnection.Poolboy.checkout/3
    (db_connection) lib/db_connection.ex:919: DBConnection.checkout/2
    (db_connection) lib/db_connection.ex:741: DBConnection.run/3
    (db_connection) lib/db_connection.ex:1132: DBConnection.run_meter/3
    (db_connection) lib/db_connection.ex:584: DBConnection.prepare_execute/4
    (ecto) lib/ecto/adapters/postgres/connection.ex:93: Ecto.Adapters.Postgres.Connection.execute/4
    (ecto) lib/ecto/adapters/sql.ex:243: Ecto.Adapters.SQL.sql_call/6
    (ecto) lib/ecto/adapters/sql.ex:193: Ecto.Adapters.SQL.query!/5
</code></pre>

<p>When I look at the Kubernetes ui, I can see the following error against my postgres pod:</p>

<pre><code>SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "db-postgresql", which is unexpected.
</code></pre>

<p>After seeing this message, I monitor the pods and everything comes good. But not before my deployment script fails.</p>

<p>My initial thought is that I could create an <code>initContainer</code> for my app that uses psql to successfully connect to the server and check for the existence of the "app_db" database. That way I don't have to worry about writing my own code for timeouts and retries - I can just take advantage of the built-in mechanism provided by Kubernetes.</p>

<p>However, I don't want to do this in my production environment (I want to run <code>mix ecto.migrate</code> on the production system manually). In this case, the <code>initContainer</code> simply serves as a waste of system resources.</p>

<p>Is there a robust way I can achieve this through the <code>gitlab-ci.yml</code> script?</p>

<p>From a conceptual point of view I would:</p>

<ol>
<li><p>Configure a <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-readiness-probes" rel="nofollow noreferrer">readiness probe</a> on my Postgres container, so that the Pod is not considered "Running" until the engine is up.</p>

<pre><code># in the Pod template:
# spec.containers['postgres']

readinessProbe:
  exec:
    command:
    - psql
    - -U
    - postgres
    - -c
    - 'SELECT 1'
  initialDelaySeconds: 5
  periodSeconds: 5
</code></pre></li>
<li><p>Wait for the Pod to transition to a "Running" state before executing my Mix task.</p>

<pre><code># in gitlab-ci.yml, before "mix ecto.migrate"

- |
    while [ "$(kubectl get pod $POD_NAME -o jsonpath='{$.status.phase}')" != "Running" ]; do
      sleep 1;
    done
</code></pre></li>
</ol>
