<p>I have database migrations which I'd like to run before deploying a new version of my app into a Kubernetes cluster. I want these migrations to be run automatically as part of a Continuous Delivery pipeline. The migration will be encapsulated as a container image. What's the best mechanism to achieve this?</p>

<p>Requirements for a solution:</p>

<ul>
<li>be able to determine if a migration failed so that we don't subsequently try to deploy a new version of the app into the cluster.</li>
<li>give up if a migration fails - don't keep retrying it.</li>
<li>be able to access logs to diagnose failed migrations.</li>
</ul>

<p>I had assumed that the Jobs functionality in Kubernetes would make this easy, but there appear to be a few challenges:</p>

<ul>
<li>Kubernetes will <a href="https://github.com/kubernetes/kubernetes/issues/20255">repeatedly re-run containers whose processes terminate with a non-zero exit code</a>, even if the Job has a <code>restartPolicy</code> of <code>never</code>.</li>
<li>blocking while waiting on the result of a queued-up job seems to require hand-rolled scripts</li>
</ul>

<p>Would using "bare pods" be a better approach? If so, how might that work?</p>

<blockquote>
  <p>blocking while waiting on the result of a queued-up job seems to require hand-rolled scripts</p>
</blockquote>

<p>This isn't necessary anymore thanks to the <code>kubectl wait</code> command.</p>

<p>Here's how I'm running db migrations in CI:</p>

<pre class="lang-sh prettyprint-override"><code>kubectl apply -f migration-job.yml
kubectl wait --for=condition=complete --timeout=60s job/migration
kubectl delete job/migration
</code></pre>

<p>In case of failure or timeout, one of the two first CLI commands returns with an erroneous exit code which then forces the rest of the CI pipeline to terminate.</p>

<p><code>migration-job.yml</code> describes a kubernetes <code>Job</code> resource configured with <code>restartPolicy: Never</code> and a reasonably low <code>activeDeadlineSeconds</code>.</p>

<p>You could also use the <code>spec.ttlSecondsAfterFinished</code> <a href="https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/" rel="noreferrer">attribute</a> instead of manually running <code>kubectl delete</code> but that's still in alpha at the time of writing and not supported by Google Kubernetes Engine at least.</p>
