<p>I read a lot about best practices applying DataBase Migrations for Kubernetes Apps. Actually there are three common solutions:</p>
<ol>
<li><strong>Separate CI/CD stage for DB Migrations</strong> (usually we run it before deploy new App version but in some cases we can deploy first, it doesn't matter for current question). Just decouple app and migrations, and run them in different stages one by one.</li>
<li><strong>InitContainer:</strong> we can run DB migraitions before main app container starts. It is a good solution, but we have to be careful to tune it to run migrations once for all pods and replicas, do not run on pod restarts and check probes (kubelet can kill container on timeout while migration still running)</li>
<li><strong>Separate Kubernetes Job or Helm Hook.</strong> Is close to initContainer, cannot share data with app container (but it is not really necessary for DB Migrations, so it's ok). Must be careful with timeout too - Helm can kill Job before migration is completed.</li>
</ol>
<p><strong>But the question - how to apply rollbacks for these solutions???</strong></p>
<p>Lets try to form my ideas:</p>
<ol>
<li>Separate stage in CI/CD: we can save previous migration name, and then rollback to it in another stage. Pipeline: Migrations -&gt; Deploy -&gt; Test -&gt; Rollback DB and ReDeploy</li>
<li>But for InitContainer and HelmHook I have to idea how to realise Rollback! Do we need additional containers for that? Does helm rollback affect DB too (don't think so). <strong>What are the best practices for that?</strong></li>
</ol>
<p>I will be very glad to any suggestions!</p>

<p>I started investigating this issue too, and it looks like all manuals, tutorials and practices are meant to be forward-only. Should you ever need to rollback a database migration, you face with the limitations of Helm rollback. There is an open issue in Helm (<a href="https://github.com/helm/helm/issues/5825" rel="nofollow noreferrer">https://github.com/helm/helm/issues/5825</a>) that addresses this very problem, but it looks like no solution so far.</p>
<p>So probably Helm rollback is not a suitable mechanism for databases as you end up with creating a batch/v1.Job with container image that knows nothing about the changes you need to rollback. I use Liquibase and it requires that the changesets you need to rollback are present in the changelog.</p>
<p>I think for now this problem can be solved with forward-only approach:</p>
<ol>
<li>Build the container image that introduces a new changeset into your database.</li>
<li>In release values for Helm specify the container image that should be run to perform a migration, and a command (like <code>liquibase update</code>).</li>
<li>If the migration fails and you see that rollback is required, you do not invoke <code>helm rollback</code>. You deploy a new release, and in its values you specify the container image that should perform the database migration rollback, and the command (like <code>liquibase rollback &lt;tag&gt;</code>). I am myself going this way now and it looks like the best solution I could came up with.</li>
</ol>
<p>Hope that helps.</p>
