<p>I have a Kubernetes based application consisting of multiple services (and pods) managed with helm chart.</p>

<p>Postgres is used as a database for all services.</p>

<p>When application is upgraded to a newer version, I am running a db migration script via <code>initContainers</code>.</p>

<p>The problem occurs when migration scripts require an exclusive access to DB (all other connections should be terminated), otherwise the script is blocked.</p>

<p>Ideal solution would be to stop all pods, run the migration and recreate them. But I am not sure how to achieve it properly with Kubernetes.</p>

<p>Tnx</p>

Meir Tseitlin
<blockquote>
<p>Ideal solution would be to stop all pods, run the migration and
recreate them. But I am not sure how to achieve it properly with
Kubernetes.</p>
</blockquote>
<p>I see from one of the comments that you use Helm, so I'd like to propose a solution leveraging Helm's hooks:</p>
<blockquote>
<p>Helm provides a hook mechanism to allow chart developers to intervene
at certain points in a release's life cycle. For example, you can use
hooks to:</p>
<ul>
<li><p>Load a ConfigMap or Secret during install before any other charts are
loaded.</p>
</li>
<li><p>Execute a Job to back up a database before installing a new
chart, and then execute a second job after the upgrade in order to
restore data.</p>
</li>
<li><p>Run a Job before deleting a release to gracefully take a
service out of rotation before removing it.</p>
</li>
</ul>
</blockquote>
<p><a href="https://helm.sh/docs/topics/charts_hooks/" rel="noreferrer">https://helm.sh/docs/topics/charts_hooks/</a></p>
<p>You could package your migration as a k8s <code>Job</code> and leverage the <code>pre-install</code> or <code>pre-upgrade</code> hook to run the job. These hooks runs after templates are rendered, but before any new resources are created in Kubernetes. Thus, your migrations will run before your Pods are deployed.</p>
<p>To delete the deployments prior to running your migrations, create a second pre-install/pre-upgrade hook with a lower <code>helm.sh/hook-weight</code> that deletes the target deployments:</p>
<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: &quot;pre-upgrade-hook1&quot;
  annotations:
    &quot;helm.sh/hook&quot;: pre-upgrade
    &quot;helm.sh/hook-weight&quot;: &quot;-1&quot;
    &quot;helm.sh/hook-delete-policy&quot;: hook-succeeded
spec:
  template:
    metadata:
      name: &quot;pre-upgrade-hook1&quot;
    spec:
      restartPolicy: Never
      serviceAccountName: &quot;&lt;an SA with delete RBAC permissions&gt;&quot;
      containers:
      - name: kubectl
        image: &quot;lachlanevenson/k8s-kubectl:latest&quot;
        command: [&quot;delete&quot;,&quot;deployment&quot;,&quot;deploy1&quot;,&quot;deploy2&quot;]
</code></pre>
<p>The lower hook-weight will ensure this job runs prior to the migration job. This will ensure the following series of events:</p>
<ol>
<li>You run <code>helm upgrade</code></li>
<li>The helm hook with the lowest hook-weight runs and deletes the relevant deployments</li>
<li>The second hook runs and runs your migrations</li>
<li>Your Chart will install with new Deployments, Pods, etc.</li>
</ol>
<p>Just make sure to keep all of the relevant Deployments in the same Chart.</p>
