<p>I have a specific scenario where I'd like to have a deployment controlled by horizontal pod autoscaling. To handle database migrations in pods when pushing a new deployment, I followed this excellent tutorial by Andrew Lock <a href="https://andrewlock.net/deploying-asp-net-core-applications-to-kubernetes-part-8-running-database-migrations-using-jobs-and-init-containers/" rel="nofollow noreferrer">here</a>.</p>
<p>In short, you must define an <code>initContainer</code> that <code>waits</code> for a Kubernetes <code>Job</code> to complete a process (like running db migrations) before the new pods can run.</p>
<p>This works well, however, I'm not sure how to handle HPA after the initial deployment because if the system detects the need to add another <code>Pod</code> in my node, the <code>initContainer</code> defined in my deployment requires a <code>Job</code> to be deployed and run, but since <code>Jobs</code> are one-off processes, the pod can not initialize and run properly (a <code>ttlSecondsAfterFinished</code> attribute removes the <code>Job</code> anyways).</p>
<p>How can I define an <code>initContainer</code> to run when I deploy my app so I can push my database migrations in a <code>Job</code>, but also allow HPA to control dynamically adding a <code>Pod</code> without needing an <code>initContainer</code>?</p>
<p>Here's what my <code>deployment</code> looks like:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: graphql-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: graphql-pod
  template:
    metadata:
      labels:
        app: graphql-pod
    spec:
      initContainers:
        - name: wait-for-graphql-migration-job
          image: groundnuty/k8s-wait-for:v1.4 # This is an image that waits for a process to complete
          args:
            - job
            - graphql-migration-job # this job is defined next
      containers:
        - name: graphql-container
          image: image(graphql):tag(graphql)
</code></pre>
<p>The following <code>Job</code> is also deployed</p>
<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: graphql-migration-job
spec:
  ttlSecondsAfterFinished: 30
  template:
    spec:
      containers:
      - name: graphql-migration-container
        image: image(graphql):tag(graphql)
        command: [&quot;npm&quot;, &quot;run&quot;, &quot;migrate:reset&quot;]
      restartPolicy: Never
</code></pre>
<p>So basically what happens is:</p>
<ul>
<li>I deploy these two resources to my node</li>
<li><code>Job</code> is initialized</li>
<li><code>initContainer</code> on <code>Pod</code> waits for <code>Job</code> to complete using an image called <code>groundnuty/k8s-wait-for:v1.4</code></li>
<li><code>Job</code> completes</li>
<li><code>initContainer</code> completes</li>
<li><code>Pod</code> initializes</li>
<li>(after 30 TTL seconds) <code>Job</code> is removed from node</li>
</ul>
<p>(LOTS OF TRAFFIC)</p>
<ul>
<li>HPA realizes a need for another pod</li>
<li><code>initContainer</code> for NEW pod is started, but cant run because <code>Job</code> doesn't exist</li>
<li>...crashLoopBackOff</li>
</ul>
<p>Would love any insight on the proper way to handle this scenario!</p>

<p>There is, unfortunately, no simple Kubernetes feature to resolve your issue.</p>
<p>I recommend extending your deployment tooling/scripts to separate the migration job and your deployment. During the deploy process, you first execute the migration job and then deploy your deployment. Without the job attached, the HPA can nicely scale your pods.</p>
<p>There is a multitude of ways to achieve this:</p>
<ul>
<li>Have a bash, etc. script first to execute the job, wait and then update your deployment</li>
<li>Leverage more complex deployment tooling like Helm, which allows you to add a <a href="https://helm.sh/docs/topics/charts_hooks/" rel="nofollow noreferrer">'pre-install hook'</a> to your job to execute them when you deploy your application</li>
</ul>
