<p>I have a rails app that is deployed on K8S. Inside my web app, there is a cronjob thats running every day at 8pm and it takes 6 hours to finish. I noticed <code>OOMkilled</code> error occurs after a few hours from cronjob started. I also increased memory of a pod but the error still happened.</p>
<p>This is my <code>yaml</code> file:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: sync-data
spec:
  schedule: &quot;0 20 * * *&quot; # At 20:00:00pm every day
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 100
      template:
        spec:
          serviceAccountName: sync-data
          containers:
            - name: sync-data
              resources:
                requests:
                  memory: 2024Mi # OOMKilled
                  cpu: 1000m
                limits:
                  memory: 2024Mi # OOMKilled
                  cpu: 1000m
              image: xxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/path
              imagePullPolicy: IfNotPresent
              command:
                - &quot;/bin/sh&quot;
                - &quot;-c&quot;
                - |
                  rake xxx:yyyy # Will take ~6 hours to finish
          restartPolicy: Never 

</code></pre>
<p>Are there any best practices to run long consuming cronjob on K8S?
Any help is welcome!</p>

Tran B. V. Son
<p>OOM Killed can happen for 2 reasons.</p>
<ol>
<li><p>Your pod is taking more memory than the limit specified. In that case, you need to increase the limit obviously.</p>
</li>
<li><p>If all the pods in the node are taking more memory than they have requested then Kubernetes will kill some pods to free up space. In that case, you can give higher priority to this pod.</p>
</li>
</ol>
<p>You should have monitoring in place to actually determine the reasons for this. Proper monitoring will show you which pods are performing as per expectations and which are not. You could also use node selectors for long-running pods and set priority class which will remove non-cron pods first.</p>
