<p>Anyone here have experience about batch processing (e.g. spring batch) on kubernetes ? Is it good idea ? How to prevent batch processing process same data if we use kubernetes auto scaling feature ? Thank you.</p>

Daniel Setiawan
<blockquote>
<p>Anyone here have experience about batch processing (e.g. spring batch) on kubernetes ? Is it good idea ?</p>
</blockquote>
<p>For Spring Batch, we (the Spring Batch team) do have some experience on the matter which we share in the following talks:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=1NZVwv1cmMc" rel="noreferrer">Cloud Native Batch Processing on Kubernetes</a>, by Michael Minella</li>
<li><a href="https://speakerdeck.com/benas/spring-batch-kubernetes" rel="noreferrer">Spring Batch on Kubernetes</a>, by me.</li>
</ul>
<p>Running batch jobs on kubernetes can be tricky:</p>
<ul>
<li>pods may be re-scheduled by k8s on different nodes in the middle of processing</li>
<li>cron jobs might be triggered <a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations" rel="noreferrer">twice</a></li>
<li>etc</li>
</ul>
<p>This requires additional non-trivial work on the developer's side to make sure the batch application is fault-tolerant (resilient to node failure, pod re-scheduling, etc) and safe against duplicate job execution in a clustered environment.</p>
<p>Spring Batch takes care of this additional work for you and can be a good choice to run batch workloads on k8s for several reasons:</p>
<ul>
<li><strong>Cost efficiency:</strong> Spring Batch jobs maintain their state in an external database, which makes it possible to restart them <strong>from the last save point</strong> in case of job/node failure or pod re-scheduling</li>
<li><strong>Robustness:</strong> Safe against duplicate job executions thanks to a centralized job repository</li>
<li><strong>Fault-tolerance:</strong> Retry/Skip failed items in case of transient errors like a call to a web service that might be temporarily down or being re-scheduled in a cloud environment</li>
</ul>
<p>I wrote a blog post in which I explain all these aspects in details with code examples. You can find it here: <a href="https://spring.io/blog/2021/01/27/spring-batch-on-kubernetes-efficient-batch-processing-at-scale" rel="noreferrer">Spring Batch on Kubernetes: Efficient batch processing at scale</a></p>
<blockquote>
<p>How to prevent batch processing process same data if we use kubernetes auto scaling feature ?</p>
</blockquote>
<p>Making each job process a different data set is the way to go (a job per file for example). But there are different patterns that you might be interested in, see <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#job-patterns" rel="noreferrer">Job Patterns</a> from k8s docs.</p>
