<p>I am working on creating reports with Spring Batch on Azure AKS.</p>
<p>We have HPA (Horizantal Pod Autoscaler). When we have multiple batches running, pods scale out due to HPA and in some cases same jobs run twice which is not what we want.</p>
<p>Spring Batch uses a database repository for job management, but we still had an instance which same job ran twice.</p>
<p>Has anyone used Spring Batch on Azure AKS? or on a kubernetes installation? what is the best way to establish synchronization? and to manage scale out?</p>

DarthVader
<blockquote>
<p>Spring Batch uses a database repository for job management, but we still had an instance which same job ran twice</p>
</blockquote>
<p>I talked about this issue in detail in the blog post here: <a href="https://spring.io/blog/2021/01/27/spring-batch-on-kubernetes-efficient-batch-processing-at-scale" rel="nofollow noreferrer">Spring Batch on Kubernetes: Efficient batch processing at scale</a>.</p>
<blockquote>
<p>what is the best way to establish synchronization? and to manage scale out?</p>
</blockquote>
<p>The deployment pattern that works best for your use case is the &quot;Kubernetes job per Spring Batch job instance&quot;, coupled with a correctly configured central job repository (ie having a high isolation level). This pattern has several advantages:</p>
<ul>
<li>Kubernetes jobs and Spring Batch job <strong>instances</strong> have the same lifecycle (once they are successfully executed, they cannot be restarted). If a job pod fails, Kubernetes will create another pod which will resume the execution of the same Spring Batch job instance. So Spring Batch job instances can be automatically restarted and continue work until completion,  since Kubernetes' job controller will execute the necessary number of pods until the job succeeds.</li>
<li>Job instances can be scaled out by submitting as many Kubernetes jobs as needed.</li>
<li>This pattern guarantees that when Kubernetes submits the same job twice (which is a limitation), Spring Batch will prevent a concurrent execution of the same job instance (since in this deployment pattern a Kubernetes job is designed to run a job instance).</li>
</ul>
<p>In your case, I would create a Spring Batch job instance per report (you did not mention what a report is about, but I am assuming here that creating a report is about a distinct criteria, which should be the identifying Spring Batch job parameter that will result in distinct job instances). With that in place, reports will be independent and can be generated in parallel at the scale you need.</p>
<p>You will find more details in the blog post that I hope you find helpful.</p>
