title: Spring Batch: Duplicate rows after job re-run
body: <p>Our Spring Batch application is, upon restart of a failed job, processing the same records again, resulting in duplicate rows, and we want to understand how to avoid this.</p>
  
  <p>The Spring Integration poller which starts the batch job is configured to run every couple of hours. When it runs a second time, the job parameters will be the same, but if the previous run failed (for example, because of a DataTruncation exception), Spring Batch will not complain that the job has already completed.</p>
  
  <p>At the point of failure, several hundred thousand records will already have been processed and copied fromn the source table to the destination table. When the job is run a subsequent time, the same rows will be copied to the destination table, resulting in duplicates. Therefore, it appears that the job is not being resumed, but restarted from the beginning.</p>
  
  <p>The Spring Batch database is Derby (file based), this is setup when the application starts, and it appears state is not maintained between restarts of the actual application (because a job can be run again with the same parameters). However, within one application run, state is maintained. For instance, if the job completes succesfully, the next time the poller runs an exception will be thrown because a job (with those parameters) has already completed.</p>
  
  <p>Our job is definition is as follows:</p>
  
  <pre><code>&lt;batch:job id="publisherJob" &gt;
     &lt;batch:step id="step1"&gt;
        &lt;batch:tasklet &gt;
      &lt;batch:chunk reader="itemReader" processor="itemProcessor"
                writer="itemWriter" commit-interval="${...}" /&gt;
          &lt;/batch:tasklet&gt;
  
          &lt;batch:listeners&gt;
          ...
          &lt;/batch:listeners&gt;
  &lt;/batch:job&gt;
  
  &lt;bean id="itemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader"&gt;
     &lt;property name="dataSource" ref="dataSource" /&gt;
     &lt;property name="sql" value="select ${...} from ${...} where ${...}" /&gt;
     &lt;property name="rowMapper" ref="rowMapper" /&gt;
  &lt;/bean&gt;
  </code></pre>
  
  <p>The WHERE clause includes ORDER BY.</p>
  
  <p>Our understanding was that Spring Batch would retain the state at which processing failed and proceed from that point (if the error in the source table has been fixed), therefore preventing duplicate rows. What has to be configured for this to happen?</p>
  
  <p>Thanks</p>