title: Spring Batch - JdbcCursorItemReader throwing OutOfMemoryError with large MySQL table
body: <p>I am writing a program using Spring Batch to process 7,637,064 rows from a MySQL database table. I've had success with smaller tables, but the large number of rows in this table is causing OutOfMemoryError exceptions when the JdbcCursorItemReader attempts to open the cursor. </p>
  
  <p>I could probably resolve this by throwing a larger Xmx at it, but it seems to me that Spring Batch should have a way to handle this and that I may simply be missing a key piece of configuration. </p>
  
  <p>Spring Batch configuration:</p>
  
  <pre><code>  &lt;job id="reportJob" xmlns="http://www.springframework.org/schema/batch"&gt;
      &lt;step id="largeTableTransfer"&gt;
        &lt;tasklet&gt;
          &lt;chunk reader="largeTableReader" processor="largeTableTransformer" writer="largeTableWriter"
            commit-interval="10" /&gt;
        &lt;/tasklet&gt;
      &lt;/step&gt;
    &lt;/job&gt;
  
    &lt;bean id="largeTableReader" class="org.springframework.batch.item.database.JdbcCursorItemReader"&gt;
      &lt;property name="dataSource" ref="inputDataSource" /&gt;
      &lt;property name="sql" value="select * from largeTable" /&gt;
      &lt;property name="rowMapper"&gt;
        &lt;bean class="myproject.reader.largeTableRowMapper" /&gt;
      &lt;/property&gt;
    &lt;/bean&gt;
    &lt;bean id="largeTableTransformer" class="myproject.transformer.LargeTableTransformer" /&gt;
    &lt;bean id="largeTableWriter" class="myproject.writer.JdbcLargeTableWriter"&gt;
      &lt;property name="dataSource" ref="outputDataSource" /&gt;
    &lt;/bean&gt;
  </code></pre>
  
  <p>Setting <em>fetchSize</em> on the JdbcCursorItemReader appears to have no effect. The only thing that allows it to run to completion is to set the <em>maxRows</em> to a small number, but then only that number of rows get processed. </p>
  
  <p>The relevant stack trace:</p>
  
  <pre><code>2012-11-21 11:25:29,931 DEBUG [org.springframework.batch.core.repository.dao.JdbcStepExecutionDao] - &lt;Truncating long message before update of StepExecution, original message is: java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:2734)
      at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
      at java.util.ArrayList.add(ArrayList.java:351)
      at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2821)
      at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
      at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
      at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
      at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
      at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
      at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
      at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
      at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
      at org.springframework.batch.item.database.JdbcCursorItemReader.openCursor(JdbcCursorItemReader.java:125)
      at org.springframework.batch.item.database.AbstractCursorItemReader.doOpen(AbstractCursorItemReader.java:401)
      at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:134)
      at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:93)
      at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:301)
      at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:192)
      at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
      at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
      at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
      at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
      at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
      at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
      at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293)
      at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
      at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
      at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:114)
      at ifpress.ams2amx.ExampleJobConfigurationTests.testLaunchJob(ExampleJobConfigurationTests.java:32)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  </code></pre>