title: How to process data in chunks in java using Multi Threading?
body: <p>I am working on a task in which I need to process data in chunks. I have a properties file in which I define the chunk size, suppose 500 and the data that I am getting form the data base is suppose 1000 records. I want to process 1000 records in chunks 500 each using Multi Threading. </p>
  
  <p>This is the first time I am implementing this so please let me know if I can achieve the same using another technique. The main purpose behind this is that I am generating an excel file in which I populate the data keeping in mind the chunk size. So probably first thread processes 500 records and second thread next 500. </p>
  
  <p>Partial Code (Rest parses the xml and writes in Excel using POI)</p>
  
  <pre><code>public List&lt;NYProgramTO&gt; getNYPPAData() throws Exception {
      this.getConfiguration();
      List&lt;NYProgramTO&gt; to = dao.getLatestNYData();
      DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
      Document document = null;
  
      // Returns chunkSize
      List&lt;NYProgramTO&gt; myList = getNextChunk(to);
      ExecutorService executor = Executors.newFixedThreadPool(myList.size());
      myList.stream()
      .forEach((NYProgramTO nyTo) -&gt; 
      {
          executor.execute(new NYExecutorThread(nyTo, migrationConfig , appContext, dao));
      });
  
      executor.shutdown();
      executor.awaitTermination(300, TimeUnit.SECONDS);
      System.gc();
  </code></pre>
  
  <p><code>dao.getLatestNYData();</code> method returns me the total number of records from the database and this is how I populate the list <code>to</code>. </p>
  
  <p>I have the following method which gives me the next set of chunk, so suppose if 500 records had processed this method should give next 500 records to process (Hope this makes sense).</p>
  
  <pre><code>private static List&lt;NYProgramTO&gt; getNextChunk(List&lt;NYProgramTO&gt; list) {
      currentIndex = 0; // This is static int class variable 
      List&lt;NYProgramTO&gt; nyList = new ArrayList&lt;&gt;();
  
      if(list.size() == 0) {
          return list;
      }
  
      int totalCount = list.size();
      for(int i = currentIndex; i &lt; (currentIndex + chunkSize); i++) {
  
          if(i == totalCount) break;
          nyList.add(list.get(i));
      }
  
      return nyList;
  }
  </code></pre>
  
  <p>In my first method I create threads now here I am not sure to how many thread do I need to create. Currently I am passing the size of the list that I receive from <code>getNextChunk();</code> method. </p>
  
  <p><code>NYExecutorThread</code> this class simply implements Runnable and I don't have any logic in it yet. Currently I simply pass parameters on the constructor to be able to get the configurations and create threads. </p>
  
  <p>It is a little confusing and I want if anyone has implemented such a logic, please let me know how can I go ahead with this?</p>
  
  <p>Thanks</p>