title: Reading from large file (2GB) and writing to another file using Java concurrency
body: <p>i have a huge file (2GB) which contains employee numbers only. i have to read this file and get the employee number and call database to get the salary of the employee and then write this in another file with employee name and salary as its row.</p>
  
  <p>Now here is the problem, By direct reading this huge file by simple nio in java my STS goes out of memory or it takes 4-5 hrs to complete the entire read-fetch-write process.</p>
  
  <p>So i thought to use the Java concurrency to rescue me.</p>
  
  <p>for that i have a class EmployeeDetails which implements Runnable, in its main method , i have Create a thread pool executor with core size and maximum size as 2,
  specify a time out and pass the blocking queue.</p>
  
  <pre><code>BlockingQueue workQueue = new LinkedBlockingQueue();
  In ThreadPoolExecutor along with corePoolSize=2 , maximumPoolSize =20 , keepAliveTime  = Long.MAX_VALUE, workQueue is been passed.
  </code></pre>
  
  <p>then created inputFile File object to getting the input file from path and outputFile Object where i need to write the data.</p>
  
  <p>then i created two object of EmployeeDetails ,</p>
  
  <pre><code>EmployeeDetails readFile = new EmployeeDetails(inputFile);
  EmployeeDetails writeFile = new EmployeeDetails (outputFile);
  </code></pre>
  
  <p>then <code>ThreadPoolExecutor. submit(readFile)</code> and <code>ThreadPoolExecutor submit (writeFile )</code> called.</p>
  
  <p>In class <code>EmployeeDetails</code> , i have created run method, 
  run method open the database connection,
  then BufferedReader get the file,
  while loop is run untill <code>br.readLine() !=null</code>
  then for each employee db is called to get the salary as employee are in different shard (MySQL) so as its salary.</p>
  
  <p>once the salary is retrieved i put it in a list and as soon as the list size it 10000. i need to wait a thread and call the wrie thread to write it into file and flush the list to get another 10000 records and thats the problem. </p>
  
  <p>Please suggest a way to read 10000 records from a file , call the db to get details and then populating a list and once he list it 10000 record calling write thread to pic the list and write into another file. </p>