title: Reading large file and keep those records for processing
body: <p>This may be duplicate question as many pages we can find.
   But still I need few expert opinion.</p>
  
  <p>I need to read a large file containing customer record name,age,income.</p>
  
  <p>I need to process this data in my application quickly and create a dashboard like similar age employees, income groups(between range) ...
  Now the challenge is to read the large file (Explicitly I set my program heap to 512 MB) and used InputStream, Scanner classes (based on my understanding these classed will not load whole file in memory, pls correct me if wrong). so I am able to read the file with 7590912 records(250 MB file). but while setting those records in my <code>arrayList &lt;Employee&gt;</code> is showing continuous spikes and Garbage collector activity (which is expected). now to reduce my Employee object I created only three fields [name(char[]), age[int], income[float]).</p>
  
  <p>finally my program is very slow and not acceptable. Any suggestion to improve the performance except increasing the memory(Keeping in mind I will perform more operation over the collection).</p>
  
  <blockquote>
    <p>EDIT- Using H2 database to flush the read data from file. created
    batch of 10000 records (still memory is 512 MB) ..but program is
    pathetically slow. but manage to alive for some time(till 300K
    records).. (utilized space 470 MB).</p>
  </blockquote>
  
  <p>Pedantic</p>