title: java efficient way to process big text files
body: <p>Im doing a frequency dictionary, in which i read 1000 files, each one with about 1000 lines. The approach i'm following is:</p>
  
  <ul>
  <li>BufferedReader to read fileByFile</li>
  <li>read the first file, get the first sentence, split the sentence to an array string, then fill in an hashmap with the values from the string array.</li>
  <li>do this for all the senteces in that file</li>
  <li>do this for all 1000 files</li>
  </ul>
  
  <p>My problem is, this is not a very efficient way to do it, i'm taking about 4 minutes to do all this. I'v increased heap size, refactored the code to make sure i'm not doind something wrong. For this approach, i'm completly sure there's nothing i can improve in the code.</p>
  
  <p>My bet is, each time a sentece is read, a split is applied, which, multiplied by 1000 sentences in a file and by 1000 files is a huge ammount of splits to process.
  My idea is, instead of read and process file-by-file, i could read each file to a char array, and then make the split only once per file. That would ease the ammount of processing times consuming with the split. Any suggestions of implementation would be appreciated.</p>