title: Optimization/Concurrency JAVA: reading CSV file, parsing data to double then doing calculations on the values
body: <p>I'm working on a data analyzing program in Java. The data is from car sensors, and it's delivered in a CSV format. It needs to the read, parsed to double then "converted" (say from a 5/12 Volt value to G-forces/mph/% throttle opening/% braking force, etc.). The data files are going to be up to about 200MB in size.</p>
  
  <p>What I'm currently doing is reading the data with a BufferedReader, splitting the tokenz with indexOf() and substring(), parsing them to double with parseDouble, adding them to an ArrayList of ArrayLists with doubles (one ArrayList for each token, there's maximum about 20 tokens). After the arraylists are made I have to convert the doubles, which means using polynomials on each value (this seems to be taking up most of the time, more than 2/3 of it).</p>
  
  <p>The whole sequence is about 7 seconds for sample data of 240 000 lines with 20 tokens. I'm wondering how I can improve this. I've been thinking about using streams and concurrency for the file-reading and parsing, but it seems the biggest problem is the polynomial-math. The code I'm using to calculate the converted values is:</p>
  
  <pre><code>pol0 + pol1 * value + pol2 * Math.pow(value, 2) + pol3 * Math.pow(value, 3)
           + pol4 * Math.pow(value, 4);
  </code></pre>
  
  <p>where polX is the polynomials and value is the double-value I'm converting. This is of course different depending on how many polynomials I have available.</p>
  
  <p>So, does any of this make any sense? And if so, any suggestions on how to improve the performance of this process?</p>
  
  <p>Thanks for a great site with many helpful users. :-)</p>