title: Reading large CSV files in Java
body: <p>I am trying to read a <strong>1,000,000</strong> lines CSV file in Java. I'm using OpenCSV library, and it works fine on a smaller file of <strong>30,000</strong> lines. Processes it in under half a second. But when I try to read from a million line file, it never finishes.</p>
  
  <p>Now I tested to see, when it will actually stop, and by using my own version of binary search, I first tried to read <strong>500k</strong> lines, then <strong>250k</strong>, and so on, and I found that it <em>easily</em> reads <strong>145k</strong> lines, in <strong>0.5-0.7sec</strong>, while <strong>150k</strong> does not even finish. </p>
  
  <p>I have searched SO thoroughly, found several solutions which I employed in my code, such as using <code>BufferedReader</code>, <code>BufferedInputStream</code> etc, but none of those solved it. Still it fails between <strong>145-150k</strong> lines.</p>
  
  <p>This is the relevant portion of my code (swapping <strong>150000 with 145000</strong> is what causes the program to execute in &lt;1 sec):</p>
  
  <pre><code>try {
         // BufferedInputStream bufferedInputStream = new BufferedInputStream(new FileInputStream("myFile.csv"));
          CSVReader csvReader = new CSVReader(new InputStreamReader
                  (new BufferedInputStream(new FileInputStream("myFile.csv"), 8192 * 32)));
          try {
              int count = 0;
              String[] line;
              long timeStart = System.nanoTime();
              while((line = csvReader.readNext()) != null){
                  count ++;
                  if(count &gt;= 150000){
                      break;
                  }
              }
              long timeEnd = System.nanoTime();
              System.out.println("Count: " + count);
              System.out.println("Time: " + (timeEnd - timeStart) * 1.0 / 1000000000 + " sec");
          } catch (IOException e) {
              e.printStackTrace();
          }
      } catch (FileNotFoundException e) {
          System.out.println("File not found");
      }
  </code></pre>
  
  <p>As you can see, I tried setting a bigger buffer size as well. I tried various combinations of <code>Readers</code>, <code>Input Streams</code> etc and nothing really made a difference.</p>
  
  <p>I'm wondering how I can do this? Is there a way to read, say 100k lines at a time, and then continue to read next 100k?</p>
  
  <p>Also, I'm open to any other solution that does not necessarily include the <code>OpenCSV</code> library. I just used that one for its simplicity to parse a csv file.</p>