title: Merging two files line by line Java
body: <p>Is there a more efficient way than i'm currently using, to merge two files line by line appending the line from file2 onto file1?</p>
  
  <p>If file1 contains</p>
  
  <pre><code>a1
  b1
  c1
  </code></pre>
  
  <p>And file2 contains</p>
  
  <pre><code>a2
  b2
  c2
  </code></pre>
  
  <p>Then the output file should contain</p>
  
  <pre><code>a1,a2
  b1,b2
  c1,c2
  </code></pre>
  
  <p>The current combineRecords method looks like </p>
  
  <pre><code>private FileSheet combineRecords(ArrayList&lt;FileSheet&gt; toCombine) throws IOException
  {
      ArrayList&lt;String&gt; filepaths = new ArrayList&lt;String&gt;();
  
      for (FileSheet sheetIterator : toCombine)
      {
          filepaths.add(sheetIterator.filepath);
      }
  
      String filepathAddition = "";
  
      for (String s : filepaths)
      {
          filepathAddition = filepathAddition + s.split(".select.")[1].replace(".csv", "")  + ".";
      }
  
      String outputFilepath = subsheetDirectory + fileHandle.getName().split(".csv")[0] + ".select." + filepathAddition +  "csv";
  
      Log.log("Output filepath "  + outputFilepath);
  
      long mainFileLength = toCombine.get(0).recordCount();
  
      for (FileSheet f : toCombine)
      {
          int ordinal = toCombine.indexOf(f);
  
          if (toCombine.get(ordinal).recordCount() != mainFileLength)
          {
              Log.log("Error : Record counts for 0 + " + ordinal);
              return null;
          }
      }
  
      FileSheet finalValues;
  
      Log.log("Starting iteration streams");
      BufferedWriter out = new BufferedWriter(new FileWriter(outputFilepath, false));
  
      List&lt;BufferedReader&gt; streams = new ArrayList&lt;&gt;();
      for (FileSheet j : toCombine)
      {
          streams.add(new BufferedReader(new FileReader(j.filepath)));
      }
  
      String finalWrite = "";
  
      for (int i = 0; i &lt; toCombine.get(0).recordCount(); i++)
      {
  
          for (FileSheet j : toCombine)
          {
              int ordinal = toCombine.indexOf(j);
  
              finalWrite = finalWrite + streams.get(ordinal).readLine();
  
              if (toCombine.indexOf(j) != toCombine.size() - 1)
              {
                  finalWrite = finalWrite + ",";
              }
              else
              {
                  finalWrite = finalWrite + "\n";
              }
          }
  
          if (i % 1000 == 0 || i == toCombine.get(0).recordCount() - 1)
          {
              // out.write(finalWrite + "\n");
              Files.write(Paths.get(outputFilepath),(finalWrite).getBytes(),StandardOpenOption.APPEND);
  
              finalWrite = "";
          }           
      }
      out.close();
  
  
      Log.log("Finished combineRecords");
  
      finalValues = new FileSheet(outputFilepath,0);
      return finalValues;
  }
  </code></pre>
  
  <p>I've tried both bufferedwriters and files.write, and they have similar times to create file3, both in the 1:30 minute range, but i'm not sure if the bottleneck is at reading or writing</p>
  
  <p>The sample files i'm using are currently at 36,000 records, but the actual file i'll be using is ~650,000 so taking (if it scales linearly) 1625 seconds is completely unfeasible for this operation</p>
  
  <p>Edit : I've modified the code to only open files once, rather than per iteration, however i'm now getting stream closed when skipping to the nth line
  I thought that by doing <code>streams.get(ordinal).skip(i).findFirst().get();</code> would return a new stream instead of skipping then closing the stream</p>
  
  <p>Edit 2 : Modified the code to use bufferedreaders instead of streams, and write to file every 1000 lines read, and thats determined that the bottleneck is reading, because it still takes ~1:30 to do</p>