title: Node-js: Processing multiple files vs processing a big file using line offset
body: <p>I'm retrieving a considerable amount of data from a NoSQL database limited by a given value (for simplicity, let's say 2 rows) and saving to a file so I could later process this data properly. </p>
  
  <p>This database returns each table result in groups of rows. For instance, if my data had 3 groups and I used limit 2, I would get this result for the first query. </p>
  
  <pre><code>// Group 1
  2020-03-09,A,1
  2020-03-09,B,6
  // Group 2
  2020-03-09,A,2
  2020-03-10,A,3
  // Group 3
  2020-03-10,A,4
  2020-03-11,A,5
  </code></pre>
  
  <p>While I keep getting the data from the database by using limits + offsets, I will write a temporary file with these jumps between groups until it ends up reading all the data. Note the dates are ordered between groups, and the same group may have the same date in different lines.</p>
  
  <pre><code>// Group 1 (lines 1-2)
  // Group 2 (lines 3-4)
  // Group 3 (lines 5-6)
  // Group 1 (lines 7-8)
  // Group 2 (lines 9-10)
  // Group 3 (lines 11-12)
  </code></pre>
  
  <p>For the parsing phase, I want to write a final processed file with the date as the primary key in order, using the following format:</p>
  
  <pre><code>date, group1_A, group1_B, group2_A, group3_A
  2020-03-09, 1, 6, 2, (null)
  2020-03-10, (null), (null), 3, 4
  2020-03-11, (null), (null), (null), 5
  </code></pre>
  
  <p>Since I know the offsets and which lines I should read for the parsing phase, assuming all the groups will have the same number of lines until the end, I need to go to each group, line by line, and compare the dates between groups to insert data in date order in a new file, like a join operation. </p>
  
  <p>I figured I could do this by using multiple files (one file per group) and multiple readstreams, but the number of groups could be more than 1000 and I'm afraid that I could have a high overhead due to opening/processing multiple files. So I thought it would be better to use a single file and point to different lines of the file by using an offset so I could make these comparisons between groups. However, I don't know if the latter approach is even possible in node.js (read a given line in a file without processing it entirely).</p>
  
  <p>If someone could give me guidance here, I would greatly appreciate.</p>