title: Efficient way of splitting large file into smaller ones based on column value
body: <p>I have a very large csv file which I want to split into smaller files so that all the entries in the large file which have same value for the ID column (second column in the csv) end up in the same file. However, I also need to have 50 different IDs in each of the smaller file.</p>
  
  <p>I have the code to do this, but for a 1 gig file, it takes around 15 - 20 mins. Is there an efficient way of doing this?</p>
  
  <p>This is what I have right now:</p>
  
  <pre><code>awk -F, '{if(NR &gt; 1) {print &gt;&gt; $2"_backfill_tmp.csv"; close($2"_backfill_tmp.csv")}}' $input_file
  counter=0
  for file in *"_backfill_tmp.csv"
  do
    file_name=${input_file%.*}"_backfill2_part_"$PART_NUMBER".csv"
    cat "$file" &gt;&gt; "$file_name"
    rm "$file"
    (( counter++ ))
    if (( $counter % 50 == 0 )) ; then
      (( PART_NUMBER++ ))
    fi
  done
  </code></pre>
  
  <p>The awk command writes each row into separate file based on value of column 2 (ignoring the first row which is a header), so that each of the rows with same ID value end up in the same file. I close the files every time because I run into a <code>Too many files open error</code> and I can't set the <code>ulimit</code> on the machine. However, this process only takes around 15 seconds, so it is not a concern.</p>
  
  <p>I then loop through each of the temporarily created files, and write them into separate file till the <code>$counter</code> reaches 50 (i.e combine 50 file together). Now this is where it is taking a lot of time. I'm guessing since there are a lot of individual ID files, opening them one by one and merging them takes a long time.</p>
  
  <p>I'm an awk beginner, so I'm pretty sure my code isn't efficient. Is there anyway I can do the whole process faster?</p>