title: Subsetting a CSV based on a percentage of unique values
body: <p>I've been reading through other similar questions. I have this working, but it is very slow due to the size of the CSV I'm working with. Are there ways to make this more efficient?</p>
  <p>My goal:
  I have an incredibly large CSV (&gt;100 GB). I would like to take all of the unique values in a column, extract 10% of these, and then use that 10% to subsample the original CSV.</p>
  <p>What I'm doing:</p>
  <p>1 - I'm pulling all unique values from column 11 and writing those to a text file:</p>
  <pre><code>cat File1.csv | cut -f11 -d , | sort | uniq &gt; uniqueValues.txt
  </code></pre>
  <p>2 - Next, I'm sampling a random 10% of the values in <code>uniqueValues.txt</code>:</p>
  <pre><code>cat uniqueValues.txt | awk 'BEGIN {srand()} !/^$/ { if (rand() &lt;= .10) print $0'} &gt; uniqueValues.10pct.txt
  </code></pre>
  <p>3 - Next, I'm pulling the rows in <code>File1.csv</code> which have column 11 matching values from <code>uniqueValues.10pct.txt</code>:</p>
  <pre><code>awk -F, 'NR==FNR{a[$1]=$0;next}($11 in a){print}' uniqueValues.10pct.txt File1.csv &gt; File1_subsample.csv 
  </code></pre>
  <p>As far as I can tell, this seems to be working. Does this seem reasonable? Any suggestions on how to improve the efficiency?</p>