title: R: efficiently identifying highest N values of variable Z by group X
body: <p>I have a datatable that looks like this.</p>
  
  <pre><code>ID &lt;- c(rep("ABC",4), rep("DEF",4), rep("GHI",5))
  X  &lt;- c(rep(c(1,2,3,4),3),5)
  set.seed(1234)
  Z  &lt;- runif(13,min=0, max =1)  
  a &lt;- data.table(ID, X, Z)
  a
  
       ID X           Z
   1: ABC 1 0.113703411
   2: ABC 2 0.622299405
   3: ABC 3 0.609274733
   4: ABC 4 0.623379442
   5: DEF 1 0.860915384
   6: DEF 2 0.640310605
   7: DEF 3 0.009495756
   8: DEF 4 0.232550506
   9: GHI 1 0.666083758
  10: GHI 2 0.514251141
  11: GHI 3 0.693591292
  12: GHI 4 0.544974836
  13: GHI 5 0.282733584
  </code></pre>
  
  <p>I'd like to produce a dataframe that has the N highest values of Z within each X subgroup. So lets say N is 2. I'd like to end up with a dataset that looks like this:</p>
  
  <pre><code>   X  ID         Z
  1: 1 DEF 0.8609154
  2: 1 GHI 0.6660838
  3: 2 DEF 0.6403106
  4: 2 ABC 0.6222994
  5: 3 GHI 0.6935913
  6: 3 ABC 0.6092747
  7: 4 ABC 0.6233794
  8: 4 GHI 0.5449748
  9: 5 GHI 0.2827336
  </code></pre>
  
  <p>I've been using this line to achive it, but I've found it to be particularly slow when the datatable is large (i.e. over 1,500,000 lines or more.)</p>
  
  <pre><code>top_n &lt;- 2
  a &lt;- a[order(a$X, -a$Z),]
  a_2 &lt;- a[, head(.SD, top_n), by=X]
  a_2
  
     X  ID         Z
  1: 1 DEF 0.8609154
  2: 1 GHI 0.6660838
  3: 2 DEF 0.6403106
  4: 2 ABC 0.6222994
  5: 3 GHI 0.6935913
  6: 3 ABC 0.6092747
  7: 4 ABC 0.6233794
  8: 4 GHI 0.5449748
  9: 5 GHI 0.2827336
  </code></pre>
  
  <p>Any help would be most appreciated! </p>
  
  <p>Thanks!</p>