title: Extracting the last n rows of each group in a large data table, scalably
body: <p>I have found several solutions on the web that can be used to accomplish this task. However, none of them seems to scale in R. What is the best way to do this?</p>
  
  <p>Problem description:<br/>
  <strong>Data:</strong> DT or DF is a table that has an id column and otherwise lots of random data
  The table is already sorted by id (due to construction)</p>
  
  <p><strong>Task:</strong> Extract the last n rows for each id in the table</p>
  
  <pre><code>library(data.table)
  library(dplyr)
  library(magrittr)
  
  # config
  set.seed(20160313)
  num.rows = 10^5
  
  # build data set
  DT &lt;- data.table( c1=runif(num.rows) )
  for( i in 2:9 )
      DT[[paste0("c",i)]] &lt;- runif(num.rows)
  
  DT$id &lt;- 1:num.rows %/% 4
  
  # solution with data.table
  setkey(DT,id)
  cat( "solution: data.table\n" )
  print(system.time(
      t1 &lt;- DT[,tail(.SD,n=n),by=id]
  ))
  
  # solution with dplyr
  DF &lt;- as.data.frame(DT)
  cat( "solution: dply\n" )
  print(system.time(
      t2 &lt;- DF %&gt;% group_by(id) %&gt;% do( tail(.,n=n) )
  ))
  
  # second solution with dplyr
  cat( "solution: dplyr 2\n" )
  print(system.time({
      t3 &lt;- DF %&gt;% group_by(id) %&gt;% filter(rank(-row_number()) &lt;= n)
  }))
  
  
  # solution with by command
  cat( "solution: by\n" )
  print(system.time( {
      temp &lt;- by( DT, DT$id, tail, n=n )
      t4 &lt;- do.call( "rbind", as.list( temp ) )
  }))
  
  # solution using split and lapply
  cat( "solution: split and lapply\n" )
  print(system.time( {
      temp &lt;- split(DT,DT$id)
      temp &lt;- lapply(temp, tail, n=n)
      t5 &lt;- do.call( "rbind", temp )
  }))
  
  cat( "solution: via data.table 3\n" )
  print(system.time( {
      t6 &lt;- DT[DT[,tail(.I,n),by=id]$V1,]
  }))
  
  
  # failsafe checks
  if( all(t1$c1 == t2$c1) )
      cat( "1==2 OK\n" )
  if( all(t1$c1 == t3$c1) )
      cat( "1==3 OK\n" )
  if( all(t1$c1 == t4$c1) )
      cat( "1==4 OK\n" )
  if( all(t1$c1 == t5$c1) )
      cat( "1==5 OK\n" )
  if( all(t1$c1 == t6$c1) )
      cat( "1==6 OK\n" )
  </code></pre>
  
  <p><strong>edit</strong><br />
  I did some testing of the answers below with 10^7 rows (caution: some of the solutions above will not work for this many rows)</p>
  
  <p><strong>Best performers</strong>:<br />
  <strong>For n=1</strong>, i.e., extracting the last row of each group</p>
  
  <pre><code>system.time( unique(DT,by="id",fromLast=T))
  #   user  system elapsed 
  #  0.376   0.036   0.411
  
  system.time( DT[,.SD[.N],by=id])
  #   user  system elapsed
  # 10.636   0.020  10.652
  </code></pre>
  
  <p><strong>For other n</strong></p>
  
  <pre><code>system.time( DT[DT[,tail(.I,n),by=id]$V1,] )
  # for n=2
  #   user  system elapsed 
  # 33.740   0.112  33.872
  
  # for n=3
  #   user  system elapsed 
  # 33.988   0.184  34.194
  </code></pre>
  
  <p>This still seems a bit much but it works for my case.</p>