title: Why does selecting column(s) from a data.table results in a copy?
body: <p>It appears that selecting column(s) from the data.table with <code>[.data.table</code> results in a copy of the underlying vector(s). I am talking about very simple column selection, by name, there are no expressions to compute in <code>j</code> and there are no rows to subset in <code>i</code>. Even more strangely, the column subsetting in a data.frame appears to not make any copies. I am using the data.table version data.table 1.10.4. A simple example with details and benchmarks is provided below. My questions are:</p>
  
  <ul>
  <li>Am I doing something wrong? </li>
  <li>Is this a bug or is this the intended behavior? </li>
  <li>If this is intended, what is the best approach to subset a data.table by columns and avoid extra copy? </li>
  </ul>
  
  <p>The intended use-case involves large dataset, so avoiding extra copies is a must (especially since base R seems to already support this).</p>
  
  <pre><code>library(data.table)
  set.seed(12345)
  cpp_dt &lt;- data.table(a = runif(1e6), b = rnorm(1e6), c = runif(1e6))
  cols=c("a","c")
  
  ## naive / data.frame style of column selection
  ## leads to a copy of the column vectors in cols
  subset_cols_1=function(dt,cols){
    return(dt[,cols,with=F])
  }
  
  ## alternative syntax, still results in a copy
  subset_cols_2=function(dt,cols){
    return(dt[,..cols])
  }
  
  ## work-around that uses data.frame column selection,
  ## appears to avoid the copy
  subset_cols_3=function(dt,cols){
    setDF(dt)
    subset=dt[,cols]
    setDT(subset)
    setDT(dt)
    return(subset)
  }
  
  ## another approach that makes a "shallow" copy of the data.table
  ## then NULLs the not needed columns by reference
  ## appears to also avoid the copy
  subset_cols_4=function(dt,cols){
    subset=dt[TRUE]
    other_cols=setdiff(names(subset),cols)
    set(subset,j=other_cols,value=NULL)
    return(subset)
  }
  
  subset_1=subset_cols_1(cpp_dt,cols)
  subset_2=subset_cols_2(cpp_dt,cols)
  subset_3=subset_cols_3(cpp_dt,cols)
  subset_4=subset_cols_4(cpp_dt,cols)
  </code></pre>
  
  <p>Now lets look at the memory allocation and compare to original data.</p>
  
  <pre><code>.Internal(inspect(cpp_dt)) # original data, keep an eye on 1st and 3d vector
  # @7fe8ba278800 19 VECSXP g1c7 [OBJ,MARK,NAM(2),ATT] (len=3, tl=1027)
  #   @10e2ce000 14 REALSXP g1c7 [MARK,NAM(2)] (len=1000000, tl=0) 0.720904,0.875773,0.760982,0.886125,0.456481,...
  #   @10f1a3000 14 REALSXP g1c7 [MARK,NAM(2)] (len=1000000, tl=0) -0.947317,-0.636669,0.167872,-0.206986,0.411445,...
  #   @10f945000 14 REALSXP g1c7 [MARK,NAM(2)] (len=1000000, tl=0) 0.717611,0.95416,0.191546,0.48525,0.539878,...
  # ATTRIB: [removed]
  </code></pre>
  
  <p>Using <code>[.data.table</code> method to subset the columns:</p>
  
  <pre><code>.Internal(inspect(subset_1)) # looks like data.table is making a copy
  # @7fe8b9f3b800 19 VECSXP g0c7 [OBJ,NAM(1),ATT] (len=2, tl=1026)
  #   @114cb0000 14 REALSXP g0c7 [MARK,NAM(2)] (len=1000000, tl=0) 0.720904,0.875773,0.760982,0.886125,0.456481,...
  #   @1121ca000 14 REALSXP g0c7 [NAM(2)] (len=1000000, tl=0) 0.717611,0.95416,0.191546,0.48525,0.539878,...
  # ATTRIB: [removed]
  </code></pre>
  
  <p>Another syntax version that still uses <code>[.data.table</code> and still making a copy:</p>
  
  <pre><code>.Internal(inspect(subset_2)) # same, still copy
  # @7fe8b6402600 19 VECSXP g0c7 [OBJ,NAM(1),ATT] (len=2, tl=1026)
  #   @115452000 14 REALSXP g0c7 [NAM(2)] (len=1000000, tl=0) 0.720904,0.875773,0.760982,0.886125,0.456481,...
  #   @1100e7000 14 REALSXP g0c7 [NAM(2)] (len=1000000, tl=0) 0.717611,0.95416,0.191546,0.48525,0.539878,...
  # ATTRIB: [removed]
  </code></pre>
  
  <p>Using a sequence of <code>setDF</code>, followed by <code>[.data.frame</code> and <code>setDT</code>. Look, the vectors <code>a</code> and <code>c</code> are no longer copied! It appears that base R method is more efficient / has smaller memory footprint?</p>
  
  <pre><code>.Internal(inspect(subset_3)) # "[.data.frame" is not making a copy!!
  # @7fe8b633f400 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=1026)
  #   @10e2ce000 14 REALSXP g1c7 [MARK,NAM(2)] (len=1000000, tl=0) 0.720904,0.875773,0.760982,0.886125,0.456481,...
  #   @10f945000 14 REALSXP g1c7 [MARK,NAM(2)] (len=1000000, tl=0) 0.717611,0.95416,0.191546,0.48525,0.539878,...
  # ATTRIB: [removed]
  </code></pre>
  
  <p>Another approach is to make a shallow copy of the data.table, then NULL all the extra columns by reference in the new data.table. Again no copies are made.</p>
  
  <pre><code>.Internal(inspect(subset_4)) # 4th approach seems to also avoid the copy
  # @7fe8b924d800 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=1027)
  #   @10e2ce000 14 REALSXP g1c7 [MARK,NAM(2)] (len=1000000, tl=0) 0.720904,0.875773,0.760982,0.886125,0.456481,...
  #   @10f945000 14 REALSXP g1c7 [MARK,NAM(2)] (len=1000000, tl=0) 0.717611,0.95416,0.191546,0.48525,0.539878,...
  # ATTRIB: [removed]
  </code></pre>
  
  <p>Now lets look at the benchmarks of these four approaches. It looks like "[.data.frame" (<code>subset_cols_3</code>) is a clear winner.</p>
  
  <pre><code>microbenchmark({subset_cols_1(cpp_dt,cols)},
                 {subset_cols_2(cpp_dt,cols)},
                 {subset_cols_3(cpp_dt,cols)},
                 {subset_cols_4(cpp_dt,cols)},
                 times=100)
  
  # Unit: microseconds
  #                                 expr      min        lq      mean   median        uq       max neval
  #  {     subset_cols_1(cpp_dt, cols) } 4772.092 5128.7395 8956.7398 7149.447 10189.397 53117.358   100
  #  {     subset_cols_2(cpp_dt, cols) } 4705.383 5107.1690 8977.1816 6680.666  9206.164 53523.191   100
  #  {     subset_cols_3(cpp_dt, cols) }  148.659  177.9595  285.4926  250.620   283.414  4422.968   100
  #  {     subset_cols_4(cpp_dt, cols) }  193.912  241.9010  531.8308  336.467   384.844 20061.864   100
  </code></pre>