title: R - Why adding 1 column to data table nearly doubles peak memory used?
body: <p>After getting <a href="https://stackoverflow.com/questions/28279078/r-how-to-run-average-max-on-different-data-table-columns-based-on-multiple-f">help</a> from 2 kind gentlemen, I managed to switch over to data tables from data frame+plyr.</p>
  
  <p><strong>The Situation and My Questions</strong></p>
  
  <p>As I worked on, I noticed that <em>peak memory usage</em> nearly doubled from 3.5GB to 6.8GB (according to Windows Task Manager) when I added 1 new column using <code>:=</code> to my data set containing ~200K rows by 2.5K columns. </p>
  
  <p>I then tried 200M row by 25 col, the increase was from 6GB to 7.6GB before dropping to 7.25GB after a <code>gc()</code>.</p>
  
  <p>Specifically regarding adding of new columns, Matt Dowle himself mentioned <a href="https://stackoverflow.com/questions/1358003/tricks-to-manage-the-available-memory-in-an-r-session">here</a> that:</p>
  
  <blockquote>
    <p>With its := operator you can :</p>
  
  <pre><code>Add columns by reference
  Modify subsets of existing columns by reference, and by group by reference
  Delete columns by reference
  </code></pre>
    
    <p>None of these operations copy the (potentially large) data.table at
    all, not even once.</p>
  </blockquote>
  
  <p>Question 1: why would adding a single column of 'NAs' for a DT with 2.5K columns <em>double</em> the peak memory used if the data.table is not copied at all?</p>
  
  <p>Question 2: Why does the doubling not occur when the DT is 200M x 25? I didn't include the printscreen for this, but feel free to change my code and try.</p>
  
  <p><strong>Printscreens for Memory Usage using Test Code</strong></p>
  
  <ol>
  <li><p>Clean re-boot, RStudio &amp; MS Word opened - 103MB used
  <img src="https://i.stack.imgur.com/ZXhTl.jpg" alt="1. Clean re-boot, RStudio &amp; MS Word opened"></p></li>
  <li><p>Aft running DT creation code but before adding column - 3.5GB used
  <img src="https://i.stack.imgur.com/xQxVb.jpg" alt="enter image description here"></p></li>
  <li><p>After adding 1 Column filled with NA, but before gc() - 6.8GB used
  <img src="https://i.stack.imgur.com/rJ93U.jpg" alt="enter image description here"></p></li>
  <li><p>After running gc() - 3.5GB used
  <img src="https://i.stack.imgur.com/HzxA7.jpg" alt="enter image description here"></p></li>
  </ol>
  
  <p><strong>Test Code</strong></p>
  
  <p>To investigate, I did up the following test code that closely mimics my data set:</p>
  
  <pre><code>library(data.table)
  set.seed(1)
  
  # Credit: Dirk Eddelbuettel's answer in 
  # https://stackoverflow.com/questions/14720983/efficiently-generate-a-random-sample-of-times-and-dates-between-two-dates
  RandDate &lt;- function(N, st="2000/01/01", et="2014/12/31") { 
    st &lt;- as.POSIXct(as.Date(st))
    et &lt;- as.POSIXct(as.Date(et))
    dt &lt;- as.numeric(difftime(et,st,unit="sec"))
    ev &lt;- runif(N, 0, dt)
    rt &lt;- as.character( strptime(st + ev, "%Y-%m-%d") )
  }
  
  # Create Sample data
  TotalNoCol &lt;- 2500
  TotalCharCol &lt;- 3
  TotalDateCol &lt;- 1
  TotalIntCol &lt;- 600
  TotalNumCol &lt;- TotalNoCol - TotalCharCol - TotalDateCol - TotalIntCol
  nrow &lt;- 200000
  
  ColNames = paste0("C", 1:TotalNoCol)
  
  dt &lt;- as.data.table( setNames( c(
  
    replicate( TotalCharCol, sample( state.name, nrow, replace = T ), simplify = F ), 
    replicate( TotalDateCol, RandDate( nrow ), simplify = F ), 
    replicate( TotalNumCol, round( runif( nrow, 1, 30 ), 2), simplify = F ), 
    replicate( TotalIntCol, sample( 1:10, nrow, replace = T ), simplify = F ) ), 
  
      ColNames ) )
  
  gc()
  
  # Add New columns, to be run separately
  dt[, New_Col := NA ]  # Additional col; uses excessive memory?
  </code></pre>
  
  <p><strong>Research Done</strong></p>
  
  <p>I didn't find too much discussion on memory usage for DT with many columns, only <a href="https://stackoverflow.com/questions/11988643/big-data-read-subsamples-r">this</a> but even then, it's not specifically about memory.</p>
  
  <p>Most discussions on large dataset + memory usage involves DTs with very large rowcount but relatively few columns.</p>
  
  <p><strong>My System</strong></p>
  
  <p>Intel i7-4700 with 4-core/8-thread; 16GB DDR3-12800 RAM; Windows 8.1 64-bit; 500GB 7200rpm HDD; 64-bit R; Data Table ver 1.9.4</p>
  
  <p><strong>Disclaimers</strong></p>
  
  <p>Please pardon me for using a 'non-R' method (i.e. Task Manager) to measure memory used. Memory measurement/profiling in R is something I still haven't figured out.</p>
  
  <hr>
  
  <p>Edit 1: After updating to data table ver 1.9.5 and re-running. Issue persisted, unfortunately.</p>
  
  <p><img src="https://i.stack.imgur.com/yx4UE.jpg" alt="enter image description here"></p>