title: working with large lists that become too big for RAM when operated on
body: <p>Short of working on a machine with more RAM, how can I work with large lists in <code>R</code>, for example put them on disk and then work on sections of it?</p>
  
  <p>Here's some code to generate the type of lists I'm using</p>
  
  <pre><code>n = 50; i = 100
  WORD &lt;- vector(mode = "integer", length = n)
  for (i in 1:n){
    WORD[i] &lt;- paste(sample(c(rep(0:9,each=5),LETTERS,letters),5,replace=TRUE),collapse='')
  }
  dat &lt;- data.frame(WORD =  WORD,
                    COUNTS = sample(1:50, n, replace = TRUE))
  dat_list &lt;- lapply(1:i, function(i) dat) 
  </code></pre>
  
  <p>In my actual use case each data frame in the list is unique, unlike the quick example here. I'm aiming for n = 4000 and i = 100,000</p>
  
  <p>This is one example of what I want to do with this list of dataframes:</p>
  
  <pre><code>FUNC &lt;- function(x) {rep(x$WORD, times = x$COUNTS)}
  la &lt;- lapply(dat_list, FUNC)
  </code></pre>
  
  <p>With my actual use case this runs for a few hours, fills up the RAM and most of the swap and then RStudio freezes and shows a message with a bomb on it (RStudio was forced to terminate due to an error in the R session). </p>
  
  <p>I see that <code>bigmemory</code> is limited to matrices and <code>ff</code> doesn't seem to handle lists. What are the other options? If <code>sqldf</code> or a related out-of-memory method possible here, how might I get started? I can't get enough out of the documentation to make any progress and would be grateful for any pointers. Note that instructions to "buy more RAM" will be ignored! This is for a package that I'm hoping will be suitable for average desktop computers (ie. undergrad computer labs). </p>
  
  <p><strong>UPDATE</strong> Followining up on the helpful comments from SimonO101 and Ari, here's some benchmarking comparing dataframes and data.tables, loops and lapply, and with and without gc</p>
  
  <pre><code># self-contained speed test of untable
  n = 50; i = 100
  WORD &lt;- vector(mode = "integer", length = n)
  for (i in 1:n){
    WORD[i] &lt;- paste(sample(c(rep(0:9,each=5),LETTERS,letters),5,replace=TRUE),collapse='')
  }
  # as data table
  library(data.table)
  dat_dt &lt;- data.table(WORD = WORD, COUNTS = sample(1:50, n, replace = TRUE))
  dat_list_dt &lt;- lapply(1:i, function(i) dat_dt)
  
  # as data frame
  dat_df &lt;- data.frame(WORD =  WORD, COUNTS = sample(1:50, n, replace = TRUE))
  dat_list_df &lt;- lapply(1:i, function(i) dat_df)
  
  # increase object size
  y &lt;- 10
  dt &lt;- c(rep(dat_list_dt, y))
  df &lt;- c(rep(dat_list_df, y))
  # untable
  untable &lt;- function(x) rep(x$WORD, times = x$COUNTS)
  
  
  # preallocate objects for loop to fill
  df1 &lt;- vector("list", length = length(df))
  dt1 &lt;- vector("list", length = length(dt))
  df3 &lt;- vector("list", length = length(df))
  dt3 &lt;- vector("list", length = length(dt))
  # functions for lapply
  df_untable_gc &lt;- function(x) { untable(df[[x]]); if (x%%10) invisible(gc()) }
  dt_untable_gc &lt;- function(x) { untable(dt[[x]]); if (x%%10) invisible(gc()) }
  # speedtests
  library(microbenchmark)
  microbenchmark(
    for(i in 1:length(df)) { df1[[i]] &lt;- untable(df[[i]]); if (i%%10) invisible(gc()) },
    for(i in 1:length(dt)) { dt1[[i]] &lt;- untable(dt[[i]]); if (i%%10) invisible(gc()) },
    df2 &lt;- lapply(1:length(df), function(i) df_untable_gc(i)),
    dt2 &lt;- lapply(1:length(dt), function(i) dt_untable_gc(i)),
    for(i in 1:length(df)) { df3[[i]] &lt;- untable(df[[i]])},
    for(i in 1:length(dt)) { dt3[[i]] &lt;- untable(dt[[i]])},
    df4 &lt;- lapply(1:length(df), function(i) untable(df[[i]]) ),
    dt4 &lt;- lapply(1:length(dt), function(i) untable(dt[[i]]) ),
  
    times = 10)
  </code></pre>
  
  <p>And here are the results, without explicit garbage collection, data.table is much faster and lapply slightly faster than a loop. With explicit garbage collection (as I think SimonO101 might be suggesting) they are all much the same speed - a lot slower! I know that using <code>gc</code> is a bit controversial and probably not helpful in this case, but I'll give it a shot with my actual use-case and see if it makes any difference. Of course I don't have any data on memory use for any of these functions, which is really my main concern. Seems that there is no function for memory benchmarking equivalent to the timing functions (for windows, anyway). </p>
  
  <pre><code>Unit: milliseconds
                                                                                                   expr
   for (i in 1:length(df)) {     df1[[i]] &lt;- untable(df[[i]])     if (i%%10)          invisible(gc()) }
   for (i in 1:length(dt)) {     dt1[[i]] &lt;- untable(dt[[i]])     if (i%%10)          invisible(gc()) }
                                              df2 &lt;- lapply(1:length(df), function(i) df_untable_gc(i))
                                              dt2 &lt;- lapply(1:length(dt), function(i) dt_untable_gc(i))
                                           for (i in 1:length(df)) {     df3[[i]] &lt;- untable(df[[i]]) }
                                           for (i in 1:length(dt)) {     dt3[[i]] &lt;- untable(dt[[i]]) }
                                              df4 &lt;- lapply(1:length(df), function(i) untable(df[[i]]))
                                              dt4 &lt;- lapply(1:length(dt), function(i) untable(dt[[i]]))
            min           lq       median           uq         max neval
   37436.433962 37955.714144 38663.120340 39142.350799 39651.88118    10
   37354.456809 38493.268121 38636.424561 38914.726388 39111.20439    10
   36959.630896 37924.878498 38314.428435 38636.894810 39537.31465    10
   36917.765453 37735.186358 38106.134494 38563.217919 38751.71627    10
      28.200943    29.221901    30.205502    31.616041    34.32218    10
      10.230519    10.418947    10.665668    12.194847    14.58611    10
      26.058039    27.103217    27.560739    28.189448    30.62751    10
       8.835168     8.904956     9.214692     9.485018    12.93788    10
  </code></pre>