title: How to monitor progress of an apply function?
body: <p>I need to work out a 2886*2886 correlation matrix, problem is that building an intermediary datatable (<code>RESULT</code>) takes a long time for it to be binded together so I would like to be able to do the following things while calling the last line <code>RESULT=rbindlist(apply(COMB, 1, append))</code> in the code below :</p>
  
  <ol>
  <li>Estimate the time it will take for the apply function to complete</li>
  <li>Monitor its progress</li>
  <li>Be able to pause and continue at later time</li>
  </ol>
  
  <p>Here is the code :</p>
  
  <pre><code>SOURCE=data.table(NAME=rep(paste0("NAME", as.character(1:2889)), each=600), VALUE=sample(c(TRUE,FALSE), 600, TRUE) )
  &gt; SOURCE
              NAME VALUE
        1:   NAME1  TRUE
        2:   NAME1  TRUE
        3:   NAME1  TRUE
        4:   NAME1  TRUE
        5:   NAME1  TRUE
       ---              
  1733396: NAME999  TRUE
  1733397: NAME999  TRUE
  1733398: NAME999  TRUE
  1733399: NAME999  TRUE
  1733400: NAME999 FALSE
  
  setkey(SOURCE,NAME)
  a=SOURCE[,unique(NAME)]
  COMB=data.table(expand.grid(a,a, stringsAsFactors=FALSE))
  &gt; COMB
               Var1    Var2
        1:    NAME1   NAME1
        2:   NAME10   NAME1
        3:  NAME100   NAME1
        4: NAME1000   NAME1
        5: NAME1001   NAME1
       ---                 
  8346317:  NAME995 NAME999
  8346318:  NAME996 NAME999
  8346319:  NAME997 NAME999
  8346320:  NAME998 NAME999
  8346321:  NAME999 NAME999
  
  append &lt;- function(X) {
  data.table(NAME1=X[1], VALUE1=SOURCE[X[1], VALUE], 
      NAME2=X[2], VALUE2=SOURCE[X[2], VALUE] )
  }
  
  RESULT=rbindlist(apply(COMB, 1, append))
  </code></pre>
  
  <p>Any idea ?</p>
  
  <p>Also do you know if there is a faster way to generate the datatable <code>RESULT</code> from  <code>SOURCE</code> ? <code>RESULT</code>is an intermediary datatable to work out the correlation values between <code>VALUE1</code> and <code>VALUE2</code> for each couple of <code>NAME</code>.</p>
  
  <p>With a subset of <code>SOURCE</code> <code>RESULT</code>looks like that :</p>
  
  <pre><code>SOURCE=SOURCE[sample(1:nrow(SOURCE), 3)]
  setkey(SOURCE,NAME)
  a=SOURCE[,unique(NAME)]
  COMB=data.table(expand.grid(a,a, stringsAsFactors=FALSE))
  RESULT=rbindlist(apply(COMB, 1, append))
  &gt; RESULT
        NAME1 VALUE1    NAME2 VALUE2
  1: NAME1859   TRUE NAME1859   TRUE
  2:  NAME768  FALSE NAME1859   TRUE
  3:  NAME795   TRUE NAME1859   TRUE
  4: NAME1859   TRUE  NAME768  FALSE
  5:  NAME768  FALSE  NAME768  FALSE
  6:  NAME795   TRUE  NAME768  FALSE
  7: NAME1859   TRUE  NAME795   TRUE
  8:  NAME768  FALSE  NAME795   TRUE
  9:  NAME795   TRUE  NAME795   TRUE
  </code></pre>
  
  <p>Later on I'm going to do <code>RESULT[,VALUE3:=(VALUE1==VALUE2)]</code> to finally get the correlation values : <code>RESULT[, mean(VALUE3), by=c("NAME1", "NAME2")]</code>
  So maybe the whole process can be done more efficiently, who knows.</p>