title: Getting the correlation with significance of one variable with the rest of the dataset, by time, in data.table
body: <p>I stole this example from the following post: <a href="https://stackoverflow.com/questions/42611836/correlationmatrix-from-data-table">LINK</a></p>
  
  <pre><code>set.seed(1)
  TDT &lt;- data.table(Group = c(rep("A",40),rep("B",60)),
                        Id = c(rep(1,20),rep(2,20),rep(3,20),rep(4,20),rep(5,20)),
                        Time = rep(seq(as.Date("2010-01-03"), length=20, by="1 month") - 1,5),
                        norm = round(runif(100)/10,2),
                        x1 = sample(100,100),
                        x2 = round(rnorm(100,0.75,0.3),2),
                        x3 = round(rnorm(100,0.75,0.3),2),
                        x4 = round(rnorm(100,0.75,0.3),2),
                        x5 = round(rnorm(100,0.75,0.3),2))
  </code></pre>
  
  <p>In order to get the correlations of x1 - x5 by time, one could use:</p>
  
  <pre><code>TDT[, x := .(list(cor(.SD))), by = Time, .SDcols = 5:9]
  </code></pre>
  
  <p>However, instead of the correlations of x1 - x5 by time, I am interested in the correlations of <code>x1</code> with ALL other variables (not only x1 - x5) over time. In addition, I somehow would like to have the significance of the correlation: <a href="https://www.youtube.com/watch?v=zMOvsBI5Txo" rel="nofollow noreferrer">for an explanation see video around 5:16</a>.</p>
  
  <p>In order to deal with non-numerical columns and <code>NA</code>'s, I tried to go this way:</p>
  
  <pre><code>numcols &lt;- which(sapply(TDT, is.numeric))
  TDTcor &lt;- TDT[, x := .(list(cor(.SD, use= "pairwise.complete.obs", method= "pearson"))), by = time, .SDcols = numcols]
  </code></pre>
  
  <p>The primary problem is that this still gives all correlations (which gives problems due to the very large datasets I use). It also does not give the significance of the correlations. </p>
  
  <p>Could anyone tell me how to proceed ?</p>