title: Filtering from changing set in dplyr / data.table instead of for loop
body: <p>I've got a <code>data.frame</code> with 2 columns: Time of the user request in seconds (ranges from <code>1</code> to <code>86400</code>) and the user ID. After counting number of requests, unique requests and IDs of unique request for every second I'm setting the time window - say 10 seconds. Now for every second I want to find a list of new  unique IDs that appeared in this specific second and didn't appeared in my time interval [x-1, x-window] (where x means specific second), count number of this IDs and number of unique IDs in time interval.</p>
  
  <p>I accomplished this using <code>for</code> loop but it's really slow. Is there a way to get the same effect using <code>data.table</code> or <code>dplyr</code> package (or some other faster technique) ?</p>
  
  <p>Simple data:</p>
  
  <pre><code>library(readr)
  library(dplyr)
  data&lt;- structure(list(Second = c(1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 
                                2L, 2L, 3L, 3L, 3L, 3L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L), UniqueId = c(1L, 
                                                                                                      1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 2L, 3L, 9L, 10L, 3L, 7L, 11L, 
                                                                                                      12L, 13L, 1L, 11L, 9L, 14L, 15L)), .Names = c("Second", "UniqueId"
                                                                                                      ), row.names = c(NA, -23L), class = c("tbl_df", "tbl", "data.frame"
                                                                                                      ))
  
  &gt;head(data)
     # A tibble: 6 × 2
    Second UniqueId
     &lt;int&gt;    &lt;int&gt;
  1      1        1
  2      1        1
  3      1        2
  4      1        3
  5      1        4
  6      1        5
  
  
  # Statistics per second
  data2&lt;- data %&gt;% group_by(Second) %&gt;%
       summarise(n_req=n(),n_uniq_req=n_distinct(UniqueId),
              Ids=list(unique(UniqueId))) %&gt;%
   arrange(Second) %&gt;%
   ungroup() 
  
  window &lt;- 2 # two second window
  
  data3&lt;-data2 %&gt;% mutate(New_Ids=NA, UniqueUsersInMemory=NA,
                      NewRequest=NA)
  
  New_Ids=list()
  UniqueUsersInMemory=c()
  NewRequest=c()
  for(i in 1:nrow(data2)){
    temp&lt;- unlist(data2[(data2$Second&lt;data2$Second[i] &amp;   data2$Second&gt;=data2$Second[i]-window),"Ids"])
    New_Ids[[i]]&lt;-setdiff(unlist(data2[i,"Ids"]),temp)
    UniqueUsersInMemory[[i]]&lt;-length(unique(temp))
    NewRequest[[i]]&lt;-length(unique(New_Ids[[i]]))
  }
  data3$New_Ids&lt;-New_Ids
  data3$UniqueUsersInMemory&lt;-UniqueUsersInMemory
  data3$NewRequest&lt;-NewRequest
  </code></pre>
  
  <p>In the end I'm getting something like this:</p>
  
  <p><a href="https://i.stack.imgur.com/B4VPa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/B4VPa.png" alt=""></a></p>