title: anomaly detection using Paralel computing in R
body: <p>I have a data frame with more then 300mil rows and I want to detect anomaly in each group, which is consist of country and IDs (Each group) then I wrote the following code to detect anomaly points however it takes a lot time. Could you please suggest any other option that make it faster. The data frame format:</p>
  <pre><code> df &lt;- data.frame(&quot;id&quot; = 1:n,&quot;country&quot;= (&quot;US&quot;,..),&quot;date&quot;=(&quot;2021-01-01&quot;,..),&quot;value&quot;=c(10,....)) 
  
  
      registerDoParallel()
  groupColumns &lt;- c(&quot;country&quot;,&quot;id&quot;)
  system.time(temp_anom &lt;- ddply(df, groupColumns, function(x){
    x &lt;- x[,c('date','value')]  
    resid.q &lt;- quantile(x$value,prob = c(0.1,0.90))
    iqr &lt;- diff(resid.q)
    limits &lt;- resid.q + 3 * iqr * c(-1,1) 
    lower_bound &lt;- limits[1]
    upper_bound &lt;- limits[2]
    outlier_dip_index &lt;- dplyr::filter(x, value &lt; lower_bound) %&gt;% data.frame() 
    if (nrow(outlier_dip_index) &gt; 0) {
      outlier_dip_index$status &lt;- &quot;dip&quot;}
    outlier_spike_index &lt;- dplyr::filter(x, value &gt; upper_bound) %&gt;% data.frame()
    if (nrow(outlier_spike_index) &gt; 0) {
      outlier_spike_index$status &lt;- &quot;spike&quot;  
      outlier &lt;- rbind(outlier_spike_index,outlier_dip_index)
      outlier
    }
  },.paralle = T))
  </code></pre>