title: Foreach Loop - Object not found
body: <p>Apologies for long delay. Some things came up and haven't had a chance to come back to this and update the post with clearer and easier to follow details/data/code.</p>
  
  <p>Here's some data.   </p>
  
  <pre><code>data &lt;- data.table(ZIP1 = c('99999', '99999', '99999', '99999', '99999'),
                        City1 = c('LOS ANGELAS', 'OAKLAND', 'SAN DIEGO', 'LOS ANGELOS', 'LOST ANGELOST'),
                        Name1 = c("JOHN", 'JOhn', 'JoN', 'JOHN', 'JOHNSON'))
  
  data2 &lt;- data.table(ZIP2 = c('99999', '12345', '99999', '99999', '99999'),
                     City2 = c('LOS ANGELAS', 'OAKLAND', 'SAN DIEGO', 'LOS ANGELOS', 'LOST ANGELOST'),
                     Name2 = c("JOHN", 'JOhn', 'JoN', 'JOHN', 'JOHNSON'))
  
  zips&lt;- data.table(zip = c('12345','45678','19899','99999','02345','98129','09101','10001','09839'))
  </code></pre>
  
  <p>Code how it works today:</p>
  
  <pre><code>library('stringr')
  library('stringdist')
  library('readr')
  library('data.table')
  
  func1 &lt;- function(df) {
    df2[] &lt;- lapply(df2, as.character)
    df2$MatchName &lt;- 1-stringdist(data$Name1, data2$Name2, method="jw", p=0.1)
    df2$MatchCity &lt;- 1-stringdist(data$City1, data2$City2, method="jw", p=0.1)
    ##df1$glm &lt;- predict(fit.glm, df1)  Overlay a model to predict if it's a match
    ##df1matches &lt;- df1[glm == '1'] And then we write it somewhere else, SqlServer, disk, etc.
    #rm(df1) then we remove it as we loop through the next zip of matches
  }                      
  
  setkey(data, ZIP1)
  setkey(data2, ZIP2)
  setkey(zips, zip)
  
  for(row in zips$zip) {
    #print(row)
    df1 &lt;- data[ZIP1 %in% row] 
    df2 &lt;- df1[data2, nomatch=0, allow.cartesian=TRUE]
    if (nrow(df1) == 0) {
      next
    }
    df2[,func1(.SD)]
  }
  </code></pre>
  
  <p>Which returns the following dataframe</p>
  
  <pre><code> ZIP1         City1   Name1       City2 Name2 MatchName MatchCity
  1: 99999   LOS ANGELAS    JOHN LOS ANGELAS  JOHN 0.7333333 0.5200216
  2: 99999       OAKLAND    JOhn LOS ANGELAS  JOHN 0.7333333 0.5200216
  3: 99999     SAN DIEGO     JoN LOS ANGELAS  JOHN 1.0000000 1.0000000
  4: 99999   LOS ANGELOS    JOHN LOS ANGELAS  JOHN 1.0000000 1.0000000
  5: 99999 LOST ANGELOST JOHNSON LOS ANGELAS  JOHN 1.0000000 1.0000000
  6: 99999   LOS ANGELAS    JOHN   SAN DIEGO   JoN 0.7333333 0.5200216
  </code></pre>
  
  <p>Essentially, I'm trying to run the loop in parallel to speed it up.  Depending on the size of "data", this process can take up to 5 hours. Due to the size and memory constraints, we are using the loop to segment the datasets into smaller manageable pieces. Just to reiterate, the process we have today does in fact work, and works well.  The hope is to potentially speed it up with foreach and using a parallel backend.  Ideally, the returned result is a dataframe/datatable as we insert each row into a database(can also write to disk and then insert).</p>
  
  <pre><code>    results = foreach(zips=iter(zips, by='row'), .combine=rbind) %dopar%  {
    df1 &lt;- data[data$ZIP1 %in% row]
    df2 &lt;- df1[data2, nomatch=0, allow.cartesian=TRUE]
    if (nrow(df1) == 0) {
    next
    }
  df2[,func1(.SD)]
  }
  </code></pre>
  
  <p>My attempt however is throwing an "undefined columns selected" error.</p>