title: Why does this R loop get iteratively more cumbersome?
body: <p><strong>Some background:</strong> I am interested in comparing results on a nurse-administered test to patients' most-recent bedside vitals, measured electronically:</p>
  
  <ul>
  <li><p>I have one dataset of nurse-administered tests that includes a datetime for when the test was administered.</p></li>
  <li><p>Additionally, I have a second very large (~1B rows) dataset of patient vitals with their own datetimes.</p></li>
  </ul>
  
  <p>My computing resources are limited, so I am interested in writing a nested loop that, for each nurse-administered test (outer loop): 1) Reads in a chunk of the vitals data (inner repeat loop), 2) Checks for data pertaining to the patient the test was administered to, 3) Isolates the row of vitals data with the closest datetime to the nurse test datetime, and 4) Writes the row out to a file.</p>
  
  <pre><code>##ned2 is the nurse test data
  
  for(i in 1:nrow(ned2)){
  
  ##'index' tracks the current chunk# of the large datafile, 
  ##'i' is the row number of the smaller datafile we are currently working with
  
  t &lt;- 999999999
  temp &lt;- NULL
  ned3 &lt;- ned2[i,]
  y &lt;- Sys.time()
  index &lt;- 0
  
  repeat{
  
  ##Read in a piece of the large file; as 'index' increases, skips and reads the next chunk
  
    vitalchunk &lt;- read.delim(file, header = F, sep = "|", fill = T, nrows = chunksize, skip = chunksize*index, col.names = vitals_colnames)
  
  ##Get rid of some useless rows, also restricts vitals data
  ##to rows having the same 'VISIT_ID'
  
    vital &lt;- vitalchunk %&gt;% 
    filter(OBSERVATION_TEXT == "HR", KIDS_VISIT_ID == ned3$KIDS_VISIT_ID)
  
  
  ##If, after filtering, there's nothing to match, skip to the next chunk  
  
    if(! nrow(vital) &gt; 0){
      index &lt;- index + 1
    } else{
  
  ##Otherwise, find the row in the current chunk of the large datafile whose datetime 
  ##is closest to that of the 'i'th row       
  
    ned3$indices &lt;- sapply(ned3$TAKEN_DTM, function(d) which.min(abs( as.POSIXct(vital$CENSUS_MINUTE_DTM) - as.POSIXct(d, format = "%m/%d/%Y %H:%M:%S"))))
    vital$indices &lt;- 1:nrow(vital)
  
  ##match them      
  
    z &lt;- merge(ned3, vital, by = c("KIDS_VISIT_ID", "indices"))
  
  ##If the datetime of the matched row is closer than the stored datetime 
  ##of the previous iteration (starts at t = 9999...), replace it with the current one  
  
    if(abs( as.POSIXct(z$CENSUS_MINUTE_DTM) - as.POSIXct(z$TAKEN_DTM, format = "%m/%d/%Y %H:%M:%S")) &lt; t){
      t &lt;- abs( as.POSIXct(z$CENSUS_MINUTE_DTM) - as.POSIXct(z$TAKEN_DTM, format = "%m/%d/%Y %H:%M:%S"))
      temp &lt;- z
    }
  
  ##Increase index: on repeat, bring in the next chunk      
    index &lt;- index + 1
  }
  
  ## If the chunk we just read in was at the end of the document, break the repeat   
  
  if(! nrow(vitalchunk) == chunksize){break}
  }
  
  ##write out the row into a tab-delimited doc, should be the closest-matched row  
    write(paste(temp, collapse = "\t"), file = "P:/Outcomes/Gabe/NPASS/output.txt", append = TRUE)
  rm(vital, vitalchunk, z)
  }
  </code></pre>
  
  <p>Sorry if the annotations make it harder to see what the code is doing.</p>
  
  <p>Anyway, my question is about what may be happening behind the scenes of this loop.  When I run it, the first few iterations of the "repeat" loop portion run fine.  However, as the loop continues and later "chunks" of the large file are brought in, it takes iteratively longer to run.  The whole point of this loop is to process the large datafile in chunks because it is too large to read into memory at once, but for some reason each iteration of the loop is longer than the previous one and it eventually takes an intractable amount of time.</p>
  
  <p>I suspect that there may be some object being carried from one iteration to the next, but I have had no luck finding the source of this odd behavior.  I'm not experienced with repeat loops, so is there perhaps something in my code that would fix this issue?  Alternatively, could this be a subtle feature of repeat loops?</p>
  
  <p>Thanks for your help, SO; let me know if there's anything I can provide to help.</p>