title: [R]: applying a function to columns based on conditional row position
body: <p>I am attempting to find the number of observations by column in a data frame that meet a certain condition after the max for that column has been encountered.</p>
  
  <p>Here is a highly simplified example:</p>
  
  <pre><code>fake.dat&lt;-data.frame(samp1=c(5,6,7,5,4,5,10,5,6,7), samp2=c(2,3,4,6,7,9,2,3,7,8), samp3=c(2,3,4,11,7,9,2,3,7,8),samp4=c(5,6,7,5,4,12,10,5,6,7))
  
         samp1 samp2 samp3 samp4
  1      5     2     2     5
  2      6     3     3     6
  3      7     4     4     7
  4      5     6    11     5
  5      4     7     7     4
  6      5     9     9    12
  7     10     2     2    10
  8      5     3     3     5
  9      6     7     7     6
  10     7     8     8     7
  </code></pre>
  
  <p>So, let's say I'm trying to find the number of observations per column that are greater than 5 after excluding all the observations in a column up to and including the row where the maximum for the column occurs.</p>
  
  <p>Expected outcome:</p>
  
  <pre><code>samp1 samp2 samp3 samp4 
     2     2     4    3 
  </code></pre>
  
  <p>I am able to get the answer I want by using nested <code>for loops</code> to exclude the observations I don't want.  </p>
  
  <pre><code>newfake.dat&lt;-data.frame()
  
  for(j in 1:length(fake.dat)){
  for(i in 1:nrow(fake.dat)){
      ifelse(i&gt;max.row[j],newfake.dat[i,j]&lt;-fake.dat[i,j],"NA")
  print(newfake.dat)
  }}
  </code></pre>
  
  <p>This creates a new data frame on which I can run an easy <code>apply</code> function.</p>
  
  <pre><code>colcount&lt;-apply(newfake.dat,2,function(x) (sum(x&gt;5,na.rm=TRUE)))
  
     V1 V2 V3 V4
  1  NA NA NA NA
  2  NA NA NA NA
  3  NA NA NA NA
  4  NA NA NA NA
  5  NA NA  7 NA
  6  NA NA  9 NA
  7  NA  2  2 10
  8   5  3  3  5
  9   6  7  7  6
  10  7  8  8  7
  
  V1 V2 V3 V4 
   2  2  4  3 
  </code></pre>
  
  <p>Which is all well and good for this tiny example dataset, but is prohibitively slow on anything approaching the size of my real datasets.  Which are large (2000 x 2000 or larger) and numerous.  I tried it with a truncated version of one of my files (fewer columns, but same number of rows) and it ran for at least 5 hours (I left it going when I left work for the day).  Also, I don't really <em>need</em> the new dataframe for anything other than to be able to run the apply function.</p>
  
  <p>Is there any way to do this more efficiently?  I tried limiting the rows that the <code>apply</code> function works on by using <code>seq</code> and the row number of the max.  </p>
  
  <pre><code>maxrow&lt;-apply(fake.dat,2,function(x) which.max(x))
  print(maxrow)
  
  seq.att&lt;-apply(fake.dat,2,function(x) {
      sum(x[which(seq(1,nrow(fake.dat))==(maxrow)):nrow(fake.dat)]&gt;5,na.rm=TRUE)})
  </code></pre>
  
  <p>Which kicks up four instances of this warning message:</p>
  
  <pre><code>1: In seq(1, nrow(fake.dat)) == (maxrow) :
    longer object length is not a multiple of shorter object length
  </code></pre>
  
  <p>If I ignore the warning message and get the output anyway it doesn't give me the answer I expected:</p>
  
  <pre><code>samp1 samp2 samp3 samp4 
      2     3     3     3 
  </code></pre>
  
  <p>I also tried using a <code>while</code> function which kept cycling so I stopped it (I misplaced the code I tried for this).  </p>
  
  <p>So far the most promising result has come from the nested <code>for loops</code>, but I know it's terribly inefficient and I'm hoping that there's a better way.  I'm still new to R, and I'm sure I'm tripping up on some syntax somewhere.  Thanks in advance for any help you can provide!</p>