title: R - Working on Speed / Efficiency - Creating + Updating a Matrix from a large file
body: <p>Ill start with my intuition - I presume there is some way to do this using "apply" without looping over every single entry in my file, but I cannot figure it out.</p>
  
  <p>I have a very large file. Around 4,000,000 rows, 6 columns.  For the sake of this discussion, only 3 of those columns are relevant.  They are <code>day</code>, <code>grade1</code> and <code>grade2</code></p>
  
  <p>I want to create a matrix with multiple fields, using those 3 columns from my .txt mentioned above.  </p>
  
  <p>I want <code>day</code>, <code>grade1Wins</code>, <code>grade2Wins</code>, <code>ties</code>, <code>grade1Score</code>, <code>grade2Score</code>, <code>grade1Avg</code> and <code>grade2Avg</code>.  These are all per day.</p>
  
  <p>So my output matrix will have those 8 columns, and rows equal to the number of days that there are.</p>
  
  <p>I define these as follows.</p>
  
  <p><code>Grade1Wins</code> : Starts at 0, increment by 1 for each row where <code>Grade1</code> > <code>|Grade2|</code>.
  <code>Grade2Wins</code> : Starts at 0, increment by 1 for each row where <code>|Grade2|</code> > <code>Grade1</code>.
  <code>ties</code> : Starts at 0, increment by 1 for each row where <code>Grade1</code> == <code>|Grade2|</code></p>
  
  <p><code>Grade1Score</code> : Summation of raw score values.
  <code>Grade2Score</code> : Summation of raw score values.</p>
  
  <p><code>Grade1Avg :</code>Grade1Score<code>/ (numRows)
  </code>Grade2Avg : <code>Grade2Score</code> / (numRows)</p>
  
  <p>I want this all as a matrix.  So, for example, my data may look like....(subset to only show the 2 relevant columns)</p>
  
  <pre><code>Day    Grade1    Grade2
  1    2         -4
  1    4         -4
  2    10        -1
  </code></pre>
  
  <p>So my output matrix would be</p>
  
  <pre><code>`day` `grade1wins` `grade2wins` `ties` `grade1score` `grade2score` `grade1avg` `grade2avg`
    1        0           1          1         6           -8             2          -4
    2        1           0          0         10          -1             10          -1
  </code></pre>
  
  <p>Currently, I do this via loops.
  My matrix is pre-defined (# cols / # rows determined in advance, have a blank matrix full of 0's waiting for editting).</p>
  
  <p>I loop through every row of my tab-delimited .txt file.</p>
  
  <p>I determine the day of the current row.  This is my row number.
  I pull the values from the row for <code>grade1</code> and <code>grade2</code>.</p>
  
  <pre><code>#Loop over every single row.
  for(i in 1:len)
  {
              entry = entries[i,]
              rowNum = entry$day
              if( entry$grade1&gt; abs(entry$grade2) )
              {
                  mat[rowNum, "grade1wins"] = mat[rowNum, "grade1wins"] + 1 ## Increment the counter 
              } else if( abs(entry$grade2) &gt; entry$grade1 ) {
                  mat[rowNum, "grade2wins"] = mat[rowNum, "grade2wins"] + 1 ## Increment the counter 
              } else {
                  mat[rowNum, "ties"] = mat[rowNum, "ties"] + 1
              }
  
              mat[rowNum, "grade1"] = mat[rowNum, "grade1"] + entry$grade1
              mat[rowNum, "grade2"] = mat[rowNum, "grade2"] + entry$grade2
  } # end loop, we went through every single entry now
  mat[, "PosAvg"] = mat[,"PosScore"] / mat[, "NumTweets"]
  mat[, "NegAvg"] = mat[,"NegScore"] / mat[, "NumTweets"]
  </code></pre>
  
  <p>I presume there is some way to do this using "apply" without looping over every single entry in my file, but I cannot figure it out.</p>