title: Iterating and modifying dataframe using Pandas groupby
body: <p>I am working with a large array of 1's and need to systematically remove 0's from sections of the array. The large array is comprised of many smaller arrays, for each smaller array I need to replace its upper and lower triangles with 0's systematically. For example we have an array with 5 sub arrays indicated by the index value (all sub-arrays have the same number of columns):</p>
  
  <pre><code>     0    1    2
  0  1.0  1.0  1.0
  1  1.0  1.0  1.0
  1  1.0  1.0  1.0
  2  1.0  1.0  1.0
  2  1.0  1.0  1.0
  2  1.0  1.0  1.0
  3  1.0  1.0  1.0
  3  1.0  1.0  1.0
  3  1.0  1.0  1.0
  3  1.0  1.0  1.0
  4  1.0  1.0  1.0
  4  1.0  1.0  1.0
  4  1.0  1.0  1.0
  4  1.0  1.0  1.0
  4  1.0  1.0  1.0
  </code></pre>
  
  <p>I want each group of rows to be modified in its upper and lower triangle such that the resulting matrix is:</p>
  
  <pre><code>      0    1    2
  0  1.0  1.0  1.0
  1  1.0  1.0  0.0
  1  0.0  1.0  1.0
  2  1.0  0.0  0.0
  2  0.0  1.0  0.0
  2  0.0  0.0  1.0
  3  1.0  0.0  0.0
  3  1.0  1.0  0.0
  3  0.0  1.0  1.0
  3  0.0  0.0  1.0
  4  1.0  0.0  0.0
  4  1.0  1.0  0.0
  4  1.0  1.0  1.0
  4  0.0  1.0  1.0
  4  0.0  0.0  1.0
  </code></pre>
  
  <p>At the moment I am using only numpy to achieve this resulting array, but I think I can speed it up using Pandas grouping. In reality my dataset is very large almost 500,000 rows long. The numpy code is below:</p>
  
  <pre><code>import numpy as np
  
  candidateLengths = np.array([1,2,3,4,5])
  centroidLength =3
  
  smallPaths = [min(l,centroidLength) for l in candidateLengths]
  
  # This is the k_values of zeros to delete. To be used in np.tri
  k_vals = list(map(lambda smallPath: centroidLength - (smallPath), smallPaths))
  maskArray = np.ones((np.sum(candidateLengths), centroidLength))
  
  startPos = 0
  endPos = 0
  for canNo, canLen in enumerate(candidateLengths):
      a = np.ones((canLen, centroidLength))
      a *= np.tri(*a.shape, dtype=np.bool, k=k_vals[canNo])
      b = np.fliplr(np.flipud(a))
      c = a*b
  
      endPos = startPos + canLen
  
      maskArray[startPos:endPos, :] = c
  
      startPos = endPos
  
  print(maskArray)
  </code></pre>
  
  <p>When I run this on my real dataset it takes nearly 5-7secs to execute. I think this is down to this massive for loop. How can I use pandas groupings to achieve  a higher speed? Thanks</p>