title: Numpy for loop vectorization: broadcasting and testing for unique elements with np.all
body: <h2>The working for loop, expected result</h2>
  
  <p>I am attempting to vectorize a slow for loop in code with a very large dataset to remove duplicates based on a test.  The result should keep only elements where the first 3 elements are unique, and the 4th element is the largest of all duplicates.  e.g.</p>
  
  <pre><code>in = np.array(((0, 12, 13, 1), (0, 12, 13, 10), (1, 12, 13, 2)))
  </code></pre>
  
  <p>should become</p>
  
  <pre><code>out = np.array(((0, 12, 13, 10), (1, 12, 13, 2)))
  </code></pre>
  
  <p>This is trivial to achieve with a for loop, but as I mentioned, it is very slow.</p>
  
  <pre><code>unique = np.unique(in[:, :3], axis=0)
  out = np.empty((0, 4))
  for i in unique:
      out = np.vstack((out, np.hstack((i[:], np.max(in[np.all(in[:, :3] == i[:], axis=1)][:, 3])))))
  </code></pre>
  
  <h2>What I've tried (1)</h2>
  
  <p>When I attempt to remove the for loop with indices by replacing each <code>i[:]</code> with <code>unique[np.arange(unique.shape[0])]</code> : </p>
  
  <pre><code>out = np.vstack((out, np.hstack((unique[np.arange(unique.shape[0])], np.max(in[np.all(in[:, :3].astype(int) == unique[np.arange(unique.shape[0])], axis=1)][:, 3])))))
  </code></pre>
  
  <p>Numpy complains about the input shape in conjunction with all:</p>
  
  <pre><code>Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
    File "&lt;__array_function__ internals&gt;", line 6, in all
    File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 2351, in all
      return _wrapreduction(a, np.logical_and, 'all', axis, None, out, keepdims=keepdims)
    File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 90, in _wrapreduction
      return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
  numpy.AxisError: axis 1 is out of bounds for array of dimension 0
  </code></pre>
  
  <h2>What I've tried (2)</h2>
  
  <p>Based on a suggestion from StackOverflow while typing this question (<a href="https://stackoverflow.com/questions/60906051/broadcasting-vectorizing-inner-and-outer-for-loops-in-python-numpy">Broadcasting/Vectorizing inner and outer for loops in python/NumPy</a>):</p>
  
  <pre><code>newout = np.vstack((newout, np.hstack((tempunique[:, None], np.max(inout[np.all(inout[:, :3].astype(int) == tempunique[:, None], axis=1)][:, 3])))))
  </code></pre>
  
  <p>I get an error complaining about the size mismatch between input and output:</p>
  
  <pre><code>Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  IndexError: boolean index did not match indexed array along dimension 0; dimension is 3 but corresponding boolean dimension is 2
  </code></pre>
  
  <h2>Restating the question</h2>
  
  <p>Is there a correct way to broadcast my indices to eliminate the for loop?</p>