Take the 2-minute tour ×

I need to filter an array to remove the elements that are lower than a certain threshold. My current code is like this:

threshold = 5
a = numpy.array(range(10)) # testing data
b = numpy.array(filter(lambda x: x >= threshold, a))

The problem is that this creates a temporary list, using a filter with a lambda function (slow).

As this is a quite simple operation, maybe there is a numpy function that does it in an efficient way, but I've been unable to find it.

I've thought that another way to achieve this could be sorting the array, finding the index of the threshold and returning a slice from that index onwards, but even if this would be faster for small inputs (and it won't be noticeable anyway), its definitively asymptotically less efficient as the input size grows.

Any ideas? Thanks!

Update: I took some measurements too, and the sorting+slicing was still twice as fast than the pure python filter when the input was 100.000.000 entries.

In [321]: r = numpy.random.uniform(0, 1, 100000000)

In [322]: %timeit test1(r) # filter
1 loops, best of 3: 21.3 s per loop

In [323]: %timeit test2(r) # sort and slice
1 loops, best of 3: 11.1 s per loop

In [324]: %timeit test3(r) # boolean indexing
1 loops, best of 3: 1.26 s per loop
share|improve this question
oh this %timeit, is it ipython tool? looks neat! –  yosukesabai Nov 3 '11 at 15:26
yeah, it's quite nice :-) it even calculates automatically how many iterations it should perform to average the measurements if the code takes very little time to execute –  fortran Nov 3 '11 at 15:32
@yosukesabai - IPython's %timeit uses the builtin timeit module. Have a look at it, as well. docs.python.org/library/timeit.html –  Joe Kington Nov 3 '11 at 16:04

1 Answer 1

up vote 39 down vote accepted

b = a[a>threshold] this should do

I tested as follows:

import numpy as np, datetime
# array of zeros and ones interleaved
lrg = np.arange(2).reshape((2,-1)).repeat(1000000,-1).flatten()

t0 = datetime.datetime.now()
flt = lrg[lrg==0]
print datetime.datetime.now() - t0

t0 = datetime.datetime.now()
flt = np.array(filter(lambda x:x==0, lrg))
print datetime.datetime.now() - t0

I got

$ python test.py


share|improve this answer
very clever way of using indexing! thanks! I'm adding a link to the reference... –  fortran Nov 3 '11 at 12:16
added test result, not just what I think it should do. :p –  yosukesabai Nov 3 '11 at 12:38
This kind of indexing does not maintain the size of the array, how is it possible to keep the same number of elements and zeroing the subthreshold values? –  linello Jul 24 '13 at 10:00
@linello, a[a<=threshold] = 0 is going to mask out the part that do not exceed the threshold –  yosukesabai Aug 17 '13 at 19:25
I ran in to the issue of filtering based on two criteria. Here is the solution: stackoverflow.com/a/3248599/1373468 –  Robin Newhouse Jan 12 at 3:29

Your Answer


