I have over half a million pairs of true labels and predicted scores (the length of each 1d array varies and can be between 10,000-30,000 in length) that I need to calculate the AUC for. Right now, I have a for-loop that calls:

# Simple Example with two pairs of true/predicted values instead of 500,000
from sklearn import metrics
import numpy as np

pred = [None] * 2
pred[0] = np.array([3,2,1])
pred[1] = np.array([15,12,14,11,13])

true = [None] * 2
true[0] = np.array([1,0,0])
true[1] = np.array([1,1,1,0,0])

for i in range(2):
    fpr, tpr, thresholds = metrics.roc_curve(true[i], pred[i])
    print metrics.auc(fpr, tpr)

However, it takes about 1-1.5 hours to process the entire dataset and calculate the AUC for each true/prediction pair. Is there a faster/better way to do this?


Each of the 500k entries can have shape (1, 10k+). I understand that I could parallelize it but I'm stuck on a machine with only two processors and so my time can really only be effectively cut down to say, 30-45, minutes which is still too long. I've identified that the AUC calculation itself is slow and was hoping to find a faster AUC algorithm than what is available in sklearn. Or, at least, find a better way to vectorize the AUC calculation so that it can be broadcasted across multiple rows.

  • so to be clear, your pred and true arrays have length 500k entries, each of which is an np.array with shape (10k, 1)? if yes, you want to calculate the auc for each of pred[i|/true[i] combinations, that is each calculation is independent? – miraculixx Aug 26 '16 at 21:02
  • how much time does the AUC calculation for one pair take? How much time t per each pair would be allowable in order to get down to what you need (i.e. t * 0.5e6 < T_max? I'm assuming you have already looked at the cost/benefit ratio of spending time to optimize this v.s. the cost to buy a larger machine (or rent one e.g. at AWS for the time of the calculation). – miraculixx Aug 27 '16 at 6:52

Is there a faster/better way to do this?

Since the calculation of each true/pred pair is independent (if I understood your setup), you should be able to reduce total processing time by using multiprocessing, effectively parallelizing the calculations:

import multiprocessing as mp

def roc(v):
    """ calculate one pair, return (index, auc) """
    i, true, pred = v
    fpr, tpr, thresholds = metrics.roc_curve(true, pred, drop_intermediate=True)
    auc = metrics.auc(fpr, tpr)
    return i, auc

pool = mp.Pool(3) 
result = pool.map_async(roc, ((i, true[i], pred[i]) for i in range(2)))
print result.get()
[(0, 1.0), (1, 0.83333333333333326)]

Here Pool(3) creates a pool of 3 processes, .map_async maps all true/pred pairs and calls the roc function, passing one pair at a time. The index is sent along to map back results.

If the true/pred pairs are too large to serialize and send to the processes, you might need to write the data into some external data structure before calling roc, passing it just the reference i and read the data for each pair true[i]/pred[i] from within roc before processing.

A Pool automatically manages the scheduling of processes. To reduce the risk of a memory hog, you might need to pass the Pool(...., maxtasksperchild=1) parameter which would start a new process for each true/pred pair (choose any other number as you see fit).


I'm stuck on a machine with only two processors

naturally this is a limiting factor. However considering the availability of cloud computing resources at very reasonable cost that you only pay for the time you actually need it, you might want to consider alternatives in hardware before you spend eons of hours optimizing a calculation that can be so effectively parallelized. That's a luxury in its own right, really.



Probably not - sklearn already uses efficient numpy operations for its calculation of relevant parts:

# -- calculate tps, fps, thresholds
# sklearn.metrics.ranking:_binary_clf_curve()
distinct_value_indices = np.where(np.logical_not(isclose(
        np.diff(y_score), 0)))[0]
threshold_idxs = np.r_[distinct_value_indices, y_true.size - 1]
# accumulate the true positives with decreasing threshold
tps = (y_true * weight).cumsum()[threshold_idxs]
if sample_weight is not None:
    fps = weight.cumsum()[threshold_idxs] - tps
    fps = 1 + threshold_idxs - tps
return fps, tps, y_score[threshold_idxs]

# -- calculate auc
# sklearn.metrics.ranking:auc()
area = direction * np.trapz(y, x)

You might be able to optimize this by profiling these functions and removing operations that you can apply more efficiently beforehand. A quick profiling of your example input scaled to 5M rows reveals a few potential bottlenecks (marked >>>):

# your for ... loop wrapped in function roc()
%prun -s cumulative roc
722 function calls (718 primitive calls) in 5.005 seconds
Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    5.005    5.005 <string>:1(<module>)
        1    0.000    0.000    5.005    5.005 <ipython-input-51-27e30c04d997>:1(roc)
        2    0.050    0.025    5.004    2.502 ranking.py:417(roc_curve)
        2    0.694    0.347    4.954    2.477 ranking.py:256(_binary_clf_curve)
     >>>2    0.000    0.000    2.356    1.178 fromnumeric.py:823(argsort)
     >>>2    2.356    1.178    2.356    1.178 {method 'argsort' of 'numpy.ndarray' objects}
        6    0.062    0.010    0.961    0.160 arraysetops.py:96(unique)
     >>>6    0.750    0.125    0.750    0.125 {method 'sort' of 'numpy.ndarray' objects}
     >>>2    0.181    0.090    0.570    0.285 numeric.py:2281(isclose)
        2    0.244    0.122    0.386    0.193 numeric.py:2340(within_tol)
        2    0.214    0.107    0.214    0.107 {method 'cumsum' of 'numpy.ndarray' objects}

Your Answer

