title: When using multiple classifiers - How to measure the ensemble's performance? [SciKit Learn]
body: <p>I have a classification problem (predicting whether a sequence belongs to a class or not), for which I decided to use multiple classification methods, in order to help filter out the false positives. </p>
  
  <p>(The problem is in bioinformatics - classifying protein sequences as being Neuropeptide precursors sequences. <a href="http://bioinformatics.oxfordjournals.org/content/early/2013/12/13/bioinformatics.btt725.abstract" rel="noreferrer">Here's the original article</a> if anyone's interested, <a href="http://bioinformatics.oxfordjournals.org/content/early/2013/12/13/bioinformatics.btt725.abstract" rel="noreferrer">and the code used to generate features and to train a single predictor</a>) .</p>
  
  <p>Now, the classifiers have roughly similar performance metrics (83-94% accuracy/precision/etc'  on the training set for 10-fold CV),  so my 'naive' approach was to simply use multiple classifiers (Random Forests, ExtraTrees, SVM (Linear kernel), SVM (RBF kernel) and GRB) , and to use a simple majority vote. </p>
  
  <p>MY question is: 
  How can I get the performance metrics for the different classifiers and/or their votes predictions? 
  That is, I want to see if using the multiple classifiers improves my performance at all, or which combination of them does.</p>
  
  <p>My intuition is maybe to use the ROC score, but I don't know how to "combine" the results and to get it from a combination of classifiers. (That is, to see what the ROC curve is just for each classifier alone [already known], then to see the ROC curve or AUC for the training data using combinations of classifiers). </p>
  
  <p>(I currently filter the predictions using "predict probabilities" with the Random Forests and ExtraTrees methods, then I filter arbitrarily for results with a predicted score below '0.85'. An additional layer of filtering is "how many classifiers agree on this protein's positive classification"). </p>
  
  <p>Thank you very much!!</p>
  
  <p>(The <a href="http://neuropid.cs.huji.ac.il/" rel="noreferrer">website</a> implementation, where we're using the multiple classifiers - <a href="http://neuropid.cs.huji.ac.il/" rel="noreferrer">http://neuropid.cs.huji.ac.il/</a> )</p>
  
  <p>The whole shebang is implemented using SciKit learn and python. Citations and all!)</p>