title: sklearn auc ValueError: Only one class present in y_true
body: <p>I searched Google, and saw a couple of StackOverflow posts about this error. They are not my cases. </p>
  
  <p>I use keras to train a simple neural network and make some predictions on the splitted test dataset. But when use <code>roc_auc_score</code> to calculate AUC, I got the following error: </p>
  
  <p><code>"ValueError: Only one class present in y_true. ROC AUC score is not defined in that case."</code>. </p>
  
  <p>I inspect the target label distribution, and they are highly imbalanced. Some labels(in the total 29 labels) have only 1 instance. So it's likely they will have no positive label instance in the test label. So the sklearn's <code>roc_auc_score</code> function reported the only one class problem. That's reasonable.</p>
  
  <p>But I'm curious, as when I use sklearn's <code>cross_val_score</code> function, it can handle the AUC calculation without error.</p>
  
  <pre><code>my_metric = 'roc_auc' 
  scores = cross_validation.cross_val_score(myestimator, data,
                                     labels, cv=5,scoring=my_metric)
  </code></pre>
  
  <p>I wonder what happened in the <code>cross_val_score</code>, is it because the <code>cross_val_score</code> use a stratified cross-validation data split?</p>
  
  <hr>
  
  <p><strong>UPDATE</strong><br>
  I continued to make some digging, but still can't find the difference behind.I see that cross_val_score call <code>check_scoring(estimator, scoring=None, allow_none=False)</code> to return a scorer, and the <code>check_scoring</code> will call <code>get_scorer(scoring)</code> which will return <code>scorer=SCORERS[scoring]</code></p>
  
  <p>And the <code>SCORERS['roc_auc']</code> is <code>roc_auc_scorer</code>；<br>
  the <code>roc_auc_scorer</code> is made by </p>
  
  <pre><code>roc_auc_scorer = make_scorer(roc_auc_score, greater_is_better=True,
                                   needs_threshold=True)
  </code></pre>
  
  <p>So, it's still using the roc_auc_score function. I don't get why cross_val_score behave differently with directly calling roc_auc_score.</p>