title: Why roc_auc produces weird results in sklearn?
body: <p>I have a binary classification problem where I use the following code to get my <code>weighted avarege precision</code>, <code>weighted avarege recall</code>, <code>weighted avarege f-measure</code> and <code>roc_auc</code>.</p>
  <pre><code>df = pd.read_csv(input_path+input_file)
  
  X = df[features]
  y = df[[&quot;gold_standard&quot;]]
  
  clf = RandomForestClassifier(random_state = 42, class_weight=&quot;balanced&quot;)
  k_fold = StratifiedKFold(n_splits=10, shuffle=True, random_state=0)
  scores = cross_validate(clf, X, y, cv=k_fold, scoring = ('accuracy', 'precision_weighted', 'recall_weighted', 'f1_weighted', 'roc_auc'))
  
  print(&quot;accuracy&quot;)
  print(np.mean(scores['test_accuracy'].tolist()))
  print(&quot;precision_weighted&quot;)
  print(np.mean(scores['test_precision_weighted'].tolist()))
  print(&quot;recall_weighted&quot;)
  print(np.mean(scores['test_recall_weighted'].tolist()))
  print(&quot;f1_weighted&quot;)
  print(np.mean(scores['test_f1_weighted'].tolist()))
  print(&quot;roc_auc&quot;)
  print(np.mean(scores['test_roc_auc'].tolist()))
   
  </code></pre>
  <p>I got the following results for the same dataset with 2 different feature settings.</p>
  <pre><code>Feature setting 1 ('accuracy', 'precision_weighted', 'recall_weighted', 'f1_weighted', 'roc_auc'):  
  0.6920, 0.6888, 0.6920, 0.6752, 0.7120
  
  Feature setting 2 ('accuracy', 'precision_weighted', 'recall_weighted', 'f1_weighted', 'roc_auc'):
  0.6806  0.6754  0.6806  0.6643  0.7233
  </code></pre>
  <p>So, we can see that in <code>feature setting 1</code> we get good results for 'accuracy', 'precision_weighted', 'recall_weighted', 'f1_weighted' compared to <code>feature setting 2</code>.</p>
  <p>However, when it comes to 'roc_auc' <code>feature setting 2</code> is better than <code>feature setting 1</code>. I found this weird becuase every other metric was better with <code>feature setting 1</code>.</p>
  <p>On one hand, I suspect that this happens since I am using <code>weighted</code> scores for <code>precision, recall and f-measure</code> and not with <code>roc_auc</code>. Is it possible to do <code>weighted roc_auc</code> for binary classification in sklearn?</p>
  <p>What is the real problem for this weird <code>roc_auc</code> results?</p>