title: Apply permutation test after nested cross validation
body: <p>I ran a <strong>nested</strong>-cross validation procedure on my data set. Now I would like to assess the significance of the average performance of all outer models. Is there any package that allows me to do this? I came across <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.permutation_test_score.html#sklearn-model-selection-permutation-test-score" rel="nofollow noreferrer"><code>sklearn.model_selection.permutation_test_score</code></a> but it seems that this function only takes one classifier as input and I also don't know how I would have to implement this function for my problem. I am looking for a way to assess the significance of the average classification performance of all surrogate models that are returned by <code>cross_validate</code>.</p>
  <p>Example:</p>
  <pre><code>from sklearn.datasets import load_iris
  from sklearn.model_selection import KFold
  from sklearn.model_selection import GridSearchCV
  from sklearn.model_selection import cross_validate
  from sklearn.svm import SVC
  import numpy as np
  import matplotlib.pyplot as plt
  
  # get iris data set
  iris = load_iris()
  X = iris.data
  y = iris.target
  
  # set random seed
  random_seed = 42
  
  # use support vector machine
  svr = SVC()
  
  # optimize C-Parameter
  p_grid = {&quot;C&quot;: [1, 10, 100]}
  
  # Define cross-validation strategies
  inner_cv = KFold(shuffle=True,random_state=random_seed)
  outer_cv = KFold(shuffle=True,random_state=random_seed)
  
  # define inner cv
  grid = GridSearchCV(svr,param_grid=p_grid,cv=inner_cv)
  
  # define outer cv
  scores = cross_validate(grid,X,y,cv=outer_cv,return_train_score=True,return_estimator=True)
  
  # get average performance across all surrogate models
  avg_test_score = np.mean(scores['test_score'])
  
  # plot results
  x = np.arange(len(scores['estimator']))
  width = 0.35  # the width of the bars
  fig, ax = plt.subplots()
  train_scores = ax.bar(x - width/2,height=scores['train_score'],width=width,label='Train Score')
  test_scores = ax.bar(x + width/2,height=scores['test_score'],width=width,label='Test Score')
  ax.axhline(y=avg_test_score, xmin=0.0, xmax=1.0, color='orange')
  ax.set_ylabel('Score')
  ax.set_xticks(x)
  ax.set_xticklabels(['Outer Fold 1','Outer Fold 2','Outer Fold 3','Outer Fold 4','Outer Fold 5'])
  ax.legend(bbox_to_anchor=(1,0.5),loc='center left')
  </code></pre>
  <p>In this example the average performance across all five surrogate models is <code>avg_test_score≈0.97</code>. Now is there a way to assess the significance of this average value by repeating the whole procedure <em>n</em> times, shuffling the labels, and obtain the frequency of this value or a higher one?</p>
  <p>times</p>
  <p><a href="https://i.stack.imgur.com/jvOLZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jvOLZ.png" alt="enter image description here" /></a></p>