title: Nested cross validation with StratifiedShuffleSplit in sklearn
body: <p>I am working on a binary classification problem and would like to perform the nested cross validation to assess the classification error. The reason why I'm doing the nested CV is due to the small sample size (N_0 = 20, N_1 = 10), where N_0, N_1 are the numbers of instances in 0 and 1 classes respectively.</p>
  
  <p>My code is quite simple:</p>
  
  <pre><code>&gt;&gt; pipe_logistic = Pipeline([('scl', StandardScaler()),('clf', LogisticRegression(penalty='l1'))])
  &gt;&gt; parameters = {'clf__C': logspace(-4,1,50)}
  &gt;&gt; grid_search = GridSearchCV(estimator=pipe_logistic, param_grid=parameters, verbose=1, scoring='f1', cv=5)
  &gt;&gt; cross_val_score(grid_search, X, y, cv=5)
  </code></pre>
  
  <p>So far, so good. If I want to change the CV scheme (from random splitting to StratifiedShuffleSplit in both, outer and inner CV loops, I face the problem: how can I pass the class vector y, as it is required by the StratifiedShuffleSplit function?</p>
  
  <p>Naively:</p>
  
  <pre><code>&gt;&gt; grid_search = GridSearchCV(estimator=pipe_logistic, param_grid=parameters, verbose=1, scoring='f1', cv=StratifiedShuffleSplit(y_inner_loop, 5, test_size=0.5, random_state=0))
  &gt;&gt; cross_val_score(grid_search, X, y, cv=StratifiedShuffleSplit(y, 5, test_size=0.5, random_state=0))
  </code></pre>
  
  <p>So, the problem is how to specify the <strong>y_inner_loop</strong> ?</p>
  
  <p>** My data set is slightly imbalanced (20/10) and I would like to keep this splitting ratio for training and assessing the model.</p>