title: Hot to use Kappa Cohen's coefficient with LOOCV in GridSearchCV?
body: <p>I want to optimize some models with GridSearchCV, using Leave One Out CV method and Kappa Cohen's coefficient as a scoring. So, I want to collect all predicted values for each combination of parameters, compute Kappa Cohen's coefficient for every one of them, and show the best combination based on Kappa Cohen's coefficient.</p>
  <p>I have something like this:</p>
  <pre><code>param_grid = dict(clf__class_weight=['balanced',None],
                    clf__probability=[True],
                    clf__C=[0.1,1,20,50,70,100,500],
                    clf__kernel=['rbf'],
                    clf__gamma=[0.01,1,5,10,'scale','auto'],
                    clf__decision_function_shape=['ovo'],
                    )
              
  pipe = Pipeline([('scale', StandardScaler()), ('clf', SVC())])
  loo=LeaveOneOut()       
  kappa=make_scorer(cohen_kappa_score)
         
  grid = GridSearchCV (pipe, param_grid=param_grid,cv=loo, n_jobs=-1, verbose=2, scoring = kappa)
  grid.fit(X, y)
  print(grid.best_params_)
  </code></pre>
  <p>Of course, this doesn't work as it tries to calculate kappa for every single prediction which is not possible, so I have a result like this:</p>
  <pre><code>...
  /usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:663: RuntimeWarning: invalid value encountered in true_divide
    k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)
  /usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:663: RuntimeWarning: invalid value encountered in true_divide
    k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)
  /usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:663: RuntimeWarning: invalid value encountered in true_divide
    k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)
  /usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:663: RuntimeWarning: invalid value encountered in true_divide
    k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)
  /usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:663: RuntimeWarning: invalid value encountered in true_divide
    k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)
  /usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:663: RuntimeWarning: invalid value encountered in true_divide
    k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)
  ...
  </code></pre>
  <p>Can anyone help me with this issue?</p>