title: cv parameter in sklearn.model_selection.GridSearchCV in regression problems: best_score_ decreases with larger cv values
body: <p>I have problems understanding the effect of the cv parameter in GridSearchCV. I'm still on Python 2.7 with sklearn 0.18</p>
  
  <p>When using GridSearchCV to tune some parameters for e.g. ridge regression, I would expect the scores to go up for a larger number of folds, as more and more data is included in training.
  However, I observed the opposite behaviour. Could anyone explain to me, whether this should be expected?</p>
  
  <p>Consider the following example</p>
  
  <pre><code>from __future__ import division, print_function
  from sklearn import linear_model
  from sklearn import preprocessing
  from sklearn import model_selection
  import numpy as np
  
  # Some example data
  X = np.ones((100, 4)) * np.arange(100).reshape((100, 1)) + (np.random.random((100, 4)) * 0.5)
  Y = np.ones((100, 1)) * np.arange(100).reshape((100, 1))
  scalerx = preprocessing.StandardScaler()
  X_ = scalerx.fit_transform(X)
  scalerx = preprocessing.StandardScaler()
  Y_ = scalerx.fit_transform(Y)
  
  for cv in np.arange(2, 100, 5):
      GS = model_selection.GridSearchCV(
          cv=cv, estimator=linear_model.Ridge(random_state=0),
          param_grid=[{'alpha': [10e-2, 10e-1, 10e0, 10e1, 10e2]}],
          )
      GS.fit(X_, Y_)
      print(cv, GS.best_score_)
  </code></pre>
  
  <p>this produces the following output</p>
  
  <pre><code>2 0.999967723551
  7 0.999700831429
  12 0.999157506389
  17 0.998278596601
  22 0.997107931494
  27 0.995490243014
  32 0.993315835556
  37 0.990634277512
  42 0.986174197011
  47 0.982697293408
  52 0.941176694311
  57 0.844045003327
  62 0.744867073869
  67 0.646916948176
  72 0.548521372859
  77 0.451907425236
  82 0.352978995707
  87 0.256264849417
  92 0.1575027865
  97 0.058902807202
  </code></pre>
  
  <p>The best score decreases with smaller test samples (larger number of folds).</p>