title: GridSearchCV parameters don't improve classification
body: <p>I have a target dataset that I divide into 5 non-overlapping folds. 
  At each iteration (total iterations == 5) I use 1 fold (let's call it <code>fold_for_tuning</code>) to do parameter tuning, and I use 4 folds for testing. </p>
  
  <p>The reason for this is that I want to do domain adaptation, and before tuning, I fit the source data to the classifier, and I am tuning using small subset of target data). </p>
  
  <p>I call GridSearchCV and fit <code>fold_for_tuning</code>, and also I pass a bunch of parameters that I want to tune:</p>
  
  <pre><code>param_test1 = {
              'max_depth': [5, 7],
              'min_child_weight': [0.5, 1, 2],
              'gamma': [0.1, 1],
              'subsample': [0.6, 0.7],
              'colsample_bytree': [0.6, 0.7],
              'reg_alpha': [0.01, 0.1]
          }
  
  gsearch = GridSearchCV(estimator=classifierXGB,
                                  param_grid=param_test1,
                                  scoring='accuracy',
                                  n_jobs=4, iid=False, cv=2)
  
  gsearch.fit(fold_for_tuning_data, fold_for_tuning_labels)
  </code></pre>
  
  <p>After each iteration, I get <code>gsearch.best_params_</code>, and I set them into <code>classifierXGB</code>(because they should give better prediction, in my understanding).
  Then, when I call 
  <code>test_y_predicted = classifierXGB.predict(4_unseen_folds)</code>
  I get not improvement:</p>
  
  <pre><code>prediction before tuning:
  acc: 0.690658872245
  auc: 0.700764301397
  f1: 0.679211922203
  prediction after tuning:
  acc: 0.691382460414
  auc: 0.701595887248
  f1: 0.680132554837
  </code></pre>
  
  <p><strong>But if I call</strong> <code>gsearch.predict(4_unseen_folds)</code>
  <strong>I get MUCH BETTER performance:</strong></p>
  
  <pre><code>prediction grid search :
  acc: 0.933313032887
  auc: 0.930058979926
  f1: 0.920623414281
  </code></pre>
  
  <p><strong>So I am confused: what is happening inside grid search? Shouldn't it be optimizing only the parameters that I pass in <code>param_grid</code>? If so, then why setting the very same parameters in <code>classifierXGB</code> doesn't result into better performance?</strong></p>