title: Why does sklearn.grid_search.GridSearchCV return random results on every execution?
body: <p>I was trying to get the optimum features for a decision tree classifier over the Iris dataset using <code>sklearn.grid_search.GridSearchCV</code>. I used StratifiedKFold (<code>sklearn.cross_validation.StratifiedKFold</code>) for cross-validation, since my data was biased. But on every execution of <code>GridSearchCV</code>, it returned a different set of parameters.<br>
  Shouldn't it return the same set of optimum parameters given that the data and the cross-validation was same every single time?</p>
  
  <p>Source code follows:</p>
  
  <pre><code>from sklearn.tree import DecisionTreeClassifier
  from sklearn.grid_search import GridSearchCV
  
  decision_tree_classifier = DecisionTreeClassifier()
  
  parameter_grid = {'max_depth': [1, 2, 3, 4, 5],
                    'max_features': [1, 2, 3, 4]}
  
  cross_validation = StratifiedKFold(all_classes, n_folds=10)
  
  grid_search = GridSearchCV(decision_tree_classifier, param_grid = parameter_grid,
                            cv = cross_validation)
  
  grid_search.fit(all_inputs, all_classes)
  
  print "Best Score: {}".format(grid_search.best_score_)
  print "Best params: {}".format(grid_search.best_params_)
  </code></pre>
  
  <p>Outputs:</p>
  
  <pre><code>Best Score: 0.959731543624
  Best params: {'max_features': 2, 'max_depth': 2}
  
  Best Score: 0.973154362416
  Best params: {'max_features': 3, 'max_depth': 5}
  
  Best Score: 0.973154362416
  Best params: {'max_features': 2, 'max_depth': 5}
  
  Best Score: 0.959731543624
  Best params: {'max_features': 3, 'max_depth': 3}
  </code></pre>
  
  <p>This is an excerpt from an Ipython notebook which I made recently, with reference to <strong>Randal S Olson's</strong> notebook, which can be found <a href="https://github.com/darthv115/Machine-Learning-and-Data-Science-Projects/blob/master/Iris_Classification/Iris%20Classification.ipynb" rel="nofollow noreferrer">here</a>.</p>
  
  <p><strong>Edit</strong>:
  Its not the <code>random_state</code> parameter of <code>StratifiedKFold</code> which results in varied results but rather the <code>random_state</code> parameter of <code>DecisionTreeClassifer</code> which randomly initializes the tree and gives varied results (refer <a href="https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier" rel="nofollow noreferrer">documentation</a>). As for <code>StratifiedKFold</code>, as long as the <code>shuffle</code> parameter is set to <code>False</code> (default), it generates the same training-test split (refer documentation).</p>