title: How to use a fixed validation set (not K-fold cross validation) in Scikit-learn for a decision tree classifier/random forest classifier?
body: <p>I am new to machine learning and data science. Sorry, if it is a very stupid question.   </p>
  
  <p>I see there is an inbuilt function for cross-validation but not for a fixed validation set. I have a dataset with 50,000 samples labeled with years from 1990 to 2010. I need to train different classifiers on 1990-2008 samples, then validate on 2009 samples, and test on 2010 samples. </p>
  
  <p>EDIT:
  After @Quan Tran's answer, I tried this. This is how it should be?</p>
  
  <pre><code># Fit a decision tree
  estimator1 = DecisionTreeClassifier( max_depth = 9, max_leaf_nodes=9)
  estimator1.fit(X_train, y_train)
  print estimator1
  
  
  # validate using validation set
  acc = np.zeros((20,20))  # store accuracy 
  for i in range(20):
       for j in range(20):
           estimator1 = DecisionTreeClassifier(max_depth = i+1, max_leaf_nodes=j+2)
           estimator1.fit(X_valid, y_valid)
           y_pred = estimator1.predict(X_valid)
           acc[i,j] = accuracy_score(y_valid, y_pred)
  
  best_mod = np.where(acc == acc.max())
  print best_mod
  print acc[best_mod]
  
  
  
   # Predict target values
  estimator1 = DecisionTreeClassifier(max_depth = int(best_mod[0]) + 1, max_leaf_nodes= int(best_mod[1]) + 2)
  estimator1.fit(X_valid, y_valid)
  y_pred = estimator1.predict(X_test)
  confusion = metrics.confusion_matrix(y_test, y_pred)
  
  TP = confusion[1, 1]
  TN = confusion[0, 0]
  FP = confusion[0, 1]
  FN = confusion[1, 0]
  
  
  # Classification Accuracy
  print "======= ACCURACY ========"
  print((TP + TN) / float(TP + TN + FP + FN))
  print accuracy_score(y_valid, y_pred)
  # store the predicted probabilities for class 
  y_pred_prob = estimator1.predict_proba(X_test)[:, 1]
  
  
  # plot a ROC curve for y_test and y_pred_prob
  fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)
  plt.plot(fpr, tpr)
  plt.xlim([0.0, 1.0])
  plt.ylim([0.0, 1.0])
  plt.title('ROC curve for DecisionTreeClassifier')
  plt.xlabel('False Positive Rate (1 - Specificity)')
  plt.ylabel('True Positive Rate (Sensitivity)')
  plt.grid(True)
  
  print("======= AUC ========")
  print(metrics.roc_auc_score(y_test, y_pred_prob))
  </code></pre>
  
  <p>I get this answer, which is <strong>not the best accuracy</strong>.</p>
  
  <pre><code>DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=9,
          max_features=None, max_leaf_nodes=9, min_samples_leaf=1,
          min_samples_split=2, min_weight_fraction_leaf=0.0,
          presort=False, random_state=None, splitter='best')
  (array([5]), array([19]))
  [ 0.8489011]
  ======= ACCURACY ========
  0.574175824176
  0.538461538462
  ======= AUC ========
  0.547632099893
  </code></pre>