### Decision Trees and Random Forests

#### Packages


```python
import sys
import sklearn
import graphviz
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt

%matplotlib inline

from sklearn import tree
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, GridSearchCV

print('The Python version is {}.\n'.format(sys.version))
print('The Numpy version is {}.\n'.format(np.__version__))
print('The Pandas version is {}.\n'.format(pd.__version__))
print('The Matplotlib version is {}.\n'.format(mpl.__version__))
print('The Scikit-Learn version is {}.\n'.format(sklearn.__version__))
```

    The Python version is 3.8.5 (default, Jul 28 2020, 12:59:40) 
    [GCC 9.3.0].
    
    The Numpy version is 1.19.1.
    
    The Pandas version is 1.1.0.
    
    The Matplotlib version is 3.3.3.
    
    The Scikit-Learn version is 0.24.0.
    


### Decision Trees

Decision trees and the machine learning models that are based on them, in particular random forests and gradient boosted trees, are fundamentally different types of models than generalized linear models, such as logistic regression. GLMs are rooted in the theories of classical statistics, which have a long history. The mathematics behind linear regression were originally developed at the beginning of the 19th century, by Legendre and Gauss. Because of this, the normal distribution is also called the Gaussian.

Decision trees have an intuitive structure and mimic the way that logical decisions might be made by humans. Therefore, they are a highly interpretable type of mathematical model, which can be a particularly desirable property depending on the audience. 

Decision trees are able to capture non-linear effects of the features, as opposed to a linear relationship.

In general, it should be clear that in a tree with $n$ levels, where every node before the final level is split,
there will be $2n$ leaf nodes. This is important to bear in mind as the maximum depth is one of the hyperparameters that you can set for a decision tree classifier in scikit-learn

#### Growing a tree for the study case

Let us load the data and select the desired columns:


```python
df = pd.read_csv('data/default_of_credit_card_clients_cleaned.csv')

features_response = df.columns.tolist()
to_remove = ['id', 'sex'] + ['pay_{}'.format(i) for i in range(2,7)] + features_response[-6:]

features_response = [feature for feature in features_response if feature not in to_remove]
```

Let us split the dataset into train and test sets:


```python
X_train, X_test, y_train, y_test = train_test_split(df[features_response].values,
                                                    df.default_payment_next_month.values,
                                                    test_size=0.2, random_state=24)
```

Now, let us instantiate the decision tree model:


```python
dt = tree.DecisionTreeClassifier(max_depth=2)

dt.fit(X_train, y_train)
```




    DecisionTreeClassifier(max_depth=2)



Using the graphviz library, let display a graphical representation of the tree:


```python
dot_data = tree.export_graphviz(dt, out_file=None, filled=True,
                                rounded=True, feature_names=features_response,
                                proportion=True, class_names=['Not Defaulted', 'Defaulted'])
```


```python
graph = graphviz.Source(dot_data)
graph
```




    

    



The training decision tree worked by starting with all the training samples in the initial node at the top of the tree, and then splited these into two groups based on a **threshold** in one of the features. The cut point was represented by a Boolean condition in the top **pay_1 <= 1.5** node. In the top node, there were all the samples ("samples = 100.0%"). Following the first split, 89.5% of the samples got sorted into the node on the left, while the remaining 10.5% went into the node on the right.

Let us confirm the proportions stated by the model:


```python
(features_response.index('pay_1'),
 X_train.shape)
```




    (4, (21331, 17))




```python
sum(X_train[:, features_response.index('pay_1')] <= 1.5)/X_train.shape[0]
```




    0.8946134733486475



Let us confirm the values line (the class proportion for the first node):


```python
np.mean(y_train)
```




    0.223102526838873



The class fractions were also colored: those with a higher proportion of the negative class than the positive class were orange, with darker orange signifying higher proportions, while those with a higher proportion of the positive class had a similar scheme using a blue color.

The line starting with "class" indicates how the decision tree made predictions from a given node, if that node were a leaf node. 

Decision trees for classification make predictions by determining which leaf node a sample will be sorted in to, given the values of the features, and then predicting the class of the majority of the training samples in that leaf node.

#### Node Impurity

The goal of training decision trees is to make splits so that the next two nodes after the split have a higher purity, or, in other words, are closer to containing either only positive or only negative samples.

In practice, decision trees are actually trained using the inverse of purity, or node impurity. This is some measure of how far the node is from having 100% of the training samples belonging to one class and is analogous to the concept of a cost function, which signifies how far a given solution is from a theoretical perfect solution. For a given node $m$, the misclassification rate is simply the proportion of the less common class in that node, since all these samples will be misclassified when the majority class in that node is taken as the prediction.

The fraction of the positive class for a node $m$ and a class $k$ is given by:

$p_{mk+1} = 1 - p_{mk}$

Let us look at the missclassification distribution:


```python
p_m0 = np.linspace(0.01, 0.99, 101)
p_m1 = 1 - p_m0

misclassification_rate = np.minimum(p_m0, p_m1)

plt.plot(p_m0, misclassification_rate, label='Misclassification rate')
plt.xlabel('$p_{m0}$')
plt.legend()
```

Every time a node is split when growing a decision tree, two new nodes are created. Since the prediction from either of these new nodes is simply the majority class, an important goal will be to reduce the misclassification rate. Therefore, we will want to find a feature, from all the possible features, and a value of this feature at which to make a cut point, so that the misclassification rate in the two new nodes will be as low as possible when averaging over all the classes.

The algorithm then considers each possible threshold for every candidate feature and chooses the one that results in the lowest impurity, calculated as the average impurity across the two possible new nodes, weighted by the number of samples in each node. This process is repeated until a stopping criterion of the tree, such as max_depth.

While the misclassification rate is an intuitive measure of impurity, it happens that there are better measures that can be used to find splits during the model training process. The two options that are available in scikit-learn for the impurity calculation, which you can specify with the criterion keyword argument, are the Gini impurity and the cross-entropy options.

#### Gini

The Gini impurity is calculated for a node $m$ using the following formula:


$$
\begin{equation}
\begin{split}
Gini = \sum_{k} p_{mk} (1 - p_{mk})
\end{split}
\end{equation}
$$

Here, the summation is taken over all classes. In the case of a binary classification problem, there are only two classes, and we can write
this programmatically as follows

#### Cross Entropy

Cross entropy is calculated using this formula:


$$
\begin{equation}
\begin{split}
\text{cross entropy} = - \sum_{k} p_{mk} log(p_{mk})
\end{split}
\end{equation}
$$

Let us see these two distributions:


```python
def gini_impurity(**kwargs):
    """
    Returns the gini impurity for a given array (or arrays)

    Arguments:
        **kwargs (array): Numpy Arrays
    Returns:
        Numpy Array
    """
    return sum([i*(1-i) for i in kwargs.values()])

def cross_entropy_impurity(**kwargs):
    """
    Returns the cross entropy a given array (or arrays)

    Arguments:
        **kwargs (array): Numpy Arrays
    Returns:
        Numpy Array
    """
    return - sum([i*np.log(i) for i in kwargs.values()])
```


```python
gini = gini_impurity(x1=p_m0, x2=p_m1)
cross_entropy = cross_entropy_impurity(x1=p_m0, x2=p_m1)
```


```python
plt.plot(p_m0, misclassification_rate, label='Misclassification rate')
plt.plot(p_m0, gini, label='Gini impurity')
plt.plot(p_m0, cross_entropy, label='Cross entropy')
plt.xlabel('$p_{m0}$')
plt.legend()
```

Like the misclassification rate, both the Gini impurity and the cross entropy are highest when the class fractions are equal at 0.5, and they decrease as the node becomes purer – in other words, when they contain a higher proportion of just one of the classes. However, the Gini impurity is somewhat steeper than the misclassification rate in certain regions of the class fraction, which enables it to more effectively find the best split. Cross-entropy looks yet steeper.

It is necessary to consider both impurity metrics in a cross-validation search for hyperparameters in order to
determine the appropriate one. Note that in scikit-learn, Gini impurity can be specified with the criterion argument using the 'gini' string, while cross entropy is just referred to as 'entropy'.

#### Greedy Algorithm

There is no guarantee that a decision tree trained by the process described previously will be the best possible decision tree for finding leaf nodes with the lowest impurity. This is because the algorithm used to train decision trees is what is called a greedy algorithm.

The reason why the algorithm is used this way is because it takes substantially longer to consider all possible splits in a way that enables finding the truly optimal tree. Despite this shortcoming of the decision tree training process, there are methods that you can use to reduce the possible harmful effects of the greedy algorithm. Instead of searching for the best split at each node, the splitter keyword argument to the decision tree class can be specified as random in order to choose a random feature to make a split on.

### Decision Trees Pros

#### No need to scale data:

It is not an algorithms based on gradient descent, neither use L1 or L2 regularization to penalize coefficients. The algorithms that use the previous estimations need to scale the features to balance their weights during the training phase.

#### Non-linear relationships and interactions

Because each successive split in a decision tree is performed on a subset of the samples resulting from previous split(s), decision trees can describe complex non-linear relationships of a single feature, as well as interactions between features.

#### Predicted Probabilities

A decision tree makes predictions based on the majority of class of the leaf nodes. The probability is based on the proportion of the majority class in the leaf node. If the leaf node consisted 75% of the positive class, for
example, the prediction for that node will be the positive class and the predicted probability will be 0.75. The predicted probabilities from decision trees are not considered to be as statistically rigorous as those from generalized linear models, but they are still commonly used to measure the performance of models by methods that depend on varying the threshold for classification, such as the ROC curve or the precision-recall curve.

### Finding the best hyperparameters for a D3 model

Let us see the number of samples in the training study case set:


```python
X_train.shape
```




    (21331, 17)



With 21,331 training samples and 4-fold cross-validation, there will be three-fourth of the samples, or about 16,000 samples, in each training fold.

A theoretical limitation is that at least one sample in each leaf is needed. The depth of the tree relates
to the number of leaves, such that a split at every node n, implies 2n leaf nodes. Therefore, a tree with $L$ leaf nodes has a depth of approximately $log_{2}(L)$. In the limiting case, if this tree grow deep enough, so that every leaf node has one training sample for a given fold, the depth will be $log_{2}(16,000) \approx 14$. So, 14 is the theoretical limit to the depth of a tree that could grow in this case.

The depth of 14 would certainly imply overfitting, so this depth is not desired. Let us define different values for the max depth parameter:


```python
params = {'max_depth': [1, 2, 4, 6, 8, 10, 12]}
```

And now instantiate the GridSearch object:


```python
cv = GridSearchCV(dt, param_grid=params, scoring='roc_auc',
                  n_jobs=None, refit=True, cv=4, verbose=1,
                  pre_dispatch=None, error_score=np.nan, return_train_score=True)
```

The other options were the ROC AUC metric (scoring='roc_auc'), the 4-fold cross-validation (cv=4), and to calculate training scores (return_train_score=True) to assess the bias-variance trade-off.

Let us fit the data:


```python
cv.fit(X_train, y_train)
```

    Fitting 4 folds for each of 7 candidates, totalling 28 fits





    GridSearchCV(cv=4, estimator=DecisionTreeClassifier(max_depth=2),
                 param_grid={'max_depth': [1, 2, 4, 6, 8, 10, 12]},
                 pre_dispatch=None, return_train_score=True, scoring='roc_auc',
                 verbose=1)



And save the dictionary of results in a DataFrame object:


```python
cv_results_df = pd.DataFrame(cv.cv_results_)
cv_results_df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>mean_fit_time</th>
      <th>std_fit_time</th>
      <th>mean_score_time</th>
      <th>std_score_time</th>
      <th>param_max_depth</th>
      <th>params</th>
      <th>split0_test_score</th>
      <th>split1_test_score</th>
      <th>split2_test_score</th>
      <th>split3_test_score</th>
      <th>mean_test_score</th>
      <th>std_test_score</th>
      <th>rank_test_score</th>
      <th>split0_train_score</th>
      <th>split1_train_score</th>
      <th>split2_train_score</th>
      <th>split3_train_score</th>
      <th>mean_train_score</th>
      <th>std_train_score</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.032530</td>
      <td>0.007054</td>
      <td>0.004060</td>
      <td>0.000895</td>
      <td>1</td>
      <td>{'max_depth': 1}</td>
      <td>0.639514</td>
      <td>0.643398</td>
      <td>0.651891</td>
      <td>0.650753</td>
      <td>0.646389</td>
      <td>0.005136</td>
      <td>7</td>
      <td>0.648680</td>
      <td>0.647384</td>
      <td>0.644553</td>
      <td>0.644934</td>
      <td>0.646388</td>
      <td>0.001712</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.051850</td>
      <td>0.001237</td>
      <td>0.003802</td>
      <td>0.000087</td>
      <td>2</td>
      <td>{'max_depth': 2}</td>
      <td>0.695134</td>
      <td>0.699022</td>
      <td>0.713376</td>
      <td>0.699510</td>
      <td>0.701761</td>
      <td>0.006917</td>
      <td>5</td>
      <td>0.704034</td>
      <td>0.702700</td>
      <td>0.698113</td>
      <td>0.702535</td>
      <td>0.701845</td>
      <td>0.002232</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.101508</td>
      <td>0.009389</td>
      <td>0.003910</td>
      <td>0.000102</td>
      <td>4</td>
      <td>{'max_depth': 4}</td>
      <td>0.732720</td>
      <td>0.741078</td>
      <td>0.746946</td>
      <td>0.743731</td>
      <td>0.741119</td>
      <td>0.005276</td>
      <td>2</td>
      <td>0.756882</td>
      <td>0.752421</td>
      <td>0.749368</td>
      <td>0.753055</td>
      <td>0.752932</td>
      <td>0.002673</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.143802</td>
      <td>0.002191</td>
      <td>0.004772</td>
      <td>0.000942</td>
      <td>6</td>
      <td>{'max_depth': 6}</td>
      <td>0.743836</td>
      <td>0.746910</td>
      <td>0.751932</td>
      <td>0.740517</td>
      <td>0.745798</td>
      <td>0.004201</td>
      <td>1</td>
      <td>0.782202</td>
      <td>0.780044</td>
      <td>0.775228</td>
      <td>0.774776</td>
      <td>0.778062</td>
      <td>0.003158</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.188230</td>
      <td>0.005357</td>
      <td>0.004744</td>
      <td>0.000730</td>
      <td>8</td>
      <td>{'max_depth': 8}</td>
      <td>0.727023</td>
      <td>0.732562</td>
      <td>0.750618</td>
      <td>0.731040</td>
      <td>0.735311</td>
      <td>0.009066</td>
      <td>3</td>
      <td>0.812061</td>
      <td>0.808245</td>
      <td>0.803415</td>
      <td>0.802370</td>
      <td>0.806523</td>
      <td>0.003890</td>
    </tr>
    <tr>
      <th>5</th>
      <td>0.225156</td>
      <td>0.002811</td>
      <td>0.004701</td>
      <td>0.000228</td>
      <td>10</td>
      <td>{'max_depth': 10}</td>
      <td>0.706871</td>
      <td>0.708786</td>
      <td>0.717045</td>
      <td>0.712074</td>
      <td>0.711194</td>
      <td>0.003857</td>
      <td>4</td>
      <td>0.848864</td>
      <td>0.853907</td>
      <td>0.841430</td>
      <td>0.836463</td>
      <td>0.845166</td>
      <td>0.006704</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.259489</td>
      <td>0.005075</td>
      <td>0.004958</td>
      <td>0.000604</td>
      <td>12</td>
      <td>{'max_depth': 12}</td>
      <td>0.676996</td>
      <td>0.661479</td>
      <td>0.676430</td>
      <td>0.675982</td>
      <td>0.672722</td>
      <td>0.006501</td>
      <td>6</td>
      <td>0.887915</td>
      <td>0.903416</td>
      <td>0.885891</td>
      <td>0.875493</td>
      <td>0.888179</td>
      <td>0.009980</td>
    </tr>
  </tbody>
</table>
</div>



The DataFrame had one row for each combination of hyperparameters in the grid. As only one hyperparameter is being investigated, there was one row for each of the seven values that were searched for. A lot of output for each row can be seen, such as the mean and standard deviation of the time in seconds that each of the four folds took for both training (fitting) and testing (scoring).


```python
cv_results_df.columns
```




    Index(['mean_fit_time', 'std_fit_time', 'mean_score_time', 'std_score_time',
           'param_max_depth', 'params', 'split0_test_score', 'split1_test_score',
           'split2_test_score', 'split3_test_score', 'mean_test_score',
           'std_test_score', 'rank_test_score', 'split0_train_score',
           'split1_train_score', 'split2_train_score', 'split3_train_score',
           'mean_train_score', 'std_train_score'],
          dtype='object')



Generally speaking, the "best" combination of hyperparameters was the one with the highest average testing score. This was an estimation of how well the model, fit using these hyperparameters, could perform when scored on new data.

Let us display how the average testing score varies with the max_depth hyperparameter, including the standard deviations of the 4-fold training and testing scores as error bars:


```python
ax = plt.axes()
ax.errorbar(cv_results_df['param_max_depth'],
            cv_results_df['mean_train_score'],
            yerr=cv_results_df['std_train_score'],
            label='Mean $\pm$ 1 SD training scores')
ax.errorbar(cv_results_df['param_max_depth'],
            cv_results_df['mean_test_score'],
            yerr=cv_results_df['std_test_score'],
            label='Mean $\pm$ 1 SD testing scores')
ax.legend()
plt.xlabel('max_depth')
plt.ylabel('ROC AUC')
```

The standard deviations of the training and testing scores were shown as vertical lines at each value of max_depth that was tried; the distance above and below the average score was 1 standard deviation (they are also in the same units - it would be different when using the variance, which is the std squared). 

The error bars indicated how variable the scores were across folds. There was not much variability between the folds, since the standard deviations were no high, so this was not an issue.

It is possible to see that as the tree's depth grown deeper and deeper, the model fitted the training data better and better. As noted previously, if the model grew trees deep enough so that each leaf node had just one training sample, a model that was very specific to the training data was spotted. In fact, it would fit the training data
perfectly. This model can be said to have an extremely high variance.

The apparent effect of increasing max_depth only increased testing scores up to a point, after which deeper trees had lower testing performance. This is another example of how the bias-variance trade-off can be used to create a better predictive model. In summary, shallower trees had more bias, since they were not fitting the training data as well. This is okay as some bias is acceptable so long a better performance on the testing data is verified.

The best depth for the study case was 6.

Comparing the results with the previous - logistic regression - model, the D3 showed a better performance with an ROC CURVE 0.745, the best performance for the logistic regression - with feature engineering - was 0.740.

### Random Forests

Random forests are examples of what are called ensemble models, because they are formed by combining other models. By combining the predictions of many models, it is possible to improve upon the deficiencies of any given one of them.

Random forests are just ensembles of many decision trees; all the models in this kind of ensemble have the same mathematical form. One hyperparameter, **n_estimators**, needs to be specified when building a random forest model. This parameter counts the number of D3 estimators in the forest. Generally speaking, the more trees, the better. As the number of trees increases, the variance of the overall ensemble will decrease. This should result in the random forest model having better generalization to new data, which will be reflected in increased testing scores.

The variance diminishes when using random forests mainly because of two principles:

#### The number of features considered in each split

By using all the features to fit the training data, overfitting is possible. By limiting the number of features considered at each split, some of the decision trees in a random forest will potentially find better splits. his is because, although they are still greedily searching for the best split, they are doing it with a limited selection of features.

#### The samples used to grow different trees

The other way that the trees in a random forest differ from each other is that they are usually grown with different training samples. To do this, a statistical procedure known as bootstrapping is used, which means to generate new synthetic datasets from the original data. The synthetic datasets are created by randomly selecting samples from the original dataset using replacement. Here, "replacement" means that if we select a certain sample, we will continue to consider it for selection, that is, it is "replaced" to the original dataset after we've sampled it.

The procedure of using random sampling to create synthetic datasets, and training models on them separately, is called bagging,which is short for bootstrapped aggregation.


### Selecting the Best Feautures

After model training, classification trees will take an input sample and a produce a predicted class, for example, whether or not a credit account in our case study problem will default. One intuitive approach to combining the predictions of these trees into the ultimate prediction of the forest is to take a majority vote. Sikit-learn uses a somewhat different approach: adding up the predicted probabilities for each class and then choosing the one with the highest probability. This captures more information from each tree than just the predicted class.

####  Predictions and Interpretability

Random forests are difficult to understand and interpret besides the ranking of the features (that is why it is used as a feature selection sometimes). Although the rank is make available, how the rank was done is difficult to know.

Let us train a Random Forest model:


```python
rf = RandomForestClassifier(n_estimators=10, criterion='gini', max_depth=3,
                            min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0,
                            max_features='auto', max_leaf_nodes=None, min_impurity_decrease=0.0,
                            min_impurity_split=None, bootstrap=True, oob_score=False, n_jobs=None,
                            random_state=4, verbose=0, warm_start=False, class_weight=None)
```


```python
rf_params = {'n_estimators': list(range(10, 110, 10))}
```

Let us instantiate the cv GridSearch object:


```python
cv_rf = GridSearchCV(rf, param_grid=rf_params, scoring='roc_auc',
                     n_jobs=None, refit=True, cv=4, verbose=1,
                     pre_dispatch=None, error_score=np.nan, return_train_score=True)
```

And train the model:


```python
cv_rf.fit(X_train, y_train)
```

    Fitting 4 folds for each of 10 candidates, totalling 40 fits





    GridSearchCV(cv=4,
                 estimator=RandomForestClassifier(max_depth=3, n_estimators=10,
                                                  random_state=4),
                 param_grid={'n_estimators': [10, 20, 30, 40, 50, 60, 70, 80, 90,
                                              100]},
                 pre_dispatch=None, return_train_score=True, scoring='roc_auc',
                 verbose=1)




```python
cv_rf_results_df = pd.DataFrame(cv_rf.cv_results_)
cv_rf_results_df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>mean_fit_time</th>
      <th>std_fit_time</th>
      <th>mean_score_time</th>
      <th>std_score_time</th>
      <th>param_n_estimators</th>
      <th>params</th>
      <th>split0_test_score</th>
      <th>split1_test_score</th>
      <th>split2_test_score</th>
      <th>split3_test_score</th>
      <th>mean_test_score</th>
      <th>std_test_score</th>
      <th>rank_test_score</th>
      <th>split0_train_score</th>
      <th>split1_train_score</th>
      <th>split2_train_score</th>
      <th>split3_train_score</th>
      <th>mean_train_score</th>
      <th>std_train_score</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.165941</td>
      <td>0.006219</td>
      <td>0.009214</td>
      <td>0.000139</td>
      <td>10</td>
      <td>{'n_estimators': 10}</td>
      <td>0.739183</td>
      <td>0.746599</td>
      <td>0.772127</td>
      <td>0.771322</td>
      <td>0.757308</td>
      <td>0.014656</td>
      <td>10</td>
      <td>0.767973</td>
      <td>0.767158</td>
      <td>0.760550</td>
      <td>0.763683</td>
      <td>0.764841</td>
      <td>0.002955</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.310278</td>
      <td>0.004693</td>
      <td>0.014494</td>
      <td>0.000648</td>
      <td>20</td>
      <td>{'n_estimators': 20}</td>
      <td>0.745612</td>
      <td>0.753712</td>
      <td>0.777616</td>
      <td>0.771737</td>
      <td>0.762169</td>
      <td>0.012998</td>
      <td>9</td>
      <td>0.772819</td>
      <td>0.771499</td>
      <td>0.765944</td>
      <td>0.765395</td>
      <td>0.768914</td>
      <td>0.003284</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.449935</td>
      <td>0.012662</td>
      <td>0.020149</td>
      <td>0.002076</td>
      <td>30</td>
      <td>{'n_estimators': 30}</td>
      <td>0.747259</td>
      <td>0.754154</td>
      <td>0.777954</td>
      <td>0.771825</td>
      <td>0.762798</td>
      <td>0.012524</td>
      <td>7</td>
      <td>0.774655</td>
      <td>0.772100</td>
      <td>0.765026</td>
      <td>0.766697</td>
      <td>0.769620</td>
      <td>0.003910</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.595663</td>
      <td>0.013271</td>
      <td>0.025158</td>
      <td>0.002189</td>
      <td>40</td>
      <td>{'n_estimators': 40}</td>
      <td>0.746682</td>
      <td>0.754553</td>
      <td>0.777709</td>
      <td>0.773134</td>
      <td>0.763020</td>
      <td>0.012813</td>
      <td>4</td>
      <td>0.774300</td>
      <td>0.772642</td>
      <td>0.765051</td>
      <td>0.765876</td>
      <td>0.769467</td>
      <td>0.004057</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.720641</td>
      <td>0.005543</td>
      <td>0.028281</td>
      <td>0.000115</td>
      <td>50</td>
      <td>{'n_estimators': 50}</td>
      <td>0.748009</td>
      <td>0.755878</td>
      <td>0.779465</td>
      <td>0.771250</td>
      <td>0.763651</td>
      <td>0.012379</td>
      <td>1</td>
      <td>0.775241</td>
      <td>0.774025</td>
      <td>0.765732</td>
      <td>0.766947</td>
      <td>0.770486</td>
      <td>0.004191</td>
    </tr>
    <tr>
      <th>5</th>
      <td>0.852607</td>
      <td>0.001184</td>
      <td>0.033063</td>
      <td>0.000228</td>
      <td>60</td>
      <td>{'n_estimators': 60}</td>
      <td>0.748437</td>
      <td>0.755604</td>
      <td>0.779478</td>
      <td>0.770978</td>
      <td>0.763624</td>
      <td>0.012252</td>
      <td>2</td>
      <td>0.775419</td>
      <td>0.773615</td>
      <td>0.766133</td>
      <td>0.767032</td>
      <td>0.770550</td>
      <td>0.004031</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.993426</td>
      <td>0.003056</td>
      <td>0.038019</td>
      <td>0.000113</td>
      <td>70</td>
      <td>{'n_estimators': 70}</td>
      <td>0.747491</td>
      <td>0.754465</td>
      <td>0.779104</td>
      <td>0.771023</td>
      <td>0.763021</td>
      <td>0.012620</td>
      <td>3</td>
      <td>0.774420</td>
      <td>0.773379</td>
      <td>0.765960</td>
      <td>0.766174</td>
      <td>0.769983</td>
      <td>0.003934</td>
    </tr>
    <tr>
      <th>7</th>
      <td>1.143173</td>
      <td>0.005024</td>
      <td>0.043118</td>
      <td>0.001129</td>
      <td>80</td>
      <td>{'n_estimators': 80}</td>
      <td>0.747424</td>
      <td>0.754488</td>
      <td>0.778753</td>
      <td>0.770872</td>
      <td>0.762884</td>
      <td>0.012502</td>
      <td>5</td>
      <td>0.774234</td>
      <td>0.773257</td>
      <td>0.765978</td>
      <td>0.766220</td>
      <td>0.769922</td>
      <td>0.003840</td>
    </tr>
    <tr>
      <th>8</th>
      <td>1.279748</td>
      <td>0.004862</td>
      <td>0.049479</td>
      <td>0.003227</td>
      <td>90</td>
      <td>{'n_estimators': 90}</td>
      <td>0.747097</td>
      <td>0.753794</td>
      <td>0.778507</td>
      <td>0.770426</td>
      <td>0.762456</td>
      <td>0.012571</td>
      <td>8</td>
      <td>0.774095</td>
      <td>0.772973</td>
      <td>0.766022</td>
      <td>0.765748</td>
      <td>0.769709</td>
      <td>0.003846</td>
    </tr>
    <tr>
      <th>9</th>
      <td>1.416619</td>
      <td>0.002662</td>
      <td>0.052569</td>
      <td>0.000116</td>
      <td>100</td>
      <td>{'n_estimators': 100}</td>
      <td>0.747221</td>
      <td>0.753924</td>
      <td>0.779195</td>
      <td>0.770937</td>
      <td>0.762819</td>
      <td>0.012811</td>
      <td>6</td>
      <td>0.774340</td>
      <td>0.773598</td>
      <td>0.766103</td>
      <td>0.766914</td>
      <td>0.770238</td>
      <td>0.003750</td>
    </tr>
  </tbody>
</table>
</div>



Let us display the performance for each gridsearch params:


```python
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(10, 4))

axs[0].plot(cv_rf_results_df['param_n_estimators'],
            cv_rf_results_df['mean_fit_time'],
            '-o')
axs[0].set_xlabel('Number of trees')
axs[0].set_ylabel('Mean fit time (seconds)')

axs[1].errorbar(cv_rf_results_df['param_n_estimators'],
                cv_rf_results_df['mean_test_score'],
                yerr=cv_rf_results_df['std_test_score'])
axs[1].set_xlabel('Number of trees')
axs[1].set_ylabel('Mean testing ROC AUC $\pm$ 1 SD ')
plt.tight_layout()
```

It is possible to see that by using a random forest, the model performance increased on the cross-validation testing folds compared to the previous models. While no attempt to tune the random forest hyperparameters to achieve the best model performance was done, the results indicated that a random forest was higher in performance compared to previous efforts.

However, along with these higher model testing scores, notice that there was also more variability between the folds saw with the decision tree; this variability was visible as larger standard deviations in model testing scores across the folds.

The left plot shows that there was a linear increase in training time as more trees were added to the forest. This was expected as more trees were evaluated. The additional computation time did not compensate in terms of performance, as beyond 20 trees it was not clear that adding more trees reliably improves testing performance. While the model with 50 trees had the highest score, the fact that adding more trees actually decreased the testing score somewhat indicated that the gain in ROC AUC for 50 trees might just was due to randomness.

Let us see the best hyperparameter as well as the best params estimated by the model:


```python
cv_rf.best_params_
```




    {'n_estimators': 50}




```python
feat_imp_df = pd.DataFrame({
    'Feature name':features_response,
    'Importance':cv_rf.best_estimator_.feature_importances_
})

feat_imp_df.sort_values('Importance', ascending=False)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Feature name</th>
      <th>Importance</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>4</th>
      <td>pay_1</td>
      <td>0.609609</td>
    </tr>
    <tr>
      <th>11</th>
      <td>pay_amt1</td>
      <td>0.094123</td>
    </tr>
    <tr>
      <th>0</th>
      <td>limit_bal</td>
      <td>0.079265</td>
    </tr>
    <tr>
      <th>13</th>
      <td>pay_amt3</td>
      <td>0.047067</td>
    </tr>
    <tr>
      <th>12</th>
      <td>pay_amt2</td>
      <td>0.035393</td>
    </tr>
    <tr>
      <th>6</th>
      <td>bill_amt2</td>
      <td>0.022422</td>
    </tr>
    <tr>
      <th>14</th>
      <td>pay_amt4</td>
      <td>0.021164</td>
    </tr>
    <tr>
      <th>15</th>
      <td>pay_amt5</td>
      <td>0.015962</td>
    </tr>
    <tr>
      <th>10</th>
      <td>bill_amt6</td>
      <td>0.014293</td>
    </tr>
    <tr>
      <th>8</th>
      <td>bill_amt4</td>
      <td>0.013454</td>
    </tr>
    <tr>
      <th>7</th>
      <td>bill_amt3</td>
      <td>0.013321</td>
    </tr>
    <tr>
      <th>16</th>
      <td>pay_amt6</td>
      <td>0.011831</td>
    </tr>
    <tr>
      <th>9</th>
      <td>bill_amt5</td>
      <td>0.011264</td>
    </tr>
    <tr>
      <th>5</th>
      <td>bill_amt1</td>
      <td>0.008574</td>
    </tr>
    <tr>
      <th>1</th>
      <td>education</td>
      <td>0.001630</td>
    </tr>
    <tr>
      <th>3</th>
      <td>age</td>
      <td>0.000459</td>
    </tr>
    <tr>
      <th>2</th>
      <td>marriage</td>
      <td>0.000170</td>
    </tr>
  </tbody>
</table>
</div>



The top 5 most important features from the random forest were the same as the top 5 chosen by an ANOVA F-test analysis previously done, though in some different order. This was a good confirmation about the importance of these features as they were evaluated by different methods.

### Cross-Validation GridSearch with Random Forest on two hyperparameters

Let us extend the previous exercise and conduct a grid search over the maximum depth of a tree. First, let us create a dictionary representing the grid for **n_estimators** and **max_depth**:


```python
rf_params = {'n_estimators': [10, 50, 100, 200],
             'max_depth': [3, 6, 9, 12]}
```

And instantiate the GridSearch object using the same options that were applied before and train the predictor (to use parallel computing set **n_jobs=-1** that will match the number of cpus available automatically):


```python
cv_rf = GridSearchCV(rf, param_grid=rf_params, scoring='roc_auc',
                     n_jobs=None, refit=True, cv=4, verbose=2,
                     pre_dispatch=None, error_score=np.nan, return_train_score=True)

cv_rf.fit(X_train, y_train)
```

    Fitting 4 folds for each of 16 candidates, totalling 64 fits
    [CV] END .......................max_depth=3, n_estimators=10; total time=   0.2s
    [CV] END .......................max_depth=3, n_estimators=10; total time=   0.2s
    [CV] END .......................max_depth=3, n_estimators=10; total time=   0.2s
    [CV] END .......................max_depth=3, n_estimators=10; total time=   0.2s
    [CV] END .......................max_depth=3, n_estimators=50; total time=   0.8s
    [CV] END .......................max_depth=3, n_estimators=50; total time=   0.8s
    [CV] END .......................max_depth=3, n_estimators=50; total time=   0.8s
    [CV] END .......................max_depth=3, n_estimators=50; total time=   0.8s
    [CV] END ......................max_depth=3, n_estimators=100; total time=   1.5s
    [CV] END ......................max_depth=3, n_estimators=100; total time=   1.5s
    [CV] END ......................max_depth=3, n_estimators=100; total time=   1.5s
    [CV] END ......................max_depth=3, n_estimators=100; total time=   1.5s
    [CV] END ......................max_depth=3, n_estimators=200; total time=   3.0s
    [CV] END ......................max_depth=3, n_estimators=200; total time=   3.0s
    [CV] END ......................max_depth=3, n_estimators=200; total time=   2.9s
    [CV] END ......................max_depth=3, n_estimators=200; total time=   2.9s
    [CV] END .......................max_depth=6, n_estimators=10; total time=   0.3s
    [CV] END .......................max_depth=6, n_estimators=10; total time=   0.3s
    [CV] END .......................max_depth=6, n_estimators=10; total time=   0.3s
    [CV] END .......................max_depth=6, n_estimators=10; total time=   0.3s
    [CV] END .......................max_depth=6, n_estimators=50; total time=   1.3s
    [CV] END .......................max_depth=6, n_estimators=50; total time=   1.3s
    [CV] END .......................max_depth=6, n_estimators=50; total time=   1.3s
    [CV] END .......................max_depth=6, n_estimators=50; total time=   1.2s
    [CV] END ......................max_depth=6, n_estimators=100; total time=   2.5s
    [CV] END ......................max_depth=6, n_estimators=100; total time=   2.5s
    [CV] END ......................max_depth=6, n_estimators=100; total time=   2.5s
    [CV] END ......................max_depth=6, n_estimators=100; total time=   2.5s
    [CV] END ......................max_depth=6, n_estimators=200; total time=   5.0s
    [CV] END ......................max_depth=6, n_estimators=200; total time=   5.0s
    [CV] END ......................max_depth=6, n_estimators=200; total time=   5.0s
    [CV] END ......................max_depth=6, n_estimators=200; total time=   5.0s
    [CV] END .......................max_depth=9, n_estimators=10; total time=   0.4s
    [CV] END .......................max_depth=9, n_estimators=10; total time=   0.4s
    [CV] END .......................max_depth=9, n_estimators=10; total time=   0.4s
    [CV] END .......................max_depth=9, n_estimators=10; total time=   0.4s
    [CV] END .......................max_depth=9, n_estimators=50; total time=   1.7s
    [CV] END .......................max_depth=9, n_estimators=50; total time=   1.7s
    [CV] END .......................max_depth=9, n_estimators=50; total time=   1.7s
    [CV] END .......................max_depth=9, n_estimators=50; total time=   1.7s
    [CV] END ......................max_depth=9, n_estimators=100; total time=   3.4s
    [CV] END ......................max_depth=9, n_estimators=100; total time=   3.4s
    [CV] END ......................max_depth=9, n_estimators=100; total time=   3.4s
    [CV] END ......................max_depth=9, n_estimators=100; total time=   3.4s
    [CV] END ......................max_depth=9, n_estimators=200; total time=   6.9s
    [CV] END ......................max_depth=9, n_estimators=200; total time=   6.9s
    [CV] END ......................max_depth=9, n_estimators=200; total time=   6.9s
    [CV] END ......................max_depth=9, n_estimators=200; total time=   6.9s
    [CV] END ......................max_depth=12, n_estimators=10; total time=   0.4s
    [CV] END ......................max_depth=12, n_estimators=10; total time=   0.4s
    [CV] END ......................max_depth=12, n_estimators=10; total time=   0.4s
    [CV] END ......................max_depth=12, n_estimators=10; total time=   0.4s
    [CV] END ......................max_depth=12, n_estimators=50; total time=   2.1s
    [CV] END ......................max_depth=12, n_estimators=50; total time=   2.1s
    [CV] END ......................max_depth=12, n_estimators=50; total time=   2.1s
    [CV] END ......................max_depth=12, n_estimators=50; total time=   2.2s
    [CV] END .....................max_depth=12, n_estimators=100; total time=   4.3s
    [CV] END .....................max_depth=12, n_estimators=100; total time=   4.3s
    [CV] END .....................max_depth=12, n_estimators=100; total time=   4.2s
    [CV] END .....................max_depth=12, n_estimators=100; total time=   4.3s
    [CV] END .....................max_depth=12, n_estimators=200; total time=   8.7s
    [CV] END .....................max_depth=12, n_estimators=200; total time=   8.5s
    [CV] END .....................max_depth=12, n_estimators=200; total time=   8.5s
    [CV] END .....................max_depth=12, n_estimators=200; total time=   8.6s





    GridSearchCV(cv=4,
                 estimator=RandomForestClassifier(max_depth=3, n_estimators=10,
                                                  random_state=4),
                 param_grid={'max_depth': [3, 6, 9, 12],
                             'n_estimators': [10, 50, 100, 200]},
                 pre_dispatch=None, return_train_score=True, scoring='roc_auc',
                 verbose=2)



Now, let store the results into a pandas DataFrame object:


```python
cv_rf_results_df = pd.DataFrame(cv_rf.cv_results_)
cv_rf_results_df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>mean_fit_time</th>
      <th>std_fit_time</th>
      <th>mean_score_time</th>
      <th>std_score_time</th>
      <th>param_max_depth</th>
      <th>param_n_estimators</th>
      <th>params</th>
      <th>split0_test_score</th>
      <th>split1_test_score</th>
      <th>split2_test_score</th>
      <th>split3_test_score</th>
      <th>mean_test_score</th>
      <th>std_test_score</th>
      <th>rank_test_score</th>
      <th>split0_train_score</th>
      <th>split1_train_score</th>
      <th>split2_train_score</th>
      <th>split3_train_score</th>
      <th>mean_train_score</th>
      <th>std_train_score</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.160572</td>
      <td>0.010295</td>
      <td>0.009034</td>
      <td>0.000137</td>
      <td>3</td>
      <td>10</td>
      <td>{'max_depth': 3, 'n_estimators': 10}</td>
      <td>0.739183</td>
      <td>0.746599</td>
      <td>0.772127</td>
      <td>0.771322</td>
      <td>0.757308</td>
      <td>0.014656</td>
      <td>15</td>
      <td>0.767973</td>
      <td>0.767158</td>
      <td>0.760550</td>
      <td>0.763683</td>
      <td>0.764841</td>
      <td>0.002955</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.739119</td>
      <td>0.014972</td>
      <td>0.028264</td>
      <td>0.000112</td>
      <td>3</td>
      <td>50</td>
      <td>{'max_depth': 3, 'n_estimators': 50}</td>
      <td>0.748009</td>
      <td>0.755878</td>
      <td>0.779465</td>
      <td>0.771250</td>
      <td>0.763651</td>
      <td>0.012379</td>
      <td>12</td>
      <td>0.775241</td>
      <td>0.774025</td>
      <td>0.765732</td>
      <td>0.766947</td>
      <td>0.770486</td>
      <td>0.004191</td>
    </tr>
    <tr>
      <th>2</th>
      <td>1.433841</td>
      <td>0.013277</td>
      <td>0.052308</td>
      <td>0.000071</td>
      <td>3</td>
      <td>100</td>
      <td>{'max_depth': 3, 'n_estimators': 100}</td>
      <td>0.747221</td>
      <td>0.753924</td>
      <td>0.779195</td>
      <td>0.770937</td>
      <td>0.762819</td>
      <td>0.012811</td>
      <td>13</td>
      <td>0.774340</td>
      <td>0.773598</td>
      <td>0.766103</td>
      <td>0.766914</td>
      <td>0.770238</td>
      <td>0.003750</td>
    </tr>
    <tr>
      <th>3</th>
      <td>2.848571</td>
      <td>0.006656</td>
      <td>0.100138</td>
      <td>0.000516</td>
      <td>3</td>
      <td>200</td>
      <td>{'max_depth': 3, 'n_estimators': 200}</td>
      <td>0.747454</td>
      <td>0.753057</td>
      <td>0.777644</td>
      <td>0.771729</td>
      <td>0.762471</td>
      <td>0.012550</td>
      <td>14</td>
      <td>0.774453</td>
      <td>0.773534</td>
      <td>0.766139</td>
      <td>0.767404</td>
      <td>0.770382</td>
      <td>0.003653</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.252126</td>
      <td>0.002849</td>
      <td>0.010875</td>
      <td>0.000209</td>
      <td>6</td>
      <td>10</td>
      <td>{'max_depth': 6, 'n_estimators': 10}</td>
      <td>0.752905</td>
      <td>0.758904</td>
      <td>0.780036</td>
      <td>0.774017</td>
      <td>0.766466</td>
      <td>0.010980</td>
      <td>11</td>
      <td>0.795911</td>
      <td>0.796476</td>
      <td>0.793695</td>
      <td>0.796446</td>
      <td>0.795632</td>
      <td>0.001141</td>
    </tr>
    <tr>
      <th>5</th>
      <td>1.219600</td>
      <td>0.008659</td>
      <td>0.036014</td>
      <td>0.000334</td>
      <td>6</td>
      <td>50</td>
      <td>{'max_depth': 6, 'n_estimators': 50}</td>
      <td>0.760089</td>
      <td>0.765251</td>
      <td>0.788567</td>
      <td>0.778669</td>
      <td>0.773144</td>
      <td>0.011193</td>
      <td>8</td>
      <td>0.807081</td>
      <td>0.805109</td>
      <td>0.799176</td>
      <td>0.801229</td>
      <td>0.803149</td>
      <td>0.003113</td>
    </tr>
    <tr>
      <th>6</th>
      <td>2.418893</td>
      <td>0.007093</td>
      <td>0.067700</td>
      <td>0.000225</td>
      <td>6</td>
      <td>100</td>
      <td>{'max_depth': 6, 'n_estimators': 100}</td>
      <td>0.760964</td>
      <td>0.765515</td>
      <td>0.788793</td>
      <td>0.778936</td>
      <td>0.773552</td>
      <td>0.011004</td>
      <td>7</td>
      <td>0.808194</td>
      <td>0.806130</td>
      <td>0.800846</td>
      <td>0.803123</td>
      <td>0.804573</td>
      <td>0.002808</td>
    </tr>
    <tr>
      <th>7</th>
      <td>4.845598</td>
      <td>0.009769</td>
      <td>0.131047</td>
      <td>0.000598</td>
      <td>6</td>
      <td>200</td>
      <td>{'max_depth': 6, 'n_estimators': 200}</td>
      <td>0.761481</td>
      <td>0.765319</td>
      <td>0.787854</td>
      <td>0.779745</td>
      <td>0.773600</td>
      <td>0.010682</td>
      <td>6</td>
      <td>0.808276</td>
      <td>0.806853</td>
      <td>0.800682</td>
      <td>0.804006</td>
      <td>0.804954</td>
      <td>0.002906</td>
    </tr>
    <tr>
      <th>8</th>
      <td>0.345823</td>
      <td>0.001968</td>
      <td>0.012519</td>
      <td>0.000102</td>
      <td>9</td>
      <td>10</td>
      <td>{'max_depth': 9, 'n_estimators': 10}</td>
      <td>0.757853</td>
      <td>0.755444</td>
      <td>0.781544</td>
      <td>0.773823</td>
      <td>0.767166</td>
      <td>0.010899</td>
      <td>10</td>
      <td>0.854161</td>
      <td>0.856197</td>
      <td>0.850317</td>
      <td>0.852039</td>
      <td>0.853178</td>
      <td>0.002211</td>
    </tr>
    <tr>
      <th>9</th>
      <td>1.677115</td>
      <td>0.008941</td>
      <td>0.045137</td>
      <td>0.000182</td>
      <td>9</td>
      <td>50</td>
      <td>{'max_depth': 9, 'n_estimators': 50}</td>
      <td>0.763956</td>
      <td>0.764420</td>
      <td>0.786690</td>
      <td>0.780255</td>
      <td>0.773830</td>
      <td>0.009908</td>
      <td>4</td>
      <td>0.867242</td>
      <td>0.871380</td>
      <td>0.868213</td>
      <td>0.867390</td>
      <td>0.868557</td>
      <td>0.001672</td>
    </tr>
    <tr>
      <th>10</th>
      <td>3.346243</td>
      <td>0.009928</td>
      <td>0.086113</td>
      <td>0.000312</td>
      <td>9</td>
      <td>100</td>
      <td>{'max_depth': 9, 'n_estimators': 100}</td>
      <td>0.763872</td>
      <td>0.765788</td>
      <td>0.789016</td>
      <td>0.781199</td>
      <td>0.774969</td>
      <td>0.010530</td>
      <td>3</td>
      <td>0.872099</td>
      <td>0.874119</td>
      <td>0.870537</td>
      <td>0.871196</td>
      <td>0.871988</td>
      <td>0.001350</td>
    </tr>
    <tr>
      <th>11</th>
      <td>6.705547</td>
      <td>0.007114</td>
      <td>0.167002</td>
      <td>0.000740</td>
      <td>9</td>
      <td>200</td>
      <td>{'max_depth': 9, 'n_estimators': 200}</td>
      <td>0.764963</td>
      <td>0.767541</td>
      <td>0.790255</td>
      <td>0.782034</td>
      <td>0.776199</td>
      <td>0.010402</td>
      <td>1</td>
      <td>0.873807</td>
      <td>0.874909</td>
      <td>0.871185</td>
      <td>0.871246</td>
      <td>0.872787</td>
      <td>0.001619</td>
    </tr>
    <tr>
      <th>12</th>
      <td>0.426109</td>
      <td>0.005192</td>
      <td>0.014213</td>
      <td>0.000121</td>
      <td>12</td>
      <td>10</td>
      <td>{'max_depth': 12, 'n_estimators': 10}</td>
      <td>0.754513</td>
      <td>0.751116</td>
      <td>0.764061</td>
      <td>0.759223</td>
      <td>0.757228</td>
      <td>0.004884</td>
      <td>16</td>
      <td>0.922544</td>
      <td>0.919157</td>
      <td>0.926191</td>
      <td>0.914636</td>
      <td>0.920632</td>
      <td>0.004263</td>
    </tr>
    <tr>
      <th>13</th>
      <td>2.093706</td>
      <td>0.010107</td>
      <td>0.053497</td>
      <td>0.000099</td>
      <td>12</td>
      <td>50</td>
      <td>{'max_depth': 12, 'n_estimators': 50}</td>
      <td>0.763750</td>
      <td>0.763663</td>
      <td>0.780781</td>
      <td>0.778086</td>
      <td>0.771570</td>
      <td>0.007921</td>
      <td>9</td>
      <td>0.940170</td>
      <td>0.943000</td>
      <td>0.942297</td>
      <td>0.937049</td>
      <td>0.940629</td>
      <td>0.002315</td>
    </tr>
    <tr>
      <th>14</th>
      <td>4.155320</td>
      <td>0.017221</td>
      <td>0.107935</td>
      <td>0.004810</td>
      <td>12</td>
      <td>100</td>
      <td>{'max_depth': 12, 'n_estimators': 100}</td>
      <td>0.765665</td>
      <td>0.765061</td>
      <td>0.783413</td>
      <td>0.781119</td>
      <td>0.773815</td>
      <td>0.008493</td>
      <td>5</td>
      <td>0.942902</td>
      <td>0.946968</td>
      <td>0.946109</td>
      <td>0.942448</td>
      <td>0.944607</td>
      <td>0.001962</td>
    </tr>
    <tr>
      <th>15</th>
      <td>8.371203</td>
      <td>0.092187</td>
      <td>0.208209</td>
      <td>0.005270</td>
      <td>12</td>
      <td>200</td>
      <td>{'max_depth': 12, 'n_estimators': 200}</td>
      <td>0.765910</td>
      <td>0.766963</td>
      <td>0.785611</td>
      <td>0.783513</td>
      <td>0.775499</td>
      <td>0.009101</td>
      <td>2</td>
      <td>0.945018</td>
      <td>0.949378</td>
      <td>0.946415</td>
      <td>0.944689</td>
      <td>0.946375</td>
      <td>0.001851</td>
    </tr>
  </tbody>
</table>
</div>



Now, let us plot a three dimensional space into a two dimensional plot using a Checkerboard Graph. A Checkerboard Graph plots a two-dimensional grid with colored squares (or other shapes on it) as a way to show three dimensions of data. First, it is necessary to create a meshgrid of the *x*-axis and *y*-axis:


```python
xx_rf, yy_rf = np.meshgrid(range(5), range(5))
```

Let us set the colour map scale and create the axes:


```python
cm = plt.cm.jet

plt.figure(figsize=(20,10))
ax = plt.axes()

pcolor_graph = ax.pcolormesh(xx_rf, 
                             yy_rf, 
                             cv_rf_results_df['mean_test_score'].values.reshape((4,4)), 
                             cmap=cm)

plt.colorbar(pcolor_graph, label='Average testing ROC AUC')

ax.set_aspect('equal')
ax.set_xticks([0.5, 1.5, 2.5, 3.5])
ax.set_yticks([0.5, 1.5, 2.5, 3.5])
ax.set_xticklabels([str(tick_label) for tick_label in rf_params['n_estimators']])
ax.set_yticklabels([str(tick_label) for tick_label in rf_params['max_depth']])
ax.set_xlabel('Number of trees')
ax.set_ylabel('Maximum depth')
```

There was certainly an advantage to using trees with a depth of more than three. Of the parameter combinations that were tried, **max_depth=9** with **200 trees** yielded the best average testing score, with ROC AUC = 0.776.

This was the best model found so far (compared to the individual D3 and logistic regression).

16ddcc92a12f76941a5715ee05076892149ec1c6
ipynb
Jupyter Notebook
decision-trees-and-random-forests.ipynb
jeantardelli/data-science
f39b7bf24b8b8d2d6f4c3048251dd56bf19354d2
MIT
2020-12-21T20:56:22.000Z
2020-12-21T20:56:22.000Z
decision-trees-and-random-forests.ipynb
jeantardelli/data-science
f39b7bf24b8b8d2d6f4c3048251dd56bf19354d2
MIT
decision-trees-and-random-forests.ipynb
jeantardelli/data-science
f39b7bf24b8b8d2d6f4c3048251dd56bf19354d2
MIT
Qwen/Qwen-72B
1. YES
2. YES