<center>
<table style="border:none">
    <tr style="border:none">
    <th style="border:none">
        <a  href='https://colab.research.google.com/github/AmirMardan/ml_course/blob/main/6_classical_machine_learning/3_classification_1.ipynb'></a>
    </th>
    <th style="border:none">
        <a  href='https://github1s.com/AmirMardan/ml_course/blob/main/6_classical_machine_learning/3_classification_1.ipynb'></a>
    </th>
    </tr>
</table>
</center>


This notebook is created by <a href='https://amirmardan.github.io'> Amir Mardan</a>. For any feedback or suggestion, please contact me via <a href="mailto:mardan.amir.h@gmail.com">email</a>, (mardan.amir.h@gmail.com).



<a name='top'></a>
# Classification 1

In a classification problem, we need to predict the label of a sample which is a discrete value (class).
There are different methods to train a model for predicting the class of data samples.
In this notebook, we learn how to use 

- Logistic regression
- Support vector machine (SVM)
- Random forest method
- Confusion matrix

This notebook will cover the following topics:

- [1. Data Preparation](#preparation)
- [2. Logistic Regression](#logistic_regression)
- [3. Support Vector Machine](#svm)
- [4. Random Forest Classifier](#random_forest)

This notebook is created to classify handwritten digits. Let's import required packages and the dataset.


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

```


```python
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True)
print('X: ', X.shape)
print('y: ', y.shape)

```

    X:  (1797, 64)
    y:  (1797,)



```python
fig, ax = plt.subplots(3, 5, figsize=(8, 5),
                       sharex=True, sharey=True)

idx = np.random.randint(0, X.shape[0], 15)
for i, ax in enumerate(ax.flat):
    
    ax.imshow(X[idx[i], :].reshape(8, 8), cmap='gray')
    ax.axis('off')
    
```

<a id='preparation'></a>
## 1. Data Preparation


```python
from sklearn.model_selection import train_test_split

X /= X.max()

X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.15,
                                                    random_state=1)

print('Train: ', X_train.shape)
print('Test: ', X_test.shape)
```

    Train:  (1527, 64)
    Test:  (270, 64)


<a id='logistic_regression'></a>
## 2. Logistic Regression

Logistic regression model takes advantage of a logistic function to put the output of a linear equation between 0 and 1.
This logistic function is called the sigmoid function.

$
\hat y = \frac{1}{1 + e^{-z}}
$

<center>

</center>


The sigmoid function returns a probability score between 0 and 1. In order to map this score to a categorical class, we define a threshold value

\begin{equation}
\begin{cases}
p \geq 0.5, \quad \text{class } 1\\
p < 0.5, \quad \text{class } 0\\
\end{cases}
\end{equation}

Logistic regression uses the cross-entropy cost function as

\begin{equation}
J(y, \hat y) = -\frac{1}{m}\sum_{i=1}^m [y^{(i)} log(\hat y^{(i)}) + (1 - y^{(i)}) log(1 - \hat y^{(i)})].
\end{equation}



```python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

log_model = LogisticRegression()
log_model.fit(X_train, y_train)
y_pred = log_model.predict(X_train)
```

    /Users/amir/repos/venv/AI/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:444: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
    
    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      n_iter_i = _check_optimize_result(


**Confusion Matrix**

Using confusion matrix, we can see how the model works and which labels are making trouble for model to predict correctly.


```python
from sklearn.metrics import confusion_matrix

def confusion_matrix_plot(y_pred, y_true):
    ax = sns.heatmap(confusion_matrix(y_true, y_pred),
                annot=True, fmt='d', cbar=False)

    ax.set_ylabel('True')
    ax.set_xlabel('Predicted')
    ax.set_title('Confusion matrix with training data');
    
    
confusion_matrix_plot(y_train, y_pred)
    


```

We can also measure the score of the model using `accuracy_score` which shows the best model with a score of 1.


```python
from sklearn.metrics import accuracy_score


print(f'Score: {accuracy_score(y_train, y_pred, normalize=True)}')
```

    Score: 0.982973149967256



```python
y_pred = log_model.predict(X_test)

confusion_matrix_plot(y_test, y_pred)


print(f'Score: {accuracy_score(y_test, y_pred)}')
```


```python
def wrong_classified_plot(X_true, y_true, y_pred):
    idx = y_true != y_pred
    X_wrong = X_true[idx]

    n_wrong = X_wrong.shape[0]


    fig, ax = plt.subplots(2, int(n_wrong/2), figsize=(8, 5),
                        sharex=True, sharey=True)

    for i, ax in enumerate(ax.flat):
        
        ax.imshow(X_wrong[i, :].reshape(8, 8), cmap='gray')
        ax.axis('off')
        ax.set_title(f'T: {y_true[idx][i]}, P: {y_pred[idx][i]}')
        
        
wrong_classified_plot(X_test, y_test, y_pred)
```

<a id='svm'></a>
## 3. Support Vector Machine

Support vector machines (SVMs) are kernel-based machine learning methods that can be used for classification and regression.
The point about SVMs is that they use the hyperplane that has the maximum distance to the data points.

<center>

</center>




```python
from sklearn.svm import SVC

svm = SVC(kernel='linear', C=.01)
svm.fit(X_train, y_train)
y_pred = svm.predict(X_train)
```


```python
confusion_matrix_plot(y_pred, y_train)

print(f'Score: {accuracy_score(y_train, y_pred)}')

```


```python
y_pred = svm.predict(X_test)

confusion_matrix_plot(y_pred, y_test)

print(f'Score: {accuracy_score(y_test, y_pred)}')
```


```python
wrong_classified_plot(X_test, y_test, y_pred)
```


```python
from sklearn.model_selection import GridSearchCV

param_grid = {
    'kernel': ['poly', 'linear'],
    'C':[0.01, 1, 10],
    'gamma':[1,0.1,0.01], 
    'degree':[2,3,4],
}

grid_search = GridSearchCV(SVC(), param_grid, verbose=2)

grid_search.fit(X_train, y_train)
```

    Fitting 5 folds for each of 54 candidates, totalling 270 fits
    [CV] END .............C=0.01, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...........C=0.01, degree=2, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=2, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=2, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=2, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=2, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=2, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=2, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=2, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=2, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=2, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END .........C=0.01, degree=2, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=2, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=2, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=2, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=2, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END ..........C=0.01, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ........C=0.01, degree=2, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=2, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=2, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=2, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=2, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END .............C=0.01, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...........C=0.01, degree=3, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=3, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=3, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=3, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=3, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=3, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=3, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=3, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=3, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=3, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END .........C=0.01, degree=3, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=3, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=3, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=3, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=3, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END ..........C=0.01, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ........C=0.01, degree=3, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=3, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=3, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=3, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=3, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END .............C=0.01, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=0.01, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...........C=0.01, degree=4, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=4, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=4, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=4, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=4, gamma=1, kernel=linear; total time=   0.1s
    [CV] END ...........C=0.01, degree=4, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=4, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=4, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=4, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END ...........C=0.01, degree=4, gamma=0.1, kernel=poly; total time=   0.1s
    [CV] END .........C=0.01, degree=4, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=4, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=4, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=4, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END .........C=0.01, degree=4, gamma=0.1, kernel=linear; total time=   0.1s
    [CV] END ..........C=0.01, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=0.01, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ........C=0.01, degree=4, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=4, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=4, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=4, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ........C=0.01, degree=4, gamma=0.01, kernel=linear; total time=   0.1s
    [CV] END ................C=1, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ............C=1, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END .............C=1, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=2, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ...........C=1, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ................C=1, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ............C=1, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END .............C=1, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ...........C=1, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ................C=1, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ................C=1, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ..............C=1, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ............C=1, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=1, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END .............C=1, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END .............C=1, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ...........C=1, degree=4, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=4, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=4, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=4, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...........C=1, degree=4, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...............C=10, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=2, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=2, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ...........C=10, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=2, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=10, degree=2, gamma=0.01, kernel=poly; total time=   0.0s
    [CV] END ............C=10, degree=2, gamma=0.01, kernel=poly; total time=   0.0s
    [CV] END ............C=10, degree=2, gamma=0.01, kernel=poly; total time=   0.0s
    [CV] END ............C=10, degree=2, gamma=0.01, kernel=poly; total time=   0.0s
    [CV] END ............C=10, degree=2, gamma=0.01, kernel=poly; total time=   0.0s
    [CV] END ..........C=10, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=2, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...............C=10, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=3, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=3, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ...........C=10, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=3, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=10, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ............C=10, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ............C=10, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ............C=10, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ............C=10, degree=3, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=10, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=3, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ...............C=10, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END ...............C=10, degree=4, gamma=1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=1, kernel=linear; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END .............C=10, degree=4, gamma=0.1, kernel=poly; total time=   0.0s
    [CV] END ...........C=10, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ...........C=10, degree=4, gamma=0.1, kernel=linear; total time=   0.0s
    [CV] END ............C=10, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ............C=10, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ............C=10, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ............C=10, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ............C=10, degree=4, gamma=0.01, kernel=poly; total time=   0.1s
    [CV] END ..........C=10, degree=4, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=4, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=4, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=4, gamma=0.01, kernel=linear; total time=   0.0s
    [CV] END ..........C=10, degree=4, gamma=0.01, kernel=linear; total time=   0.0s





<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: "▸";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: "▾";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: "";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: "";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: "";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id="sk-container-id-1" class="sk-top-container"><div class="sk-text-repr-fallback"><pre>GridSearchCV(estimator=SVC(),
             param_grid={&#x27;C&#x27;: [0.01, 1, 10], &#x27;degree&#x27;: [2, 3, 4],
                         &#x27;gamma&#x27;: [1, 0.1, 0.01],
                         &#x27;kernel&#x27;: [&#x27;poly&#x27;, &#x27;linear&#x27;]},
             verbose=2)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class="sk-container" hidden><div class="sk-item sk-dashed-wrapped"><div class="sk-label-container"><div class="sk-label sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually" id="sk-estimator-id-1" type="checkbox" ><label for="sk-estimator-id-1" class="sk-toggleable__label sk-toggleable__label-arrow">GridSearchCV</label><div class="sk-toggleable__content"><pre>GridSearchCV(estimator=SVC(),
             param_grid={&#x27;C&#x27;: [0.01, 1, 10], &#x27;degree&#x27;: [2, 3, 4],
                         &#x27;gamma&#x27;: [1, 0.1, 0.01],
                         &#x27;kernel&#x27;: [&#x27;poly&#x27;, &#x27;linear&#x27;]},
             verbose=2)</pre></div></div></div><div class="sk-parallel"><div class="sk-parallel-item"><div class="sk-item"><div class="sk-label-container"><div class="sk-label sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually" id="sk-estimator-id-2" type="checkbox" ><label for="sk-estimator-id-2" class="sk-toggleable__label sk-toggleable__label-arrow">estimator: SVC</label><div class="sk-toggleable__content"><pre>SVC()</pre></div></div></div><div class="sk-serial"><div class="sk-item"><div class="sk-estimator sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually" id="sk-estimator-id-3" type="checkbox" ><label for="sk-estimator-id-3" class="sk-toggleable__label sk-toggleable__label-arrow">SVC</label><div class="sk-toggleable__content"><pre>SVC()</pre></div></div></div></div></div></div></div></div></div></div>




```python
svm = grid_search.best_estimator_

y_pred = svm.predict(X_test)

confusion_matrix_plot(y_pred, y_test)

print(f'Score: {accuracy_score(y_test, y_pred)}')
```


```python
wrong_classified_plot(X_test, y_test, y_pred)
```

<a id='random_forest'></a>
## 4. Random Forest Classifier

Random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.

<center>
    
</center>


```python
from sklearn.ensemble import RandomForestClassifier

rfc_model = RandomForestClassifier()
rfc_model.fit(X_train, y_train)
y_pred = rfc_model.predict(X_train)
```


```python
confusion_matrix_plot(y_pred, y_train)

print(f'Score: {accuracy_score(y_train, y_pred)}')
```


```python
y_pred = rfc_model.predict(X_test)

confusion_matrix_plot(y_pred, y_test)

print(f'Score: {accuracy_score(y_test, y_pred)}')
```


```python
wrong_classified_plot(X_test, y_test, y_pred)
```

### [TOP ☝️](#top)

4d7805b31c79bce968c6598d80b18fb1760503f4
ipynb
Jupyter Notebook
6_classical_machine_learning/3_classification_1.ipynb
AmirMardan/ml_course
0af36720cb010ae7acc942820b611217272a9386
MIT
2022-03-14T00:33:25.000Z
2022-03-26T17:14:32.000Z
6_classical_machine_learning/3_classification_1.ipynb
AmirMardan/ml_course
0af36720cb010ae7acc942820b611217272a9386
MIT
6_classical_machine_learning/3_classification_1.ipynb
AmirMardan/ml_course
0af36720cb010ae7acc942820b611217272a9386
MIT
2022-03-12T19:52:46.000Z
2022-03-25T12:39:31.000Z
Qwen/Qwen-72B
1. YES
2. YES