<h1> Data Transformation </h1>

## Logistic Regression - on [Titanic Dataset](https://www.kaggle.com/c/titanic)

- Models the probability an object belongs to a class
- Values ranges from 0 to 1
- Can use threshold to classify into which classes a class belongs
- An S-shaped curve

$
\begin{align}
\sigma(t) = \frac{1}{1 + e^{-t}}
\end{align}
$


#### Read the data


```python
import pandas as pd
df_train = pd.read_csv('../data/titanic_train.csv')
```


```python
df_train.head(8)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>PassengerId</th>
      <th>Survived</th>
      <th>Pclass</th>
      <th>Name</th>
      <th>Sex</th>
      <th>Age</th>
      <th>SibSp</th>
      <th>Parch</th>
      <th>Ticket</th>
      <th>Fare</th>
      <th>Cabin</th>
      <th>Embarked</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>0</td>
      <td>3</td>
      <td>Braund, Mr. Owen Harris</td>
      <td>male</td>
      <td>22.0</td>
      <td>1</td>
      <td>0</td>
      <td>A/5 21171</td>
      <td>7.2500</td>
      <td>NaN</td>
      <td>S</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>1</td>
      <td>1</td>
      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>
      <td>female</td>
      <td>38.0</td>
      <td>1</td>
      <td>0</td>
      <td>PC 17599</td>
      <td>71.2833</td>
      <td>C85</td>
      <td>C</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>1</td>
      <td>3</td>
      <td>Heikkinen, Miss. Laina</td>
      <td>female</td>
      <td>26.0</td>
      <td>0</td>
      <td>0</td>
      <td>STON/O2. 3101282</td>
      <td>7.9250</td>
      <td>NaN</td>
      <td>S</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4</td>
      <td>1</td>
      <td>1</td>
      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>
      <td>female</td>
      <td>35.0</td>
      <td>1</td>
      <td>0</td>
      <td>113803</td>
      <td>53.1000</td>
      <td>C123</td>
      <td>S</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5</td>
      <td>0</td>
      <td>3</td>
      <td>Allen, Mr. William Henry</td>
      <td>male</td>
      <td>35.0</td>
      <td>0</td>
      <td>0</td>
      <td>373450</td>
      <td>8.0500</td>
      <td>NaN</td>
      <td>S</td>
    </tr>
    <tr>
      <th>5</th>
      <td>6</td>
      <td>0</td>
      <td>3</td>
      <td>Moran, Mr. James</td>
      <td>male</td>
      <td>NaN</td>
      <td>0</td>
      <td>0</td>
      <td>330877</td>
      <td>8.4583</td>
      <td>NaN</td>
      <td>Q</td>
    </tr>
    <tr>
      <th>6</th>
      <td>7</td>
      <td>0</td>
      <td>1</td>
      <td>McCarthy, Mr. Timothy J</td>
      <td>male</td>
      <td>54.0</td>
      <td>0</td>
      <td>0</td>
      <td>17463</td>
      <td>51.8625</td>
      <td>E46</td>
      <td>S</td>
    </tr>
    <tr>
      <th>7</th>
      <td>8</td>
      <td>0</td>
      <td>3</td>
      <td>Palsson, Master. Gosta Leonard</td>
      <td>male</td>
      <td>2.0</td>
      <td>3</td>
      <td>1</td>
      <td>349909</td>
      <td>21.0750</td>
      <td>NaN</td>
      <td>S</td>
    </tr>
  </tbody>
</table>
</div>



## Data Statistics

#### Describing the statistics for numerical features


```python
df_train.describe()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>PassengerId</th>
      <th>Survived</th>
      <th>Pclass</th>
      <th>Age</th>
      <th>SibSp</th>
      <th>Parch</th>
      <th>Fare</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>891.000000</td>
      <td>891.000000</td>
      <td>891.000000</td>
      <td>714.000000</td>
      <td>891.000000</td>
      <td>891.000000</td>
      <td>891.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>446.000000</td>
      <td>0.383838</td>
      <td>2.308642</td>
      <td>29.699118</td>
      <td>0.523008</td>
      <td>0.381594</td>
      <td>32.204208</td>
    </tr>
    <tr>
      <th>std</th>
      <td>257.353842</td>
      <td>0.486592</td>
      <td>0.836071</td>
      <td>14.526497</td>
      <td>1.102743</td>
      <td>0.806057</td>
      <td>49.693429</td>
    </tr>
    <tr>
      <th>min</th>
      <td>1.000000</td>
      <td>0.000000</td>
      <td>1.000000</td>
      <td>0.420000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>223.500000</td>
      <td>0.000000</td>
      <td>2.000000</td>
      <td>20.125000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>7.910400</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>446.000000</td>
      <td>0.000000</td>
      <td>3.000000</td>
      <td>28.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>14.454200</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>668.500000</td>
      <td>1.000000</td>
      <td>3.000000</td>
      <td>38.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
      <td>31.000000</td>
    </tr>
    <tr>
      <th>max</th>
      <td>891.000000</td>
      <td>1.000000</td>
      <td>3.000000</td>
      <td>80.000000</td>
      <td>8.000000</td>
      <td>6.000000</td>
      <td>512.329200</td>
    </tr>
  </tbody>
</table>
</div>



#### Find the count of the non-NaN values per feature


```python
df_train.count()
```




    PassengerId    891
    Survived       891
    Pclass         891
    Name           891
    Sex            891
    Age            714
    SibSp          891
    Parch          891
    Ticket         891
    Fare           891
    Cabin          204
    Embarked       889
    dtype: int64



## What features can be removed?

### Remove features that are not related to your outcome


```python
df_train.drop(['Name', 'Ticket'], axis=1, inplace=True)
```

### Remove column with missing data


```python
df_train.drop(['Cabin'], axis=1, inplace=True)
```

## Data Imputation - Filling in missing values
- Select a percentage threshold that you would want to accomodate
- Around 1/5th to 1/3rd of the data (20% to 33.3%)
- if more than 50% of the data is missing, you will be generating data for the majority of your dataset - Not a good thing to do


```python
from matplotlib import pyplot as plt
import seaborn as sns
plt.figure(figsize=(7,5))
sns.boxplot(x='Pclass',y='Age',data=df_train)
```


```python
from matplotlib import pyplot as plt
import seaborn as sns
plt.figure(figsize=(7,5))
sns.boxplot(x='Sex',y='Age',data=df_train)
```


```python
from matplotlib import pyplot as plt
import seaborn as sns
plt.figure(figsize=(7,5))
sns.boxplot(x='Embarked',y='Age',data=df_train)
```


```python
def add_age(cols):
    Age = cols[0]
    Pclass = cols[1]
    if pd.isnull(Age):
        return int(df_train[df_train["Pclass"] == Pclass]["Age"].mean())
    else:
        return Age
```


```python
df_train['Age'] = df_train[['Age', 'Pclass']].apply(add_age,axis=1)
```


```python
df_train.count()
```




    PassengerId    891
    Survived       891
    Pclass         891
    Sex            891
    Age            891
    SibSp          891
    Parch          891
    Fare           891
    Embarked       889
    dtype: int64



### Drop Rows


```python
df_train.dropna(inplace=True)
```


```python
df_train.count()
```




    PassengerId    889
    Survived       889
    Pclass         889
    Sex            889
    Age            889
    SibSp          889
    Parch          889
    Fare           889
    Embarked       889
    dtype: int64



## Categorical to Numerical

#### Convert the categorical values to numeric
- Find the columns that are explicitly categorical - like male, female
- Find the columns that are although numerical, represent categorical features

### One-Hot Encoding
- A technique to create multiple feature for each corrsponding value




```python
import numpy as np
col = 'Sex'
print(np.unique(df_train[col]))
```

    ['female' 'male']



```python
import numpy as np
col = 'Embarked'
print(np.unique(df_train[col]))
```

    ['C' 'Q' 'S']



```python
import numpy as np
col = 'Pclass'
print(np.unique(df_train[col]))
```

    [1 2 3]


#### pd.get_dummies()


```python
# sex = pd.get_dummies(df_train["Sex"],drop_first=True)
# embarked = pd.get_dummies(df_train["Embarked"],drop_first=True)
# pclass = pd.get_dummies(df_train["Pclass"],drop_first=True)

sex = pd.get_dummies(df_train["Sex"])
embarked = pd.get_dummies(df_train["Embarked"])
pclass = pd.get_dummies(df_train["Pclass"])
```

### Drop the columns that were used for transformation


```python
df_train.drop(['Sex', 'Embarked', 'Pclass', 'PassengerId'], axis=1, inplace=True)
```


```python
df_train.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Survived</th>
      <th>Age</th>
      <th>SibSp</th>
      <th>Parch</th>
      <th>Fare</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>22.0</td>
      <td>1</td>
      <td>0</td>
      <td>7.2500</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>38.0</td>
      <td>1</td>
      <td>0</td>
      <td>71.2833</td>
    </tr>
    <tr>
      <th>2</th>
      <td>1</td>
      <td>26.0</td>
      <td>0</td>
      <td>0</td>
      <td>7.9250</td>
    </tr>
    <tr>
      <th>3</th>
      <td>1</td>
      <td>35.0</td>
      <td>1</td>
      <td>0</td>
      <td>53.1000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0</td>
      <td>35.0</td>
      <td>0</td>
      <td>0</td>
      <td>8.0500</td>
    </tr>
  </tbody>
</table>
</div>



### Add encoded columns to the training dataset


```python
df_train = pd.concat([df_train,pclass,sex,embarked],axis=1)
```


```python
df_train.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Survived</th>
      <th>Age</th>
      <th>SibSp</th>
      <th>Parch</th>
      <th>Fare</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>female</th>
      <th>male</th>
      <th>C</th>
      <th>Q</th>
      <th>S</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>22.0</td>
      <td>1</td>
      <td>0</td>
      <td>7.2500</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>38.0</td>
      <td>1</td>
      <td>0</td>
      <td>71.2833</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>1</td>
      <td>26.0</td>
      <td>0</td>
      <td>0</td>
      <td>7.9250</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>1</td>
      <td>35.0</td>
      <td>1</td>
      <td>0</td>
      <td>53.1000</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0</td>
      <td>35.0</td>
      <td>0</td>
      <td>0</td>
      <td>8.0500</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>



# Save the transformed file as a pickle file


```python
df_train.shape
```




    (889, 13)




```python
import pickle as pkl
df_train.to_pickle('../data/titanic_tansformed.pkl')
```

## Logistic Regression


```python
data = df_train.drop("Survived",axis=1)
label = df_train["Survived"]
```


```python
data.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Age</th>
      <th>SibSp</th>
      <th>Parch</th>
      <th>Fare</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>female</th>
      <th>male</th>
      <th>C</th>
      <th>Q</th>
      <th>S</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>22.0</td>
      <td>1</td>
      <td>0</td>
      <td>7.2500</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>38.0</td>
      <td>1</td>
      <td>0</td>
      <td>71.2833</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>26.0</td>
      <td>0</td>
      <td>0</td>
      <td>7.9250</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>35.0</td>
      <td>1</td>
      <td>0</td>
      <td>53.1000</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>35.0</td>
      <td>0</td>
      <td>0</td>
      <td>8.0500</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>




```python
from sklearn.cross_validation import train_test_split
data_train, data_test, label_train, label_test = train_test_split(data, label, test_size = 0.3, random_state = 101)
```

    /Users/talat/anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
      "This module will be removed in 0.20.", DeprecationWarning)



```python
from sklearn.linear_model import LogisticRegression

# Run Logistic Regression
log_regr = LogisticRegression()
log_regr.fit(data_train, label_train)
predictions = log_regr.predict(data_test)
```

### Accuracy


```python
print('Accuracy', log_regr.score(data_test, label_test))
print('Coefficients', log_regr.coef_)
print('Intercept', log_regr.intercept_)
```

    Accuracy 0.8277153558052435
    Coefficients [[-0.03855005 -0.24328718 -0.10076536  0.00231142  1.18429317  0.2997331
      -0.8678696   1.56887514 -0.95271847  0.39520621  0.20701836  0.0139321 ]]
    Intercept [0.61615666]


### Precision Recall


```python
from sklearn.metrics import classification_report
print(classification_report(label_test, predictions))
```

                 precision    recall  f1-score   support
    
              0       0.82      0.92      0.87       163
              1       0.85      0.68      0.76       104
    
    avg / total       0.83      0.83      0.82       267
    


## Cross Validation


```python
from sklearn.model_selection import StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
# skf = StratifiedKFold(n_splits=5)

log_regr = LogisticRegression()
log_regr.fit(data_train, label_train)
score = log_regr.score(data_train, label_train)
print('Train accuracy score', score)


score_cv = cross_val_score(log_regr, data_train, label_train, cv=10, scoring='accuracy')
print('Cross Val Accuracy for each run', score_cv)
print('CrossVal Accuracy', score_cv.mean())

```

    Train accuracy score 0.8086816720257235
    Cross Val Accuracy for each run [0.76190476 0.6984127  0.79365079 0.87301587 0.80952381 0.77777778
     0.78688525 0.81967213 0.91803279 0.7704918 ]
    CrossVal Accuracy 0.8009367681498828


## AUC - Receiver Operating Characteristics
- How much a model is capable of distinguishing between classes
- Higher the AUC, better the model is

$
\begin{align}
True Positive Rate = \frac{TP}{TP + FN}
\end{align}
$

<br>
$
\begin{align}
\ False Positive Rate = 1 - \frac{TN}{TN + FP} = \frac{FP}{TN + FP}
\end{align}
$


```python
from sklearn import metrics

fpr, tpr, threshold = metrics.roc_curve(label_test, log_regr.predict(data_test))
roc_auc = metrics.auc(fpr, tpr)
print('AUCROC Stage1 vs Healthy: ' , roc_auc)
```

    AUCROC Stage1 vs Healthy:  0.8014688532326569



```python
import matplotlib.pyplot as plt

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()
```


```python

```

cd8438a471ec419fb1ff2f5b36aacce015db6ceb
ipynb
Jupyter Notebook
classification/notebooks/04 - Data Transformation.ipynb
pshn111/Machine-Learning-Package
fbbaa44daf5f0701ea77e5b62eb57ef822e40ab2
MIT
classification/notebooks/04 - Data Transformation.ipynb
pshn111/Machine-Learning-Package
fbbaa44daf5f0701ea77e5b62eb57ef822e40ab2
MIT
classification/notebooks/04 - Data Transformation.ipynb
pshn111/Machine-Learning-Package
fbbaa44daf5f0701ea77e5b62eb57ef822e40ab2
MIT
Qwen/Qwen-72B
1. YES
2. YES