<jupyter_start><jupyter_text>Seminar 1. Machine learning on Titanic dataThe notebook provides an intro to the exploratory analysis of the data, data preprocessing and application of machine learning methods.The notebook is based on kaggle kernel "Titanic: Machine Learning from Disaster"  https://www.kaggle.com/omarelgabry/a-journey-through-titanic by <NAME>.Data is from a toy competition on kaggle named Titanic.The goal of the competition is to predict who survived and who died during the sinking of the RMS Titanic.  Documentation to go through:* https://docs.python.org/3/* https://pandas.pydata.org/docs* https://matplotlib.org/contents.html* https://docs.scipy.org/doc/* http://scikit-learn.org/stable/documentation.html  Some additional info:* http://www.scipy-lectures.org/* https://www.kaggle.com/* https://pydata.org/<jupyter_code># importing data processing tools: pandas and numpy
import numpy as np
import pandas as pd
from pandas import Series, DataFrame<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code># get titanic files as a DataFrame
titanic_dataframe = pd.read_csv("https://raw.githubusercontent.com/adasegroup/ML2022_seminars/master/seminar1/titanic/train.csv", index_col='PassengerId')<jupyter_output><empty_output><jupyter_text>Look through the data<jupyter_code># preview the data
titanic_dataframe.head()
# list the features
print(titanic_dataframe.keys())
# column selection by name
titanic_dataframe['Age']
# row selection by id
titanic_dataframe.loc[1]
# column selection by index
titanic_dataframe.iloc[:, 0]
# row selection by index
titanic_dataframe.iloc[0, :]<jupyter_output><empty_output><jupyter_text>Hints and tipsYou can use ```%time``` or ```tqdm``` to track the code timing.Note that ```pandas``` is column oriented data structure.<jupyter_code>%time titanic_dataframe['Fare'].mean()
data_titanic_transpose = titanic_dataframe.T
%time data_titanic_transpose.loc['Fare'].mean()
from tqdm import tqdm
for i in tqdm(range(100000000)):
    pass<jupyter_output>100%|████████████████████████| 100000000/100000000 [00:14<00:00, 6700120.11it/s]<jupyter_text>Data Dictionary| Variable | Definition | Key || ------------- |:-------------|: -----|| survival | Survival   | 0 = No, 1 = Yes | | pclass   | Ticket class | 1 = 1st, 2 = 2nd, 3 = 3rd || sex | Sex | || Age | Age in years | || sibsp |  of siblings / spouses aboard the Titanic | || parch |  of parents / children aboard the Titanic | || ticket | Ticket number  | || fare   | Passenger fare | || cabin  | Cabin number | || embarked | Port of Embarkation | C = Cherbourg, Q = Queenstown, S = Southampton |<jupyter_code>titanic_dataframe.info()
titanic_dataframe.describe()<jupyter_output><empty_output><jupyter_text>Hints and tipsWrite ```?``` after the function you are interested in or just press ``` Shift + Tab```  for the function short referense. Double ``` Shift + Tab``` will expand to the full reference.<jupyter_code># call information for the function 
titanic_dataframe.drop?
# drop unnecessary columns, these columns won't be useful in analysis and prediction
titanic_dataframe.drop(['Name','Ticket'], axis=1, inplace=True)
titanic_dataframe.info()
for column_name in titanic_dataframe.columns:
    print(column_name, 'null', titanic_dataframe[column_name].isnull().sum())

# It has a lot of NaN values, so it won't cause a remarkable impact on prediction
titanic_dataframe.drop("Cabin", axis=1, inplace=True)
# Count various embarked values
print(titanic_dataframe["Embarked"].value_counts())

# Fill the two missing values with the most occurred value, which is "S".
titanic_dataframe["Embarked"] = titanic_dataframe["Embarked"].fillna("S")
print(titanic_dataframe["Embarked"].value_counts())
# Groupby 
titanic_dataframe.groupby("Survived").count()<jupyter_output><empty_output><jupyter_text>Tasks:1. What is the mean value and stds of ages for every passanger class?2. In what port of embarked the absolute difference between the amount men and women was the greatest?3. What is a number of NaN values in every column?4. Replace NaN values in age with median value and calculate the std value.<jupyter_code>titanic_dataframe.groupby("Pclass").Age.mean()
titanic_dataframe.groupby("Pclass").Age.std()
titanic_dataframe.groupby(["Embarked", "Sex"]).count()
titanic_dataframe.isnull().sum()
median_age = titanic_dataframe["Age"].median()
median_age
titanic_dataframe["Age"] = titanic_dataframe["Age"].fillna(median_age)
titanic_dataframe.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 891 entries, 1 to 891
Data columns (total 8 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Survived  891 non-null    int64  
 1   Pclass    891 non-null    int64  
 2   Sex       891 non-null    object 
 3   Age       891 non-null    float64
 4   SibSp     891 non-null    int64  
 5   Parch     891 non-null    int64  
 6   Fare      891 non-null    float64
 7   Embarked  891 non-null    object 
dtypes: float64(2), int64(4), object(2)
memory usage: 94.9+ KB<jupyter_text>Plotting<jupyter_code># visualization tools: matplotlib, seaborn
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('whitegrid')
%matplotlib inline
# Simple plot
x = titanic_dataframe['Age']
y = titanic_dataframe['Fare']
plt.plot(x, y, 'o')
plt.xlabel('Age')
plt.ylabel('Fare')<jupyter_output><empty_output><jupyter_text>![matplotlib_figure_axes_axis.png](attachment:matplotlib_figure_axes_axis.png)<jupyter_code># Catplot plot represents share of survived passengers for different embarkment ports
sns.catplot?
sns.catplot(x = 'Embarked', y = 'Survived', data=titanic_dataframe, height=4, aspect=3, kind = 'point')

figure_handle, (axis1, axis2, axis3) = plt.subplots(1, 3, figsize=(15, 5))

sns.countplot(x='Embarked', data=titanic_dataframe, ax=axis1)
sns.countplot(x='Survived', hue="Embarked", data=titanic_dataframe, order=[1, 0], ax=axis2)

# group by embarked, and get the mean for survived passengers for each value in Embarked
sns.barplot(x='Embarked', y='Survived', data=titanic_dataframe[["Embarked", "Survived"]], order=['S','C','Q'], ax=axis3)<jupyter_output><empty_output><jupyter_text>![1_3hdYEX5eixaV4F3wT5OmBg.png](attachment:1_3hdYEX5eixaV4F3wT5OmBg.png)<jupyter_code># consider Embarked column in predictions,
# and remove "S" dummy variable, 
# and leave "C" & "Q", since they seem to have a good rate for Survival.

# OR, don't create dummy variables for Embarked column, just drop it, 
# because logically, Embarked doesn't seem to be useful in prediction.

embark_dummies_titanic  = pd.get_dummies(titanic_dataframe['Embarked'])
embark_dummies_titanic
embark_dummies_titanic.drop(['S'], axis=1, inplace=True)

titanic_dataframe = titanic_dataframe.join(embark_dummies_titanic)

titanic_dataframe.drop(['Embarked'], axis=1, inplace=True)
# Examine fare variable

# convert from float to int
titanic_dataframe['Fare'] = titanic_dataframe['Fare'].astype(int)

# get fare for survived & didn't survive passengers 
fare_not_survived = titanic_dataframe["Fare"][titanic_dataframe["Survived"] == 0]
fare_survived = titanic_dataframe["Fare"][titanic_dataframe["Survived"] == 1]

# get average and std for fare of survived/not survived passengers
avgerage_fare = DataFrame([fare_not_survived.mean(), fare_survived.mean()])
std_fare = DataFrame([fare_not_survived.std(), fare_survived.std()])

# plot
titanic_dataframe['Fare'].plot(kind='hist', figsize=(15, 3), bins=100, xlim=(0, 50))

std_fare.index.names = ["Survived"]
avgerage_fare.index.names = ["Survived"]
avgerage_fare.plot(yerr=std_fare, kind='bar', legend=False)
# Do the same thing for pclass variable with no confidence interval visible
print(titanic_dataframe[["Pclass", "Survived"]].groupby(['Pclass'], as_index=True))
# adjust the figure size
plt.figure(figsize=[10,5])
sns.barplot(x='Pclass', y='Survived', data=titanic_dataframe[["Pclass", "Survived"]], order=[1, 2, 3])
# Age 
fig, (axis1, axis2) = plt.subplots(1, 2, figsize=(15, 4))
axis1.set_title('Original Age values - Titanic')
axis2.set_title('New Age values - Titanic')

# get average, std, and number of NaN values in titanic_df
average_age_titanic = titanic_dataframe["Age"].mean()
std_age_titanic = titanic_dataframe["Age"].std()
count_nan_age_titanic = titanic_dataframe["Age"].isnull().sum()

# generate random numbers between (mean - std) & (mean + std)
random_ages = np.random.randint(average_age_titanic - std_age_titanic, 
                                average_age_titanic + std_age_titanic, 
                                size=count_nan_age_titanic)

# plot original Age values
# NOTE: drop all null values, and convert to int
titanic_dataframe['Age'].dropna().astype(int).hist(bins=70, ax=axis1)

# fill NaN values in Age column with random values generated
titanic_dataframe.loc[np.isnan(titanic_dataframe["Age"]), "Age"] = random_ages

# convert from float to int
titanic_dataframe['Age'] = titanic_dataframe['Age'].astype(int)

# plot new Age Values
titanic_dataframe['Age'].hist(bins=70, ax=axis2)
# .... continue with plotting of Age column

# peaks for survived/not survived passengers by their age
facet = sns.FacetGrid(titanic_dataframe, hue="Survived", aspect=3)
facet.map(sns.kdeplot, 'Age', shade=True)
facet.set(xlim=(0, titanic_dataframe['Age'].max()))
facet.add_legend()

# average survived passengers by age
figure_handle, axis1 = plt.subplots(1, 1, figsize=(18, 4))
average_age = titanic_dataframe[["Age", "Survived"]].groupby(['Age'], as_index=False).mean()
sns.barplot(x='Age', y='Survived', data=average_age)
# Instead of having two columns Parch & SibSp, 
# we can have only one column that represents if a passenger had any family member aboard or not,
# Meaning, if having any family member (whether parent, brother, ...etc) increases chances of Survival or not.
titanic_dataframe['Family'] = titanic_dataframe["Parch"] + titanic_dataframe["SibSp"]
titanic_dataframe.loc[titanic_dataframe['Family'] > 0, 'Family'] = 1
titanic_dataframe.loc[titanic_dataframe['Family'] == 0, 'Family'] = 0

# drop Parch & SibSp
titanic_dataframe.drop(['SibSp','Parch'], axis=1, inplace=True)
# plot Family
figure_handle, (axis1, axis2) = plt.subplots(1, 2, sharex=True, figsize=(10, 5))

sns.countplot(x='Family', data=titanic_dataframe, order=[1, 0], ax=axis1)
axis1.set_xticklabels(["With Family", "Alone"], rotation=0)

# average of survived for those who had/didn't have any family member
sns.barplot(x='Family', y='Survived', data=titanic_dataframe[["Family", "Survived"]], order=[1, 0], ax=axis2)
# Sex variable

# As we see, children(age < ~16) on aboard seem to have a high chances for Survival.
# So, we can classify passengers as males, females, and child
def get_person(passenger):
    age, sex = passenger
    return 'child' if age < 16 else sex

titanic_dataframe['Person'] = titanic_dataframe[['Age','Sex']].apply(get_person, axis=1)

# No need to use Sex column since we created Person column
#titanic_dataframe.drop(['Sex'], axis=1, inplace=True)

# create dummy variables for Person column, & drop Male as it has the lowest average of survived passengers
person_dummies_titanic = pd.get_dummies(titanic_dataframe['Person'])
person_dummies_titanic.columns = ['Child', 'Female', 'Male']
person_dummies_titanic.drop(['Male'], axis=1, inplace=True)

titanic_dataframe = titanic_dataframe.join(person_dummies_titanic)

figure_handle, (axis1, axis2) = plt.subplots(1, 2, figsize=(10, 5))

sns.countplot(x='Person', data=titanic_dataframe, ax=axis1)

# average of survived for each Person(male, female, or child)
sns.barplot(x='Person', y='Survived', data=titanic_dataframe[["Person", "Survived"]], 
            ax=axis2, order=['male', 'female', 'child'])

# we don't need person variable after introduction of the corresponding dummy variables
titanic_dataframe.drop(['Person'], axis=1, inplace=True)
# Pclass
sns.catplot('Pclass', 'Survived', order=[1, 2, 3], data=titanic_dataframe, height=5, kind = 'point')

# The goal is to create dummy variables for class and joint it to the initial dataframe
# create dummy variables for Pclass column, & drop 3rd class as it has the lowest average of survived passengers
pclass_dummies_titanic  = pd.get_dummies(titanic_dataframe['Pclass'])
pclass_dummies_titanic.columns = ['Class_1', 'Class_2', 'Class_3']
pclass_dummies_titanic.drop(['Class_3'], axis=1, inplace=True)

titanic_dataframe = titanic_dataframe.join(pclass_dummies_titanic)<jupyter_output>/home/aaptss/.local/lib/python3.8/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(<jupyter_text>Task1. Is distribution of age similar for men and women?2. Compare Age distribution for all three classes.<jupyter_code>gender_titanic1 = titanic_dataframe[titanic_dataframe.Sex == "male"]
gender_titanic2 = titanic_dataframe[titanic_dataframe.Sex == "female"]

fig, (axis1, axis2) = plt.subplots(1, 2, figsize=(15, 4))

axis1.set_title('Age histogram - Titanic, male')
axis2.set_title('Age histogram - Titanic, female')

gender_titanic1["Age"].hist(ax = axis1, bins = 50)
gender_titanic2["Age"].hist(ax = axis2, bins = 50)
cls_titanic1 = titanic_dataframe.loc[(titanic_dataframe.Class_1 == 1) & (titanic_dataframe.Class_2 == 0)]
cls_titanic2 = titanic_dataframe.loc[(titanic_dataframe.Class_2 == 1) & (titanic_dataframe.Class_1 == 0)]
cls_titanic3 = titanic_dataframe.loc[(titanic_dataframe.Class_1 == 0) & (titanic_dataframe.Class_2 == 0)]

fig, (axis1, axis2, axis3) = plt.subplots(1, 3, figsize=(15, 4))

axis1.set_title('Age histogram - Titanic, Class1')
axis2.set_title('Age histogram - Titanic, Class2')
axis3.set_title('Age histogram - Titanic, Class3')

cls_titanic1["Age"].hist(ax = axis1, bins = 50)
cls_titanic2["Age"].hist(ax = axis2, bins = 50)
cls_titanic3["Age"].hist(ax = axis3, bins = 50)<jupyter_output><empty_output><jupyter_text>It's time for Machine learning! ![MLearning](https://media.giphy.com/media/BdrSy2gqURFEk/giphy.gif)<jupyter_code># machine learning tools: various methods from scikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC, LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
titanic_dataframe.head()
titanic_dataframe.drop('Person', axis=1, inplace=True)
# titanic_dataframe.drop("Person")
titanic_dataframe.head()
train, test = train_test_split(titanic_dataframe, train_size=0.5, test_size=0.5)
train_x = train.drop(['Survived'], axis=1)
train_y = train['Survived']
test_x = test.drop(['Survived'], axis=1)
test_y = test['Survived']
# Logistic Regression

logistic_regression_model = LogisticRegression(solver='liblinear')
logistic_regression_model.fit(train_x, train_y)
train_prediction = logistic_regression_model.predict(train_x)
test_prediction = logistic_regression_model.predict(test_x)
train_accuracy = accuracy_score(train_y, train_prediction)
test_accuracy = accuracy_score(test_y, test_prediction)

print('Train Accuracy:', train_accuracy)
print('Test Accuracy:', test_accuracy)
# get Correlation Coefficient for each feature using Logistic Regression
coeff_df = DataFrame(titanic_dataframe.columns.delete(0))
coeff_df.columns = ['Features']
coeff_df["Coefficient Estimate"] = pd.Series(logistic_regression_model.coef_[0])

# preview
coeff_df
# Support Vector Machines

svm_model = SVC(C=1.0, gamma=0.5)
svm_model.fit(train_x, train_y)
train_prediction = svm_model.predict(train_x)
test_prediction = svm_model.predict(test_x)
train_accuracy = accuracy_score(train_y, train_prediction)
test_accuracy = accuracy_score(test_y, test_prediction)

print('Train Accuracy:', train_accuracy)
print('Test Accuracy:', test_accuracy)
# Random Forests

random_forest_model = RandomForestClassifier(n_estimators=10)
random_forest_model.fit(train_x, train_y)
train_prediction = random_forest_model.predict(train_x)
test_prediction = random_forest_model.predict(test_x)
train_accuracy = accuracy_score(train_y, train_prediction)
test_accuracy = accuracy_score(test_y, test_prediction)

print('Train Accuracy:', train_accuracy)
print('Test Accuracy:', test_accuracy)
# K nearest neighbours 

knn_model = KNeighborsClassifier(n_neighbors=1)
knn_model.fit(train_x, train_y)
train_prediction = knn_model.predict(train_x)
test_prediction = knn_model.predict(test_x)
train_accuracy = accuracy_score(train_y, train_prediction)
test_accuracy = accuracy_score(test_y, test_prediction)

print('Train Accuracy:', train_accuracy)
print('Test Accuracy:', test_accuracy)<jupyter_output>Train Accuracy: 0.9797752808988764
Test Accuracy: 0.6771300448430493