<reponame>Murdism/Covid_Tracking-Linear-polynomial-
# -*- coding: utf-8 -*-
"""Assignment_1_Machine Learning.ipynb

Automatically generated by Colaboratory.

Date: 11-02-2021

# Tasks
Covid data: https://covidtracking.com/
1. Consider as training data set the first 70% of the available data
2. Perform linear or polynomial regression until you are happy with the results of the training data.
3. Introduce regularization and redo the step 2 and provide a short discussion of the improvements vs. performance.
4. Evaluate how your hypothesis is performing on the prediction using the remaining 30% of the data (test set).
5. Provide short discussion of the results and your insights.
 The challenging parts of this homework are how to quantify the short discussion of the improvements vs. performance in the Step 3, and to quantitativelly evaluate the performance in Step 4. (You're allowed to search for the appropriate way to handle these issues.)

# Libraries
"""

#Import all the necessary libraries
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score
#Seaborn is used to pairplot 
import seaborn as sns
import matplotlib.pyplot as plt

from google.colab import drive
drive.mount('/content/drive')

"""# Code

"""

#Data Input ( Source: " https://covidtracking.com/data/national" ) # 08-Feb-2021
mydata = pd.read_csv("/content/drive/MyDrive/ML Excercises/national-history-latest.csv")
#Print Info
print(mydata.info())

"""To select specific columns

mydata = mydata(1:3)  
mydataframe=pd.DataFrame(mydata,columns=["totalTestResultsIncrease","positiveIncrease","states"])

"""

# Data Cleaning
# Lets take data where covid tests are taken in all states , i.e 56 states. # To make sure there is conssistency in the data
covidStat= mydata [mydata.states == 56]
# "Drop NA" can be used to remove null data entries .."how= "all" " drops a row if all are Nan but "how =" Any"  "drops it if one is NA
# inplace= True gives equal size columns
covidStat.dropna(inplace=True)
print(covidStat.info())

# select the numeric features to compare them and see if there is any correlation
data_numeric = covidStat.select_dtypes(include=['float64', 'int64'])
plt.figure(figsize=(20, 10))
#sns.pairplot(data_numeric)
sns.heatmap(data_numeric.corr(),annot = True,vmin=-1, vmax=1, center= 0,cmap= 'coolwarm') # ,fmt='.1g,' cmap= 'magma'
plt.show()

# Based on the results of the correlation graph above ,
# In this assignement i will predict the number of deaths based on the following features:
   #1: hospitalizedIncrease 
   #2:  onVentilatorCurrently
   #3: positiveIncrease   
   # 4: number of tests increase
covid_selected_data= mydata [mydata.states == 56]
covid_selected_data=pd.DataFrame(covidStat,columns=["deathIncrease","onVentilatorCurrently","hospitalizedIncrease","totalTestResultsIncrease","positiveIncrease"])
covid_selected_data.dropna(inplace=True)
print(covid_selected_data.info())

y = covid_selected_data['deathIncrease'].values.reshape(-1,1)
x = covid_selected_data[['totalTestResultsIncrease','onVentilatorCurrently','hospitalizedIncrease','positiveIncrease']].values.reshape(-1,4)
x_one_feature = covid_selected_data[['positiveIncrease']]
x_train,x_test,y_train,y_test= train_test_split(x,y,test_size=0.30)
x_one_feature_train,x_one_feature_test,y_one_feature_train,y_one_feature_test= train_test_split(x_one_feature,y,test_size=0.30)
print("Shape of training data")
print("x_train : %s  y_train: %s " % (x_train.shape, y_train.shape))

# multivariate model and prediciton ---> features:"Postive Increase"  output: "number of deaths" 
prediction_model= LinearRegression(normalize=True)#copy_x,fit_intercept ,n_jobs ,normalize 
prediction_model.fit(x_train,y_train) 
prediction_train=prediction_model.predict(x_train)

# Univarient model and prediciton ---> feature:'totalTestResultsIncrease','onVentilatorCurrently','hospitalizedIncrease','positiveIncrease'  output: "number of deaths" 
prediction_one_feature_model = LinearRegression(normalize=True).fit(x_one_feature_train,y_one_feature_train) 
prediction_one_feature_train= prediction_one_feature_model.predict(x_one_feature_train)
#prediction on test
prediction_one_feature_test=prediction_one_feature_model.predict(x_one_feature_test)

#This is to see what would happen if it was trained using all the available traininhg data
prediction_model_for_all= LinearRegression(normalize=True)#copy_x,fit_intercept ,n_jobs ,normalize 
prediction_model_for_all.fit(x,y) 
prediciton_all=prediction_model_for_all.predict(x)
error_mse_all=mean_squared_error(y,prediciton_all)
root_error_mse_all=np.sqrt(error_mse_all)
print("square root of MSE: %s" % root_error_mse_all)

#calculate error
# univarient error on tested on training data
error_mse_one_feature_train=mean_squared_error(y_one_feature_train,prediction_one_feature_train)
root_error_mse_0ne_feature_train=np.sqrt(error_mse_one_feature_train)
# univarient error on tested on testing data
error_mse_one_feature_test=mean_squared_error(y_one_feature_test,prediction_one_feature_test)
root_error_mse_one_feature_test=np.sqrt(error_mse_one_feature_test)
print("Root Error of MSE on univarient model:  ")
print("Error on training data = %s " % root_error_mse_0ne_feature_train)
print("Error on testing data = %s " % root_error_mse_one_feature_test)
score_one_univarient= r2_score(y_one_feature_train, prediction_one_feature_train)
print("R2 Score on training data = %s "%(score_one_univarient))
score_one_univarient= r2_score(y_one_feature_test, prediction_one_feature_test)
print("R2 Score on testing data = %s "%(score_one_univarient))

print("The linear model is: Y = {:.5} + {:.5}X".format(prediction_one_feature_model.intercept_[0], prediction_one_feature_model.coef_[0][0]))
plt.figure(figsize=(16, 8))
plt.scatter(
    x_one_feature_train,
    y_one_feature_train,
    c='black'
)
plt.plot(
    x_one_feature_train,
    prediction_one_feature_train,
    c='blue',
    linewidth=2
)
plt.xlabel("Tests")
plt.ylabel("Postive Cases")
plt.show()

# Polynomial regression on univarient model 
poly_one_feature= PolynomialFeatures(degree=2)
poly_one_feature.fit(x_one_feature_train,y_one_feature_train)
x_poly_one_feature=poly_one_feature.fit_transform(x_one_feature_train)
lin2=LinearRegression()
lin2.fit(x_poly_one_feature, y_one_feature_train)
x_poly_test_one_feature=poly_one_feature.fit_transform(x_one_feature_test)
pred_one_feature_poly = lin2.predict(x_poly_test_one_feature)
error_mse_one_feature_poly=mean_squared_error(y_one_feature_test,pred_one_feature_poly)
root_error_one_feature_poly=np.sqrt(error_mse_one_feature_poly)
print("Root Error of MSE on univarient model ->Polynomial degree 2:  ")
print("Error on testing data = %s " % root_error_one_feature_poly)
score_one= r2_score(y_one_feature_test, pred_one_feature_poly)
print("R2 Score on testing data = %s "%(score_one))

# Linear  regression on multivariate input
prediction_test=prediction_model.predict(x_test)
#calculate error
error_mse_train=mean_squared_error(y_train,prediction_train)
root_error_mse_train=np.sqrt(error_mse_train)
error_mse_test=mean_squared_error(y_test,prediction_test)
root_error_mse_test=np.sqrt(error_mse_test)
print("Root Error of MSE on  multivariate linear model:  ")
print("Error on traing data = %s " %root_error_mse_train)
print("Error on testing data = %s " %root_error_mse_test)
score_linear_multivariate= r2_score(y_test, prediction_test)
print("R2 Score on testing data = %s "%(score_linear_multivariate))
score_linear_multivariate_training= r2_score(y_train, prediction_train)
print("R2 Score on training data = %s "%(score_linear_multivariate_training))
# print(y_test,prediction_test)

# Polynomial regression on multivariate input
def polynomial_regression_model(degree):
      poly= PolynomialFeatures(degree=degree)
      poly.fit(x_train,y_train)
      x_poly=poly.fit_transform(x_train)
      line3=LinearRegression()
      line3.fit(x_poly, y_train)
      x_poly_test=poly.fit_transform(x_test)
      x_poly_train=poly.fit_transform(x_train)
      pred_multivariant_poly = line3.predict(x_poly_test)
      pred_multivariant_poly_train = line3.predict(x_poly_train)
      error_mse_multivariant_poly=mean_squared_error(y_test,pred_multivariant_poly)
      root_error_multivariant_poly=np.sqrt(error_mse_multivariant_poly)
      print("Root Error of MSE on  multivariate Polynomial degree %s model:  " %degree)
      print("Error on testing data = %s " %root_error_multivariant_poly)
      score_multivariate_poly_train = r2_score(y_train, pred_multivariant_poly_train)
      print("R2 Score on training data = %s "%score_multivariate_poly_train)
      score_multivariate_poly = r2_score(y_test, pred_multivariant_poly)
      print("R2 Score on testing data = %s "%score_multivariate_poly)
      print("/////////////////////////////////////////////////////////////////////////////////////")

for degree in range (2,6):
      polynomial_regression_model(degree)

"""After checking and comapring different models and number of features to predict the number of deaths, finally, a polynomial of degree 3 model with the following features: 
 #1: hospitalizedIncrease 
 #2:  onVentilatorCurrently
 #3: positiveIncrease   
 #4: number of tests increase

 was selected as the final model. This is the best model because it is moderatly accurate but has overfitting problem. In this modeltraining score =82% while test score= 56%. The noticeable difference in trainng and testing score could be the sign of overfitting. But with the help of regularization this could be improved therefore this  model is selected.

"""

# Polynomial regression on multivariate input
from sklearn.linear_model import Ridge
def RigeModel(alpha,pdegree):
      poly= PolynomialFeatures(degree=pdegree)
      poly.fit(x_train,y_train)
      x_poly=poly.fit_transform(x_train)
      line3=Ridge(alpha=alpha,normalize=True)
      line3.fit(x_poly, y_train)
      x_poly_test=poly.fit_transform(x_test)
      pred_multivariant_poly = line3.predict(x_poly_test)
      error_mse_multivariant_poly=mean_squared_error(y_test,pred_multivariant_poly)
      root_error_multivariant_poly=np.sqrt(error_mse_multivariant_poly)
      #print("For Alpha= ",alpha)
      #print("Root Error of MSE on  multivariate Polynomial degree 2 model:  ")
      #print("Error on testing data = %s " %root_error_multivariant_poly)
      score_multivariate_poly = r2_score(y_test, pred_multivariant_poly)
     # print("R2 Score on testing data = %s    alpha= %s"%(score_multivariate_poly,alpha))
      return score_multivariate_poly  , root_error_multivariant_poly   
best_ridge= 0.0
best_alpha= 0.001
b_error=0
for alpha in range (1,1000): #a=0.032
     acuridge,err=RigeModel(alpha/1000,2)
     if acuridge > best_ridge:
          best_ridge=acuridge
          best_alpha=alpha/1000
          b_error=err
          
print ("Best (test)accuracy For degree 2: %s       Alpha: %s     Square Root of MSE: %s " %((best_ridge*100),best_alpha,b_error))
best_ridge_d3= 0.0
best_alpha_d3= 0.001
bb_error=0
for alpha_s in range (1,1000): #a=0.032
     acuridges,b_err=RigeModel((alpha_s/1000),3)
     if acuridges > best_ridge_d3:
          best_ridge_d3=acuridges
          best_alpha_d3=alpha_s/1000
          bb_error=b_err

print ("Best (test) accuracy For degree 3: %s       Alpha: %s     Square Root of MSE: %s " %((best_ridge_d3*100),best_alpha_d3,  bb_error))

"""After using regulariazation on the degree 3 polynomial regression model, the test score improved from 63% to 80.2%. This is very drastic improvement to the model.

3. Introduce regularization and redo the step 2 and provide a short discussion of the improvements vs. performance.

**Ans**: Before using regularization, a polynomial of order 3 model was used to determine the number of deaths based on the number of hospitalizedIncrease, onVentilatorCurrently,positiveIncrease,and number of tests increase.
Based on the score of the model and the error it can be seen that there is a noticable difference when the model is predicting a training data and testing data. 
This huge difference might arise from the fact that the training data is not enough to the system and it is overfitting. This means the model has high variance and is not able to generalize. 
Regularization is one solution to such system. By using L2 regularization the score of predicting new (test data) data can be improved. 
This is done by adding extra  penality to  features (input features) and reducing their effect on the model, a process called Regualarization. 
This  lowers the power of each penalized feature to affect the output of the model.  After using L2 regularization the overall test data prediction score should be increased and the mean squared error (MSE) should be decreased. 



4. Evaluate how your hypothesis is performing on the prediction using the remaining 30% of the data (test set).

**Ans**: As expected, using L2 regularization improved the model score and error rate on the test dataset. 
The training data prediction score (Coefficient of determination) was 82% and the testing data prediction score was 56%. 
Similarly the sqaure root of MSE was 641.8. After using ridge regularization or L2 regularization the score on the testing data improved from 56% to 81%,
while the sqaure root of mean squared error decreased to 420.05. This is a huge improvement in the predicting power of the model.


5. Provide short discussion of the results and your insights.

**Ans**:  As explained in number 3, regularization is used to reduce overfiting and sometimes feature selection.
It penalizes the model features so as to discourage learning to some extent that allows generalization of the data. 
The additional penality is to decrease the variance of the model without removing features and adding a big bias. 
Regularization is therfore a compromise in a sense between losing important features( less variance but data loss) and overfitting (high variance in the model but important features are available).
in the first model used in this assignment, there was high varinance in the model which is evident in the difference between training data score and test data score. The Model is a polynomial of degree 3 with 4 input features. After using Regularization, however, the test dataset prediction jumped 25 points to reach 81%. This shows that the system was overfit and/or had very correlated features.
"""

# -*- coding: utf-8 -*-
"""Assignment_1_Machine Learning.ipynb

Automatically generated by Colaboratory.

Date: 11-02-2021

# Tasks
Covid data: https://covidtracking.com/
1. Consider as training data set the first 70% of the available data
2. Perform linear or polynomial regression until you are happy with the results of the training data.
3. Introduce regularization and redo the step 2 and provide a short discussion of the improvements vs. performance.
4. Evaluate how your hypothesis is performing on the prediction using the remaining 30% of the data (test set).
5. Provide short discussion of the results and your insights.
 The challenging parts of this homework are how to quantify the short discussion of the improvements vs. performance in the Step 3, and to quantitativelly evaluate the performance in Step 4. (You're allowed to search for the appropriate way to handle these issues.)

# Libraries
"""

#Import all the necessary libraries
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score
#Seaborn is used to pairplot 
import seaborn as sns
import matplotlib.pyplot as plt

from google.colab import drive
drive.mount('/content/drive')

"""# Code

"""

#Data Input ( Source: " https://covidtracking.com/data/national" ) # 08-Feb-2021
mydata = pd.read_csv("/content/drive/MyDrive/ML Excercises/national-history-latest.csv")
#Print Info
print(mydata.info())

"""To select specific columns

mydata = mydata(1:3)  
mydataframe=pd.DataFrame(mydata,columns=["totalTestResultsIncrease","positiveIncrease","states"])

"""

# Data Cleaning
# Lets take data where covid tests are taken in all states , i.e 56 states. # To make sure there is conssistency in the data
covidStat= mydata [mydata.states == 56]
# "Drop NA" can be used to remove null data entries .."how= "all" " drops a row if all are Nan but "how =" Any"  "drops it if one is NA
# inplace= True gives equal size columns
covidStat.dropna(inplace=True)
print(covidStat.info())

# select the numeric features to compare them and see if there is any correlation
data_numeric = covidStat.select_dtypes(include=['float64', 'int64'])
plt.figure(figsize=(20, 10))
#sns.pairplot(data_numeric)
sns.heatmap(data_numeric.corr(),annot = True,vmin=-1, vmax=1, center= 0,cmap= 'coolwarm') # ,fmt='.1g,' cmap= 'magma'
plt.show()

# Based on the results of the correlation graph above ,
# In this assignement i will predict the number of deaths based on the following features:
   #1: hospitalizedIncrease 
   #2:  onVentilatorCurrently
   #3: positiveIncrease   
   # 4: number of tests increase
covid_selected_data= mydata [mydata.states == 56]
covid_selected_data=pd.DataFrame(covidStat,columns=["deathIncrease","onVentilatorCurrently","hospitalizedIncrease","totalTestResultsIncrease","positiveIncrease"])
covid_selected_data.dropna(inplace=True)
print(covid_selected_data.info())

y = covid_selected_data['deathIncrease'].values.reshape(-1,1)
x = covid_selected_data[['totalTestResultsIncrease','onVentilatorCurrently','hospitalizedIncrease','positiveIncrease']].values.reshape(-1,4)
x_one_feature = covid_selected_data[['positiveIncrease']]
x_train,x_test,y_train,y_test= train_test_split(x,y,test_size=0.30)
x_one_feature_train,x_one_feature_test,y_one_feature_train,y_one_feature_test= train_test_split(x_one_feature,y,test_size=0.30)
print("Shape of training data")
print("x_train : %s  y_train: %s " % (x_train.shape, y_train.shape))

# multivariate model and prediciton ---> features:"Postive Increase"  output: "number of deaths" 
prediction_model= LinearRegression(normalize=True)#copy_x,fit_intercept ,n_jobs ,normalize 
prediction_model.fit(x_train,y_train) 
prediction_train=prediction_model.predict(x_train)

# Univarient model and prediciton ---> feature:'totalTestResultsIncrease','onVentilatorCurrently','hospitalizedIncrease','positiveIncrease'  output: "number of deaths" 
prediction_one_feature_model = LinearRegression(normalize=True).fit(x_one_feature_train,y_one_feature_train) 
prediction_one_feature_train= prediction_one_feature_model.predict(x_one_feature_train)
#prediction on test
prediction_one_feature_test=prediction_one_feature_model.predict(x_one_feature_test)

#This is to see what would happen if it was trained using all the available traininhg data
prediction_model_for_all= LinearRegression(normalize=True)#copy_x,fit_intercept ,n_jobs ,normalize 
prediction_model_for_all.fit(x,y) 
prediciton_all=prediction_model_for_all.predict(x)
error_mse_all=mean_squared_error(y,prediciton_all)
root_error_mse_all=np.sqrt(error_mse_all)
print("square root of MSE: %s" % root_error_mse_all)

#calculate error
# univarient error on tested on training data
error_mse_one_feature_train=mean_squared_error(y_one_feature_train,prediction_one_feature_train)
root_error_mse_0ne_feature_train=np.sqrt(error_mse_one_feature_train)
# univarient error on tested on testing data
error_mse_one_feature_test=mean_squared_error(y_one_feature_test,prediction_one_feature_test)
root_error_mse_one_feature_test=np.sqrt(error_mse_one_feature_test)
print("Root Error of MSE on univarient model:  ")
print("Error on training data = %s " % root_error_mse_0ne_feature_train)
print("Error on testing data = %s " % root_error_mse_one_feature_test)
score_one_univarient= r2_score(y_one_feature_train, prediction_one_feature_train)
print("R2 Score on training data = %s "%(score_one_univarient))
score_one_univarient= r2_score(y_one_feature_test, prediction_one_feature_test)
print("R2 Score on testing data = %s "%(score_one_univarient))

print("The linear model is: Y = {:.5} + {:.5}X".format(prediction_one_feature_model.intercept_[0], prediction_one_feature_model.coef_[0][0]))
plt.figure(figsize=(16, 8))
plt.scatter(
    x_one_feature_train,
    y_one_feature_train,
    c='black'
)
plt.plot(
    x_one_feature_train,
    prediction_one_feature_train,
    c='blue',
    linewidth=2
)
plt.xlabel("Tests")
plt.ylabel("Postive Cases")
plt.show()

# Polynomial regression on univarient model 
poly_one_feature= PolynomialFeatures(degree=2)
poly_one_feature.fit(x_one_feature_train,y_one_feature_train)
x_poly_one_feature=poly_one_feature.fit_transform(x_one_feature_train)
lin2=LinearRegression()
lin2.fit(x_poly_one_feature, y_one_feature_train)
x_poly_test_one_feature=poly_one_feature.fit_transform(x_one_feature_test)
pred_one_feature_poly = lin2.predict(x_poly_test_one_feature)
error_mse_one_feature_poly=mean_squared_error(y_one_feature_test,pred_one_feature_poly)
root_error_one_feature_poly=np.sqrt(error_mse_one_feature_poly)
print("Root Error of MSE on univarient model ->Polynomial degree 2:  ")
print("Error on testing data = %s " % root_error_one_feature_poly)
score_one= r2_score(y_one_feature_test, pred_one_feature_poly)
print("R2 Score on testing data = %s "%(score_one))

# Linear  regression on multivariate input
prediction_test=prediction_model.predict(x_test)
#calculate error
error_mse_train=mean_squared_error(y_train,prediction_train)
root_error_mse_train=np.sqrt(error_mse_train)
error_mse_test=mean_squared_error(y_test,prediction_test)
root_error_mse_test=np.sqrt(error_mse_test)
print("Root Error of MSE on  multivariate linear model:  ")
print("Error on traing data = %s " %root_error_mse_train)
print("Error on testing data = %s " %root_error_mse_test)
score_linear_multivariate= r2_score(y_test, prediction_test)
print("R2 Score on testing data = %s "%(score_linear_multivariate))
score_linear_multivariate_training= r2_score(y_train, prediction_train)
print("R2 Score on training data = %s "%(score_linear_multivariate_training))
# print(y_test,prediction_test)

# Polynomial regression on multivariate input
def polynomial_regression_model(degree):
      poly= PolynomialFeatures(degree=degree)
      poly.fit(x_train,y_train)
      x_poly=poly.fit_transform(x_train)
      line3=LinearRegression()
      line3.fit(x_poly, y_train)
      x_poly_test=poly.fit_transform(x_test)
      x_poly_train=poly.fit_transform(x_train)
      pred_multivariant_poly = line3.predict(x_poly_test)
      pred_multivariant_poly_train = line3.predict(x_poly_train)
      error_mse_multivariant_poly=mean_squared_error(y_test,pred_multivariant_poly)
      root_error_multivariant_poly=np.sqrt(error_mse_multivariant_poly)
      print("Root Error of MSE on  multivariate Polynomial degree %s model:  " %degree)
      print("Error on testing data = %s " %root_error_multivariant_poly)
      score_multivariate_poly_train = r2_score(y_train, pred_multivariant_poly_train)
      print("R2 Score on training data = %s "%score_multivariate_poly_train)
      score_multivariate_poly = r2_score(y_test, pred_multivariant_poly)
      print("R2 Score on testing data = %s "%score_multivariate_poly)
      print("/////////////////////////////////////////////////////////////////////////////////////")

for degree in range (2,6):
      polynomial_regression_model(degree)

"""After checking and comapring different models and number of features to predict the number of deaths, finally, a polynomial of degree 3 model with the following features: 
 #1: hospitalizedIncrease 
 #2:  onVentilatorCurrently
 #3: positiveIncrease   
 #4: number of tests increase

 was selected as the final model. This is the best model because it is moderatly accurate but has overfitting problem. In this modeltraining score =82% while test score= 56%. The noticeable difference in trainng and testing score could be the sign of overfitting. But with the help of regularization this could be improved therefore this  model is selected.

"""

# Polynomial regression on multivariate input
from sklearn.linear_model import Ridge
def RigeModel(alpha,pdegree):
      poly= PolynomialFeatures(degree=pdegree)
      poly.fit(x_train,y_train)
      x_poly=poly.fit_transform(x_train)
      line3=Ridge(alpha=alpha,normalize=True)
      line3.fit(x_poly, y_train)
      x_poly_test=poly.fit_transform(x_test)
      pred_multivariant_poly = line3.predict(x_poly_test)
      error_mse_multivariant_poly=mean_squared_error(y_test,pred_multivariant_poly)
      root_error_multivariant_poly=np.sqrt(error_mse_multivariant_poly)
      #print("For Alpha= ",alpha)
      #print("Root Error of MSE on  multivariate Polynomial degree 2 model:  ")
      #print("Error on testing data = %s " %root_error_multivariant_poly)
      score_multivariate_poly = r2_score(y_test, pred_multivariant_poly)
     # print("R2 Score on testing data = %s    alpha= %s"%(score_multivariate_poly,alpha))
      return score_multivariate_poly  , root_error_multivariant_poly   
best_ridge= 0.0
best_alpha= 0.001
b_error=0
for alpha in range (1,1000): #a=0.032
     acuridge,err=RigeModel(alpha/1000,2)
     if acuridge > best_ridge:
          best_ridge=acuridge
          best_alpha=alpha/1000
          b_error=err
          
print ("Best (test)accuracy For degree 2: %s       Alpha: %s     Square Root of MSE: %s " %((best_ridge*100),best_alpha,b_error))
best_ridge_d3= 0.0
best_alpha_d3= 0.001
bb_error=0
for alpha_s in range (1,1000): #a=0.032
     acuridges,b_err=RigeModel((alpha_s/1000),3)
     if acuridges > best_ridge_d3:
          best_ridge_d3=acuridges
          best_alpha_d3=alpha_s/1000
          bb_error=b_err

print ("Best (test) accuracy For degree 3: %s       Alpha: %s     Square Root of MSE: %s " %((best_ridge_d3*100),best_alpha_d3,  bb_error))

"""After using regulariazation on the degree 3 polynomial regression model, the test score improved from 63% to 80.2%. This is very drastic improvement to the model.

3. Introduce regularization and redo the step 2 and provide a short discussion of the improvements vs. performance.

**Ans**: Before using regularization, a polynomial of order 3 model was used to determine the number of deaths based on the number of hospitalizedIncrease, onVentilatorCurrently,positiveIncrease,and number of tests increase.
Based on the score of the model and the error it can be seen that there is a noticable difference when the model is predicting a training data and testing data. 
This huge difference might arise from the fact that the training data is not enough to the system and it is overfitting. This means the model has high variance and is not able to generalize. 
Regularization is one solution to such system. By using L2 regularization the score of predicting new (test data) data can be improved. 
This is done by adding extra  penality to  features (input features) and reducing their effect on the model, a process called Regualarization. 
This  lowers the power of each penalized feature to affect the output of the model.  After using L2 regularization the overall test data prediction score should be increased and the mean squared error (MSE) should be decreased. 



4. Evaluate how your hypothesis is performing on the prediction using the remaining 30% of the data (test set).

**Ans**: As expected, using L2 regularization improved the model score and error rate on the test dataset. 
The training data prediction score (Coefficient of determination) was 82% and the testing data prediction score was 56%. 
Similarly the sqaure root of MSE was 641.8. After using ridge regularization or L2 regularization the score on the testing data improved from 56% to 81%,
while the sqaure root of mean squared error decreased to 420.05. This is a huge improvement in the predicting power of the model.


5. Provide short discussion of the results and your insights.

**Ans**:  As explained in number 3, regularization is used to reduce overfiting and sometimes feature selection.
It penalizes the model features so as to discourage learning to some extent that allows generalization of the data. 
The additional penality is to decrease the variance of the model without removing features and adding a big bias. 
Regularization is therfore a compromise in a sense between losing important features( less variance but data loss) and overfitting (high variance in the model but important features are available).
in the first model used in this assignment, there was high varinance in the model which is evident in the difference between training data score and test data score. The Model is a polynomial of degree 3 with 4 input features. After using Regularization, however, the test dataset prediction jumped 25 points to reach 81%. This shows that the system was overfit and/or had very correlated features.
"""
# -*- coding: utf-8 -*- Assignment_1_Machine Learning.ipynb  Automatically generated by Colaboratory.  Date: 11-02-2021  # Tasks Covid data: https://covidtracking.com/ 1. Consider as training data set the first 70% of the available data 2. Perform linear or polynomial regression until you are happy with the results of the training data. 3. Introduce regularization and redo the step 2 and provide a short discussion of the improvements vs. performance. 4. Evaluate how your hypothesis is performing on the prediction using the remaining 30% of the data (test set). 5. Provide short discussion of the results and your insights.  The challenging parts of this homework are how to quantify the short discussion of the improvements vs. performance in the Step 3, and to quantitativelly evaluate the performance in Step 4. (You're allowed to search for the appropriate way to handle these issues.)  # Libraries #Import all the necessary libraries #Seaborn is used to pairplot # Code #Data Input ( Source: " https://covidtracking.com/data/national" ) # 08-Feb-2021 #Print Info To select specific columns  mydata = mydata(1:3)   mydataframe=pd.DataFrame(mydata,columns=["totalTestResultsIncrease","positiveIncrease","states"]) # Data Cleaning # Lets take data where covid tests are taken in all states , i.e 56 states. # To make sure there is conssistency in the data # "Drop NA" can be used to remove null data entries .."how= "all" " drops a row if all are Nan but "how =" Any"  "drops it if one is NA # inplace= True gives equal size columns # select the numeric features to compare them and see if there is any correlation #sns.pairplot(data_numeric) # ,fmt='.1g,' cmap= 'magma' # Based on the results of the correlation graph above , # In this assignement i will predict the number of deaths based on the following features: #1: hospitalizedIncrease #2:  onVentilatorCurrently #3: positiveIncrease # 4: number of tests increase # multivariate model and prediciton ---> features:"Postive Increase"  output: "number of deaths" #copy_x,fit_intercept ,n_jobs ,normalize # Univarient model and prediciton ---> feature:'totalTestResultsIncrease','onVentilatorCurrently','hospitalizedIncrease','positiveIncrease'  output: "number of deaths" #prediction on test #This is to see what would happen if it was trained using all the available traininhg data #copy_x,fit_intercept ,n_jobs ,normalize #calculate error # univarient error on tested on training data # univarient error on tested on testing data # Polynomial regression on univarient model # Linear  regression on multivariate input #calculate error # print(y_test,prediction_test) # Polynomial regression on multivariate input After checking and comapring different models and number of features to predict the number of deaths, finally, a polynomial of degree 3 model with the following features:   #1: hospitalizedIncrease   #2:  onVentilatorCurrently  #3: positiveIncrease     #4: number of tests increase   was selected as the final model. This is the best model because it is moderatly accurate but has overfitting problem. In this modeltraining score =82% while test score= 56%. The noticeable difference in trainng and testing score could be the sign of overfitting. But with the help of regularization this could be improved therefore this  model is selected. # Polynomial regression on multivariate input #print("For Alpha= ",alpha) #print("Root Error of MSE on  multivariate Polynomial degree 2 model:  ") #print("Error on testing data = %s " %root_error_multivariant_poly) # print("R2 Score on testing data = %s    alpha= %s"%(score_multivariate_poly,alpha)) #a=0.032 #a=0.032 After using regulariazation on the degree 3 polynomial regression model, the test score improved from 63% to 80.2%. This is very drastic improvement to the model.  3. Introduce regularization and redo the step 2 and provide a short discussion of the improvements vs. performance.  **Ans**: Before using regularization, a polynomial of order 3 model was used to determine the number of deaths based on the number of hospitalizedIncrease, onVentilatorCurrently,positiveIncrease,and number of tests increase. Based on the score of the model and the error it can be seen that there is a noticable difference when the model is predicting a training data and testing data.  This huge difference might arise from the fact that the training data is not enough to the system and it is overfitting. This means the model has high variance and is not able to generalize.  Regularization is one solution to such system. By using L2 regularization the score of predicting new (test data) data can be improved.  This is done by adding extra  penality to  features (input features) and reducing their effect on the model, a process called Regualarization.  This  lowers the power of each penalized feature to affect the output of the model.  After using L2 regularization the overall test data prediction score should be increased and the mean squared error (MSE) should be decreased.     4. Evaluate how your hypothesis is performing on the prediction using the remaining 30% of the data (test set).  **Ans**: As expected, using L2 regularization improved the model score and error rate on the test dataset.  The training data prediction score (Coefficient of determination) was 82% and the testing data prediction score was 56%.  Similarly the sqaure root of MSE was 641.8. After using ridge regularization or L2 regularization the score on the testing data improved from 56% to 81%, while the sqaure root of mean squared error decreased to 420.05. This is a huge improvement in the predicting power of the model.   5. Provide short discussion of the results and your insights.  **Ans**:  As explained in number 3, regularization is used to reduce overfiting and sometimes feature selection. It penalizes the model features so as to discourage learning to some extent that allows generalization of the data.  The additional penality is to decrease the variance of the model without removing features and adding a big bias.  Regularization is therfore a compromise in a sense between losing important features( less variance but data loss) and overfitting (high variance in the model but important features are available). in the first model used in this assignment, there was high varinance in the model which is evident in the difference between training data score and test data score. The Model is a polynomial of degree 3 with 4 input features. After using Regularization, however, the test dataset prediction jumped 25 points to reach 81%. This shows that the system was overfit and/or had very correlated features.