<h1 align=center> Home Quiz 1 - Logistic Regression</h1>
<br>
$$
\text{Chatziefraimidis Lefteris 2209}\\
$$

## Problem 1: Gradient Descent

We will estimate the parameters $w_{0},w_{1},w_{2}$ using gradient descent for the following prediction model:
<br>
<br>
$$ y = w_{0} + w_{1}x_{1} + w_{2}x_{2} + w_{3}x_{1}^2 + \epsilon \space \text{  ,where }\space\epsilon\text{ ~ } N(0,\sigma^2)$$
<br>
$
\text{The error of the approximation: } \epsilon = y - y_{predicted} = y - w_{0} + w_{1}x_{1} + w_{2}x_{2} + w_{3}x_{1}^{2} 
$

### $\triangleright$ Exercise (a) :

<strong>Gaussian Distribution :</strong>
<br>
$X$ is distributed according to normal (or Gaussian) distribution with mean $\mu$ and variance $\sigma^2$

$$ 
\begin{aligned}
X &\sim \mathcal{N}(\mu,\sigma^2) \\ \\
p(X = x|\mu,\sigma^2)&= \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{1}{2\sigma^2}(x-\mu)^2}
\end{aligned}
$$

So,for our model we have:
<br>
<br>
$$P(y|x_{1},x_{2}) = \frac{1}{\sqrt{2\pi\sigma^2}}e^-{\frac{(y-(w_{0} + w_{1}x_{1} + w_{2}x_{2} + w_{3}x_{1}^2))^2}{2\sigma^2}}$$
<br>
and the above expression its called likelihood function.

### $\triangleright$ Exercise (b) :
Assume you are given a set of training observations $(x_{1}^{(i)},x_{2}^{(i)},y^{(i)})$ for $i=1,....,n$
<br>
<br>
Log-likelihood of this training data: 
<br>
$$LogP(y|x_{1},x_{2}) =  \sum_{i=1}^n \log \frac{1}{\sqrt{2\pi\sigma^2}} e^-{\frac{(y-(w_{0} + w_{1}x_{1}^{(i)} + w_{2}x_{2}^{(i)} + w_{3}x_{1}^{(i)2}))^2}{2\sigma^2}}$$
<br>
$$ = \sum_{i=1}^n \left[-\frac{1}{2}\log 2\pi\sigma^2 - \frac{1}{2\sigma^2}(y-(w_{0} + w_{1}x_{1}^{(i)} + w_{2}x_{2}^{(i)} + w_{3}x_{1}^{(i)2}))^2\right]$$
<br>
$$ = \sum_{i=1}^n \left[- \frac{1}{2\sigma^2}(y-(w_{0} + w_{1}x_{1}^{(i)} + w_{2}x_{2}^{(i)} + w_{3}x_{1}^{(i)2}))^2\right]$$


### $\triangleright$ Exercise (c) :
Based on your answer above, we can write a loss function $f(w_{0},w_{1},w_{2},w_{3})$ that can be minimized to find the desired parameter estimates:
<br>
<br>
$$f(w_{0},w_{1},w_{2},w_{3}) = \frac{1}{n}\sum_{i=1}^n \left[(y-(w_{0} + w_{1}x_{1}^{(i)} + w_{2}x_{2}^{(i)} + w_{3}x_{1}^{(i)2}))^2\right] , \frac{1}{n} \text{ makes the loss interpretable}$$
<br>
<br>
This particular loss function is also known as Mean squared error(MSE).We can use gradient descent to optimize it.

### $\triangleright$ Exercise (d) :
Lets calculate the gradient of $f(w)$ with respect to the parameter vector $w = [w_{0},w_{1} ,w_{2},w_{3}]^{T} :$
<br>
<br>
$$\nabla_w f(w) = \begin{bmatrix}
\frac{\partial f(w)}{\partial w_0} &\frac{\partial f(w)}{\partial w_1}  &\frac{\partial f(w)}{\partial w_2}  & \frac{\partial f(w)}{\partial w_3}
\end{bmatrix}^{T}$$
<br>
<br>
Our goal was to find $w_0,w_1,w_2,w_3$ such that:
<br>
<br>
$$\nabla_w f(w) = \begin{bmatrix}
\frac{\partial f(w)}{\partial w_0} &\frac{\partial f(w)}{\partial w_1}  &\frac{\partial f(w)}{\partial w_2}  & \frac{\partial f(w)}{\partial w_3} 
\end{bmatrix}^{T} = \text{[ 0 0 0 0 ]}$$
<br>
because this guarantees that $f(w)$ is minimized.
<br>
<br>
$$
\nabla_w f(w) = 
\begin{bmatrix}
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-1))
\\ 
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,1})
\\ 
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,2})
\\ 
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,1}^{2})
\\
\\
\end{bmatrix} =
\begin{bmatrix}
0
\\
0 
\\
0 
\\ 
0
\end{bmatrix}
$$


### $\triangleright$ Exercise (e) :
Gradient descent update rule for $w$ in terms of $\nabla_w f(w)$:
<br>
<br>
$$
w^{(\textrm{iteration}+1)} = w^{(\textrm{iteration})} - \alpha\nabla f(w^{(\textrm{iteration})})
\text{ , }\alpha\text{ : learning rate}
$$
<br>
<br>
$$
w^{(\textrm{iteration}+1)} = w^{(\textrm{iteration})} - \alpha\begin{bmatrix}
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-1))
\\ 
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,1})
\\ 
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,2})
\\ 
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,1}^{2})
\\
\\
\end{bmatrix}
$$

### $\triangleright$ Exercise (f) :


```python
from sympy import *

#Define the symbols
x1 = Symbol('x1')
x2 = Symbol('x2')

w0 = Symbol('w0')
w1 = Symbol('w1')
w2 = Symbol('w2')
w3 = Symbol('w3')
y = Symbol('y')
n = Symbol('n')
a =Symbol('a')

#Define the expression y prediction
ypred = w0 + w1*x1 + w2*x2 + w3*(x1**2) 

#Define the loss function f
f = (1/n)*(y - ypred)**2

#Calculate the derivatives in order to calculate the nabla
df_dw0 = diff(f,w0)
df_dw1 = diff(f,w1)
df_dw2 = diff(f,w2)
df_dw3 = diff(f,w3)

#Nabla of loss function
nabla_f = Matrix([[df_dw0],[df_dw1],[df_dw2],[df_dw3]])

#Calculate the new weights
weights = Matrix([[w0],[w1],[w2],[w3]])
weights = weights - a*nabla_f

print("Nabla: ")
print(nabla_f)
print("Weights: ")
print(weights)
```

    Nabla: 
    Matrix([[(2*w0 + 2*w1*x1 + 2*w2*x2 + 2*w3*x1**2 - 2*y)/n], [-2*x1*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n], [-2*x2*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n], [-2*x1**2*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n]])
    Weights: 
    Matrix([[-a*(2*w0 + 2*w1*x1 + 2*w2*x2 + 2*w3*x1**2 - 2*y)/n + w0], [2*a*x1*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n + w1], [2*a*x2*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n + w2], [2*a*x1**2*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n + w3]])


## Problem 2: Logistic Regression

### $\triangleright$ Exercise (a) :

#### Some theory will we need:
Sigmoid (Logistic) Function:
$$
\sigma(z) = \frac{1}{1 + \exp(-z)}
$$
<br>
<br>
In logistic regression we model a binary variable $y \in \{0,1\}(Bernoulli)$
<br>
<br>
$$
p(y|x,\beta_0,\beta) = \sigma(\beta_0 + x^{T}\beta) = \frac{1}{1 + e^{-(\beta_0 + x^{T}\beta)}} \\
$$
<br>
<br>
Likelihood function is:
<br>
<br>
$$
L(\beta_0,\beta|y,x) = \prod_i p(y_i | x_i,\beta_0,\beta)^{y_i}(1 - p(y_i| x_i,\beta_0,\beta))^{1-y_i}
$$
<br>
<br>
Log-likelihood function is:
<br>
<br>
$$
\log L(\beta_0,\beta|y,x) = \sum_i y_i\log  p(y_i| x_i,\beta_0,\beta) + (1 - y_i)log(1 -  p(y_i| x_i,\beta_0,\beta)) \\
=\sum_i y_i\log  p(y_i| x_i,\beta_0,\beta) + (1 - y_i)log(p(y_i| x_i,\beta_0,\beta)) \\
=\sum_i y_i\log1 - y_i\log(1 + e^{-(\beta_0 + x_i\beta)}) + \log(\frac{e^{-(\beta_0 + x_i\beta)}}{1 + e^{-(\beta_0 + x_i\beta)}}) -y_i\log(e^{-(\beta_0 + x_i\beta)}) +y_i\log(1 + e^{-(\beta_0 + x_i\beta)}) \\
=\sum_i   \log(\frac{e^{-(\beta_0 + x_i\beta)}}{1 + e^{-(\beta_0 + x_i\beta)}}) -y_i\log(e^{-(\beta_0 + x_i\beta)}) \\
=\sum_i  \log(\frac{1}{1 + e^{(\beta_0 + x_i\beta)}}) -y_i\log(e^{-(\beta_0 + x_i\beta)}) \\
=\sum_i  -\log(1 + e^{(\beta_0 + x_i\beta)}) + y_i(\beta_0 + x_i\beta)
$$
<br>
<br>
Our objective function for gradient ascent:
<br>
<br>
$$
f(w_0,w) = \sum_i  -\log(1 + e^{(w_0 + x_iw)}) + y_i(w_0 + x_iw)
$$

### $\triangleright$ Exercise (b) :
We compute the partial derivative of the objective function with respect to $w_0$ and with respect to an
arbitrary $w_j$:
<br>
<br>
$$
\frac{\partial f}{\partial w_j} = - \sum_i \frac{e^{(w_0 +x_iw)}x_i}{1 + e^{(w_0 +x_iw)}} + \sum_i y_ix_i \\
=\sum_i (y_i - p(y_i|x_i,\beta_0,\beta) ) x_{ij}
$$

##  Logistic Regression Implementation


```python
import numpy as np
import pandas as pd
import numpy.matlib
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d

############################################## Functions we will use #############################################

#Define the sigmoid function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

#Calculate the logistic regression objective value
def LR_CalcObj(XTrain,yTrain,wHat):
    
    #Get the dimensions
    [n,p] = XTrain.shape
    
    #Add one's to Xtrain
    XTrain=np.c_[np.ones((n,1)),XTrain]
    
    #Calculate X*w abd exp(X*w)
    Xw = np.dot(XTrain,wHat)
    eXw = np.exp(Xw)
    
    #Calculate objective value
    return np.sum(yTrain*Xw - np.log(1+eXw))

#Check whether the objective value has converged 
def LR_CheckConvg(old,new,tol):
    
    #Compute difference between objectives
    diff = np.abs(old-new);
    
    #Compare difference to tolerance
    if(diff < tol):
        return True
    else:
        return False

#Calculate the new value of wHat using the gradient
def LR_UpdateParams(wHat,grad,eta):
    
    #Update value of w
    wHat = wHat + eta*grad
    return wHat

#Calculate the gradient of the logistic regression
def LR_CalcGrad(XTrain,yTrain,wHat):
    
    #Get the dimensions
    [n,p] = XTrain.shape
    
    #Add one's to Xtrain
    XTrain=np.c_[np.ones((n,1)),XTrain]
    
    #Calculate X*w abd exp(X*w)
    z = np.dot(XTrain,wHat)
    h = sigmoid(z)

    #Return gradient
    return np.dot(XTrain.T, (yTrain - h))

#Run the gradient ascent algorithm for logistic regression
def LR_GradientAscent(XTrain,yTrain):
    
    #Define step size
    eta = 0.01
    
    #Define the covergence tolerance
    tol = 0.001
    
    #Get the dimensions 
    [n,p] = XTrain.shape
    
    #Initialize wHat
    wHat = np.zeros((p+1,1))
    
    #Initialize objVal
    objVals=[]
    objVals.append(LR_CalcObj(XTrain,yTrain,wHat))
    
    #Initialize convergence flag
    hasConverged = False
    
    while(not hasConverged):
        
        #Calculate gradient
        grad = LR_CalcGrad(XTrain,yTrain,wHat)
        
        #Update parameter estimate
        wHat = LR_UpdateParams(wHat,grad,eta)
        
        #Calculate new objective
        newObj = LR_CalcObj(XTrain,yTrain,wHat)
        
        #Check convergence
        hasConverged = LR_CheckConvg(objVals[-1],newObj,tol)
        
        #Store new objective
        objVals.append(newObj)
        
    return wHat,objVals

#Predict the labls for a test set using logistic regression
def LR_PredictLabels(XTest,yTest,wHat):
    
    #Get dimensions
    [n,p] = XTest.shape
    
    #Add one's to XTest
    XTest=np.c_[np.ones((n,1)),XTest]
    
    #Calculate X*w abd exp(X*w)
    Xw = np.dot(XTest,wHat)
    eXw = np.exp(Xw)
    
    #Calculate p(Y = 0)
    pY0 = 1/(1 + eXw)
    
    #Calculate p(Y = 1)
    pY1 = eXw/(1 + eXw)
    
    yHat =[]
    #Choose best propability
    for i in range(0,len(pY0)):
        if(pY1[i] > pY0[i]):
            yHat.append([1])
        else:
            yHat.append([0])
    
    yHat=np.array(yHat)
    #Calculate error
    numErrors = np.sum(yHat!= yTest)

    return yHat,numErrors
def PlotDB():
    #Load the data 

    #Training
    XTrain = np.array(pd.read_csv('XTrain.csv',header=None))
    yTrain = np.array(pd.read_csv('yTrain.csv',header=None))

    #Testing
    XTest = np.array(pd.read_csv('XTest.csv',header=None))
    yTest = np.array(pd.read_csv('yTest.csv',header=None))

    #Train logistic regression
    [wHat,objVals] = LR_GradientAscent(XTrain,yTrain)
    
    ind0 = []
    ind1 = []
    
    for i in range(len(yTest)):
        
        if(yTest[i] == 0):
            ind0.append(i)
        else:
            ind1.append(i)
            
    #Calculate decision boundary
    dbDimJ = np.arange(np.min(XTest[:,0]),np.max(XTest[:,0]),step=.01)
    dbDimK = (-(wHat[0] + wHat[1])*dbDimJ)/wHat[2]
    
    plt.plot(XTest[ind0,0],XTest[ind0,1],'r.')
    plt.plot(XTest[ind1,0],XTest[ind1,1],'b.')
    plt.plot(dbDimJ,dbDimK,'k-')
    plt.xlabel('Dimension 1')
    plt.ylabel('Dimension 2')
    plt.title('Logistic Regression Decision Boundary')
    plt.show()
############################################## Start of the program #############################################

#Load the data 

#Training
XTrain = np.array(pd.read_csv('XTrain.csv',header=None))
yTrain = np.array(pd.read_csv('yTrain.csv',header=None))

#Testing
XTest = np.array(pd.read_csv('XTest.csv',header=None))
yTest = np.array(pd.read_csv('yTest.csv',header=None))

#Train Phase
wHat,objVals = LR_GradientAscent(XTrain,yTrain)

#Test Phase 
yHat,numErrors = LR_PredictLabels(XTest,yTest,wHat)

#Print the number of misclassified examples
print('There were %d misclassified examples in the test set\n'%(numErrors))


#Plot the objective values
plt.plot(objVals)
plt.xlabel('Gradient Ascent Iteration')
plt.ylabel('Logistic Regression Objective Value')
plt.title('Convergence of Gradient Ascent for Logistic Regression')
plt.show()

print('Gradient ascent coverges after %d iterations\n'%(len(objVals)-1))

#2D Plot
PlotDB()


#Evaluate the training set and test error as a function of training set size
n = XTrain.shape[0]
kVals =np.arange(10,n+10,step=10)
m = XTest.shape[0]

#Errors for test,train
trainingError = np.zeros((len(kVals),1));
testError = np.zeros((len(kVals),1));

for i in range(len(kVals)):
    
    #Set k 
    k=kVals[i]
    
    #Generate trainingset
    subsetsInds = np.random.randint(0,n,size=k)
    XTrainSubset = XTrain[subsetsInds,:]
    yTrainSubset = yTrain[subsetsInds,:]
    
    #Train logistic regression
    wHat,objVals = LR_GradientAscent(XTrainSubset,yTrainSubset)
    
    #Test classifier on training set 
    [yHatTrain,numErrorsTrain] = LR_PredictLabels(XTrainSubset,yTrainSubset,wHat)
    trainingError[i] = numErrorsTrain/k;
    
    #Test classifier on test set
    [yHatTest,numErrorsTest] = LR_PredictLabels(XTest,yTest,wHat)
    testError[i] = numErrorsTest/m; 

#Plot the above 
plt.plot(kVals,trainingError)
plt.plot(kVals,testError)
plt.xlabel('Training Set Size')
plt.ylabel('Prediction Error')
plt.title('Logistic Regression Performance by Training Set Size')
plt.legend(['Training Error','Test Error'])
plt.show()


#Perform the same experiment but average over multiple random training sets
n = XTrain.shape[0]
kVals =np.arange(10,n+10,step=10)
m = XTest.shape[0]

#Errors for test,train
trainingError = np.zeros((len(kVals),1));
testError = np.zeros((len(kVals),1));

for i in range(len(kVals)):
    
    #Set k 
    k=kVals[i]
    
    for j in range(0,10):
        
        #Generate trainingset
        subsetsInds = np.random.randint(0,n,size=k)
        XTrainSubset = XTrain[subsetsInds,:]
        yTrainSubset = yTrain[subsetsInds,:]
        
         #Train logistic regression
        wHat,objVals = LR_GradientAscent(XTrainSubset,yTrainSubset)

        #Test classifier on training set 
        [yHatTrain,numErrorsTrain] = LR_PredictLabels(XTrainSubset,yTrainSubset,wHat)
        trainingError[i] += numErrorsTrain/k;

        #Test classifier on test set
        [yHatTest,numErrorsTest] = LR_PredictLabels(XTest,yTest,wHat)
        testError[i] += numErrorsTest/m;
    
    trainingError[i]/= 10;
    testError[i] /= 10;


#Plot the above 
plt.plot(kVals,trainingError)
plt.plot(kVals,testError)
plt.xlabel('Training Set Size')
plt.ylabel('Prediction Error')
plt.title('Logistic Regression Performance by Training Set Size')
plt.legend(['Training Error','Test Error'])
plt.show()

```

### $\triangleright$ Exercise (g) :

$\bullet$ As the training set size increases our model overfits.That happens because we increase the training set.
Imagine if you only had 2 points for the training set. It would be easy for almost any model to match this set exactly. However, it's very likely that the model would fail horribly on the test set. It hasn't really seen enough to learn. With small training sets, the model is likely to underfit and perform poorly on the test set. This shows a high variance in the model.
<br>

$\bullet$ The more data points you add to the training set, the more it can overcome the overfitting, and will perform better on the test set. However, it can perform worse on the training set, because the model may not be able to fit each example to the model. This can be a good thing, as noise in the training set can be ignored.



### $\triangleright$ Exercise (h) :

$\bullet$ In our classification problem with two classes, a decision boundary or decision surface is a hypersurface that partitions the underlying vector space into two sets, one for each class. The classifier will classify all the points on one side of the decision boundary as belonging to one class and all those on the other side as belonging to the other class.
<br>

$\bullet$ XTrain is an $n × p$ dimensional matrix that contains one training instance per row.So in order to classify each row you need a hypersurface to separate those data.

How to find the decision boundary: 
* Assume a 2D problem with two features $x_1$ and $x_2$, then
$$
p(y=1|x,\beta_0,\beta) = 
\sigma{(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}= 
\frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}
$$
<br>
<br>
The decision boundary between 0 and 1 is 0.5.So:
$$
\frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}} = \frac{1}{2}
$$
<br>
$$
e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)} = 1
$$
<br>
$$
\beta_0 + \beta_1 x_1 + \beta_2 x_2 = 0 \rightarrow  x_2 = \beta_0 - \frac{\beta_1}{\beta_2} x_1
$$

d1239f05d6223b09b8c49b317973aab9eb4814ed
ipynb
Jupyter Notebook
Logistic Regression - Quiz 1/Quiz_1.ipynb
echatzief/Machine-Learning-Projects
900493e4f14ef4faf1a794d85d103d966dca5be5
MIT
Logistic Regression - Quiz 1/Quiz_1.ipynb
echatzief/Machine-Learning-Projects
900493e4f14ef4faf1a794d85d103d966dca5be5
MIT
Logistic Regression - Quiz 1/Quiz_1.ipynb
echatzief/Machine-Learning-Projects
900493e4f14ef4faf1a794d85d103d966dca5be5
MIT
Qwen/Qwen-72B
1. YES
2. YES