<jupyter_start><jupyter_text>Logistic Regression With Gradient Descent<jupyter_code>import math #I import math to help with the entropy loss calculation involving logarithm 
from math import exp
import numpy as np #I import numpy to help with array manipulation
import sys

def sigmoid(x):
    try:
    
        sig= exp(x) / (1 + exp(x))

    except OverflowError as err:
        if (x>0):
            return 1
        else: 
            return 0
    
    return sig

def linearCombination(x, w):
    sum = 0
    for i in range(len(w)):
        sum += w[i] * x[i]
    return sum
def p1(x, w):
    return sigmoid(linearCombination(w,x))<jupyter_output><empty_output><jupyter_text>I added a funtion to help compute the entropy loss Entropy_loss<jupyter_code>def Entropy_loss(x,w,y): #This will enable me to answer the question of whether the resulting model is appropraite for the task being learned. Note that me major aim is to minimize the entropy loss. 
    sum = 0
    for i in range(len(x)):
        P_1 = (math.log(p1(x[i],w)))
        P_2 = (math.log(1-p1(x[i], w)))
        sum += -((y[i] * P_1) + ((1-y[i]) * P_2))
    return sum
from numpy import zeros

def gradient(X, y, w, lam):
    sum = zeros(len(w))
    for i in range(len(X)):
        for j in range(len(w)):
            sum[j] += (p1(X[i],w) - y[i]) * X[i][j] 
            
    for j in range(len(w)):
        sum[j] += lam * w[j]
        
    return sum
def gradientDescent(X,y,lr,lam,maxIt):
    w = zeros(len(X[0]))
    for gen in range(maxIt):
        grad = gradient(X,y,w,lam);
        for i in range(len(w)):
            w[i] = w[i] - lr * grad[i]
    return w
w = gradientDescent([[1,0.3,0.1],[1,0.3,0.2],[1,0.6,0.7],[1,0.8,0.2]], [0,0,1,1], 5, 0.1, 1000)
x = [[1,0.3,0.1],[1,0.3,0.2],[1,0.6,0.7],[1,0.8,0.2]]
y =  [0,0,1,1]
w = np.array(w)
print(w)
P_x1= p1(x[0], w) #The value of y = 0, for the given x([1,0.3,0.1])
P_x2 = p1(x[1], w) #The value of y = 0, for the given x([1,0.3,0.2])
P_x3  = p1(x[2], w) # The value of y = 1, for the given x([1,0.6,0.7])
P_x4 = p1(x[3], w) #the value of y=1, for the given x([1,0.8,0.2])
print("Probability of x[0] is " +str('{0:.6f}'.format(P_x1)))
print("Probability of x[1] is " +str('{0:.6f}'.format(P_x2)))
print("Probability of x[2] is " +str('{0:.6f}'.format(P_x3)))
print("Probability of x[3] is " +str('{0:.6f}'.format(P_x1)))<jupyter_output>Probability of x[0] is 0.001652
Probability of x[1] is 0.001823
Probability of x[2] is 0.003625
Probability of x[3] is 0.001652<jupyter_text>The above probability calcuations was only correct for P_x1 and P_x2 since the probabilities were less than zero and it predicted class zero (0). However, the probabilities for P_x3 and P_x4 were less than zero and it belongs to class one (1).<jupyter_code>entroy_loss = Entropy_loss(x,w,y)
print('The Entropy loss is calculated to be ' +str('{0:.4f}'.format(entroy_loss)))<jupyter_output>The Entropy loss is calculated to be 11.6064<jupyter_text>The entropy loss function shows that the resulting model is not appropriate for the task being learned. Since our aim is to minimize the entropy loss function. A model with an entropy loss of zero is said to be a perfect model.  question 2One positive one negative regarding the choice of argument from this python statement gradientDescent([[1,0.3,0.1],[1,0.3,0.2],[1,0.6,0.7],[1,0.8,0.2]], [0,0,1,1], 5, .1, 1000).If we play around with the learning rate lr = 5, without appying the regularization then we will notice that a learning rate of 5 is very appropriate for the model as it minimize the the entropy loss.<jupyter_code>from numpy import zeros

def gradient(X, y, w, lam):
    sum = zeros(len(w))
    for i in range(len(X)):
        for j in range(len(w)):
            sum[j] += (p1(X[i],w) - y[i]) * X[i][j] 
            
    #for j in range(len(w)):
        #sum[j] += lam * w[j]
        
    return sum 
    #case1
    #If we change the above gradient function with this code structure we will an entropy loss of 0.004538105532145029. That is we, silence the effect of regularization (lam)
    # Entropy loss = 0.004538105532145029. 
     
    #case 2
    # If We silence the effect of regularization and increase our learnig rate lr to a large extend like say lr = 50, the model begins to experience the issue overfitting and we get an entropy loss of 1.1909078857919278e-05
    # Entropy loss = 1.1909078857919278e-05

    #Case 3
    #If we silence the effect of regularization and decrease our learning rate lr to a large extend like say lr = 0.005, the model begins to experience the issue of underfitting and our entropy loss begins to increase. we get an entropy loss of 2.021615680201623
    # Entropy loss = 2.021615680201623

    # We can conclude an lr of 5 was a very good choice and its a positive point regarding the choice of argument in the python statement. 

    #Please you can change the indicated parameters and arguments, and rerun the jupyter notebook to confirms this values<jupyter_output><empty_output><jupyter_text>If we play around with the regularization term (lam) while keeping every other argument same. we observe the following.<jupyter_code>results = gradientDescent([[1,0.3,0.1],[1,0.3,0.2],[1,0.6,0.7],[1,0.8,0.2]], [0,0,1,1], 5, 0.01, 1000)

#If we play around with the regularization term (lam) while keeping every other argument same. we observe the following:

#Case1
#if we reduce the regularization term (lam) to like say lam = 0.01. Our model perfomance increases considerabily and we get en entroy loss of 0.40635578009422885.
#  Entropy loss = 0.40635578009422885
#if we decide to reduce further to like say lam = 0.001 our model performs even better and we get an entropy loss of 0.08866936520064983
# Entropy loss = 0.08866936520064983

#Case2
#If we increase the regularization term lam = 0.2. model performnace get even worse. we get an entropy loss of 25.54995971940941
# Entropy loss = 25.54995971940941

# So we can conclude that our lam was one negative choice regarding argument in the python statement. 

#Please you can change the indicated parameters and arguments, and rerun the jupyter notebook to confirms this values<jupyter_output><empty_output>