# In-class exercise 9: Deep Learning (Part 1B)
In this notebook we will gain some hands-on experience with backpropagation


```python
import numpy as np
import matplotlib.pyplot as plt
import time

%matplotlib inline

from sklearn.datasets import load_digits
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import label_binarize
```


```python
X, y = load_digits(return_X_y=True)
# Convert labels into one-hot format
Y = label_binarize(y, classes=np.unique(y))
K = Y.shape[1]  # number of classes
D = X.shape[1]  # number of features

np.random.seed(123)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3)
```


```python
# Dataset consists of 1797 samples. Each sample is represented with a 64-dimensional vector and belongs to one of 10 classes
X.shape, Y.shape
```




    ((1797, 64), (1797, 10))



# 1. Simple backpropagation example

Addition of two vectors


```python
class Add:
    def forward(self, x, y):
        return x + y
    
    def backward(self, d_out):
        d_x = d_out
        d_y = d_out
        return d_x, d_y
```

Elementwise multiplication of two vectors


```python
class Multiply:
    def forward(self, x, y):
        self.cache = (x, y)
        return x * y
    
    def backward(self, d_out):
        x, y = self.cache
        d_x = d_out * y
        d_y = d_out * x
        return d_x, d_y
```

Sum of a vector


```python
class Sum:
    def forward(self, x):
        self.cache = x
        return np.sum(x)
    
    def backward(self, d_out):
        x = self.cache
        d_x = d_out * np.ones_like(x)
        return d_x
```

### Dot product of two vectors as composition of multiplication and summation


```python
x = np.arange(1, 5, dtype=np.float32)
y = np.arange(-1, 3, dtype=np.float32)
```


```python
mult = Multiply()
vec_sum = Sum()

w = mult.forward(x, y)
z = vec_sum.forward(w)

d_w = vec_sum.backward(1.0)  #因为是导数的最开始，所以没有dout了，乘以1就是自己
d_x, d_y = mult.backward(d_w)
```


```python
w, z, d_x, d_y
```




    (array([-1.,  0.,  3.,  8.], dtype=float32),
     10.0,
     array([-1.,  0.,  1.,  2.], dtype=float32),
     array([1., 2., 3., 4.], dtype=float32))



### Dot product of two vectors as one operation

Dot product of two vectors


```python
class DotProduct:
    def forward(self, x, y):
        self.cache = x, y
        return np.dot(x, y)
    
    def backward(self, d_out):
        x, y = self.cache
        d_x = d_out * y
        d_y = d_out * x
        return d_x, d_y
```


```python
x = np.arange(1, 5, dtype=np.float32)
y = np.arange(-1, 3, dtype=np.float32)
```


```python
dp = DotProduct()
z = dp.forward(x, y)
d_x, d_y = dp.backward(1.0)
```


```python
z, d_x, d_y
```




    (10.0,
     array([-1.,  0.,  1.,  2.], dtype=float32),
     array([1., 2., 3., 4.], dtype=float32))



**Lessons:**
1. By implementing `forward` and `backward` method we can compute gradients of an arbitrary composition of functions
2. Use `cache` to store values that will be needed in the backward pass
3. Multiple operations can be combined into a single module (i.e. function)
4. Pass `1.0` as `d_out` for the terminal node in our computational graph

# 2. Multi-class logistic regression (without backprop)

Multi-class logistic regression model

Data:
* Data matrix $\mathbf{X} \in \mathbb{R}^{N \times D}$.
* Target labels in one-hot format $\mathbf{Y} \in \mathbb{R}^{N \times K}$.
$Y_{nk} = 1$ if sample $n$ belongs to class $k$, $Y_{nk} = 0$ otherwise.

Model parameters:
* Weight matrix $\mathbf{W} \in \mathbb{R}^{D \times K}$.
* Bias vector $\mathbf{b} \in \mathbb{R}^{K}$.

Making predictions with the model:
* Logits 
$$\mathbf{a}_n = \mathbf{W x}_n + \mathbf{b}$$
* Denote the matrix of logits as 
$$\mathbf{A} = \mathbf{XW} +  \mathbf{1}_N \mathbf{b}^T \in \mathbb{R}^{N \times K}$$
* Convert logits to probabilities using softmax function
$$p(Y_{nk} = 1 \mid \mathbf{x}_n, \mathbf{W}, \mathbf{b}) = \frac{\exp(A_{nk})}{\sum_{c = 1}^{K} \exp(A_{nc})}$$

Negative log-likelihood


\begin{align}
-\log p(\mathbf{Y} \mid \mathbf{X}, \mathbf{W}, \mathbf{b}) &= - \frac{1}{N}\sum_{n=1}^{N} \sum_{k=1}^{K} Y_{nk} \log p(Y_{nk} = 1 \mid \mathbf{x}_n, \mathbf{W}, \mathbf{b})\\
&= \frac{1}{N} \sum_{n=1}^{N} \sum_{k=1}^{K} Y_{nk} \left(-A_{nk} + \log \left( \sum_{c=1}^{C} \exp(A_{nc}) \right) \right)\\
%&= \frac{1}{N} \sum_{n=1}^{N} \left(\sum_{k=1}^{K} -Y_{nk} A_{nk} \right) + \log \left( \sum_{c=1}^{C} \exp(A_{nc}) \right)
\end{align}



```python
from scipy.special import softmax
```


```python
def predict(X, W, b):
    """Generate predictions for a multi-class logistic regression model.
    
    Args:
        X: data matrix, shape (N, D)
        W: weight matrix, shape (D, K)
        b: bias vector, shape (K)
        
    Returns:
        Y_pred: Predicted class probabilities, shape (N, K).
            Y_pred[n, k] = probability that sample n belongs to class k.
    """
    return softmax(X @ W + b, axis=1)
```

Negative log-likelihood of multiclass logistic regression 


```python
def nll_loss(X, W, b, Y):
    """Compute negative log-likelihood of a logistic regression model.
    
    Also known as categorical cross entropy loss.
    
    Args:
        X: data matrix, shape (N, D)
        W: weight matrix, shape (D, K)
        b: bias vector, shape (K)
        Y: true labels in one-hot format, shape (N, K)
    
    Returns:
        loss: loss of the logistic regression model, shape ()
    """
    N = X.shape[0]
    logits = X @ W + b
    
    logits_shifted = logits - logits.max(axis=1, keepdims=True)
    log_sum_exp = np.log(np.sum(np.exp(logits_shifted), axis=1, keepdims=True))
    log_probs = logits_shifted - log_sum_exp
    loss = -np.sum(Y * log_probs) / N
    return loss
```


```python
def nll_grad(X, W, b, Y):
    """Compute gradient of the NLL loss w.r.t. W and b.
    
    Args:
        X: data matrix, shape (N, D)
        W: weight matrix, shape (D, K)
        b: bias vector, shape (K)
        Y: true labels in one-hot format, shape (N, K)
    
    Returns:
        d_W: gradient of the los w.r.t. W, shape (D, K)
        d_b: gradient of the los w.r.t. b, shape (K)
    """
    N = X.shape[0]
    probas = softmax(X @ W + b, axis=1) - Y
    d_W = X.T @ probas / N
    d_b = probas.sum(axis=0) / N
    return d_W, d_b
```


```python
# Initialize learnable model parameters
W = np.zeros([D, K])
b = np.zeros([K])
```


```python
# Specify optimization parameters
learning_rate = 1e-2
max_epochs = 301
report_frequency = 25
```


```python
for epoch in range(max_epochs):
    # Compute train loss
    loss = nll_loss(X_train, W, b, Y_train)
    
    if epoch % report_frequency == 0:
        print(f'Epoch {epoch:4d}, loss = {loss:.4f}')
        
    # Perform the update
    d_W, d_b = nll_grad(X_train, W, b, Y_train)
    
    W = W - learning_rate * d_W
    b = b - learning_rate * d_b
```

    Epoch    0, loss = 2.3026
    Epoch   25, loss = 0.3411
    Epoch   50, loss = 0.2293
    Epoch   75, loss = 0.1862
    Epoch  100, loss = 0.1610
    Epoch  125, loss = 0.1440
    Epoch  150, loss = 0.1314
    Epoch  175, loss = 0.1215
    Epoch  200, loss = 0.1136
    Epoch  225, loss = 0.1069
    Epoch  250, loss = 0.1013
    Epoch  275, loss = 0.0964
    Epoch  300, loss = 0.0921



```python
loss_test = nll_loss(X_test, W, b, Y_test)
y_pred_test = predict(X_test, W, b).argmax(axis=1)
y_test = Y_test.argmax(axis=1)
acc_test = accuracy_score(y_test, y_pred_test)
print(f'loss_test = {loss_test:.4f}, accuracy_test = {acc_test:.3f}')
```

    loss_test = 0.1511, accuracy_test = 0.956


# 3. Multi-class logistic regression (with backprop)


```python
import nn
```


```python
class LogisticRegression:
    """Logistic regression model.
    
    Gradients are computed with backpropagation.
    """
    def __init__(self, num_features, num_classes, learning_rate=1e-2):
        self.learning_rate = learning_rate
        
        # Initialize the model parameters
        self.params = {
            'W': np.zeros([num_features, num_classes]),
            'b': np.zeros([num_classes])
        }
        
        # Define layers
        self.affine = nn.Affine()
        self.cross_entropy = nn.CategoricalCrossEntropy()
    
    def predict(self, X):
        """Generate predictions for one minibatch.
        
        Args:
            X: data matrix, shape (N, D)
            
        Returns:
            Y_pred: predicted class probabilities, shape (N, D)
            Y_pred[n, k] = probability that sample n belongs to class k
        """
        logits = self.affine.forward(X, 
                                     self.params['W'], 
                                     self.params['b'])
        return softmax(logits, axis=1)
    
    def step(self, X, Y):
        """Perform one step of gradient descent on the minibatch of data."""
        # Forward pass - compute the loss on training data
        logits = self.affine.forward(X, 
                                     self.params['W'], 
                                     self.params['b'])
        loss = self.cross_entropy.forward(logits, Y)
        
        # Backward pass - compute the gradients of loss w.r.t. all the model parameters
        grads = {}
        d_logits, _ = self.cross_entropy.backward()
        _, grads['W'], grads['b'] = self.affine.backward(d_logits)
        
        # Apply the gradients
        for p in self.params:
            self.params[p] = self.params[p] - self.learning_rate * grads[p]
        return loss
```


```python
# Specify optimization parameters
learning_rate = 1e-2
max_epochs = 301
report_frequency = 25
```


```python
log_reg = LogisticRegression(num_features=D, num_classes=K, learning_rate=learning_rate)
```


```python
for epoch in range(max_epochs):
    loss = log_reg.step(X_train, Y_train)
    
    if epoch % report_frequency == 0:
        print(f'Epoch {epoch:4d}, loss = {loss:.4f}')
```

    Epoch    0, loss = 2.3026
    Epoch   25, loss = 0.3411
    Epoch   50, loss = 0.2293
    Epoch   75, loss = 0.1862
    Epoch  100, loss = 0.1610
    Epoch  125, loss = 0.1440
    Epoch  150, loss = 0.1314
    Epoch  175, loss = 0.1215
    Epoch  200, loss = 0.1136
    Epoch  225, loss = 0.1069
    Epoch  250, loss = 0.1013
    Epoch  275, loss = 0.0964
    Epoch  300, loss = 0.0921



```python
y_pred_test = log_reg.predict(X_test).argmax(axis=1)
loss_test = log_reg.step(X_test, Y_test)
y_test = Y_test.argmax(axis=1)
acc_test = accuracy_score(y_test, y_pred_test)
print(f'loss_test = {loss_test:.4f}, accuracy_test = {acc_test:.3f}')
```

    loss_test = 0.1511, accuracy_test = 0.956


11663a7374bbdb7365d7967c1312426e9d6a08f1
ipynb
Jupyter Notebook
inclass_07_backpropagation.ipynb
vingoh/Machine-learning-exercises
0b847cbe66a531bb6488d7eebbbae51fe5515e8b
MIT
inclass_07_backpropagation.ipynb
vingoh/Machine-learning-exercises
0b847cbe66a531bb6488d7eebbbae51fe5515e8b
MIT
inclass_07_backpropagation.ipynb
vingoh/Machine-learning-exercises
0b847cbe66a531bb6488d7eebbbae51fe5515e8b
MIT
Qwen/Qwen-72B
1. YES
2. YES