# Calculus and Machine learning

In deep learning, we typically choose **loss functions** that are **differentiable** with respect to our modelʼs parameters. Put simply, this means that for each parameter, we can determine how rapidly the loss would increase or decrease, were we to increase or decrease that parameter by an infinitesimally small amount.

1. Getting better on training datasets means minimizing a loss function. 
2. Producing a model that performs well on data that we have never seen before, that means what's performance of the model on testing dataset.

Thus we can decompose the task of fitting models into two key concerns:
1. **Optimization**:  the process of fitting our models to observed data (like training and valid datasets)
2. **Generalization**: produce models whose validity extends beyond the exact set of data examples used to train them, like testing datasets

Suppose that we have a deep neural network where the weights are $w=(w_1, w_2, \ldots, w_n)$. Given a training dataset, we consider to minimize the loss of our neural network on this dataset: $\mathcal{L}(w)$:

**Solution**: 
1. we often start by initializing our weights randomly. 
2. We iteratively take small steps in the direction which makes the loss decrease as rapidly as possible

# Differential Calculus (微分)

## Derivative(导数) of a function

Suppose that we have a function ($y=f(x)$) $f: \mathbb{R} \rightarrow \mathbb{R}$, whose input and output are both scalars. The **derivative** of f is defined as:
\begin{equation}
\begin{aligned}
f^{'}(x) &= \lim_{h \rightarrow 0} \frac{f(x+h) - f(x)}{h} = \lim_{\Delta_x \rightarrow 0} \frac{f(x+\Delta_x) -f(x)}{\Delta_x}\;\textrm{if this limit exists} \\
&=\frac{dy}{dx} = \frac{df}{dx} = \frac{d}{dx}f(x) = Df(x) = D_xf(x)
\end{aligned}
\end{equation} where symbols $\frac{d}{dx}$ and $D$ are **differentiation** operators that indicate operation of differentiation.

If $f^{'}(a)$ exists, $f$ is said to be **differentiable** at $a$, that is defined as:
\begin{equation}
f^{'}(a) = \lim_{x \rightarrow a} \frac{f(x) - f(a)}{x-a}
\end{equation}

If $f$ is **differentiable** at every number of an interval ($x\in [a,b]$), then this function is differentiable on this interval ($[a,b]$).

## Linear Approximation

In particular, note that the following equation approximates the value of f by a line which passes through the point (x, f(x)) and has slope $f'$. In this way we say that the derivative gives a linear approximation to the function f.
\begin{equation}
\begin{aligned}
\frac{df(x)}{dx} = \lim_{\epsilon \rightarrow 0} \frac{f(x+\epsilon) - f(x))}{\epsilon } &\Rightarrow \frac{df(x)}{dx} \approx \frac{f(x+\epsilon) - f(x))} {\epsilon } \\
\Rightarrow \epsilon \frac{df(x)}{dx} \approx f(x+\epsilon) - f(x) \\
\Rightarrow f(x+\epsilon) \approx f(x) + \epsilon \frac{df(x)}{dx} = f(x) + \epsilon f^{'}
\end{aligned}
\end{equation}


```python
# f(x) = 3x^2 - 4x
def f(x):
    return 3*x**2 - 4*x

# Functions to calcuate limits
def numerical_limit(f, x, h):
    return (f(x + h) - f(x)) / h

h = 0.0001
x= 1
for i in range(5):
    print(f'h={h:.5f}, numerical limit={numerical_limit(f, x, h):.5f}')
    h *= 0.1
```

    h=0.00010, numerical limit=2.00030
    h=0.00001, numerical limit=2.00003
    h=0.00000, numerical limit=2.00000
    h=0.00000, numerical limit=2.00000
    h=0.00000, numerical limit=2.00000


## Common Derivatives

- $(x^n)' = nx^{n-1}$ (the power rule, $n$ is any real number)
- $(e^x)' = e^x$
- $(\sin(x))' = \cos(x)$
- $(\cos(x))' = -\sin(x)$
- $\tan(x)' = \sec^2(x)$
- $\cot(x)' = -\csc^2(x)$
- $\sec(x)' = \sec x \tan x$
- $\csc x' = -\csc x \cot x$
- $(a^x)' = a^x\ln x (a \gt 0, a \neq 1)$
- $(\log_ax)' = \frac{1}{x\ln a} (a \gt 0, a \neq 1)$
- $(\ln(x))' = \frac{1}{x}$

## Derivative Rules

Assume $u=u(x), v=v(x)$ are differentiable at an interval $x \in [a,b]$, and $C$ is a constant number
1. Constant Multiple Rule: (Cu)^' = Cu'
2. Sum Rule: $(u \pm v) = u' \pm v'$
3. Product Rule $(uv)' = u'v + uv'$
4. Quotient Rule: $(\frac{u}{v})' = \frac{u'v - uv'}{v^2} \; (v \neq 0 )$

## Partial Derivatives

So far we have dealt with the differentiation of functions with just **one variable**. In deep learning, functions often depend on **many variables**. Thus, we need to extend the ideas of differentiation to these **multivariate functions**.

Let $y = f(x_1, x_2, \ldots, x_n)$ be a function with $n$ variables. The partial derivative of $y$ with respect to its $i^{th}$ parameter $x_i$ is:
\begin{equation}
\frac{\partial y}{\partial x_i} = \frac{\partial f}{\partial x_i} = f_{x_i} = f_i = D_if = D_{x_i} f = \lim_{\epsilon \rightarrow 0} \frac{f(x_1, x_2, \ldots, x_i + \epsilon, \ldots, x_n) - f(x_1, x_2, \ldots, x_i, \ldots, x_n)}{\epsilon}
\end{equation}

To calculate $\frac{\partial y}{\partial x_i}$, we can simply treat all variables except $x_i$ as constants and calculate the derivative of $y$ with respect to $x_i$. 

\begin{equation}
f(x+\epsilon) \approx f(x) + \epsilon  \frac{df}{dx}(x) \\ \Downarrow \\
L(x_1 + \epsilon_1, x_2, \ldots, x_n) \approx L(x_1, x_2, \ldots, x_n) + \epsilon_1 \frac{\partial L}{\partial x_1} (x_1, x_2, \ldots, x_n) \\ \Downarrow \\
L(x_1 + \epsilon_1, x_2 + \epsilon_2, \ldots, x_n + \epsilon_n) \approx L(x_1, x_2, \ldots, x_n) + \sum_{i=1}^{n} \epsilon_i \frac{\partial L}{\partial x_i} (x_1, x_2, \ldots, x_n) 
\end{equation}

This may look like a mess, but we can make this more familiar by noting that the sum on the right looks exactly like a dot product, so if we let:

\begin{equation}
\epsilon = [\epsilon_1, \ldots, \epsilon_n]^T \; \textrm{and} \; \Delta_x L = [\frac{\partial L}{\partial x_1}, \ldots, \frac{\partial L}{\partial x_n}]  \\ \Downarrow \\
L(X + \epsilon) \approx L(X) + \epsilon \cdot \Delta_x L(X)
\end{equation}
We will call the vector $\Delta_x L$ the gradient of $L$. This equation allows us to tell approximately how the function L will change given any perturbation to the input.


```python
%matplotlib inline
import numpy as np
import torch
from IPython import display
def f(x, y):
    return torch.log(torch.exp(x) + torch.exp(y))
def grad_f(x, y):
    return torch.tensor([torch.exp(x) / (torch.exp(x) + torch.exp(y)), torch.exp(y) / (torch.exp(x) + torch.exp(y))])

epsilon = torch.tensor([0.01, -0.03])

grad_approx = f(torch.tensor([0.]), torch.log(torch.tensor([2.]))) + epsilon.dot(grad_f(torch.tensor([0.]), torch.log(torch.tensor(2.))))

true_value = f(torch.tensor([0.]) + epsilon[0], torch.log(torch.tensor([2.])) + epsilon[1])

f'approximation: {grad_approx}, true Value: {true_value}'
```




    'approximation: tensor([1.0819]), true Value: tensor([1.0821])'



## Gradients

We can concatenate partial derivatives of a multivariate function with respect to all its variables to obtain the **gradient** vector of the function

Suppose that the input of function $f: \mathbb{R}^n \rightarrow \mathbb{R}$ is an n-dimensional vector $x = [x_1, x_n, \ldots, x_n]^T$ and the output is a scalar. The gradient of the function $f(x)$ with respect to $x$ is a vector of n partial derivatives:
\begin{equation}
\Delta_xf(x) = \Delta f(x) = [\frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \ldots, \frac{\partial y}{\partial x_n}]^T  = [\frac{\partial f(x)}{\partial x_1}, \frac{\partial f(x)}{\partial x_2}, \ldots, \frac{\partial f(x)}{\partial x_n}]^T
\end{equation}

Let x be an n-dimensional vector $x = [x_1, x_n, \ldots, x_n]^T$, The following rules are often used when differentiating multivariate functions:

- For all $A \in \mathbb{R}^{m \times n}$, $\Delta_x A x = A^T$
- For all $A \in \mathbb{R}^{n \times m}$, $\Delta_x x^T A = A$
- For all $A \in \mathbb{R}^{n \times n}$, $\Delta_x x^T A x = (A + A^T) x$
- $\Delta_x ||x||^2 = \Delta_x x^T x = 2x$
- $\forall X, \Delta_X ||X||^2_F = \Delta_x x^T x = 2X$

## Geometry of Gradients and Gradient Descent
Consider an input vecotr $X=[x_1, \ldots, x_n]$ and its corresponding loss function $L(X)$, $L$ can be approxmated and minimized by following equation:
\begin{equation}
L(X + \epsilon) \approx L(X) + \epsilon \cdot \Delta_x L(X)
\end{equation} 
where $\epsilon = [\epsilon_1, \ldots, \epsilon_n]^T$ and gradients $\Delta_x L = [\frac{\partial L}{\partial x_1}, \ldots, \frac{\partial L}{\partial x_n}]$.

Let us suppose that I want to use this to help minimize our loss L. Let us understand geometrically the algorithm of gradient descent first:

1. Start with a random choice for the initial parameters $X$
2. Find the direction $v$ that makes $L$ decrease the most rapidly at $X$.
3. Take a small step in that direction: $X \rightarrow X + \epsilon v$
4. Repeat

The only thing we do not know exactly how to do is to compute the vector $v$ in the second step. We will call such a direction **the direction of steepest descent**.

Using the geometric understanding of dot products $\cos(\theta) = \frac{u \cdot v}{ ||u||*||v||}$ where $\theta$ is the angle between two vectors $u$ and $v$, we can rewrite the above equation as:
\begin{equation}
\begin{aligned}
L(X + v) {}&\approx L(X) + v \cdot \Delta_x L(X) \\
&= L(X) + ||v|| * || \Delta_x L(X)|| \cos(\theta) \\
&= L(X) + || \Delta_x L(X)|| \cos(\theta)
\end{aligned}
\end{equation} Note that we have taken our direction to have length one for convenience, and used $\theta$ for the angle
between $v$ and $\Delta_XL(X)$.

Intuitively, we want to find the direction that decreases L as rapidly as possible, that means $L(X + v) - L(X) = || \Delta_x L(X)|| \cos(\theta) \leq 0$. Therefore, the only way the direction we pick enters into this equation is through $\cos(\theta)$, and thus we wish to make this cosine as negative as possible. Now, recalling the shape of cosine, we can make this as negative as possible by making $\cos(\theta) = -1$ or equivalently making the angle between the gradient and our chosen direction to be $\pi$ radians or equivalently 180 degrees. 
- The only way to achieve this is to head in the exact opposite direction: pick $v$ to point in the exact opposite direction to $\Delta_XL(X)$.
- That means the direction of steepest decent points in the direction of $-\Delta_XL(X)$.

Thus our informal algorithm can be rewritten as follows.

1. Start with a random choice for the initial parameters $X$
2. Find the direction $v$ that makes $L$ decrease the most rapidly at $X$, that's $v = -\Delta_XL(X)$.
3. Take a small step in the opposite of that direction: $X \rightarrow X - \epsilon \Delta_XL(X)$
4. Repeat

**Summary**

 - Use the gradient to find the direction that decreases the loss as rapidly as possible, and update the parameters to take a step in that direction
 - The only possible points where we can minimize (or maximize) a function will have gradient equal to zero, however, not every point with gradient zero is the true global minimum (or maximum).

## Multivariate Chain Rule

The chain rule enables us to differentiate composite functions. 

Let us first consider functions of a single variable. Suppose that functions $y = f(u)$ and $u = g(x)$ are both differentiable, then the chain rule states that:
\begin{equation}
\frac{dy}{dx} = \frac{dy}{du}*\frac{du}{dx}
\end{equation}


Next, suppose that the differentiable function y has variables $y = f(x_1, x_2, \ldots, x_n) = f(u_1, u_2, \ldots, u_m)$, where each differentiable function $u_i = g_i(x_1, x_2, \ldots, x_n)$. Then the chain rule gives for a variable $x_i$:
\begin{equation}
\begin{aligned}
\frac{dy}{d{x_i}} &= \sum_{j = 1}^m \frac{dy}{d{u_j}}*\frac{d{u_j}}{d{x_i}}\\
&=\frac{dy}{d{u_1}}*\frac{d{u_1}}{d{x_i}} + \frac{dy}{d{u_2}}*\frac{d{u_2}}{d{x_i}} + \ldots + \frac{dy}{d{u_m}}*\frac{d{u_m}}{d{x_i}}
\end{aligned}
\end{equation}

### Example
Let us suppose that we have a function of four variables $(w, x, y$ and $z)$, as shown in following figure


we can make by composing many terms:
\begin{equation}
y = f(u, v) = (u + v)^2 \\
u = u(a, b) = (a + b)^2, \; v = v(a,b) = (a - b)^2 \\
a = a(w, x, y, z) = (w +x + y + z)^2, b = b(w, x, y, z) = (w + x - y - z)^2
\end{equation}

Such chains of equations are common when working with neural networks, so trying to understand how to compute gradients of such functions is key.

If we want to compute say $\frac{\partial f}{\partial w}$, we may apply the multi-variate chain rule to see:
\begin{equation}
\frac{\partial f}{\partial w} = \frac{\partial f}{\partial u}\frac{\partial u}{\partial w} + \frac{\partial f}{\partial v}\frac{\partial v}{\partial w} \\
\frac{\partial u}{\partial w} = \frac{\partial u}{\partial a}\frac{\partial a}{\partial w} + \frac{\partial u}{\partial b}\frac{\partial b}{\partial w} \\
\frac{\partial v}{\partial w} = \frac{\partial v}{\partial a}\frac{\partial a}{\partial w} + \frac{\partial v}{\partial b}\frac{\partial b}{\partial w} \\
\end{equation}

Let us try using this decomposition to compute $\frac{\partial f}{\partial w}$. Notice that all we need here are the various single step partials:
\begin{equation}
\frac{\partial f}{\partial u} = 2(u+v), \frac{\partial f}{\partial v} = 2(u+v) \\
\frac{\partial u}{\partial a} = 2(a+b), \frac{\partial u}{\partial b} = 2(a+b) \\
\frac{\partial v}{\partial a} = 2(a-b), \frac{\partial v}{\partial b} = -2(a+b) \\
\frac{\partial a}{\partial w} = 2(w+x+y+z), \frac{\partial b}{\partial w} = 2(w+x-y-z)
\end{equation}

If we write this out into code this becomes a fairly manageable expression.


```python
# Compute the value of the function from inputs to outputs
w, x, y, z = -1, 0, -2, 1
a, b = (w + x + y + z)**2, (w + x - y - z)**2
u, v = (a + b)**2, (a - b)**2
f = (u + v)**2
print(f' f at {w}, {x}, {y}, {z} is {f}')
```

     f at -1, 0, -2, 1 is 1024



```python
# Compute the single step partials
df_du, df_dv = 2 * (u + v), 2 * (u + v)
du_da, du_db = 2 * (a + b), 2 * (a + b)
dv_da, dv_db = 2 * (a - b), -2 * (a -b)

da_dw, db_dw = 2 * (w + x + y + z), 2 * (w + x - y - z)
da_dx, db_dx = 2 * (w + x + y + z), 2 * (w + x - y - z)
da_dy, db_dy = 2 * (w + x + y + z), -2 * (w + x - y - z)
da_dz, db_dz = 2 * (w + x + y + z), -2 * (w + x - y - z)

# Compute the final result from inputs to outputs
du_dw = du_da * da_dw + du_db * db_dw 
dv_dw = dv_da * da_dw + dv_db * db_dw
df_dw = df_du * du_dw + df_dv * dv_dw

print(f'df/dw at {w}, {x}, {y}, {z} is {df_dw}')
```

    df/dw at -1, 0, -2, 1 is -4096



```python
# Now compute how f changes when we change any value from output to input
df_da = df_du * du_da + df_dv * dv_da
df_db = df_du * du_db + df_dv * dv_db

df_dw = df_da * da_dw + df_db * db_dw
df_dx = df_da * da_dx + df_db * db_dx
df_dy = df_da * da_dy + df_db * db_dy 
df_dz = df_da * da_dz + df_db * db_dz

print(f'df/dw at {w}, {x}, {y}, {z} is {df_dw}')
print(f'df/dx at {w}, {x}, {y}, {z} is {df_dx}')
print(f'df/dy at {w}, {x}, {y}, {z} is {df_dy}')
print(f'df/dz at {w}, {x}, {y}, {z} is {df_dz}')
```

    df/dw at -1, 0, -2, 1 is -4096
    df/dx at -1, 0, -2, 1 is -4096
    df/dy at -1, 0, -2, 1 is -4096
    df/dz at -1, 0, -2, 1 is -4096


The fact that we compute derivatives from f back towards the inputs rather than from the inputs forward to the outputs (as we did in the first code snippet above) is what gives this algorithm its name: **backpropagation**. Note that there are two steps: 
1. Compute the value of the function, and the single step partials from front to back. While not done above, this can be combined into a single forward pass. 
2. Compute the gradient of f from back to front. We call this the backwards pass.


This is precisely what every deep learning algorithm implements to allow the computation of the gradient of the loss with respect to every weight in the network at one pass. It is an astonishing fact that we have such a decomposition.


```python
# Initialize as ndarrays, then attach gradients
w = torch.tensor([-1.], requires_grad=True)
x = torch.tensor([0.], requires_grad=True)
y = torch.tensor([-2.], requires_grad=True)
z = torch.tensor([1.], requires_grad=True)
# Do the computation like usual, tracking gradients
a, b = (w + x + y + z)**2, (w + x - y - z)**2
u, v = (a + b)**2, (a - b)**2
f = (u + v)**2
print(f' f: {f} at \n w: {w}, \n x: {x}, \n y: {y}, \n z: {z}')
```

     f: tensor([1024.], grad_fn=<PowBackward0>) at 
     w: tensor([-1.], requires_grad=True), 
     x: tensor([0.], requires_grad=True), 
     y: tensor([-2.], requires_grad=True), 
     z: tensor([1.], requires_grad=True)



```python
# Execute backward pass
#All of what we did above can be done automatically by calling f.backwards().
f.backward()

print(f'df/dw at {w.data.item()}, {x.data.item()}, {y.data.item()}, '
      f'{z.data.item()} is {w.grad.data.item()}')
print(f'df/dx at {w.data.item()}, {x.data.item()}, {y.data.item()}, '
      f'{z.data.item()} is {x.grad.data.item()}')
print(f'df/dy at {w.data.item()}, {x.data.item()}, {y.data.item()}, '
      f'{z.data.item()} is {y.grad.data.item()}')
print(f'df/dz at {w.data.item()}, {x.data.item()}, {y.data.item()}, '
      f'{z.data.item()} is {z.grad.data.item()}')
```

    df/dw at -1.0, 0.0, -2.0, 1.0 is -4096.0
    df/dx at -1.0, 0.0, -2.0, 1.0 is -4096.0
    df/dy at -1.0, 0.0, -2.0, 1.0 is -4096.0
    df/dz at -1.0, 0.0, -2.0, 1.0 is -4096.0


## How to calucate differentiation in Pytorch: Automatic Differentiation

### Backward for a scalar variable returned value y

Suppose a differentiating function $y = f(x) = 2*x^T*x$ where $x$ is a n-dimensional vectors:

\begin{equation}
y = f(x) = 2*x^T*x = 2 (x^2_1 + x^2_2 + \ldots + x^2_n)
\end{equation}

The gradients of this function is: 
\begin{equation}
\Delta_x f(x) = [\frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \ldots, \frac{\partial y}{\partial x_n}]^T = [4x_1, 4x_2, \ldots, 4x_n]^T
\end{equation}


```python
import torch

# A differentiating function f(x) = 2X^T * X with a scalar value returned
def f(x):
    return 2*torch.dot(x, x)

# let us create the variable x and assign it an initial value
x = torch.arange(4.0)
x
```




    tensor([0., 1., 2., 3.])




```python
# It is important that we do not allocate new memory every time we take a derivative with respect 
# to a parameter because we will often update the same parameters thousands or millions of times and
# could quickly run out of memory.
# The fact that gradients need to be computed for a Tensor do not mean that the grad attribute will be populated
x.requires_grad_(True) # Same as `x = torch.arange(4.0, requires_grad=True)`
x.grad
```


```python
# Now let us calculate y.
# Since x is a vector of length 4, an dot product of x and x is performed, yielding the scalar output that we assign to y.
y = f(x)
y
```




    tensor(28., grad_fn=<MulBackward0>)




```python
# At this time, there is no gradients are calcuated for x
x.grad
```


```python
# Next, we can automatically calculate the gradient of y with respect to each
# component of x by calling the function for backpropagation and printing the gradient.
y.backward()
x.grad
```




    tensor([ 0.,  4.,  8., 12.])




```python
# The gradient of the function y = 2x^⊤*x with respect to x should be 4x.
x.grad == 4 * x
```




    tensor([True, True, True, True])




```python
# PyTorch accumulates the gradient in default, we need to clear the previous values
x.grad.zero_()
x.grad
```




    tensor([0., 0., 0., 0.])




```python
# Now let us calculate another function of y = sum(x) = x_1 + x_n + ... + x_n.
y = x.sum()
y.backward()
x.grad
```




    tensor([1., 1., 1., 1.])



### Backward for Non-Scalar returned Variables Y

The most natural interpretation of the differentiation of a vector y with respect to a vector x is a matrix. For higher-order and higher-dimensional y and x, the differentiation result could be a high-order tensor.


```python

import torch

# A differentiating function f(x) = 2X^T * X with a scalar value returned
def f(x):
    return x * x

# let us create the variable x and assign it an initial value
x = torch.arange(4.0, requires_grad=True)
print(f"The input x: {x}")

y = f(x)
print(f"The output Y: {y}")

# Invoking `backward` on a non-scalar requires passing in a `gradient` argument
# which specifies the gradient of the differentiated function w.r.t `self`.
# In our case, we simply want to sum the partial derivatives, so passing
# in a gradient of ones is appropriate
y.sum().backward() # or y.backward(torch.ones(len(x))) equivalent to the below

print(f"The x grad: {x.grad}")
```

    The input x: tensor([0., 1., 2., 3.], requires_grad=True)
    The output Y: tensor([0., 1., 4., 9.], grad_fn=<MulBackward0>)
    The x grad: tensor([0., 2., 4., 6.])


### Computing the Gradient of Python Control Flow


```python
def f(a):
    b = a * 2
    while b.norm() < 1000:
        b = b * 2
    if b.sum() > 0:
        c = b
    else:
        c = 100 * b
    return c
```


```python
a = torch.randn(size=(), requires_grad=True)
d = f(a)
d.backward()
print (f"a: {a}")
print (f"a grad: {a.grad}")
print (f"d: {d}")
```

    a: -0.18225865066051483
    a grad: 819200.0
    d: -149306.28125



```python

a.grad == d / a
```




    tensor(True)



# Integral Calculus （积分）


```python

```

a52b6ad5c4f4601a244a7737d0d411ae1148d994
ipynb
Jupyter Notebook
deep_learning/Calculus.ipynb
bingrao/notebook
4bd74a09ffe86164e4bd318b25480c9ca0c6a462
MIT
deep_learning/Calculus.ipynb
bingrao/notebook
4bd74a09ffe86164e4bd318b25480c9ca0c6a462
MIT
deep_learning/Calculus.ipynb
bingrao/notebook
4bd74a09ffe86164e4bd318b25480c9ca0c6a462
MIT
Qwen/Qwen-72B
1. YES
2. YES