# Mathematical Preliminaries

**DIVE into Deep Learning**
___


```python
from IPython import display
from manim import *
```

The following is a lecture series that introduces the basic theory of deep learning.


```python
display.IFrame(src="https://www.youtube.com/embed/videoseries?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi", width="100%", height=450)
```

Understanding the lectures requires some background on vector calculus and probability theory. $\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}
$

**What to know about vector calculus?**


```python
%%html
<b>Notation:</b><br>

```

- [Vectors](https://en.wikipedia.org/wiki/Vector_(mathematics_and_physics)) in lowercase boldface font: 

$$\M{x}=\begin{bmatrix}x_1 \\ x_2 \\ \vdots \end{bmatrix}$$

- [Matrices](https://en.wikipedia.org/wiki/Matrix_(mathematics)) in uppercase boldface font:

$$\M{W}=\begin{bmatrix}w_{11} & w_{12} & \cdots \\
w_{21} & \ddots &  \\
\vdots &  &  \end{bmatrix}$$

- [Matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication):

$$\M{W}\M{x} = \begin{bmatrix}w_{11} & w_{12} & \cdots \\
w_{21} & \ddots &  \\
\vdots &  &  \end{bmatrix}
\begin{bmatrix}x_1 \\ x_2 \\ \vdots \end{bmatrix}
=
\begin{bmatrix}w_{11}x_1 + w_{12}x_2 + \cdots \\ w_{21}x_1+\cdots \\ \vdots \end{bmatrix}
$$

**What to know about Probability Theory?**


```python
%%html
<b>Distribution:</b><br>

```

[Joint distribution](https://en.wikipedia.org/wiki/Joint_probability_distribution#Mixed_case):  

$$p_{\RM{x}\R{y}}(\M{x},y)= \underbrace{p_{\R{y}|\RM{x}}(y|\M{x})}_{\underbrace{\Pr}_{\text{probability measure}\kern-3em}\Set{\R{y}=y|\RM{x}=\M{x}}} \cdot \underbrace{p_{\RM{x}}(\M{x})}_{(\underbrace{\partial_{x_1}}_{\text{partial derivative w.r.t. $x_1$}\kern-5em} \partial_{x_2}\cdots)\Pr\Set{\RM{x} \leq \M{x}}\kern-4em}\kern1em \text{where}$$    
- $p_{\R{y}|\RM{x}}(y|\M{x})$ is the *probability mass function [(pmf)](https://en.wikipedia.org/wiki/Probability_mass_function)* of $\R{y}=y\in \mc{Y}$ [conditioned](https://en.wikipedia.org/wiki/Conditional_probability_distribution) on $\RM{x}=\M{x}\in \mc{X}$, and
- $p_{\RM{x}}(\M{x})$ is the *(multivariate) probability density function [(pdf)](https://en.wikipedia.org/wiki/Probability_density_function#Densities_associated_with_multiple_variables)* of $\RM{x}=\M{x}\in \mc{X}$.


```python
%%html
<b>Expectation:<br>

```

For any function $g$ of $(\RM{x},y)$, the expectations are:  
  
$$
\begin{align}
\E[g(\RM{x},\R{y})|\RM{x}]&=\sum_{y\in \mc{Y}} g(\RM{x},y)\cdot p_{\R{y}|\RM{x}}(y|\RM{x})\tag{conditional exp.}
\\
\E[g(\RM{x},\R{y})] &=\int_{\mc{X}} \underbrace{\sum_{y\in \mc{Y}} g(\RM{x},y)\cdot \underbrace{p_{\RM{x},\R{y}}(\M{x},y)}_{p_{\R{y}|\RM{x}}(y|\M{x}) p_{\R{x}}(\M{x})}\kern-1.7em}_{\E[g(\RM{x},\R{y})|\RM{x}]}\kern1.4em\,\d \M{x} \tag{exp.}\\
&= \E[\E[g(\RM{x},\R{y})|\RM{x}]] \tag{iterated exp.}
\end{align}
$$

All the videos were created using the python library [mathematical animation engine (`manim`)](https://github.com/3b1b/manim).


```python
from manim import *
```

**Exercise** Run the following cell and see the effect when changing

- Mobjects `Text('Hello, World!')` to 
  - `MathTex(r'E=mc^2')` or
  - `Circle()` or
  - `Square()`...
- Animation objects `Write` to 
  - `FadeIn` or 
  - `GrowFromCenter`...

You may take a look at the documentation [here](https://docs.manim.community/en/v0.2.0/index.html) and a more detailed [tutorial](https://talkingphysics.wordpress.com/2019/01/08/getting-started-animating-with-manim-and-python-3-7/) here to create more complicated animations. Since rending videos take time, create a separate notebook for more intensive work without affecting the autograding of this notebook.


```python
%%manim -ql --progress_bar=none --disable_caching --flush_cache -v ERROR HelloWorld
class HelloWorld(Scene):
    def construct(self):
        self.play(Write(Text('Hello, World!')))
```

To show all the options available to `manim`:


```python
%manim render --help
```

aecc6622c72ac352744ff1957e5e1a787110e7db
ipynb
Jupyter Notebook
part1/theory.ipynb
ccha23/divedeep
dd9c5e0a589613fa37c467b7863e58c5d22d8d3f
MIT
2021-06-29T00:46:39.000Z
2021-06-29T00:46:39.000Z
part1/theory.ipynb
ccha23/divedeep
dd9c5e0a589613fa37c467b7863e58c5d22d8d3f
MIT
part1/theory.ipynb
ccha23/divedeep
dd9c5e0a589613fa37c467b7863e58c5d22d8d3f
MIT
2021-07-03T02:44:06.000Z
2021-07-03T02:44:06.000Z
Qwen/Qwen-72B
1. YES
2. YES