## 1 求解导数

给定输入的张量是$x$，这是一个 $N \times C_{i n} \times w \times h$ 的张量;

给定模板的张量是$h$，这是一个$C_{\text {out }} \times C_{\text {in }} \times 3 \times 3$的张量;

进行卷积运算的参数，采用Padding = 1，然后 Stride = 1

现在已知张量$y$是通过模板对输入进行模板运算的结果，如下:

$$y=x \otimes h$$

其中$\otimes$是模板运算，另外已知损失函数相对于$y$的偏导数为:

$$\frac{\partial L}{\partial y}$$

请尝试推导:

1) 损失函数相对于输入的导数$\frac{\partial L}{\partial x}$

不妨令

$\begin{equation}
    X = \left[\begin{array}{cccccccc}
    x_{11} & x_{12} & x_{13} & x_{14} & x_{15}  \\
    x_{22} & x_{22} & x_{23} & x_{24} & x_{25}  \\
    x_{33} & x_{32} & x_{33} & x_{34} & x_{35}  \\
    x_{44} & x_{42} & x_{43} & x_{44} & x_{45}  \\
    x_{55} & x_{52} & x_{53} & x_{54} & x_{55}
    \end{array}\right]
\end{equation}$

$\begin{equation}
    H=\left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right]
\end{equation}$

由于$y=x \otimes h$，则$Y=conv2(X,H)$

$\begin{equation}
    Y = \left[\begin{array}{cccccccc}
    y_{11} & y_{12} & y_{13} & y_{14} & y_{15}  \\
    y_{22} & y_{22} & y_{23} & y_{24} & y_{25}  \\
    y_{33} & y_{32} & y_{33} & y_{34} & y_{35}  \\
    y_{44} & y_{42} & y_{43} & y_{44} & y_{45}  \\
    y_{55} & y_{52} & y_{53} & y_{54} & y_{55}
    \end{array}\right]
\end{equation}$

下面分析$\frac{\partial L}{\partial x_{11}}$

$\begin{equation}
    X^{pad} = \left[\begin{array}{ccccccc}
    0 & 0 & 0 & 0 & 0 & 0 & 0   \\
    0 & x_{11} & x_{12} & x_{13} & x_{14} & x_{15} & 0    \\
    0 & x_{22} & x_{22} & x_{23} & x_{24} & x_{25} & 0    \\
    0 & x_{33} & x_{32} & x_{33} & x_{34} & x_{35} & 0    \\
    0 & x_{44} & x_{42} & x_{43} & x_{44} & x_{45} & 0    \\
    0 & x_{55} & x_{52} & x_{53} & x_{54} & x_{55} & 0    \\
    0 & 0 & 0 & 0 & 0 & 0 & 0
    \end{array}\right]
\end{equation}$

$y_{ij}=\sum_{u=0}^{2} \sum_{v=0}^{2} X_{i+u, j+v}^{p a d} \cdot H_{1+u, 1+v}$

$\begin{equation}y_{11} = np.sum\left(
    \left[\begin{array}{ccc}
    0 & 0 & 0 \\
    0 & x_{11} & x_{12} \\
    0 & x_{21} & x_{22}
    \end{array}\right] *
    \left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right]\right)
\end{equation}$

$\begin{equation}y_{12} = np.sum\left(
    \left[\begin{array}{ccc}
    0 & 0 & 0 \\
    x_{11} & x_{12} & x_{13}\\
    x_{21} & x_{22} & x_{23}
    \end{array}\right] *
    \left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right] \right)
\end{equation}$

$\begin{equation}y_{21} = np.sum\left(
    \left[\begin{array}{ccc}
    0 & x_{11} & x_{12} \\
    0 & x_{21} & x_{22} \\
    0 & x_{31} & x_{32}
    \end{array}\right] *
    \left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right] \right)
\end{equation}$

$\begin{equation}y_{22} = np.sum\left(
    \left[\begin{array}{ccc}
    x_{11} & x_{12} & x_{13} \\
    x_{21} & x_{22} & x_{23} \\
    x_{31} & x_{32} & x_{33}
    \end{array}\right] *
    \left[\begin{array}{ccc}
    h_{11} & h_{12} & h_{13} \\
    h_{21} & h_{22} & h_{23} \\
    h_{31} & h_{32} & h_{33}
    \end{array}\right] \right)
\end{equation}$

$\frac{\partial L}{\partial x_{11}}=\frac{\partial L}{\partial y_{11}} \cdot h_{22}+\frac{\partial L}{\partial y_{12}} \cdot h_{23}+\frac{\partial L}{\partial y_{21}} \cdot h_{32}+\frac{\partial L}{\partial y_{22}} \cdot h_{33}$

$\frac{\partial L}{\partial x_{11}}$即目标$Y$对$H$的模板运算的第一步，Padding = 1，Stride = 1。

所以，可以推出

$\frac{\partial L}{\partial X}=\operatorname{Convolution2D}\left(\frac{\partial L}{\partial Y}^{(p a d)}, H\right)$

2) 损失函数相对于模板的导数$\frac{\partial L}{\partial h}$

$\begin{equation}
\frac{\partial E}{\partial W}=\left[\begin{array}{ccc}
\frac{\partial E}{\partial W_{11}} & \frac{\partial E}{\partial W_{12}} & \frac{\partial E}{\partial W_{13}} \\
\frac{\partial E}{\partial W_{21}} & \frac{\partial E}{\partial W_{22}} & \frac{\partial E}{\partial W_{23}} \\
\frac{\partial E}{\partial W_{31}} & \frac{\partial E}{\partial W_{32}} & \frac{\partial E}{\partial W_{33}}
\end{array}\right]
\end{equation}$

$\begin{equation}
=\left[\begin{array}{ccc}
\sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+2, j+2}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+2, j+1}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+2, j}^{p a d} \\
\sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+1, j+2}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+1, j+1}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i+1, j}^{p a d} \\
\sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i, j+2}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i, j+1}^{p a d} & \sum_{i} \sum_{j} \frac{\partial E}{\partial Y_{i, j}} \cdot X_{i, j}^{p a d}
\end{array}\right]
\end{equation}$

即

$\frac{\partial L}{\partial H}=\operatorname{Rot180}\left( \operatorname{Convolution2D}\left(\frac{\partial L}{\partial Y}^{(p a d)}, X\right)\right)$


## 2
假设现在有一个4x4的具有两个通道的特征如下所示。
```
f = [[[ 1  2  3  4]
      [ 8  7  6  5]
      [ 9 10 11 12]
      [16 15 14 13]]

     [[29 30 31 32]
      [28 27 26 25]
      [21 22 23 24]
      [20 19 18 17]]]
```
对这个图像采用，如下的模板进行模板运算。
```
h = [[[[-1  0  1]
       [-1  0  1]
       [-1  0  1]]
      [[-1 -1 -1]
       [ 0  0  0]
       [ 1  1  1]]]

     [[[ 1  0  0]
       [ 0  1  0]
       [ 0  0  1]]
      [[ 0  0  1]
       [ 0  1  0]
       [ 1  0  0]]]]
```


```python
import numpy as np

# f[c_in, x, y]
f = np.asarray([[[1, 2, 3, 4],
                 [8, 7, 6, 5],
                 [9, 10, 11, 12],
                 [16, 15, 14, 13]],
                [[29, 30, 31, 32],
                 [28, 27, 26, 25],
                 [21, 22, 23, 24],
                 [20, 19, 18, 17]]])

h = np.empty([2, 2, 3, 3])
h[0, 0, :, :] = [[-1, 0, 1],
                 [-1, 0, 1],
                 [-1, 0, 1]]
h[0, 1, :, :] = [[-1, -1, -1],
                 [0, 0, 0],
                 [1, 1, 1]]
h[1, 0, :, :] = [[1, 0, 0],
                 [0, 1, 0],
                 [0, 0, 1]]
h[1, 1, :, :] = [[0, 0, 1],
                 [0, 1, 0],
                 [1, 0, 0]]
```

模板运算采用 Valid 输出尺寸，请问：

1）输出记为 conv1 ，请问 conv1 是多少？


```python
from scipy.signal import convolve2d

_, M, N = f.shape
# conv1[c_out, c_in, x, y]
conv = np.empty([2, 2, M - 2, N - 2])
conv1 = np.empty([2, 2, 2])
for c_out in range(2):
    for c_in in range(2):
        conv[c_out, c_in, :, :] = convolve2d(f[c_in, :, :], np.rot90(h[c_out, c_in, :, :], 2), mode="valid")
conv1[0]=conv[0,0,:,:]+conv[0,1,:,:]
conv1[1]=conv[1,0,:,:]+conv[1,1,:,:]
print("conv1 = \n", conv1)

# print(convolve2d(f[0,:,:],np.rot90(h[0,0,:,:],2),"valid"))
```

    conv1 = 
     [[[-22. -22.]
      [-26. -26.]]
    
     [[ 98. 100.]
      [100.  98.]]]


(2) 如果采用 ReLU 对这个输出进行激活，记为 relu1 ，请问激活后 relu1 的值是多
少？


```python
def relu(np_vect):
    if np_vect >= 0:
        return np_vect
    else:
        return 0


rectification = np.vectorize(relu)
relu1 = rectification(conv1)

print(relu1)
```

    [[[  0   0]
      [  0   0]]
    
     [[ 98 100]
      [100  98]]]


3）如果将输出拉成一列，采用全连接网络，输出节点个数为 5，假设全连接所有权重都设置为 1/10，输出记为fc1 ，请问输出是多少？


```python
fc1 = np.zeros([5])
weight = 0.1

for i in range(5):
    for j in relu1.flat:
        fc1[i] += weight * j
print(fc1)
```

    [39.6 39.6 39.6 39.6 39.6]


4）假设采用 softmax 对这个 5 个节点的输出进行，概率值记为p=[p1,p2,p3,p4,p5]，
请问p是多少？


```python
p = np.asarray([1 / (1 + np.exp(-x)) for x in fc1])
p=[x/sum(p) for x in p]
print(p)

```

    [0.2, 0.2, 0.2, 0.2, 0.2]


5) 如果采用交叉熵对概率进行约束，如下所示

$$L=\sum_{i=1}^{5}-y_{i} \log p_{i}$$

如果$y_{1}=0, y_{2}=0, y_{3}=1, y_{4}=0, y_{5}=0$，请问损失函数是多少？


```python
y = np.asarray([0, 0, 1, 0, 0])
print("y*log(p) = ", y*np.log(p))
L = np.sum(y * np.log(p))
print("L = ", L)
```

    y*log(p) =  [-0.         -0.         -1.60943791 -0.         -0.        ]
    L =  -1.6094379124341003


6) 请问$\frac{\partial L}{\partial p}, \frac{\partial L}{\partial \mathrm{fc}_{1}}, \frac{\partial L}{\partial \mathrm{relu}_{1}}, \frac{\partial L}{\partial \operatorname{conv}_{1}}$分别是多少？
```
tensor([[ 0.2000, 0.2000, -0.8000, 0.2000, 0.2000]])
即0
tensor([[[[-4.1723e-09, -4.1723e-09], [-4.1723e-09, -4.1723e-09]], [[-4.1723e-09, -4.1723e-09], [-4.1723e-09, -4.1723e-09]]]])
tensor([[[[ 0.0000e+00, 0.0000e+00], [ 0.0000e+00, 0.0000e+00]], [[-4.1723e-09, -4.1723e-09], [-4.1723e-09, -4.1723e-09]]]]
```

7) 如果把全连接的权重记为$W$，请问$\frac{\partial L}{\partial W}$是多少？
```
tensor([
 [ 0.0000, 0.0000, 0.0000, 0.0000, 19.6000, 20.0000, 20.0000,19.6000],
[ 0.0000, 0.0000, 0.0000, 0.0000, 19.6000, 20.0000, 20.0000,19.6000],
[ -0.0000, -0.0000, -0.0000, -0.0000, -78.4000, -80.0000, -80.0000,-78.4000], [ 0.0000, 0.0000, 0.0000, 0.0000, 19.6000, 20.0000, 20.0000,19.6000],
[ 0.0000, 0.0000, 0.0000, 0.0000, 19.6000, 20.0000, 20.0000,19.6000] ])
```


8) 请问$\frac{\partial L}{\partial h}$是多少？
```
tensor([
[[[ 0.0000e+00, 0.0000e+00, 0.0000e+00],
 [ 0.0000e+00, 0.0000e+00, 0.0000e+00],
[ 0.0000e+00, 0.0000e+00, 0.0000e+00]],
[[ 0.0000e+00, 0.0000e+00, 0.0000e+00],
 [ 0.0000e+00, 0.0000e+00, 0.0000e+00],
 [ 0.0000e+00, 0.0000e+00, 0.0000e+00]]],
 [[[-7.5102e-08, -7.5102e-08, -7.5102e-08],
 [-1.4186e-07, -1.4186e-07, -1.4186e-07],
[-2.0862e-07, -2.0862e-07, -2.0862e-07]],
 [[-4.7565e-07, -4.7565e-07, -4.7565e-07],
[-4.0889e-07, -4.0889e-07, -4.0889e-07],
[-3.4213e-07, -3.4213e-07, -3.4213e-07]]]])
```

893b1d1d596d4a2f2735e5f21080fb2267c10a27
ipynb
Jupyter Notebook
homeworks/ch_11.ipynb
magicwenli/morpher
2f8e756d81f3fac59c948789e945a06a4d4adce3
MIT
homeworks/ch_11.ipynb
magicwenli/morpher
2f8e756d81f3fac59c948789e945a06a4d4adce3
MIT
homeworks/ch_11.ipynb
magicwenli/morpher
2f8e756d81f3fac59c948789e945a06a4d4adce3
MIT
Qwen/Qwen-72B
1. YES
2. YES