$*$ gives $$\begin{aligned}
\label{onederivative}
\frac{\partial}{\partial *} L(p) = \sum_{\alpha=1}^n 2 (f_p(x_\alpha) - y_\alpha) \frac{\partial}{\partial *} f_p(x_\alpha).\end{aligned}$$

In the deep nonlinear case, the function computed by the neural network is of the following form:

$$\label{f}
\begin{aligned}
f_p(z) = 
\begin{pmatrix}
w_{11}^\ell &...& w_{1 m_{\ell-1}}^\ell &b_1^\ell \\
&\vdots & & \\
w_{m_\ell 1}^\ell &...& w_{m_\ell m_{\ell-1}}^\ell &b_{m_\ell}^\ell \\
\end{pmatrix}
\circ \sigma \circ
\begin{pmatrix}
w_{11}^{\ell-1} &...& w_{1 m_0}^1 &b_1^{\ell-1}\\
&\vdots & & \\
w_{m_1 1}^{\ell-1} &...& w_{m_1 m_0}^1 &b_{m_1}^{\ell-1} \\
0 & ... & 0 & 1 \\
\end{pmatrix}&
\circ \sigma \circ \\
...
\circ \sigma \circ
\begin{pmatrix}
w_{11}^1 &...& w_{1 m_0}^1 &b_1^1\\
&\vdots & & \\
w_{m_1 1}^1 &...& w_{m_1 m_0}^1 &b_{m_1}^1 \\
0 & ... & 0 & 1 \\
\end{pmatrix}
\begin{pmatrix}
z_1 \\
\vdots \\
z_{m_0}\\
1 \\
\end{pmatrix}&
\end{aligned}$$

Thus an entry of $f_p(z)$ will have the form

$$\begin{aligned}
\label{fentry}
\sum w..^\ell \sigma \left( \sum w..^{\ell-1} ... \sigma \left( \sum w..^2 \sigma \left( \sum w..^1 z. + b.^1 \right) + b.^2 \right) + ... + b.^{\ell-1} \right) + b.^\ell.\end{aligned}$$

For us, it will be sufficient to keep track of the shapes of the terms that appear, without keeping track of the indices.

\[notell\] Any derivative of any order of $f_p(z)$ 