with respect to the parameter variables that does not involve $\frac{\partial}{\partial w_{..}^\ell}$ derivatives vanishes. The only exception is the first derivative $\frac{\partial}{\partial b_{.}^\ell} f_p(z)$.

Let $D$ be a derivative satisfying the assumptions. Using expression \[fentry\], $$\begin{aligned}
D f_p(z) &= D \left( \sum w..^\ell \sigma \left( \sum w..^{\ell-1} ... \sigma \left( \sum w..^1 z. + b.^1 \right) + ... + b.^{\ell-1} \right) + b.^\ell \right)  \\
& = D \left(\sum w..^\ell \sigma \left( \sum w..^{\ell-1} ... \sigma \left( \sum w..^1 z. + b.^1 \right) + ... + b.^{\ell-1} \right) \right) + D( b.^\ell)\end{aligned}$$ Since $D$ does not contain any $w..^\ell$ derivatives, the first term vanishes. And except for the derivative $\frac{\partial}{\partial b.^\ell}$, any derivative of $b.^\ell$ vanishes, so the second term vanishes as well.

To show that every point in $S = \bigcup S_k$ is a critical point, we take any $k$ and any $p \in S_k$ and show that $\frac{\partial}{\partial *} L(p) = 0$ for any parameter $*$.

For most parameters, we will show that $\frac{\partial}{\partial *} f_p(x_\alpha) = 0$ for all $x_\alpha$.

For all $i \neq \ell$, the derivatives $$\frac{\partial}{\partial w..^i} f_p(x_\alpha)$$ and $$\frac{\partial}{\partial b.^i} f_p(x_\alpha)$$ vanish, by Lemma \[notell\].

Next, we check the derivative $$\frac{\partial}{\partial w..^\ell} f_p(x_\alpha).$$

The 