width at least $m$. Let the width of the $i^{th}$ layer be denoted $m_i$, so we assume each $m_i \geq m$ for $1 \leq i \leq \ell-1$. Let $m_0 = a$ and $m_\ell = b$. Let the number of parameters of the network be denoted by $$d = \sum_{i = 1}^{\ell} (m_{i-1} + 1)m_i.$$

\[netdiagram\]

A fully connected feedforward network of this form parameterizes functions of the form $$\label{basicformula}
M_\ell(\sigma(M_{\ell-1}( ... ( \sigma (M_1 x + b_1) ... ) + b_\ell$$ where each $M_i x + b_i$ is an affine linear transformation from ${\mathbb R}^{m_{i-1}}$ to ${\mathbb R}^{m_i}$.

We sometimes denote the affine linear transformation $x \rightarrow M_i x + b_i$ by $A_i$. In this notation (\[basicformula\]) can be expressed as $$A_\ell \circ \sigma \circ A_{\ell-1} \circ ... \circ \sigma \circ A_1(x).$$

In Figure \[netdiagram\], we diagram the feedforward neural network we have just described.

The space of all weights and biases for all the layers is isomorphic to ${\mathbb R}^d$. It will be helpful to distinguish the weights and biases for each layer, so let $${\mathbb R}^{d_i} = \{ (M_{i-1}, b_{i-1} )\}$$ denote the space of weights and biases that determine the affine map from the $(i-1)^{st}$ to the 