in \[tab:recurrences\] (see appendix for proofs). They leverage certain integral transforms [^4] as in the following

\[defn:integralTransform\] Define the transforms ${\mathrm{V}}$ and ${\mathrm{W}}$ by ${\mathrm{V}}\phi(q) := \EV[\phi(z)^2: z \sim {\mathcal{N}}(0, q)]$ and ${\mathrm{W}}\phi(\rho, \nu) := \EV[\phi(z)\phi(z'): (z, z') \sim {\mathcal{N}}(0, \begin{pmatrix}\rho & \nu \\ \nu & \rho \end{pmatrix})]$.

These recurrences are able to track the corresponding quantities in practice very well. For example, \[fig:theory\_tracks\_pratice\] compares theory vs experiments for the tanh/FRN pair. The agreement is very good for tanh/RRN (not shown, but similar to the case of tanh/FRN with $\sigma_v = 1$ and $\sigma_a = 0$) and $\alpha$-ReLU/FRN as well (see \[fig:alphaReLUTheoryVsEmpirics\]).

As mentioned in previous sections, we seek to characterize the long term/high depth behavior of all of the quantities defined in \[sec:background\]. To do so, we solve for the asymptotics of the recurrences in \[tab:recurrences\], where $\phi$ is instantiated with tanh or $\alpha$-ReLU. Our main dynamics results are summarized in \[tab:dynamics\].

                                        Tanh/RRN                                                Tanh/FRN                                             ReLU/FRN                                             $\alpha$-ReLU/FRN, $\alpha<1$                                                           
  ------------------------------------- ------------------------------------------------------- ---------------------------------------------------- ---------------------------------------------------- --------------------------------------------------------------------------------------- --
  $\p {\mathbf{p}}l$                    $\Theta(l)$, \[thm:p\_q\_linear\]                       $\Theta(l)$, \[thm:pIsLinearTanh\]                   $\exp(\Theta(l))$, \[thm:pDynamicAlphaReLU\]         $\Theta(l^{1/(1-\alpha)})$, \[thm:pDynamicAlphaReLU\]                                   
  $\p {\mathbf{s}}l$                    $\Theta(l)$, \[thm:edynamics\]                          $\Theta(l)$, \[thm:eDynamicsFullResTanh\]            $\exp(\Theta(l))$, \[thm:ReLUSquaredConvergence\]    $\Theta(l^{1/(1-\alpha)})$, \[thm:alphaReLUeConvergence\]                               
  $\p {\mathbf{e}}l - {\mathbf{e}}^*$   ${{\check\Theta}}(l^{\f2 \pi -1})$, \[thm:edynamics\]   ${\mathsf{poly}}(l)$, \[thm:eDynamicsFullResTanh\]   $\Theta(l^{-2})$, \[thm:ReLUSquaredConvergence\]     ${\mathsf{poly}}(l)$, \[thm:alphaReLUeConvergence\]                                     
  $\p {{\boldsymbol{\oldchi}}}l$        