into numerical issues (with floating point computation). See \[sec:experiments\] for details.

Over the course of our investigation of $\alpha$-ReLU, we derived several new identities involving the associated kernel functions, first defined in [@cho_kernel_2009], which relate them to the zeroth Bessel functions (\[lemma:JalphaBessel,lemma:LAlphaRec,lemma:JalphaRec,lemma:JalphaGrad\]).

Theoretical Results
===================

In what follows in the main text, we assume $\sigma_\bullet > 0$ for all $\bullet = w, v, b, a$; in the appendix, the formal statement of each main theorem will contain results for other cases. We are interested in the two major categories of nonlinearities used today: tanh-like and rectified units. We make the following formal definitions as a foundation for further consideration.

We say a function $\phi$ is [**tanh-like**]{} if $\phi$ is antisymmetric ($\phi(-x) = -\phi(x)$), $|\phi(x)| \le 1$ for all $x$, $\phi(x) \ge 0, \forall x \ge 0$, and $\phi(x)$ monotonically increases to 1 as $x \to \infty$.

\[defn:alphaReLU\] Define the $\alpha$-ReLU $\psi_\alpha(x) = x^\alpha$ if $x > 0$ and 0 otherwise. [^3]

  Antisymmetric/RRN                                                                            Any/FRN                                                                                 
  -------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------- --
                                                                                                                                                                                       
  Theorems \[thm:p\_q\_linear\], \[thm:lambda\_gamma\_recurrence\], \[thm:dalethRecReduced\]   Theorems \[thm:fullResPQRec\], \[thm:full\_res\_l\_g\_recurr\], \[thm:dalethRecFull\]   

  : Main Recurrences[]{data-label="tab:recurrences"}

By applying the central limit theorem as described in the last section, we derive a set of recurrences for different activation/architecture pairs, shown 