Convolutional unitary or orthogonal recurrent neural networks
Recurrent neural networks are extremely powerful yet hard to train. One of
their issues is the vanishing gradient problem, whereby propagation of training
signals may be exponentially attenuated, freezing training. Use of orthogonal
or unitary matrices, whose powers neither explode nor decay, has been proposed
to mitigate this issue, but their computational expense has hindered their use.
Here we show that in the specific case of convolutional RNNs, we can define a
convolutional exponential and that this operation transforms antisymmetric or
anti-Hermitian convolution kernels into orthogonal or unitary convolution
kernels. We explicitly derive FFT-based algorithms to compute the kernels and
their derivatives. The computational complexity of parametrizing this subspace
of orthogonal transformations is thus the same as the networks' iteration.
Marcelo O. Magnasco
# Convolutional _unitary_ or _orthogonal_ recurrent neural networks.

###### Abstract

Recurrent neural networks are extremely powerful yet hard to train. One of their issues is the _vanishing gradient problem_, whereby propagation of training signals may be exponentially attenuated, freezing training. Use of _orthogonal_ or _unitary_ matrices, whose powers neither explode nor decay, has been proposed to mitigate this issue, but their computational expense has hindered their use. Here we show that in the specific case of convolutional RNNs, we can define a _convolutional exponential_ and that this operation transforms antisymmetric or anti-Hermitian convolution kernels into _orthogonal_ or _unitary convolution kernels_. We explicitly derive FFT-based algorithms to compute the kernels and their derivatives. The computational complexity of parametrizing this subspace of orthogonal transformations is thus the same as the networks' iteration.

Lab of Integrative Neuroscience, Rockefeller University

## 1 tl;dr

This is an extremely terse synopsis for quick reference. All proofs, explanations and misguided attempts at clarity are in subsequent chapters, to which you're welcome to skip to.

Given a layer \(X\) in \(D\) dimensions, a spatial convolution operation \(\otimes\) and a convolution kernel \(K\) acting on \(X\) as \(K\otimes X\), we formally define the _convolutional exponential_\(e_{\otimes}^{K}\) as the kernel defined by the series

\[e_{\otimes}^{K}\otimes X\equiv X+K\otimes X+\frac{1}{2!}K\otimes K\otimes X+ \frac{1}{3!}K\otimes K\otimes K\otimes X+\frac{1}{4!}K\otimes K\otimes K\otimes K \otimes X+\cdots \tag{1.1}\]

so the linear operator defined by \(e_{\otimes}^{K}\otimes\) is _quite literally_ the matrix exponential of the linear operator defined by \(K\otimes\). This exponential can be computed in Fourier space through

\[e_{\otimes}^{K}\equiv\mathscr{F}^{-1}\left[\exp(\mathscr{F}\left[K\right])\right] \tag{1.2}\]

where the right-hand exponential is element-wise and where \(\mathscr{F}\) and \(\mathscr{F}^{-1}\) are the forward and inverse Fourier transforms in \(D\) dimensions, and hence an \(N\log N\) operation. This can easily be generalized to any operation defined through convergent power series, for example the _convolutional sine and cosine_ of \(K\) are defined through

\[\begin{array}{l}\cos_{\otimes}(K)\equiv\mathscr{F}^{-1}\left[\cos(\mathscr{ F}\left[K\right])\right]\\ \sin_{\otimes}(K)\equiv\mathscr{F}^{-1}\left[\sin(\mathscr{F}\left[K\right])\right] \end{array} \tag{1.3}\]

Given a complex-valued kernel \(K\), we define an _anti-Hermitian kernel_ as one that satisfies \(K=-\overline{K^{*}}\) where \(\bar{K}\) is the spatial flip operation and \(K^{*}\) the elementwise complex conjugate, because then the linear operator given by \(K\otimes\) is an anti-Hermitian operator. Then \(e_{\otimes}^{K}\) is _unitary_ in the sense that the linear operator \(e_{\otimes}^{K}\otimes\) is a _unitary operator_: it is the matrix exponential of the anti-Hermitian operator \(K\otimes\), and as such has eigenspectrum on the unit circle.

Given a complex-valued layer \(Z\) in \(D\) dimensions, an anti-Hermitian kernel \(K\) acting on \(X\), an input \(I_{n}\), and element-wise complex-valued activation function \(\phi\), we define a _convolutional unitary recurrent neural network_ (cuRNN?) as the iterated recursion

\[Z_{n+1}=\phi\left(e_{\otimes}^{K}\otimes Z_{n}+I_{n}\right) \tag{1.4}\]

where the subindex \(n\) represents the passage of time in the recurrence and \(Z_{0}\) is the initialization value of the layer.