Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients.   Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs).   We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices. This is done by parametrizing with a skew-symmetric matrix using the Cayley transform. Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones.   The proposed training scheme involves a straightforward gradient calculation and update step. In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs. Deep neural networks have been used to solve numerical problems of varying complexity. RNNs have parameters that are reused at each time step of a sequential data point and have achieved state of the art performance on many sequential learning tasks. Nearly all optimization algorithms for neural networks involve some variant of gradient descent. One major obstacle to training RNNs with gradient descent is due to vanishing or exploding gradients, as described in BID1 and BID14 . This problem refers to the tendency of gradients to grow or decay exponentially in size, resulting in gradient descent steps that are too small to be effective or so large that the network oversteps the local minimum. This issue significantly diminishes RNNs' ability to learn time-based dependencies, particularly in problems with long input sequences.A variety of architectures have been introduced to overcome this difficulty. The current preferred RNN architectures are those that introduce gating mechanisms to control when information is retained or discarded, such as LSTMs BID6 and GRUs BID3 , at the cost of additional trainable parameters. More recently, the unitary evolution RNN (uRNN) BID0 ) uses a parametrization that forces the recurrent weight matrix to remain unitary throughout training, and exhibits superior performance to LSTMs on a variety of synthetic and real-world tasks. For clarity, we follow the convention of BID21 and refer to this network as the restricted-capacity uRNN.Since the introduction of uRNNs, orthogonal and unitary RNN schemes have increased in both popularity and complexity. BID21 use a multiplicative update method detailed in Tagare (2011) and BID20 to expand uRNNs' capacity to include all unitary matrices. These networks are referred to as full-capacity uRNNs. BID7 's EURNN parametrizes this same space with Givens rotations, while BID8 's GORU introduces a gating mechanism for unitary RNNs to enable short term memory. BID19 introduced modified optimization and regularization methods that restrict singular values of the recurrent matrix to an interval around 1. Each of these methods involve complex valued recurrent weights. For other work in addressing the vanishing and exploding gradient problem, see BID5 and BID11 .In this paper, we consider RNNs with a recurrent weight matrix taken from the set of all orthogonal matrices. To construct the orthognal weight matrix, we parametrize it with a skew-symmetric matrix through a scaled Cayley transform. This scaling allows us to avoid the singularity issue occuring for −1 eigenvalues that may arise in the standard Cayley transform. With the parameterization, the network optimization involves a relatively simple gradient descent update. The resulting method achieves superior performance on sequential data tasks with a smaller number of trainable parameters and hidden sizes than other unitary RNNs and LSTMs.The method we present in this paper works entirely with real matrices, and as such, our results deal only with orthogonal and skew-symmetric matrices. However , the method and all related theory remain valid for unitary and skew-Hermitian matrices in the complex case. The experimental results in this paper indicate that state of the art performance can be achieved without the increased complexity of optimization along the Stiefel manifold and using complex matrices.
A novel approach to maintain orthogonal recurrent weight matrices in a RNN. Introduces a scheme for learning the recurrent parameter matrix in a neural network that uses the Cayley transform and a scaling weight matrix.  This paper suggests an RNN reparametrization of the recurrent weights with a skew-symmetric matrix using Cayley transform to keep the recurrent weight matrix orthogonal. Novel parametrization of RNNs allows representing orthogonal weight matrices relatively easily.