The Tensor-Train factorization (TTF) is an efficient way to compress large weight matrices of fully-connected layers and recurrent layers in recurrent neural networks (RNNs). However, high Tensor-Train ranks for all the core tensors of parameters need to be element-wise fixed, which results in an unnecessary redundancy of model parameters. This work applies Riemannian stochastic gradient descent (RSGD) to train core tensors of parameters in the Riemannian Manifold before finding vectors of lower Tensor-Train ranks for parameters. The paper first presents the RSGD algorithm with a convergence analysis and then tests it on more advanced Tensor-Train RNNs such as bi-directional GRU/LSTM and Encoder-Decoder RNNs with a Tensor-Train attention model. The experiments on digit recognition and machine translation tasks suggest the effectiveness of the RSGD algorithm for Tensor-Train RNNs. Recurrent Neural Networks (RNNs) are typically composed of large weight matrices of fullyconnected and recurrent layers, thus massive training data as well as exhaustive computational resources are required. The Tensor-Train factorization (TTF) aims to reduce the redundancy of RNN parameters by reshaping large weight matrices into high-dimensional tensors before factorizing them in a Tensor-Train format BID10 . The notation of Tensor-Train usually suggests that TTF is applied for the tensor representation of model parameters. Tensor-Train was initially applied to fully-connected layers BID8 , and it has been recently generalized to recurrent layers in RNNs such as LSTM and GRU BID13 . Compared with other tensor decomposition techniques like the CANDECOMP/PARAFAC decomposition BID5 and Tucker decomposition BID4 , Tensor-Train can be easily scaled to arbitrarily high dimensions and have the advantage of computational tractability to significantly large weight matrices.Given a vector of Tensor-Train ranks r = (r 1 , r 2 , · · ·, r d+1 ), TTF decomposes a d-dimensional tensor W ∈ R (m1·n1)×(m2·n2)×···×(m d ·n d ) into a multiplication of core tensors according to (1),where the k-th core tensor C [k] ∈ R r k ×m k ×n k ×r k+1 , and any index pair (i k , j k ) satisfies 1 ≤ i k ≤ m k , 1 ≤ j k ≤ n k . Additionally, the ranks r 1 and r d+1 are fixed to 1. This paper presents the RSGD algorithm for training Tensor-Train RNNs including the related properties, implementations, and convergence analysis. Our experiments on digit recognition and machine translation tasks suggest that RSGD can work effectively on the Tensor-Train RNNs regarding performance and model complexity, although the convergence speed is relatively slower in the beginning stages. Our future work will consider two directions: one is to apply the RSGD algorithm to more Tensor-Train models and test it on larger datasets of other fields; and the second one is to generalize Riemannian optimization to the variants of the SGD algorithms and study how to speed up the convergence rate.
Applying the Riemannian SGD (RSGD) algorithm for training Tensor-Train RNNs to further reduce model parameters. The paper proposes to use Riemannian stochastic gradient algorithm for low-rank tensor train learning in deep networks. Proposes an algorithm for optimizing neural networks parametrized by Tensor Train decomposition based on the Riemannian optimization and rank adaptation, and designs a bidirectional TT LSTM architecture.