In this paper, we propose a novel regularization method, RotationOut, for neural networks. 
 Different from Dropout that handles each neuron/channel independently, RotationOut regards its input layer as an entire vector and introduces regularization by randomly rotating the vector. 
 RotationOut can also be used in convolutional layers and recurrent layers with a small modification.
 We further use a noise analysis method to interpret the difference between RotationOut and Dropout in co-adaptation reduction. 
 Using this method, we also show how to use RotationOut/Dropout together with Batch Normalization. 
 Extensive experiments in vision and language tasks are conducted to show the effectiveness of the proposed method. 
 Codes will be available. Dropout (Srivastava et al., 2014 ) has proven to be effective for preventing overfitting over many deep learning areas, such as image classification (Shrivastava et al., 2017) , natural language processing (Hu et al., 2016) and speech recognition (Amodei et al., 2016) . In the years since, a wide range of variants have been proposed for wider scenarios, and most related work focus on the improvement of Dropout structures, i.e., how to drop. For example, drop connect (Wan et al., 2013) drops the weights instead of neurons, evolutional dropout (Li et al., 2016) computes the adaptive dropping probabilities on-the-fly, max-pooling dropout (Wu & Gu, 2015) drops neurons in the max-pooling kernel so smaller feature values have some probabilities to to affect the activations. These Dropout-like methods process each neuron/channel in one layer independently and introduce randomness by dropping. These architectures are certainly simple and effective. However, randomly dropping independently is not the only method to introduce randomness. Hinton et al. (2012) argues that overfitting can be reduced by preventing co-adaptation between feature detectors. Thus it is helpful to consider other neurons' information when adding noise to one neuron. For example, lateral inhibition noise could be more effective than independent noise. In this paper, we propose RotationOut as a regularization method for neural networks. RotationOut regards the neurons in one layer as a vector and introduces noise by randomly rotating the vector. Specifically, consider a fully-connected layer with n neurons: x ∈ R n . If applying RotationOut to this layer, the output is Rx where R ∈ R n×n is a random rotation matrix. It rotates the input with random angles and directions, bringing noise to the input. The noise added to a neuron comes not only from itself, but also from other neurons. It is the major difference between RotationOut and Dropout-like methods. We further show that RotationOut uses the activations of the other neurons as the noise to one neuron so that the co-adaptation between neurons can be reduced. RotationOut uses random rotation matrices instead of unrestricted matrices because the directions of feature vectors are important. Random rotation provides noise to the directions directly. Most neural networks use dot product between the feature vector and weight vector as the output. The network actually learns the direction of the weights, especially when there is a normalization layer (e.g. Batch Normalization (Ioffe & Szegedy, 2015) or Weight Normalization (Salimans & Kingma, 2016) ) after the weight layer. Random rotation of feature vecoters introduces noise into the angle between the feature and the weight, making the learning of weights directions more stable. Sabour et al. (2017) also uses the orientation of feature vectors to represent the instantiation parameters in capsules. Another motivation for rotating feature vectors comes from network dissection. Bau et al. (2017) finds that random rotations of a learned representation can destroy the interpretability which is axis-aligned. Thus random rotating the feature during training makes the network more robust. Even small rotations can be a strong regularization. We study how RotationOut helps prevent neural networks from overfitting. Hinton et al. (2012) introduces co-adaptation to interpret Dropout but few literature give a clear concept of co-adaptation. In this paper,we provide a metric to approximate co-adaptations and derive a general formula for noise analysis. Using the formula, we prove that RotationOut can reduce co-adaptations more effectively than Dropout and show how to combine Dropout and Batch Normalization together. In our experiments, RotationOut can achieve results on par with or better than Dropout and Dropoutlike methods among several deep learning tasks. Applying RotationOut after convolutional layers and fully connected layers improves image classification accuracy of ConvNet on CIFAR100 and ImageNet datasets. On COCO datasets, RotationOut also improves the generalization of object detection models. For LSTM models, RotationOut can achieve competitive results with existing RNN dropout method for speech recognition task on Wall Street Journal (WSJ) corpus. The main contributions of this paper are as follows: We propose RotationOut as a regularization method for neural networks which is different from existing Dropout-like methods that operate on each neuron independently. RotationOut randomly rotates the feature vector and introduces noise to one neuron with other neurons' information. We present a theoretical analysis method for general formula of noise. Using the method, we answer two questions: 1) how noise-based regularization methods reduce co-adaptions and 2) how to combine noise-based regularization methods with Batch Normalization. Experiments in vision and language tasks are conducted to show the effectiveness of the proposed RotationOut method. Related Work Dropout is effective for fully connected layers. When applied to convolution layers, it is less effective. Ghiasi et al. (2018) argues that information about the input can still be sent to the next layer even with dropout, which causes the networks to overfit (Ghiasi et al., 2018) . SpatialDropout (Tompson et al., 2015) drops the entire channel from the feature map. Shake-shake regularization (Gastaldi, 2017) drops the residual branches. Cutout (DeVries & Taylor, 2017) and Dropblock (Ghiasi et al., 2018 ) drop a continuois square region from the inputs/feature maps. Applying standard dropout to recurrent layers also results in poor performance (Zaremba et al., 2014; Labach et al., 2019) , since the noise caused by dropout at each time step prevents the network from retaining long-term memory. Gal & Ghahramani (2016) ; Moon et al. (2015) ; Merity et al. (2017) generate a dropout mask for each input sequence, and keep it the same at every time step so that memory can be retained. Batch Normalization (BN) (Ioffe & Szegedy, 2015) accelerates deep network training. It is also a regularization to the network, and discourage the strength of dropout to prevent overfitting (Ioffe & Szegedy, 2015) . Many modern ConvNet architectures such as ResNet (He et al., 2016) and DenseNet (Huang et al., 2017) do not apply dropout in convolutions. Li et al. (2019) is the first to argue that it is caused by the a variance shift. In this paper, we use the noise analysis method to further explore this problem. There is a lot of work studying rotations in networks. Rotations on the images (Lenc & Vedaldi, 2015; Simard et al., 2003) are important data augmentation methods. There are also studies about rotation equivalence. Worrall et al. (2017) uses an enriched feature map explicitly capturing the underlying orientations. Marcos et al. (2017) applies multiple rotated versions of each filter to the input to solve problems requiring different responses with respect to the inputs' rotation. The motivations of these work are different from ours. The most related work is network dissection (Bau et al., 2017) . They discuss the impact on the interpretability of random rotations of learned features, showing that rotation in training can be a strong regularization. In this work, we introduce RotationOut as an alternative for dropout for neural network. RotationOut adds continuous noise to data/features and keep the semantics. We further establish an analysis of noise to show how co-adaptations are reduced in neural network and why dropout is more effective than dropout. Our experiments show that applying RotationOut in neural network helps training and increase the accuracy. Possible direction for further work is the theoretical analysis of co-adaptations. As discussed earlier, the proposed correlation analysis is not optimal. It cannot explain the difference between standard Dropout and Gaussian dropout. Also it can not ex-plain some methods such as Shake-shake regularization. Further work on co-adaptation analysis can help better understand noise-based regularization methods. One example of such a matrix that rotates the (1, 3) dimensions and (2, 4) dimensions can be: In Section 2, we mentioned the complexity of RotationOut is O(D). It is because we can avoid matrix multiplications to get Rx. For example, let the R be the operator generated by Equation 17, we have: The sparse matrix in Equation 18 is similar to a combine of permutation matrix, and we do not need matrix multiplications to get the output. The output can be get by slicing and an elementwise multiplication: x[3, 4, 1, 2] * [−1, 1, 1, −1].
We propose a regularization method for neural network and a noise analysis method This paper proposes a new regularization method to mitigate the overfitting issue of deep neural networks by rotating features with a random rotation matrix to reduce co-adaptation. This paper proposes a novel regularization method for training neural networks, which adds noise neurons in an inter-dependent fashion.