We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium. We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet. The introduction of normalizing layers to neural networks has in no small part contributed to the deep learning revolution in machine learning. The most successful of these techniques in the image classification domain is the batch normalization (BatchNorm) layer BID4 , which works by normalizing the univariate first and second order statistics between layers.Batchnorm has seen near universal adoption in image classification tasks due to its surprisingly multifaceted benefits. Compared to an unnormalized network, its has been widely observed that using batch norm empirically results in:• Stability over a wide range of step sizes • Faster convergence (particularly with larger step sizes)• Improved generalizationThe multiple effects of BatchNorm make it both hard to replace and hard to analyze. In this paper we introduce Equilibrium Normalization (EquiNorm), a normalization that works in weight space and still uses a form of batch statistics unlike previous weight space approaches. EquiNorm results in very rapid convergence, even more so than BatchNorm, however as we will show in our experiments, this also results in a tendency to overfit. When combined with additional regularisation, EquiNorm can significantly outperform BatchNorm, which benefits less from this additional regularisation.
An alternative normalization technique to batch normalization Introduces a normalization technique, which normalizes the weights of convolutional layers.  This manuscript introduces a new layer-wise transform, EquiNorm, to improve upon batch normalization that does not modify the inputs to the layers but rather the layer weights.