Activation is a nonlinearity function that plays a predominant role in the convergence and performance of deep neural networks. While Rectified Linear Unit (ReLU) is the most successful activation function, its derivatives have shown superior performance on benchmark datasets. In this work, we explore the polynomials as activation functions (order ≥ 2) that can approximate continuous real valued function within a given interval. Leveraging this property, the main idea is to learn the nonlinearity, accepting that the ensuing function may not be monotonic. While having the ability to learn more suitable nonlinearity, we cannot ignore the fact that it is a challenge to achieve stable performance due to exploding gradients - which is prominent with the increase in order. To handle this issue, we introduce dynamic input scaling, output scaling, and lower learning rate for the polynomial weights. Moreover, lower learning rate will control the abrupt fluctuations of the polynomials between weight updates. In experiments on three public datasets, our proposed method matches the performance of prior activation functions, thus providing insight into a network’s nonlinearity preference. Deep learning methods have achieved excellent results in visual understanding, visual recognition, speech, and natural language processing tasks (Krizhevsky et al. (2012) , Lee et al. (2014) , Goodfellow et al. (2014) , Hochreiter & Schmidhuber (1997) , Oord et al. (2016) , Vaswani et al. (2017) ). The convolutional neural networks (CNNs) first introduced in LeCun et al. (1999) , is the foundation for numerous vision tasks. While recurrent neural networks, wavenet and the recent transformers with attention mechanism are the core algorithms used in speech and natural language processing. The commonality is the importance of deeper architectures that has both theoretical and empirical evidence (Serre et al. (2007) , Simonyan & Zisserman (2015) , Lee et al. (2014) ). One essential component for deep neural networks is the activation function that enables nonlinearity. While ReLUs are the most used nonlinearity, sigmoid and hyperbolic tangent are the traditional functions. Several derivatives of ReLU are presented in recent years that further improve the performance and minimize vanishing gradients issue (Maas et al. (2013) , He et al. (2015a) , Clevert et al. (2015) , Ramachandran et al. (2019) ). While most are fixed functions, the negative slope for Leaky ReLUs can be adjusted during the network design, and remains constant while training. Parametric ReLU adaptively changes the negative slope during training using a trainable parameter and demonstrate a significant boost in performance (He et al. (2015a) ). A relatively new activation function, Swish, is derived by an automated search techniques (Ramachandran et al. (2019) ). While the parameter β enables learning, the performance difference reported in the study between parametric and non-parametric versions is minimal. To this end, rather than using a fixed or heavily constrained nonlinearity, we believe that the nonlinearity learned by the deep networks can provide more insight on how they can be designed. In this work, we focus on the use of polynomials as nonlinearity functions. We demonstrate the stability of polynomial of orders 2 to 9 by introducing scaling functions and initialization scheme that approximates well known activation functions. Experiments on three public datasets show that our method competes with state-of-the-art activation functions on a variety of deep architectures. Despite their imperfections, our method allows each layer to find their preferred nonlinearity during training. Finally, we show the learned nonlinearities that are both monotonic and non-monotonic. We proposed a polynomial activation function that learns the nonlinearity using trainable coefficients. Our contribution is stabilizing the networks with polynomial activation as a nonlinearity by introducing scaling, initilization technique and applying a lower learning rate for the polynomial weights, which provides more insight about the nonlinearity prefered by networks. The resulting nonlinearities are both monotonic and non-monotonic in nature. In our MNIST experiments, we showed the stability of our method with orders 2 to 9 and achieved superior perfromance when compared to ReLUs, LReLUs, PReLUs, ELUs, GELUs, SELUs and Swish. In our CIFAR experiments, the performance by replacing ReLUs with polynomial activations using DenseNet, Residual Networks and Wide Residual Networks is on par with eight state-of-the-art activation functions. While the increase of parameters is negligible, our method is computationally expensive. We believe that by designing networks with simpler activations like ReLU for the initial layers, followed by layers with polynomial activations can further improve accuracies.
We propose polynomial as activation functions. The authors introduce learnable activation functions that are parameterized by polynomial functions and show results slightly better than ReLU.