ENN: A Neural Network with DCT Adaptive Activation Functions
The expressiveness of neural networks highly depends on the nature of the
activation function, although these are usually assumed predefined and fixed
during the training stage. Under a signal processing perspective, in this paper
we present Expressive Neural Network (ENN), a novel model in which the
non-linear activation functions are modeled using the Discrete Cosine Transform
(DCT) and adapted using backpropagation during training. This parametrization
keeps the number of trainable parameters low, is appropriate for gradient-based
schemes, and adapts to different learning tasks. This is the first non-linear
model for activation functions that relies on a signal processing perspective,
providing high flexibility and expressiveness to the network. We contribute
with insights in the explainability of the network at convergence by recovering
the concept of bump, this is, the response of each activation function in the
output space. Finally, through exhaustive experiments we show that the model
can adapt to classification and regression tasks. The performance of ENN
outperforms state of the art benchmarks, providing above a 40% gap in accuracy
in some scenarios.
Marc Martinez-Gost, Ana Pérez-Neira, Miguel Ángel Lagunas
# ENN: A Neural Network with DCT-Adaptive Activation Functions

###### Abstract

The expressiveness of neural networks highly depends on the nature of the activation function, although these are usually assumed predefined and fixed during the training stage. In this paper we present Expressive Neural Network (ENN), a novel architecture in which the non-linear activation functions are modeled using the Discrete Cosine Transform (DCT) and adapted using backpropagation during training. This parametrization keeps the number of trainable parameters low, is appropriate for gradient-based schemes, and adapts to different learning tasks. This is the first non-linear model for activation functions that relies on a signal processing perspective, providing high flexibility and expressiveness to the network. We contribute with insights in the explainability of the network at convergence by recovering the concept of bump, this is, the response of each activation function in the output space to provide insights. Finally, through exhaustive experiments we show that the model can adapt to classification and regression tasks. The performance of ENN outperforms state of the art benchmarks, providing up to a 40% gap in accuracy in some scenarios.

 Neural networks, adaptive activation functions, discrete cosine transform, explainable machine learning.

## I Introduction

Function approximation is a fundamental problem across many domains, such as data analysis, control systems and communications. When the explicit function is not available but input-output data pairs, the function can be uncovered by minimizing a criterion loss in a supervised setting. The problem increases in complexity when the function is non-linear, for which many signal processing techniques have been developed. For instance, least squares [1], orthogonal function approximation [2, 3], kernel methods [4] and neural networks [5, 6, 7], among others. The last decades have suffered an unprecedented growth in the development of artificial neural networks for function approximation due to its empirical success. The inception of neural networks as universal approximators boosted its development across many fields and applications, with different architectures built according to the task and data types to handle. Nevertheless, the expressiveness of the neural network is related to the non-linear activation function, which is usually assumed fixed. An overlooked field of research is in adaptive activation functions (AAF), where not only the weights of the neural networks are trained, but the non-linearities too [8].

In our previous work [9] we introduced the Discrete Cosine Transform (DCT) to approximate an univariate non-linear function in a joint communication and computing setting. Further, in [10] we show how a gradient-based algorithm can be used to tune the DCT coefficients to approximate a function in a supervised setting. However, extending the results to multivariate functions is not trivial: The number of required parameters to approximate the function increases exponentially with the number of variables, and their corresponding indexes are unknown when the explicit function is not available. This is, it is cumbersome how the top relevant coefficients can be learnt in a supervised fashion using labeled data.

In this work we propose to extend the capabilities of the DCT with a novel neural network architecture that integrates the DCT to represent and adapt the activation functions. We call this architecture Expressive Neural Network (ENN). We exploit the fact that a 2-layer neural network can theoretically represent any function and expand the representation capabilities of the network by adapting the activation functions at each neuron. The advantage of approximating an univariate function with the DCT is twofold: a small number of coefficients is required due to its high energy compaction, and the approximation error is easily controlled by the magnitude of the disregarded coefficients. In this work we also show that the DCT coefficients can be learnt using backpropagation in a supervised fashion and in the same pass as the standard network linear weights. In this way, the architecture is no different from a standard feed-forward neural network with fixed activation functions. From the learning perspective, using the DCT to model the activation functions brings the following benefits:

* Network size: The number of parameters in the network grows linear with the number of neurons, which is small due to energy compaction of DCT.
* Backpropagation: The algorithm can be implemented because the DCT coefficients are real and ordered in decreasing magnitude. Besides, there exist analytical closed-form solutions for backpropagation.
* Gradient behavior: The basis functions (cosines) are real and bounded, which prevents exploding gradients. Likewise, since the Fourier representation creates a periodic function that does not saturate, it also prevents vanishing gradients.
* Task adaptability: The output non-linearity is automati
cally adapted depending on the task (e.g., classification or regression).

While we provide a general formulation for multivariate real functions, we constraint the analysis to bivariate functions. This allows to visualize the results and intuitively understand how the network is adapting the activation functions. In this respect, we recover the concept of _bump_, which is the non-linear enclosure that each activation function generates in the output space [11]. The output space corresponds to a weighted sum of all bumps generated at the hidden layer. This concept allows to gain insights in how the network decides to exploit the periodic nature of the DCT model and create the boundaries for classification problems. In this work we focus on two general problems of function approximation, namely classification and regression. In the former there are two hypothesis associated to a function, this is, \(\mathcal{H}_{0}\) when \(f(x_{1},x_{2})<0\) and \(\mathcal{H}_{1}\) otherwise. In regression the goal is to approximate the function \(f(x_{1},x_{2})\).

The main contributions of this paper are described in the following:

1. We define ENN, a novel neural network architecture where the non-linear activation functions are parameterized by the DCT. This allows to adapt DCT coefficients in a supervised fashion and learn specific non-linearities according to the task. This results in a highly flexible and expressive architecture.
2. We provide insights in the field of explainable machine learning (XML). We recover the concept of bump, which allows to interpret how the non-linearities are adapted and what is the output response of the neural network. Furthermore, the concept allows to intuitively control several parameters (e.g., weight initialization).
3. We develop analytical closed-form expressions to adapt the non-linearities with backpropagation. While we choose the Least Mean Squares (LMS) algorithm to update the network parameters, the architecture remains a feed-forward neural network and any alternative algorithm can be used.
4. We provide extensive experiments in both classification and regression setups for which ENN outperforms all the benchmarks. In classification tasks, ENN outperforms fixed activation functions up to 40% in accuracy. With that we show how the expressiveness of network highly depends on the activation function, without the need of increasing the size of the network.
5. With respect to XML, we show how the DCT model helps to dimension the network width, this is, the number of hidden neurons. Particularly, the network converges to duplicated activation functions with opposed linear weights in the output layer. This is, the network cancels out the information coming from several branches when the task does not require that much expressiveness.

The remainder of this paper is organized as follows. In Section II, we present a literature review on neural networks for function approximation. The Fourier models for non-linear representation are presented in Section III. We present ENN in Section IV and propose the learning procedure for supervised tasks in Section V. The simulation results are shown in Section VI and we conclude the paper in Section VII.

**Notation**: Lowercase and uppercase bold symbols correspond to vectors and matrices, respectively; \(\mathbf{s}[m]\) corresponds to the \(m\)-th entry of vector \(\mathbf{s}\); \(\mathbb{R}\) stands for the set of real numbers and \(\nabla\) for the gradient.

## II Literature Review

The universal approximation theorem is a well-known result in mathematics stating that a 2-layer neural network can represent any continuous function with an arbitrary number of neurons [12, 13]. The theory behind neural networks has been further developed, providing bounds on the number of required neurons.

In a different line of research, the Kolmogorov-Arnold (KA) representation theorem shows how a multivariate function can be represented by functions of only univariate functions [14]:

\[f(x_{1},\ldots,x_{n})=\sum_{i=1}^{2n+1}\Phi_{i}\left(\sum_{j=1}^{n}\phi_{ij}( x_{j})\right), \tag{1}\]

where \(\Phi_{i}\) and \(\phi_{j}\) are termed the outer and inner functions, respectively. This result seems to be tightly connected to a 2-layer neural network, since the inner functions correspond to the hidden layer transformation and the outer functions to the output neuron. What is more, the inner functions do not depend on the function \(f\) to implement, which resembles the activation functions in neural network architectures. However, the formulation is not exactly identical: the KA representation requires \(n\) inner functions for each input variable, while a neural network implements a unique function for each linear combination of inputs. In this way, although there is extensive literature motivating the development of neural networks with the KA theorem [15, 16, 17], there are still many gaps to be resolved. What is more, the theorem is not constructive and the functions in (1) are highly non-linear.

Despite its success in many applications, neural networks undergo several shortcomings that limit the interpretability of the results: optimization algorithms are easily trapped in local minima, convergence heavily depends on initialization and fail to converge when high non-linearities exist. Usually, each neuron in feedforward neural networks implements a linear combination and a non-linear mapping. The former is usually trained, while the latter remains fixed. The non-linear activation function allows to generate non-linear mappings, which increases the expressiveness of the network. A common choice is the sigmoid function, although it exhibits well-known issues in its implementation: since the sigmoid saturates at large magnitudes the propagated gradients vanish, slowing down the learning process. The rectified linear unit (ReLU) replaced the sigmoid activation function because it does not suffer from vanishing gradients and it is computationally efficient, which results in faster convergence. Despite its popularity, ReLU also experience several weaknesses that hinder the learning capacity of the architecture: the corresponding neuron can become dead when the output remains negative for a long
time; likewise, the output is unbounded, which may produce the opposite effect of exploding gradient, affecting the stability of the network.

Many variations and novel activation functions have been designed to enhance the performance of neural networks, although this highly depends on the application and the statistics of the data. In this respect, few authors have tackled the problem of adaptive activation functions (AAF), in which the non-linear mapping is also trained to enhance the expressiveness of the network. In [8] the saturation level and slope of an S-shaped activation function can be tuned independently, which increases the expressiveness with respect to the sigmoid function. In a different vein, several authors proposed to approximate an arbitrary activation function using cubic splines [18, 19]. Later on, some authors introduced an activation function as a weighted sum of basis, such as sine, Gaussian or sigmoid [20, 21]. More recently, the authors in [22] propose an asymmetric S-shaped activation function, but only for regression tasks. The issues derived from these perspective is that they either constraint the geometry of the activation function, or the parametrization is too complex. In this work we empirically prove that having pre-specified activation functions result in suboptimal performance.

In the context of deep learning, several AAFs have been designed as well. In [23] the authors propose a gradient adaptive piecewise linear activation function as a sum of hinge-shaped functions. Although they theoretically prove that any piecewise linear function can be expressed as so, this constraints the function to be linear at both extremes. In other words, it does not prevent the neural network from exploding gradients. In [24] an S-shaped rectified linear activation function is proposed. While this model can learn non-linear and non-convex representations, it is not smooth and still constraints the activation to be defined in a piece-wise fashion. The authors in [25] develop a piecewise polynomial form which can approximate any continuous function. However, the exponential nature of the polynomials may increase the dynamic range of the parameters, which is a non-desirable property for gradient-based procedures. In [26], the ReLU is substituted by an S-shaped activation, which slightly outperforms the ReLU and its variants in different deep learning tasks. In [27, 28, 29, 30] different authors propose to model the activation as a sum of basis functions.

In the following section we propose a Fourier-based parametrization for non-linear functions. This is the first approach to model AAFs from a signal processing perspective. The proposed model does not constraint the shape of the activation function and keeps the number of trainable parameters low while circumventing the shortcomings of the previously proposed non-linear models.

## III Fourier Models for Non-linear Functions

Consider a scalar univariate function \(f(x)\) to be approximated over the input variable range \(x\in[-1,1]\). One of the most popular approximations for non-linear systems is the Volterra model. Given the function representation by the inverse Fourier transform,

\[f(x)=\frac{1}{2\pi}\int_{-\infty}^{\infty}F(w)e^{jwx}\,dw, \tag{2}\]

where \(w\) is the frequency variable and \(F(w)\) are the Fourier coefficients, the Volterra model is obtained by using the Taylor series expansion of the exponential family:

\[f(x) =\frac{1}{2\pi}\int_{-\infty}^{\infty}F(w)\left(\sum_{n=0}^{ \infty}\frac{(jw)^{n}}{n!}x^{n}\right)\,dw\] \[=\sum_{n=0}^{\infty}\left(\frac{1}{2\pi}\frac{1}{n!}\int_{- \infty}^{\infty}F(w)(jw)^{n}\,dw\right)x^{n}\] \[=\sum_{n=0}^{\infty}c_{n}x^{n} \tag{3}\]

The building blocks of (3) are power functions of the input variable, which generate a polynomial approximation. Notice that the coefficients \(c_{n}\) correspond to the \(n\)-th derivative at the origin divided by the factorial of the index. Volterra has not been widely used in practice because the exponential nature of the basis functions heavily increase the dynamic range, even when the input is bounded. Moreover, outside the dynamic range the approximation is unbounded and cannot be controlled. This happens because Volterra is a Taylor approximation and it presents high accuracy near the origin only. All these concerns make this approximation not suitable for gradient-based learning algorithms.

One possible approach to solve these issues is the representation using a finite number of terms in the Fourier approximation in (2), which corresponds to the Discrete Fourier Transform (DFT). Outside the dynamic range of \(x\) the function is periodically extended, generating discontinuities at the border. As a result, the number of coefficients required to approximate the function is highly sensitive to these discontinuities. In fact the oscillatory behavior around the discontinuity has an amplitude proportional to the number of continuous derivatives of \(f(x)\). To prevent this phenomenon the function can be extended with even symmetry and, then, periodically extended. This smooths out the edges and its corresponding derivatives, reducing the number of required coefficients with respect to the DFT. This is known as the Discrete Cosine Transform (DCT), which has the following expression:

\[f(x)\approx\sum_{q=1}^{Q}g_{q}F_{q}\cos\left(\frac{\pi(q-1)(2z+1)}{2N}\right), \tag{4}\]

with \(z=\frac{N}{2}(1+x)\), \(g_{1}=1/\sqrt{N}\) and \(g_{q}=\sqrt{2/N}\) otherwise. The \(F_{q}\in\mathbb{R}\) are termed the DCT coefficients. Notice that for functions with odd symmetry, only the odd coefficients are retained. In general, the quality of approximation is more than sufficient for \(Q=12\) (i.e., 6 coefficients in odd functions). For the sake of brevity, the following definition will be used when needed:

\[cos_{i}(x)=\cos\left(\frac{\pi}{2N}\left(2i-1\right)\left(N(x+1)+1\right)\right) \tag{5}\]

In [10] we propose an adaptive design in which the DCT coefficients of (4) are tuned using the LMS algorithm in a
supervised setting to approximate an univariate function. There are plenty of advantages in using the DCT representation: The required coefficients to provide the same quality are fewer and these are real and ordered in decreasing magnitude. Since the basis functions are orthogonal, the approximation error can be easily controlled by the magnitude of the disregarded coefficients, and it simplifies the convergence of the learning procedure. Furthermore, see that the index appears in the phase of (4), so the approximation is real and bounded, even when the input exceeds the dynamic range. All these features make the DCT an appropriate function approximation, whose coefficients can be learnt by a gradient-based rule.

### _Extension to several inputs with the DCT_

The extension of the DCT representation to multiple variables is not trivial. Consider a bivariate function, that could be parameterized by a 2-dimensional (2D) DCT as

\[f(x_{1},x_{2}) =\sum_{n=1}^{N}\sum_{m=1}^{N}F_{nm}\cos_{n}(x_{1})\cos_{m}(x_{2})\] \[\approx\sum_{n=1}^{Q}\sum_{m=1}^{Q}F_{nm}\cos_{n}(x_{1})\cos_{m}(x_ {2}) \tag{6}\]

Note that this architecture has the following drawbacks: In general the number of coefficients grows exponentially with the number of inputs \(M_{0}\) as \(Q^{M_{0}}\), along with the complexity of the DCT in several dimensions; while the coefficients in the DCT are ordered in decreasing magnitude, the structure of the indexes is broken in the 2D-DCT. This is, the location \((n,m)\) of the relevant coefficients changes with the function of interest; this implies that, unless the function is known a priori, the indexes \((n,m)\) are unknown and a supervised learning procedure is hard to implement. This happens for instance in classification problems, where the function is unknown and to be discovered.

In the following we will present ENN, a neural network architecture integrating the DCT in a single dimension and whose coefficients can be trained in a supervised fashion.

## IV DCT-Adaptive Activation Functions

A perceptron (or neuron) is a non-linear processor involving a weighted sum and a non-linear function. It is described by the following expressions:

\[z =a_{0}+\sum_{i=1}^{M}a_{i}x_{i}=\mathbf{a}^{T}[1\ \mathbf{x}^{T}]^{T} \tag{7}\] \[\overline{z} =\frac{N}{2}(z+1)\] (8) \[\hat{y} =\sigma(\overline{z}) \approx\sum_{q=1}^{Q/2}F_{q}\cos\left(\frac{\pi(2q-1)(2\overline {z}+1)}{2N}\right)\] \[=\sum_{q=1}^{Q/2}F_{q}\cos_{q}(z) \tag{9}\]

In (7), \(\mathbf{x}\) contains \(M_{0}\) inputs and a 1 is appended for the bias term \(a_{0}\). The normalization in (8) is needed to map the input to \([0,N]\), assuming the input is confined to \([-1,1]\). Nevertheless, as it will be seen later, the first linear transformation may map \(\overline{z}\) to a different range, providing expressiveness to the network. Finally, the non-linear activation function \(\sigma(\cdot)\) is approximated by the DCT with \(Q/2\) coefficients. Without loss of generality, \(g_{q}\) is assumed to be integrated in the DCT coefficient \(F_{q}\). In (9) we assume the non-linearity to have odd symmetry, so that only the odd coefficients are retained. As it will be explained later on in Sec. IV-B and shown empirically in Sec. VI, this does not prevent the network from learning only odd activation functions. A total of \(M_{0}+Q/2+1\) parameters per perceptron are to be trained, which only represents an increment of \(Q/2\) coefficients with respect to a standard perceptron with a fixed activation functions. As mentioned in the previous section, \(Q\) is small due to the energy compaction property of the DCT (e.g., around \(Q/2=6\) coefficients). Fig. 1 shows the block diagram of the perceptron. The normalization is not shown and assumed to be a part of the non-linear transformation \(\sigma(\cdot)\).

**Example 1** (Linear discriminant).: _Assume \(M_{0}=2\), we want to discriminate when \(x_{1}>x_{2}\). This can be framed as a classification problem, for which we can take \(\mathbf{a}=a[0\,1-1]^{T}\), where \(a\) is a positive constant. Then, any odd monotone increasing function will discriminate the two hypothesis. For the sake of simplicity, take \(a=1\) and the non-linearity approximated with only one coefficient (i.e., \(F_{1}=-1\)). This results in_

\[z =x_{1}-x_{2} \tag{10}\] \[\overline{z} =\frac{N}{2}(x_{1}-x_{2}+1)\] (11) \[\hat{y} =-\cos\left(\frac{\pi(2\overline{z}+1)}{2N}\right)\approx\sin \left(\frac{\pi}{2}(x_{1}-x_{2})\right) \tag{12}\]

_The solution in (12) provides a soft-decision, whereas a hard-decision would take the sign, i.e., \(\text{sign}(\hat{y})\). While there are infinite solutions for this example, the proposed scheme only increases the complexity by one extra coefficient (i.e., \(F_{1}\)). Implementing a linear \(\sigma(\cdot)\) would require many coefficients and provide no further benefit in terms of performance._

In Example 1 there are infinite minimums, all optimal. Nevertheless, in more complex problems (e.g., high-order discriminants) the expressiveness of a single perceptron is not enough. In light of the universal approximation theorem, in the

Fig. 1: One perceptron with \(M_{0}\) inputs and activation function \(\sigma(\cdot)\).
following we will increase the capabilities of the network by including a hidden layer of several neurons. This architecture, termed ENN, will be fully-adaptive as both the linear weights and the activation functions will be trained in a supervised fashion.

### _Expressive Neural Network (ENN)_

As a single layer perceptron limits the capabilities of the network, we will include another processing layer to increase its expressiveness. For a general multi-layer perceptron of \(L\) layers, the following expressions show the perceptron signals at the \(l\)-th layer:

\[\mathbf{z}_{l} =\mathbf{A}_{l}^{T}[1\ \mathbf{s}_{l-1}^{T}]^{T} \tag{13}\] \[\mathbf{\overline{z}}_{l} =\frac{N}{2}(\mathbf{z}_{l}+1)\] (14) \[\mathbf{s}_{l} =\sigma_{l,m}(\mathbf{\overline{z}}_{l}) \tag{15}\]

for \(l=1,\ldots,L\). The first input and last output correspond to \(\mathbf{s}_{0}=\mathbf{x}\) and \(\hat{y}=\mathbf{s}_{L}\), respectively. The matrix of linear weights is \(\mathbf{A}_{l}=[\mathbf{a}_{i}^{(1)}\ \ldots\ \mathbf{a}_{i}^{(M_{l})}]\) and \(M_{l}\) stands for the number of perceptrons at layer \(l\), with \(M_{0}\) being the number of inputs. The operations in (14) and (15) are performed element-wise. The AAF in the ENN, this is, the \(m\)-th element of (15) is computed as

\[\mathbf{s}_{l}[m]=\sum_{q=1}^{Q/2}F_{q,l}^{(m)}\cos_{q}(\mathbf{z}_{l}[m]), \tag{16}\]

where \(F_{q,l}^{(m)}\) corresponds to the \(q\)-th coefficient of the \(m\)-th perceptron at the \(l\)-th layer. As explicitly shown in (15) and (16) the activation function is not necessarily the same at each neuron, although the number of DCT coefficients \(Q\) is kept constant for the whole network. The last activation function could be substituted by an step function in the classification setup and a linear for regression. However, this will be maintained so that the network can self-adapt depending on the nature of the problem.

As opposed to the \(M_{0}\)-dimensional DCT in Sec. III-A, the proposed architecture does not require implementing the DCT in \(M_{0}\) dimensions and can be trained in a supervised fashion. Furthermore, the number of coefficients grows linearly as \(M_{0}Q\). Throughout the rest of the paper we assume a two input variable, namely \(M_{0}=2\), and \(L=2\) layers. These layers are sequentially termed hidden and output layer. Fig. 2 shows the architecture of the two-layer perceptron with \(M_{1}\) neurons in the hidden layer and expression (15) shows the input-output relationship.

### _Expressiveness of periodic activation functions_

As it will be shown in the experimental results, the expressiveness of the DCT comes from its periodic nature. Fig. 3 shows the sigmoid function and the corresponding DCT representation in the \([-1,1]\) range using \(Q/2=6\) coefficients. While the sigmoid function saturates outside the range, the DCT approximation offers a periodic infinite non-linearity. This offers more capacity to the activation function, as the network may choose to work at an increasing range (e.g., \([-1,0]\)), decreasing range (e.g., \([-3,-1]\)), bump range (e.g., \([-1,3]\)), valley range (e.g., \([-3,1]\)), or even with several periods simultaneously (e.g., \([-5,3]\)). While imposing odd symmetry in the DCT representation allows to have a continuous derivative, it does not constraint the resulting activation function to be odd. This Fourier model for non-linearities is the first one to provide such flexibility to activation functions, which is not possible with a fixed non-linearity not implemented with the DCT.

To understand how the non-linear activation function maps the input to the output space, we use the concept of _bump_. Consider the perceptron in Fig. 1 with \(M_{0}=2\). Notice that all input pairs satisfying the following equality are mapped to the same non-linear output \(\sigma(c)\):

\[a_{0}+a_{1}x_{1}+a_{2}x_{2}=c, \tag{16}\]

where \(c\) is a constant value. Fig. 4a shows how expression (16) corresponds to a line in the input space. Depending on the linear weights, only a certain range of the non-linearity is covered (or multiple ranges). In Fig. 4b it shown the mapping \(\sigma(c)\). It also displays to what extent the function is used between dashed lines, this is, for a minimum and maximum \(c\)

Fig. 3: Sigmoid function (dashed) and its DCT representation in the \([-1,1]\) range (solid).

Fig. 2: A 2-layer perceptron with \(M_{1}\) neurons in the hidden layer.
Fig. 4c shows the generated bump, by evaluating all the input data pairs through the linear transformation and the non-linear mapping.

## V Supervised Learning

### _Backpropagation rules_

Consider a dataset \(\mathcal{D}\), consisting of \((\mathbf{x}_{i},y_{i})\in\mathcal{D}\), where the former is a \(M_{0}\)-sized vector training sample and the latter is the associated reference (i.e., label in classification, or function value in neuromorphic computing). Given \(y_{i}\) and \(\hat{y}_{i}\), the error can be computed and propagated throughout the network to adjust the learnable parameters. We assume the loss to be the mean squared error (MSE),

\[\varepsilon^{2}=(y_{i}-\hat{y}_{i}(\mathbf{x}_{i}))^{2}, \tag{17}\]

for both classification and regression problems. Notice in (17) we explicitly write the output as a function of the input data. Given a learnable parameter \(w\), the chosen algorithm to update it is LMS. The update rule at a given iteration corresponds to

\[w\gets w-\mu\nabla\varepsilon=w-\mu\varepsilon\left(-\frac{\partial\tilde {y}}{\partial w}\right), \tag{18}\]

this is, the parameter is updated by the product of the error by the gradient of the output with respect to the parameter. The hyperparameter \(\mu\) is the step size, which controls the convergence speed of LMS. We choose the simplest algorithm to learn the parameters in the neural network because the focus of this work is on the architecture, not on the algorithm. In this respect, the experimental results show that even using the instantaneous gradient, LMS converges to a minimum and outperforms state-of-the-art models. Thus, the LMS may be replaced by any other algorithm to speed convergence, which is out of the scope of this work. Due to the same reason, we do not include the iteration index in the parameter updates.

For the 2-layer architecture in Fig. 2, there are 2 set of parameters per layer that need to be updated, namely, the DCT coefficients and the weights from the linear transformation. Starting at the output, the DCT coefficients of the layer \(l=2\) are updated as

\[F_{m,2}\gets F_{m,2}+\mu\varepsilon\frac{\partial\tilde{y}}{\partial F_{ m,2}}=F_{m,2}+\frac{4\alpha}{Q}\varepsilon\cos_{m}\left(z_{2}\right), \tag{19}\]

for \(m=1,\ldots,Q/2\). The superscript in the parameter is omitted because there is only one perceptron at the output. In general, the step size is modeled as twice the mismatch \(\alpha\) divided by the power of the corresponding input. Since the \(Q/2\) cosines used to approximate the non-linearity are orthonormal, the total power is \(Q/2\). This exhibits another advantage of this parametrization as the power remains always constant.

In order to update the linear weights from the output layer, the backpropagation procedure allows to propagate the error across the network with respect to previously computed derivatives, which makes it a very efficient algorithm. Therefore, these are updates as

\[a_{2}[k] \leftarrow a_{2}[k]+\mu\varepsilon\frac{\partial\tilde{y}}{\partial a_{2}[k ]}=a_{2}[k]+\mu\varepsilon\frac{\partial\tilde{y}}{\partial z_{2}}\frac{ \partial\tilde{z}_{2}}{\partial a_{2}[k]}\] \[= a_{2}[k]-\frac{2\alpha}{P_{1}}\varepsilon\frac{\pi}{2}s_{1}[k] \sum_{m=1}^{Q/2}F_{m,2}(2m-1)\cos_{m}(z_{2}), \tag{20}\]

for \(k=0,\ldots,M_{1}\), and \(P_{1}=\mathbf{s}_{1}^{T}\mathbf{s}_{1}\). The superscript has also been suppressed in this case. Accordingly, the DCT coefficients in the first layer are updated as

\[F_{q,1}^{(k)} \leftarrow F_{q,1}^{(k)}+\mu\varepsilon\frac{\partial\tilde{y}}{\partial F_ {q,1}^{(k)}}=F_{q,1}^{(k)}+\mu\varepsilon\frac{\partial\tilde{y}}{\partial z _{2}}\frac{\partial z_{2}}{\partial s_{1}[k]}\frac{\partial s_{1}[k]}{ \partial F_{q,1}^{(k)}}\] \[= F_{q,1}^{(k)}-\] \[\frac{4\alpha}{Q}\varepsilon\frac{\pi}{2}a_{2}[k]\cos_{q}(z_{1} [k])\sum_{m=1}^{Q/2}F_{m,2}(2m-1)\sin_{m}(z_{2}), \tag{21}\]

for \(q=1,\ldots,Q/2\) and \(k=1,\ldots,M_{1}\). And the linear combination of the first layer is updates as

\[a_{1}^{(k)}[m] \leftarrow a_{1}^{(k)}[m]+\mu\varepsilon\frac{\partial\tilde{y}}{\partial a _{1}^{(k)}[m]}\] \[= a_{1}^{(k)}[m]+\mu\varepsilon\frac{\partial\tilde{y}}{\partial z _{2}}\frac{\partial z_{2}}{\partial s_{1}[k]}\frac{\partial s_{1}[k]}{ \partial z_{1}[k]}\frac{\partial z_{1}[k]}{\partial a_{1}^{(k)}[m]}\] \[= a_{1}^{(k)}[m]+\] \[\frac{2\alpha}{P_{0}}\varepsilon\frac{\pi^{2}}{4}a_{2}[k]s_{0}[ m]\sum_{p=1}^{Q/2}F_{p,2}(2p-1)\sin_{p}(z_{2})\] \[\sum_{q=1}^{Q/2}F_{q,1}^{(k)}(2q-1)\sin_{q}(z_{1}[k]), \tag{22}\]

Fig. 4: Bump generation. Mapping of a linear combination of two inputs over the response of a non-linear function.
for \(m=1,\ldots,M_{0}\), \(k=1,\ldots,M_{1}\), and \(P_{0}=\mathbf{s}_{0}^{T}\mathbf{s}_{0}\). Without loss of generality, \(s_{0}[0]=s_{1}[0]=1\), which correspond to the bias terms in the linear combination at each layer. Notice that in all the update rules, the derivative exists and inherits the periodic nature of the cosine transform.

To enhance the stability of the convergence, the power of each input, namely \(P_{0}\) and \(P_{1}\), can be computed with a dampening effect. For instance,

\[P_{0}\leftarrow\beta P_{\mathrm{o}}+(1-\beta)\mathbf{s}_{0}^{T}\mathbf{s}_{0} \tag{23}\]

at each iteration. The \(\alpha\) parameter is set smaller in the output layer, which intuitively reduces the propagation of artifacts to previous layers during training.

### _Initialization and Training Procedure_

Instead of learning all the parameters from scratch, we devise a sequence of training stages to speed up the convergence of ENN. For both classification and regression problems, the training procedure is split into two phases:

* **Phase 1**: train the linear weights with fixed sigmoid activation functions. This is well-known to converge for a two-layer neural network. The difference with respect to the literature is that the activation function is defined using the DCT expression in (4), this is, the function is periodic. In the hidden layer, the linear weights are initialized to generate bump diversity. These weights can be interpreted as the orientation of the activation functions in the output space. See Fig. 5 with an example for \(M_{1}=4\), with linear weights corresponding to \([0,0,1]\), \([0,1,0]\), \([0,1,-1]\) and \([0,-1,1]\) and a logarithmic compander approximated with \(Q=12\) coefficients. Finally, the output layer is initialized with linear weights in a uniform distribution in \([-0.5,0.5]\). This allows to constraint the dynamics and show that the convergence does not depend on the initialization.
* **Phase 2**: using the linear weights from Phase 1 as the initialization, train both the linear weights and the activation functions.

## VI Experimental results

In the following we will test the ENN for both binary classification and regression problems. As benchmarks, we will test also the following models, in which they all have the same architecture and differ in the activation functions:

* **ReLU**: it uses a fixed non-trainable ReLU, this is, \(\sigma(z)=\max\left\{0,z\right\}\).
* **Sigm**: it uses a fixed non-trainable sigmoid activation function. Notice that the output saturates for an input of large magnitude.
* **F-DCT**: it uses a fixed non-trainable sigmoid activation function, but modeled with the DCT. Thus, the function does not saturate and it is periodic.

Although not an odd function, we include the ReLU because it is the standard activation function in current neural network architectures. The Sigm model is also included to compare it with F-DCT and assess the gains of a periodic activation function.

All architectures are built with \(M_{1}=6\) neurons in the hidden layer, and ENN with \(Q=12\) parameters in all the non-linearities. However, recall that only \(Q/2\) coefficients are different from zero because we impose odd symmetry. Regarding the LMS setup, for the non-linearities we set \(\alpha=10^{-3}\) in the hidden layer and \(\alpha=10^{-4}\) in the output layer. For the linear weights we set \(\alpha=5\cdot 10^{-5}\) in the hidden layer and \(\alpha=5\cdot 10^{-3}\) in the output layer. We keep the step-size constant for all neurons in the same layer. The dampening parameter is set to \(\beta=0.999\) for both layers.

For both classification and regression problems, a synthetic dataset is generated. All samples come from uniform distributions in the \([-1,1]\) range for each input variable. The train and test sets contain 400.000 and 50.000 samples, respectively. For comparison fairness all the models are trained with the same sequences in all the setups.

Notice that the output activation functions in ReLU and Sigm has to be specifically selected to be either sigmoid in classification or linear in regression. Conversely, in ENN it will be automatically adapted to approach the required function, which is an advantage of this architecture with respect to the benchmarks.

### _Classification_

Table I shows different classification problems that have been evaluated. It displays the ideal decision map along with the order of the discriminant. The metric chosen to compare the different models is the test accuracy, this is, the percentage of correctly classified samples in the test set.

As expected, the complexity of the problem increases with the order of the discriminant. As seen in Example 1, a single neuron is sufficient to implement a linear discriminant. This goes to show that all models are capable of producing an almost perfect boundary. With no surprise, for quadratic and cubic discriminants all models are capable of finding the appropriate boundary. When moving to high-order classification

Fig. 5: 4 different non-linearities with corresponding linear weights such that they are oriented in different directions.
problems is when the non-adaptive models are not capable of providing a satisfactory solution. Conversely, the ENN model manages to find a local minimum with high accuracy and with very consistent results for a wide variety of problems.

Fig. 6 shows the AAFs for the 8th classification problem. Notice that only two of out the six activation functions have been adapted, namely Fig. 5(a) and Fig. 5(e). The corresponding bumps are shown in Fig. 7, where the different orientations are exploited to generate diversity and, ultimately, non-linear boundaries. Another advantage of ENN is that it allows to model the network width, this is, the required number of hidden neurons. The bumps in Fig. 6(b) and 6(f) are identical, while the corresponding linear weights in the output layer have the same magnitude and opposite sign. This means that the network neglects the information that comes from these two branches. This also happens for bumps in Fig. 6(c) and 6(d). In conclusion, the network only needs two neurons to generate the non-linear boundary. However this is not the case for the non-adaptive models, which have a very small accuracy even with 6 neurons. ENN exploits the periodic nature of the Fourier representation to generate expressive non-linearities.

Fig. 8 shows the AAFs for the 7th problem, along with the corresponding bumps in Fig. 9. As expected, the architecture exploits the different orientations and periodic nature of the DCT. Notice that there is almost no drop in accuracy between the ring and the face classification problem, although the latter is more complex. This is due to the expressiveness of the DCT, which allows to generate several peaks per bump. In other words, by allowing the activation to be non-monotone it can be used to approximate the ring and both circles still with 6 neurons. Fig. 10 shows the different decision boundaries for the 7th problem. Despite the accuracy of the non-adaptive models in Table I being around 80%, the quality is very poor. Only the ENN boundary resembles the ideal map.

The output activation function converges to the same configuration for all classification problems and it is shown in Fig. 10(a). As anticipated theoretically, the learnt AAF approaches a step function. The fact of constraining the model to \(Q=12\) coefficients makes it infeasible to generate an ideal step function. Nevertheless, this solution is steeper than a standard sigmoid function, which reduces the error at data close to the boundary and increases the accuracy.

### _Regression_

The network can also be trained for regression problems, this is, to approximate a function. Particularly, we compute bivariate scalar functions. Table II shows the MSE achieved by the four different models for three different experiments.

Fig. 12 shows the different bumps for the regression problem \((x_{1}^{2}+x_{2}^{2})/2\). Notice that this time the network decides to work at different ranges of the sigmoid function. Particularly, Fig. 10(b) corresponds to a symmetric extension of the function,

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline \(\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{ \mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{ \mathbf{ }}}}}}}}}}}}}}}\) & ReLU & Sign & F-DCT & ENN \\ \hline \(\frac{x_{1}+x_{2}}{2}\) & \(2.9\cdot 10^{-3}\) & \(2.1\cdot 10^{-2}\) & \(1.9\cdot 10^{-2}\) & \(\mathbf{1.3\cdot 10^{-4}}\) \\ \(\frac{x_{1}^{2}+x_{2}^{2}}{4}\) & \(2.2\cdot 10^{-1}\) & \(2.2\cdot 10^{-1}\) & \(2.3\cdot 10^{-1}\) & \(\mathbf{1.4\cdot 10^{-4}}\) \\ \(x_{1}x_{2}\) & \(5.1\cdot 10^{-2}\) & \(3.6\cdot 10^{-2}\) & \(3.6\cdot 10^{-2}\) & \(\mathbf{1.4\cdot 10^{-4}}\) \\ \hline \hline \end{tabular}
\end{table} TABLE II: MSE in regression tasks.

\begin{table}
\begin{tabular}{c c|c c c c} \hline \hline \(\mathbf{\mathbf{\mathbf{N}}}^{\text{e}}\) & Map & Order & ReLU & Sign & F-DCT & ENN \\ \hline (1) & \multirow{2}{*}{1} & \multirow{2}{*}{99.97\%} & \multirow{2}{*}{99.90\%} & \multirow{2}{*}{99.84\%} & \multirow{2}{*}{99.85\%} \\ \cline{1-1} \cline{5-6} (2) & & & & & \\ \cline{1-1} \cline{5-6} (3) & & & & & \\ \cline{1-1} \cline{5-6} (4) & & & & & \\ \cline{1-1} \cline{5-6} (5) & & & & & \\ \cline{1-1} \cline{5-6} (6) & & & & & \\ \cline{1-1} \cline{5-6} (7) & & & & & \\ \cline{1-1} \cline{5-6} (8) & & & & & \\ \hline \hline \end{tabular}
\end{table} TABLE I: Accuracy for different binary classification problems. For each case, the ideal map and the order of the discriminant are shown.

Fig. 6: AAFs for the 8th classification problem.

Fig. 7: Bumps corresponding to the AAFs in Fig. 6.
while Fig. (f)f is intrinsically a bump. As expected, the output activation is adapted to the learning task and different from classification. As seen in Fig. (b)b the activation function approaches a linear function (in \([-0.25,0.25]\)) for all regression problems. Notice that in a non-adaptive setting one needs to choose the last non-linearity according to the task. This represents an improvement with respect to non-adaptive architectures as it is general enough to encompass different tasks.

### _Effect of the number of coefficients_

Increasing the number of coefficients \(Q\) in the DCT representation theoretically provides more expressiveness to the network. However, in practice, increasing the number of trainable parameters hinders the convergence of the algorithm because there are more local minima. Fig. 13 shows the map for the 7th classification problem trained with \(Q=\{10,12,20\}\) coefficients. As expected, the accuracy improves from 10 to 12 coefficients, but it does not improve beyond that, probably because the algorithm gets trapped at a local minimum. To prevent this from happening, we first train the network with \(Q=12\) and, once it converges, we allow the network to train up to \(Q=20\) coefficients. We term this procedure _annealing_ because we slowly adapt the optimization space to avoid suboptimal local minima.

Fig. 14 shows the output AAF for the cases Fig. (b)b and (d)d. Increasing the number of coefficients allows to create a steeper transition, reducing the number of errors around the boundary. Doing so increases the Gibbs effect, although it produces no harm to the learning process because those samples are far from the boundary and always correctly classified as either 1 or -1.

## VII Conclusions

In this paper we have presented ENN, a novel neural network architecture with adaptive activation functions. Under a signal processing perspective we use the DCT to design the non-linear functions, whose coefficients can be trained using backpropagation. The formulation is general enough to encompass different learning tasks. Specifically, ENN is able to adapt the activation functions for classification and regression problems. A two-step training procedure is also proposed, which speeds up the convergence. We provide insights in the interpretability of the network by recovering the concept of bump, this is, the response of each neuron in the output space. Through extensive experiments we determine that the expressiveness of a neural network highly depends on the activation function. Particularly, the key strength of ENN is the periodic nature of the DCT, providing high accuracy for a wide range of non-linear classification problems. In some cases outperforming state of the art fixed activation functions up to 40% in accuracy.
