Extended Deep Adaptive Input Normalization for Preprocessing Time Series
  Data for Neural Networks
Data preprocessing is a crucial part of any machine learning pipeline, and it
can have a significant impact on both performance and training efficiency. This
is especially evident when using deep neural networks for time series
prediction and classification: real-world time series data often exhibit
irregularities such as multi-modality, skewness and outliers, and the model
performance can degrade rapidly if these characteristics are not adequately
addressed. In this work, we propose the EDAIN (Extended Deep Adaptive Input
Normalization) layer, a novel adaptive neural layer that learns how to
appropriately normalize irregular time series data for a given task in an
end-to-end fashion, instead of using a fixed normalization scheme. This is
achieved by optimizing its unknown parameters simultaneously with the deep
neural network using back-propagation. Our experiments, conducted using
synthetic data, a credit default prediction dataset, and a large-scale limit
order book benchmark dataset, demonstrate the superior performance of the EDAIN
layer when compared to conventional normalization methods and existing adaptive
time series preprocessing layers.
Marcus A. K. September, Francesco Sanna Passino, Leonie Goldmann, Anton Hinel
# Extended Deep Adaptive Input Normalization for Preprocessing Time Series Data for Neural Networks

###### Abstract

Data preprocessing is a crucial part of any machine learning pipeline, and it can have a significant impact on both performance and training efficiency. This is especially evident when using deep neural networks for time series prediction and classification: real-world time series data often exhibit irregularities such as multi-modality, skewness and outliers, and the model performance can degrade rapidly if these characteristics are not adequately addressed. In this work, we propose the EDAIN (Extended Deep Adaptive Input Normalization) layer, a novel adaptive neural layer that learns how to appropriately normalize irregular time series data for a given task in an end-to-end fashion, instead of using a fixed normalization scheme. This is achieved by optimizing its unknown parameters simultaneously with the deep neural network using back-propagation. Our experiments, conducted using synthetic data, a credit default prediction dataset, and a large-scale limit order book benchmark dataset, demonstrate the superior performance of the EDAIN layer when compared to conventional normalization methods and existing adaptive time series preprocessing layers.

## 1 Introduction

There are many steps required when applying deep neural networks or, more generally, any machine learning model, to a problem. First, data should be gathered, cleaned, and formatted into machine-readable values. Then, these values need to be preprocessed to facilitate learning. Next, features are designed from the processed data, and the model architecture and its hyperparameters are chosen. This is followed by parameter optimisation and evaluation using suitable metrics. These steps may be iterated several times.

A step that is often overlooked in the literature is _preprocessing_ (see, for example, Koval, 2018), which consists in operations used to transform raw data in a format that is suitable for further modeling, such as detecting outliers, handling missing data and normalizing features. Applying appropriate preprocessing to the data can have significant impact on both performance and training efficiency (Cao et al., 2018; Nawi et al., 2013; Passalis et al., 2020; Sola and Sevilla, 1997; Tran et al., 2021). However, determining the most suitable preprocessing method usually requires a substantial amount of time and relies on iterative training and performance testing. Therefore, the main objective of this work is to propose a _novel efficient automated data preprocessing method for optimising the predictive performance of neural networks_, with a focus on normalization of multivariate time series data.

### Preprocessing multivariate time series

Let \(\mathcal{D}=\left\{\mathbf{X}^{(i)}\in\mathbb{R}^{d\times T},\ i=1,\ldots,N\right\}\) denote a dataset containing \(N\) time series, where each time series \(\mathbf{X}^{(i)}\in\mathbb{R}^{d\times T}\) is composed of \(T\)\(d\)-dimensional feature vectors. The integers \(d\) and \(T\) refer to the feature and temporal dimensions of the data, respectively.
Also, we use \(\mathbf{x}_{t}^{(i)}\in\mathbb{R}^{d},\ t=1,\ldots,T\) to refer to the \(d\) features observed at timestep \(t\) in the \(i\)-th time series. Before feeding the data into a model such as a deep neural network, it is common for practitioners to perform \(z\)-score normalization (see, for example, Koval, 2018) on \(\mathbf{x}_{t}^{(i)}=(x_{t,1}^{(i)},\ldots,x_{t,d}^{(i)})\in\mathbb{R}^{d}\), obtaining

\[\tilde{x}_{t,k}^{(i)}=\frac{x_{t,k}^{(i)}-\mu_{k}}{\sigma_{k}},\ k=1,\ldots,d,\]

where \(\mu_{k}\) and \(\sigma_{k}\) denote the mean and standard deviation of the measurements from the \(k\)-th predictor variable. Another commonly used method is min-max scaling, where the observations for each predictor variable are transformed to fall in the value range \([0,1]\) (see, for example, Koval, 2018). In this work, we refer to these conventional methods as _static preprocessing_ methods, as the transformation parameters are fixed statistics that are computed through a single sweep of the training data. Most of these transformations only change the location and scale of the observations, but real-world data often contains additional irregularities such as skewed distributions, outliers, extreme values, heavy tails and multiple modes (Cao et al., 2023; Nawi et al., 2013), which are not mitigated by transformations such as \(z\)-score and min-max normalization. Employing static normalization in such cases may lead to sub-optimal results, as demonstrated in Passalis et al. (2020, 2021); Tran et al. (2021) and in the experiments on real and synthetic data in Section 4 of this work.

In contrast, better results are usually obtained by employing _adaptive_ preprocessing methods (see, for example Lubana et al., 2021; Passalis et al., 2020, 2021; Tran et al., 2021), where the preprocessing is integrated into the deep neural network by augmenting its architecture with additional layers. Both the transformation parameters and the neural network model parameters are then jointly optimised in an end-to-end fashion as part of the objective function of interest. The main contribution of this paper belongs to this class of methods: we propose a novel adaptive normalization approach, called EDAIN (Extended Deep Adaptive Input Normalization), which can appropriately handle irregularities in the input data, without making any assumption on their distribution.

### Contributions

The main contribution of our work is EDAIN, displayed in Figure 1, a neural layer that can be added to any neural network architecture for preprocessing multivariate time series. This method complements the shift and scale layers proposed in DAIN (Passalis et al., 2020, described in details in Section 2) with an adaptive outlier mitigation sublayer and a power transform sublayer, used to handle common irregularities observed in real-world data, such as outliers, heavy tails, extreme values and skewness.

Additionally, our EDAIN method can be implemented in two versions, named _global-aware_ and _local-aware_, suited to unimodal and multi-modal data respectively. Furthermore, we propose a computationally efficient variation of EDAIN, trained via the Kullback-Leibler divergence, named EDAIN-KL. Like EDAIN, this method can normalize skewed data with outliers, but in an unsupervised fashion, and can be used in conjunction with non-neural-network models.

EDAIN is described in details in Section 3, after a discussion on related methods in Section 2. The proposed methodology is extensively evaluated on synthetic and real-world data in Section 4, followed by a discussion on its performance and a conclusion. An open-source implementation of the EDAIN layer, along with code for reproducing the experiments, is available in the GitHub repository marcusGH/edain_paper.

## 2 Related Work

Several works consider adaptive normalization methods, but they all apply the normalization transformation to the outputs of the inner layers within the neural network, known as activations in the literature. A well-known example of these transformations is batch normalization (Ioffe and Szegedy, 2015), which applies \(z\)-score normalization to the output of each inner layer, but several alternatives and extensions exist (see, for example, Huang et al., 2023; Lubana et al., 2021; Yu and Spiliopoulos, 2023). To the best of our knowledge, there are only three other methods where the deep neural network is augmented by inserting the adaptive preprocessing layer as the first step, transforming the data before it enters the network: the Deep Adaptive Input Normalization (DAIN) layer Passalis et al. (2020), the Robust Deep Adaptive Input Normalization (RDAIN) layer Passalis et al. (2021), and the Bilinear Input Normalization (BIN) layer Tran et al. (2021). We will describe the DAIN layer in more detail in the next paragraph, as this method resembles our proposed EDAIN method the most.

The DAIN layer normalizes each time series \(\mathbf{X}^{(i)}\) using three sublayers: each time series is first shifted, then scaled, and finally passed through a gating layer that can suppress irrelevant features. The unknown parameters are the weight matrices \(\mathbf{W}_{a}\), \(\mathbf{W}_{b}\), \(\mathbf{W}_{c}\in\mathbb{R}^{d\times d}\), and the bias term \(\mathbf{d}\in\mathbb{R}^{d}\), and are used for the shift, scale, and gating sublayer, respectively. The first two layers together, perform the operation

\[\tilde{\mathbf{x}}_{t}^{(i)}=\left(\mathbf{x}_{t}^{(i)}-\mathbf{W}_{a}\mathbf{ a}^{(i)}\right)\oslash\mathbf{W}_{b}\mathbf{b}^{(i)},\]
where \(\mathbf{x}_{t}^{(i)}\in\mathbb{R}^{d}\) is the input feature vector at timestep \(t\) of time series \(i\), \(\odot\) denotes element-wise division, and \(\mathbf{a}^{(i)}\in\mathbb{R}^{d}\) and \(\mathbf{b}^{(i)}\in\mathbb{R}^{d}\) are summary statistics that are computed for the \(i\)-th time series as follows:

\[\mathbf{a}^{(i)}=\frac{1}{T}\sum_{t=1}^{T}\mathbf{x}_{t}^{(i)},\ \mathbf{b}^{(i)}= \sqrt{\frac{1}{T}\sum_{t=1}^{T}\Big{(}\mathbf{x}_{t}^{(i)}-\mathbf{W}_{a} \mathbf{a}^{(i)}\Big{)}^{2}}. \tag{1}\]

In Equation (1), the power operations are applied element-wise. The third sublayer, the gating layer, performs the operation

\[\tilde{\mathbf{x}}_{t}^{(i)}=\tilde{\mathbf{x}}_{t}^{(i)}\odot S\left( \mathbf{W}_{c}\mathbf{c}^{(i)}+\mathbf{d}\right).\]

Here, \(\odot\) is the element-wise multiplication operator, \(S:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}\) denotes the logistic function applied element-wise, and \(\mathbf{c}^{(i)}\) is the summary statistic

\[\mathbf{c}^{(i)}=\frac{1}{T}\sum_{t=1}^{T}\tilde{\mathbf{x}}_{t}^{(i)}.\]

The final output of the DAIN layer is thus

\[\tilde{\tilde{\mathbf{X}}}^{(i)}=\left[\tilde{\tilde{\mathbf{x}}}_{1}^{(i)}, \ldots,\tilde{\tilde{\mathbf{x}}}_{T}^{(i)}\right]\in\mathbb{R}^{d\times T}.\]

In the RDAIN layer proposed by Passalis et al. (2021), a similar 3-stage normalization pipeline as that of the DAIN layer is used, but a residual connection across the shift and scale sublayers is also introduced. The BIN layer (Tran et al., 2021) has two sets of linear shift and scale sublayers that work similarly to the DAIN layer, which are applied across columns and rows of each time series \(\mathbf{X}^{(i)}\in\mathbb{R}^{d\times T}\). The output of the BIN layer is a trainable linear combination of the two.

In real world applications, data often present additional irregularities, such as outliers, extreme values, heavy tails and skewness, which the aforementioned adaptive preprocessing methods are not designed to handle. Therefore, this work proposes EDAIN, a layer which comprises two novel sublayers that can appropriately treat skewed and heavy-tailed data with outliers and extreme values, resulting in significant improvements in performance metrics on real and simulated data. Also, the DAIN, RDAIN and BIN adaptive preprocessing methods are only primarily designed to handle multi-modal and non-stationary time series, which are common in financial forecasting tasks (Pasalsis et al., 2020). They do this by making the shift and scale parameters a parameterised function of each \(\mathbf{X}^{(i)}\), allowing a transformation specific to each time series data point, henceforth referred to as _local-aware_ preprocessing. However, these normalization schemes do not necessarily preserve the relative ordering between time series data points, which can degrade performance on unimodal datasets. As discussed in the next section, we address this drawback by proposing a novel _global-aware_ version for our proposed EDAIN layer, which preserves ordering by learning a monotonic transformation. It must be remarked that the EDAIN layer can also be fitted in local-aware fashion, to address multi-modality when present, providing additional flexibility when modelling real-world data.

## 3 Extended Deep Adaptive Input Normalization

In this section, we describe in detail the novel EDAIN preprocessing layer, which can be added to any deep learning architecture for time series data. EDAIN adaptively applies local transformations specific to each time series, or global transformations across all the observed time series \(\mathbf{X}^{(i)}\), \(i=1,\ldots,N\) in the dataset \(\mathcal{D}\). These transformations are aimed at appropriately preprocessing the data, mitigating the effect of skewness, outliers, extreme values and heavy-tailed distributions. This section discusses in details the different sublayers of EDAIN, its global-aware and local

Figure 1: Architecture of the proposed EDAIN (Extended Deep Adaptive Input Normalization) layer. The layout and color choices of the diagram are based on Figure 1 from Passalis et al. (2020).
aware versions, and training strategies via stochastic gradient descent or Kullback-Leibler divergence minimization.

An overview of the EDAIN layer's architecture is shown in Figure 1. Given some input time series \(\mathbf{X}^{(i)}\in\mathbb{R}^{d\times T}\), each feature vector \(\mathbf{x}^{(i)}_{t}\in\mathbb{R}^{d}\) is independently transformed sequentially in four stages: an outlier mitigation operation \(\mathbf{h}_{1}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}\), a shift operation: \(\mathbf{h}_{2}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}\), a scale operation: \(\mathbf{h}_{3}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}\), and a power transformation operation: \(\mathbf{h}_{4}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}\).

Outlier mitigation sublayer.In the literature, it has been shown that an appropriate treatment of outliers and extreme values can increase predictive performance (Yin and Liu, 2022). The two most common ways of doing this are omission and winsorization (Nyitrai and Virag, 2019). The former corresponds to removing the outliers from further analysis, whereas the latter seeks to replace outliers with a censored value corresponding to a given percentile of the observations. In this work we propose the following smoothed winsorization operation obtained via the \(\tanh(\cdot)\) function:

\[\ddot{\mathbf{x}}^{(i)}_{t}=\boldsymbol{\beta}\odot\tanh\left\{\left(\mathbf{ x}^{(i)}_{t}-\hat{\boldsymbol{\mu}}\right)\oslash\boldsymbol{\beta}\right\}+\hat{ \boldsymbol{\mu}}, \tag{2}\]

where the parameter \(\boldsymbol{\beta}\in[\beta_{\min},\infty)^{d}\) controls the range to which the measurements are restricted to, and \(\hat{\boldsymbol{\mu}}\in\mathbb{R}^{d}\) is the global mean of the data, considered as a fixed constant. In this work, we let \(\beta_{\min}=1\). Additionally, we consider a ratio of winsorization to apply to each predictor variable, controlled by an unknown parameter vector \(\boldsymbol{\alpha}\in[0,1]^{d}\), combined with the smoothed winsorization operator (2) via a residual connection. This gives the following adaptive outlier mitigation operation for an input time series \(\mathbf{x}^{(i)}_{t}\in\mathbb{R}^{d}\):

\[\mathbf{h}_{1}\left(\mathbf{x}^{(i)}_{t}\right)=\boldsymbol{\alpha}\odot \ddot{\mathbf{x}}^{(i)}_{t}+(\mathbf{1}_{d}-\boldsymbol{\alpha})\odot\mathbf{ x}^{(i)}_{t},\]

where \(\mathbf{1}_{d}\) is a \(d\)-dimensional vector of ones. Both \(\boldsymbol{\alpha}\) and \(\boldsymbol{\beta}\) are considered as unknown parameters as part of the full objective function optimised during training.

Shift and scale sublayers.The adaptive shift and scale layer, combined, perform the operation

\[\mathbf{h}_{3}\left\{\mathbf{h}_{2}\left(\mathbf{x}^{(i)}_{t}\right)\right\}= (\mathbf{x}^{(i)}_{t}-\boldsymbol{m})\oslash\boldsymbol{s},\]

where the unknown parameters are \(\boldsymbol{m}\in\mathbb{R}^{d}\) and \(\boldsymbol{s}\in(0,\infty)^{d}\). Note that the EDAIN scale and shift sublayers generalise \(z\)-score scaling, which does not treat \(\boldsymbol{m}\) and \(\boldsymbol{s}\) as unknown parameters, but it sets them to the mean and standard deviation instead:

\[\boldsymbol{m}=\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}\mathbf{x}^{(i)}_{t},\]

and

\[\boldsymbol{s}=\sqrt{\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}\left(\mathbf{x}^ {(i)}_{t}-\boldsymbol{m}\right)^{2}},\]

where the power operations are applied element-wise.

Power transform sublayer.Many real-world datasets exhibit significant skewness, which is often corrected using power transformations (Schroth and Muma, 2021), such as the commonly used Box-Cox transformation (Box and Cox, 1964). One of the main limitations of the Box-Cox transformation is that it is only valid for positive values. A more general alternative which is more widely applicable is the Yeo-Johnson (YJ) transform (Yeo and Johnson, 2000):

\[f^{\lambda}_{\mathrm{YJ}}(x)=\left\{\begin{array}{ll}\frac{(x+1)^{\lambda}-1 }{\lambda}&\text{if }\lambda\neq 0,\ x\geq 0,\\ \log(x+1)&\text{if }\lambda=0,\ x\geq 0,\\ \frac{(1-x)^{2-\lambda}-1}{\lambda-2}&\text{if }\lambda\neq 2,\ x<0,\\ -\log(1-x)&\text{if }\lambda=2,\ x<0.\end{array}\right. \tag{3}\]

The transformation \(f^{\lambda}_{\mathrm{YJ}}\) only has one unknown parameter, \(\lambda\in\mathbb{R}\), and it can be applied to any \(x\in\mathbb{R}\), not just positive values (Yeo and Johnson, 2000). The power transform sublayer of EDAIN simply applies the transformation in Equation (3) along each dimension of the input time series \(\mathbf{X}^{(i)}\). That is, for each \(i=1,\ldots,N\) and \(t=1,\ldots,T\), the sublayer outputs

\[\mathbf{h}_{4}\left(\mathbf{x}^{(i)}_{t}\right)=\left[f^{\lambda_{1}}_{ \mathrm{YJ}}\left(x^{(i)}_{t,1}\right),\ldots,f^{\lambda_{d}}_{\mathrm{YJ}} \left(x^{(i)}_{t,d}\right)\right],\]

where the unknown quantities to be optimised are the power parameters \(\boldsymbol{\lambda}=(\lambda_{1},\ldots,\lambda_{d})\in\mathbb{R}^{d}\).

### Global- and local-aware normalization

For highly multi-modal and non-stationary time series data, Passalis et al. (2020, 2021) and Tran et al. (2021) observed significant performance improvements when using local-aware preprocessing methods, as these allow forming a unimodal representation space from predictor variables with multi-modal distributions. Therefore, we also propose a _local-aware_ version of the EDAIN layer in addition to the _global-aware_ version we presented earlier. In the local-aware version of EDAIN, the shift and scale operations also depend on a summary representation of the current time series \(\mathbf{X}^{(i)}\) to be preprocessed:

\[\mathbf{h}_{3}\left\{\mathbf{h}_{2}\left(\mathbf{x}^{(i)}_{t}\right)\right\}= \left\{\mathbf{x}^{(i)}_{t}-\left(\boldsymbol{m}\odot\boldsymbol{\mu}^{(i)}_{ \mathbf{X}}\right)\right\}\oslash\left(\boldsymbol{s}\odot\boldsymbol{\sigma}^ {(i)}_{\mathbf{X}}\right).\]

The summary representations \(\boldsymbol{\mu}^{(i)}_{\mathbf{X}},\boldsymbol{\sigma}^{(i)}_{\mathbf{X}}\in \mathbb{R}^{d}\), are computed through a reduction along the temporal dimension of each time series \(\mathbf{X}^{(i)}\) (_cf._ Figure 2):

\[\boldsymbol{\mu}^{(i)}_{\mathbf{x}}=\frac{1}{T}\sum_{t=1}^{T}\mathbf{x}^{(i)}_{t},\ \boldsymbol{\sigma}^{(i)}_{\mathbf{x}}=\sqrt{\frac{1}{T}\sum_{t=1}^{T}\left( \mathbf{x}^{(i)}_{t}-\boldsymbol{\mu}^{(i)}_{\mathbf{x}}\right)^{2}}, \tag{4}\]
where the power operations are applied element-wise in (4). The outlier mitigation and power transform sublayers remain the same for the local-aware version, except the \(\hat{\mathbf{\mu}}\) statistic in Equation (2) is no longer fixed, but rather the mean of the input time series:

\[\hat{\mathbf{\mu}}^{(i)}=\frac{1}{T}\sum_{t=1}^{T}\mathbf{x}_{t}^{(i)}.\]

Order preservation.Adaptive local-aware preprocessing methods work well on multi-modal data such as financial forecasting datasets (Passalis et al., 2020, 2021; Tran et al., 2021). However, as we will experimentally demonstrate in the next section, local-aware preprocessing methods might perform worse than conventional methods such as \(z\)-score normalization when applied to unimodal data. This is because local-aware methods are not _order preserving_, since the shift and scale amount is different for each time series. In many applications in which the features have a natural and meaningful ordering, it would be desirable to have

\[\tilde{x}_{t,k}^{(i)}<\tilde{x}_{t,k}^{(j)}\text{ if }x_{t,k}^{(i)}<x_{t,k}^{(j)}, \tag{5}\]

for all \(i,j\in\{1,\dots,N\}\), \(t=1,\dots,T\) and \(k=1,\dots,d\), where \(\mathbf{X}^{(i)}\) denotes the output of a transformation with \(\mathbf{X}^{(i)}\) as input. This property does not necessarily hold for the local-aware methods (DAIN, RDAIN, BIN, and local-aware EDAIN). For unimodal features, the qualitative interpretation of the predictor variables might dictate that property (5) should be maintained (for example, consider the case of credit scores for a default prediction application, _cf._ Section 4.2).

As a solution, the proposed _global-aware_ version of the proposed EDAIN layer does not use any time series specific summary statistics, which makes each of the four sublayers monotonically non-decreasing. This ensures that property (5) is maintained by the global-aware EDAIN transformation, providing additional flexibility for applications on real world data where ordering within features should be preserved.

### Optimising the EDAIN layer

The output of the proposed EDAIN layer is obtained by feeding the input time series \(\mathbf{X}^{(i)}\) through the four sublayers in a feed-forward fashion, as shown in Figure 1. The output is then fed to the deep neural network used for the task at hand. Letting \(\mathbf{W}\) denote the weights of the deep neural network model, the weights of both the deep model and the EDAIN layer are simultaneously optimised in an end-to-end manner using stochastic gradient descent, with the update equation:

\[\Delta\,(\mathbf{\alpha},\mathbf{\beta},\mathbf{m},\mathbf{s},\mathbf{\lambda}, \mathbf{W})=\\ -\eta\bigg{(}\eta_{1}\frac{\partial\mathcal{L}}{\partial\mathbf{ \alpha}},\eta_{1}\frac{\partial\mathcal{L}}{\partial\mathbf{\beta}},\eta_{2}\frac{ \partial\mathcal{L}}{\partial\mathbf{m}},\eta_{3}\frac{\partial\mathcal{L}}{ \partial\mathbf{s}},\eta_{4}\frac{\partial\mathcal{L}}{\partial\mathbf{\lambda}}, \frac{\partial\mathcal{L}}{\partial\mathbf{W}}\bigg{)},\]

where \(\eta\in(0,\infty)\) is the base learning rate, whereas \(\eta_{1},\dots,\eta_{4}\) correspond to sublayer-specific corrections to the global learning rate \(\eta\). As Passalis et al. (2020) observed when training their DAIN layer, the gradients of the unknown parameters for the different sublayers might have vastly different magnitudes, which prevents a smooth convergence of the preprocessing layer. Therefore, they proposed using separate learning rates for the different sublayers. We therefore introduce corrections \(\eta_{\ell}\in\mathbb{R},\ \ell=\{1,2,3,4\}\) as additional hyperparameters that modify the learning rates for each of the four different EDAIN sublayers.

Furthermore, note that computing the fixed constant \(\hat{\mathbf{\mu}}\) in the outlier mitigation sublayer (2) would require a sweep on the entire dataset before training the EDAIN-augmented neural network architecture, which could be computationally extremely expensive. As a solution to circumvent this issue, we propose to calculate \(\hat{\mathbf{\mu}}\) iteratively during training, updating it using a cumulative moving average estimate at each forward pass of the sublayer. We provide more details on this in Section 1 of the supplementary material.

### EdaIN-Kl

In addition to the EDAIN layer, we also propose another novel preprocessing method, named EDAIN-KL (Extended Deep Adaptive Input Normalization, optimised with Kullback-Leibler divergence). This approach uses a similar neural layer architecture as the EDAIN method, but modifies it to ensure the transformation is invertible. Its unknown parameters are then optimised with an approach inspired by normalizing flows (see, for example, Kobyzev et al., 2021).

The EDAIN-KL layer is used to transform a Gaus

Figure 2: Visual comparison of the local- and global-aware versions of adaptive preprocessing schemes.
sian base distribution \(\mathbf{Z}\sim\mathcal{N}(\mathbf{0},I_{dT})\) via a composite function \(\mathbf{g}_{\boldsymbol{\theta}}=\mathbf{h}_{1}^{-1}\circ\mathbf{h}_{2}^{-1} \circ\mathbf{h}_{3}^{-1}\circ\mathbf{h}_{4}^{-1}\) comprised of the inverses of the operations in the EDAIN sublayers, applied sequentially with parameter \(\boldsymbol{\theta}=(\boldsymbol{\alpha},\boldsymbol{\beta},\boldsymbol{m}, \boldsymbol{s},\boldsymbol{\lambda})\). The parameter \(\boldsymbol{\theta}\) is chosen to minimize the KL-divergence between the resulting distribution \(\mathbf{g}_{\boldsymbol{\theta}}(\mathbf{Z})\) and the empirical distribution of the dataset \(\mathcal{D}\):

\[\hat{\boldsymbol{\theta}}=\operatorname*{argmin}_{\boldsymbol{\theta}}\operatorname {KL}\left\{\mathcal{D}\ ||\ \mathbf{g}_{\boldsymbol{\theta}}(\mathbf{Z})\right\}.\]

Note that we apply all the operations in reverse order, compared to the EDAIN layer, because we use \(\mathbf{g}_{\boldsymbol{\theta}}\) to transform a base distribution \(\mathbf{Z}\) into a distribution that resembles the training dataset \(\mathcal{D}\). To normalize the dataset after fitting the EDAIN-KL layer, we apply

\[\mathbf{g}_{\hat{\boldsymbol{\theta}}}^{-1}=\mathbf{h}_{4}\circ\mathbf{h}_{3 }\circ\mathbf{h}_{2}\circ\mathbf{h}_{1}\]

to each \(\mathbf{X}^{(i)}\in\mathbb{R}^{d\times T}\), similarly to the EDAIN layer. The main advantage of the EDAIN-KL approach over standard EDAIN is that it allows training in an unsupervised fashion, separate from the deep model. This enables its usage for preprocessing data in a wider set of tasks, including non-deep-neural-network models. An exhaustive description of the EDAIN-KL method is provided in Section 2 of the supplementary material.

## 4 Experimental Evaluation

For evaluating the proposed EDAIN layer we consider a synthetic dataset, a large-scale default prediction dataset, and a large-scale financial forecasting dataset. We compare the two versions of the EDAIN layer (global-aware and local-aware) and the EDAIN-KL layer to \(z\)-score normalization, to the DAIN (Passalis et al., 2020) layer and to the BIN (Tran et al., 2021) layer. For all experiments, we use a recurrent neural network (RNN) model composed of gated recurrent unit (GRU) layers, followed by a classifier head with fully connected layers. Categorical features, when present, are passed through an embedding layer, whose output is combined with the output of the GRU layers and then fed to the classifier head. Full details on the model architectures, optimization procedures, including learning rates and number of epochs, can be found in Section 3 of the supplementary material and in the code repository associated with this work.

### Synthetic Datasets

Before considering real-world data, we evaluate our method on synthetic data, where we have full control over the data generating process. To do this, we develop a synthetic time series data generation algorithm which allows specifying arbitrary unnormalized probability density functions (PDFs) for each of the \(d\) predictor variables. It then generates \(N\) time series of the form \(\mathbf{X}^{(i)}\in\mathbb{R}^{d\times T}\), along with \(N\) binary response variables \(y^{(i)}\in\{0,1\}\). We present a detailed description of the algorithm in Section 4 of the supplementary material.

For our experiments, we generated \(N_{\mathcal{D}}=100\) datasets, each with \(N=50\,000\) time series of length \(T=10\) and dimensionality \(d=3\). The three predictor variables were configured to be distributed as follows:

\[f_{1}(x) =10\cdot\Phi_{\mathcal{N}}\left\{10\left(x+4\right)\right\}\cdot p _{\mathcal{N}}(x+4)\] \[\quad+\mathbb{I}_{(8,9.5)}(x)\cdot e^{x-8}/10, \tag{6}\] \[f_{2}(x) =\left\{\begin{array}{ll}20\cdot p_{\mathcal{N}}(x-20),&\text{ if }x>\pi,\\ e^{x/6}\cdot\left\{10\sin(x)+10\right\},&\text{ if }x\leq\pi,\end{array}\right.\] (7) \[f_{3}(x) =2\cdot\Phi_{\mathcal{N}}\left\{-4(x-4)\right\}\cdot p_{\mathcal{ N}}(x-4), \tag{8}\]

where \(p_{\mathcal{N}}(\cdot)\) and \(\Phi_{\mathcal{N}}(\cdot)\) denote the PDF and cumulative distribution function (CDF) of the standard normal distribution, and \(\mathbb{I}_{\mathcal{A}}(\cdot)\) is the indicator function on the set \(\mathcal{A}\). Samples from the dataset are visualised in Figure 3. We train and evaluate a RNN model with the architecture described earlier on each of the \(N_{\mathcal{D}}\) datasets using a 80%-20% train-validation split. Our results are presented in Table 1, where the binary cross-entropy (BCE) loss and the accuracy on the validation set are used as evaluation metrics.

From our experiments on the synthetic datasets, we observe that the model performance is more unstable when no preprocessing is applied, as seen from the increased variance in Table 1. We also observe that \(z\)-score normalization only gives minor performance improvements when compared to no preprocessing, aside from reducing the variance. As we have perfect information about the underlying data generation mechanism from Equations (6), (7) and (8), we also compared our methods to what we refer to as _CDF inversion_, where each observation is transformed by the CDF of its corresponding distribution, and then transformed via \(\Phi_{\mathcal{N}}^{-1}(\cdot)\), giving predictor variables with standard normal distributions. We also apply this method to the real-world datasets in Section 4.2 and Section 4.3, but since the true PDFs are unknown in those settings, we estimate the CDFs using quantiles

Figure 3: Histogram across timesteps \(t=1,\dots,T\) of the \(d=3\) predictor variables from the synthetic data.
from the distribution of the training data. Out of all the methods not using information about the underlying data generation mechanism, the global-aware version of EDAIN demonstrates superior performance. It also almost performs as well as CDF inversion, which is able to perfectly normalize each predictor variable via its data generation mechanism. Finally, we observe that the local-aware methods perform even worse than no preprocessing: this might be due to the ordering not being preserved, as discussed in Section 3.1.

### Default Prediction Dataset

The first real-world dataset we consider is the publicly available default prediction dataset published by American Express Howard et al. (2022), which contains data from \(N=458\,913\) credit card customers. For each customer, a vector of \(d=188\) aggregated profile features has been recorded at \(T=13\) different credit card statement dates, producing a multivariate time series of the form \(\mathbf{X}^{(i)}\in\mathbb{R}^{d\times T}\), \(i=1,2,\ldots,N\). Given \(\mathbf{X}^{(i)}\), the task is to predict a binary label \(y^{(i)}\in\{0,1\}\) indicating whether the \(i\)-th customer definitely at any point within 18 months after the last observed data point in the time series. A default event is defined as not paying back the credit card balance amount within 120 days after the latest statement date Howard et al. (2022). Out of the \(d=188\) features, only the 177 numerical variables are preprocessed.

To evaluate the different preprocessing methods, we perform 5-fold cross validation, which produces evaluation metrics for five different 20% validation splits. The evaluation metrics we consider are the validation BCE loss and a metric that was proposed by Howard et al. (2022) for use with this dataset, and we refer to it as the _Amerx metric_. This metric is calculated as \(M=0.5\cdot(G+D)\), where \(D\) is the default rate captured at 4% (corresponding to the proportion of positive labels captured within the highest-ranked 4% of the model predictions) and \(G\) is the normalized Gini coefficient (see Section 5 of the supplementary material and, for example, Lerman and Yitzhaki, 1989).

Our results are reported in Table 2. From the table, it can be inferred that neglecting the preprocessing step deteriorates the performance significantly. Moreover, we observe that the local-aware methods (local-aware EDAIN, BIN, and DAIN) all perform worse than \(z\)-score normalization. This might be because the data in the default prediction dataset is mostly distributed around one central mode, and the local-aware methods discard this information in favour of forming a common representation space. Another possible reason is that the local-aware methods do not preserve the relative ordering between data points \(\mathbf{X}^{(i)}\) as per Equation (5), which might be detrimental for these types of datasets. As discussed in the previous section, predictor variables such as credit scores may be present in the dataset, which should only be preprocessed via monotonic transformations. From the results presented in Table 2, we can also conclude that the proposed global-aware version of the EDAIN layer shows superior average performance when compared to all alternative preprocessing methods. Additionally, it must be re

\begin{table}
\begin{tabular}{l|c c} \hline Preprocessing method & BCE loss & Binary accuracy (\%) \\ \hline No preprocessing & \(0.1900\pm 0.0362\) & \(91.68\pm 1.68\) \\ \(z\)-score normalization & \(0.1873\pm 0.0108\) & \(91.73\pm 0.65\) \\ CDF inversion & \(0.1627\pm 0.0094\) & \(92.89\pm 0.55\) \\ EDAIN-KL & \(0.1760\pm 0.0094\) & \(92.24\pm 0.59\) \\ EDAIN (local-aware) & \(0.2099\pm 0.0095\) & \(90.71\pm 0.57\) \\
**EDAIN (global-aware)** & \(\mathbf{0.1636\pm 0.0086}\) & \(\mathbf{92.83\pm 0.51}\) \\ BIN & \(0.2191\pm 0.0103\) & \(90.36\pm 0.59\) \\ DAIN & \(0.2153\pm 0.0146\) & \(90.48\pm 0.78\) \\ \hline \end{tabular}
\end{table}
Table 1: Experimental results on synthetic data, with 95% normal confidence intervals (\(\mu\pm 1.96\sigma\)) calculated across \(N_{\mathcal{D}}=100\) datasets. The gold-standard method for this simulation (CDF inversion) is underlined.

\begin{table}
\begin{tabular}{l|c c} \hline Preprocessing method & BCE loss & Amerx metric \\ \hline No preprocessing & \(0.3242\pm 0.0148\) & \(0.6430\pm 0.0195\) \\ \(z\)-score normalization & \(0.2213\pm 0.0039\) & \(0.7872\pm 0.0068\) \\ CDF inversion & \(0.2215\pm 0.0040\) & \(0.7861\pm 0.0072\) \\ EDAIN-KL & \(0.2218\pm 0.0040\) & \(0.7858\pm 0.0060\) \\ EDAIN (local-aware) & \(0.2245\pm 0.0035\) & \(0.7813\pm 0.0057\) \\
**EDAIN (global-aware)** & \(\mathbf{0.2199\pm 0.0034}\) & \(\mathbf{0.7890\pm 0.0078}\) \\ BIN & \(0.2237\pm 0.0038\) & \(0.7829\pm 0.0064\) \\ DAIN & \(0.2224\pm 0.0035\) & \(0.7847\pm 0.0054\) \\ \hline \end{tabular}
\end{table}
Table 2: Experimental results on the Amerx default prediction dataset, with 95% normal asymptotic confidence intervals (\(\mu\pm 1.96\sigma\)) calculated across folds.

Figure 4: BCE cross-validation loss across different folds in the Amerx default prediction dataset.
marked that most of the variance in the methods' performance arise from the folds themselves, as seen in Figure 4, and the global-aware EDAIN method consistently shows superior performance across all cross-validation folds when compared to the other methods.

### Financial Forecasting Dataset

For evaluating the proposed methods, we also considered a limit order book (LOB) time series dataset (FI-2010 LOB Ntakaris et al., 2018), used as a benchmark dataset in Passalis et al. (2020) and Tran et al. (2021). The data was collected across 10 business days in June 2010 from five Finnish companies (Ntakaris et al., 2018), and it was cleaned and features were extracted based on the pipeline of Kercheval and Zhang (2015). This resulted in \(N=453\,975\) vectors of dimensionality \(d=144\). The task is predicting whether the mid price will increase, decrease or remain stationary with a prediction horizon of \(H=10\) timesteps, where a stock is labelled as stationary if the mid price changes by less than \(0.01\%\). More details on the FI-2010 benchmark dataset can be found in Ntakaris et al. (2018).

For training, we use the anchored cross-validation scheme of Ntakaris et al. (2018). We sequentially increase the size of the training set, starting with a single day and extending it to nine days, while reserving the subsequent day for evaluation in each iteration, obtaining nine different evaluation folds. For evaluating the model performance, we look at the Cohen's \(\kappa\)(Artstein and Poesio, 2008) and macro-\(F_{1}\) score, obtained by averaging class-specific \(F_{1}\) scores across the three possible outcomes (increase, decrease or stationary).

We can draw several conclusions from the results reported in Table 3. Firstly, employing some form of preprocessing is essential, as not applying any preprocessing gives \(\kappa\) values close to 0, which is what is expected to occur by random guessing. Secondly, applying a local-aware normalization scheme (local-aware EDAIN, BIN and DAIN) greatly improves performance when compared to conventional preprocessing methods such as \(z\)-score scaling. Our third observation is that the proposed local-aware EDAIN method outperforms both BIN and DAIN on average. Finally, we note that most of the variability in the evaluation metrics arises from the folds themselves, similarly to the default prediction example (_cf._ Figure 4).

## 5 Conclusion

In this work, we proposed EDAIN, an adaptive data preprocessing layer which can be used to augment any deep neural network architecture that takes multivariate time series as input. EDAIN has four adaptive sublayers (outlier mitigation, shift, scale, and power transform). It also has two versions (local-aware and global-aware), which apply local or global transformations to each time series. Also, we proposed a computationally efficient variant of EDAIN, optimised via the Kullback-Leibler divergence, named EDAIN-KL. The EDAIN layer's ability to increase the predictive performance of the deep neural network was evaluated on a synthetic dataset, a default prediction dataset, and a financial forecasting dataset. On all datasets considered, either the local-aware or global-aware version of the proposed EDAIN layer consistently demonstrated superior performance.

In Section 4.3, we observed that the local-aware preprocessing methods gave significantly better performance than the global-aware version of EDAIN and \(z\)-score normalization. However, in Section 4.1 and Section 4.2 the opposite is observed, with the global-aware version of EDAIN demonstrating superior performance and the local-aware methods being outperformed by \(z\)-score normalization. We hypothesize these differences occur because local-aware methods do not preserve the relative ordering between observations, while the global-aware EDAIN method does. In the financial forecasting dataset, which is highly multi-modal, it appears that the observations' feature values relative to their mode are more important than their absolute ordering. Such considerations should be taken into account when deciding what adaptive preprocessing method is most suitable for application on new data.

There are several directions for future work. Passalis et al. (2021) observed that the performance improvements from DAIN differed greatly between different deep neural network architectures. Therefore, the effectiveness of EDAIN with other architectures could be further explored as only GRU-based RNNs were considered in this work. Additionally, with the proposed EDAIN method, one has to manually decide whether to apply local-aware or global-aware preprocessing. This drawback could be eliminated by extending the proposed neural layer to apply both schemes

\begin{table}
\begin{tabular}{l|c c} \hline \hline Preprocessing method & Cohen’s \(\kappa\) & Macro-\(F_{1}\)-score \\ \hline No preprocessing & \(0.0035\pm 0.0049\) & \(0.2859\pm 0.0228\) \\ \(z\)-score normalization & \(0.2772\pm 0.0550\) & \(0.5047\pm 0.0403\) \\ CDF inversion & \(0.3618\pm 0.0598\) & \(0.5798\pm 0.0373\) \\ EDAIN-KL & \(0.2870\pm 0.0642\) & \(0.5104\pm 0.0519\) \\
**EDAIN (local-aware)** & \(0.3836\pm 0.0554\) & \(0.5946\pm 0.0431\) \\ EDAIN (global-aware) & \(0.2820\pm 0.0706\) & \(0.5111\pm 0.0648\) \\ BIN & \(0.3670\pm 0.0640\) & \(0.5889\pm 0.0479\) \\ DAIN & \(0.3588\pm 0.0506\) & \(0.5776\pm 0.0341\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Experimental results on the FI-2010 LOB dataset, with 95% normal asymptotic confidence intervals (\(\mu\pm 1.96\sigma\)) calculated across 9 anchored folds.
to each feature and adaptively learning which version is most suitable. Another common irregularity in real-world data is missing values (Nawi et al., 2013; Cao et al., 2018). A possible direction would be to extend EDAIN with an adaptive method for treating missing values that makes minimal assumptions on the data generation mechanism (for example, Cao et al., 2018).
