Look Into the LITE in Deep Learning for Time Series Classification
Ali Ismail-Fawaz Maxime Devanne Stefano Berretti Jonathan Weber Germain Forestier


Look Into the LITE]Look Into the LITE in Deep Learning for Time Series Classification













[1]Ali Ismail-Fawazali-el-hadi.ismail-fawaz@uha.fr

1]Maxime Devannemaxime.devanne@uha.fr


2]Stefano Berrettistefano.berretti@unifi.it


1]Jonathan Weberjonathan.weber@uha.fr


1,3]Germain Forestiergermain.forestier@uha.fr


*[1]IRIMAS, Universite de Haute-Alsace, Mulhouse, France

[2]MICC, University of Florence, Florence, Italy

[3]DSAI, Monash University, Melbourne, Australia



Deep learning models have been shown to be a powerful solution for Time Series Classification (TSC). State-of-the-art architectures, while producing promising results on the UCR

and the UEA archives

, present a high number of trainable parameters.
This can lead to long training with high CO2 emission, power consumption and possible increase in the number of FLoating-point Operation Per Second (FLOPS).
In this paper, we present a new architecture for TSC, the Light Inception with boosTing tEchnique (LITE) with only 2.34% of the number of parameters of the state-of-the-art InceptionTime model, while preserving performance.
This architecture, with only 9,814 trainable parameters due to the usage of DepthWise Separable Convolutions (DWSC), is boosted by three techniques: multiplexing, custom filters, and dilated convolution. 
The LITE architecture, trained on the UCR, is 2.78 times faster than InceptionTime and consumes 2.79 times less CO2 and power.


To evaluate the performance of the proposed architecture on multivariate time series data, we adapt LITE to handle multivariate time series, we call this version LITEMV.
To bring theory into application, we also conducted experiments using LITEMV on multivariate time series representing human rehabilitation movements, showing that LITEMV not only is the most efficient model but also the best performing for this application on the Kimore dataset, a skeleton based human rehabilitation exercises dataset.
Moreover, to address the interpretability of LITEMV, we present a study

using Class Activation Maps to understand the classification decision taken by the model during evaluation.










[
    [
    September 9, 2024
=====================

This is the author’s version of an article under review in the International Journal of Data Science and Analytics.



§ INTRODUCTION




Time Series Classification (TSC) has been widely investigated by researchers in recent years.
Some TSC tasks include the classification of surgical evaluation <cit.>, action recognition of human motion <cit.>, cheat detection in video games <cit.>, interpretability <cit.>, Entomology <cit.>,  
Thanks to the availability of the UCR archive <cit.>, 

and the UEA archive <cit.>, the largest archives for univariate and multivariate

TSC datasets, a significant amount of work has been done in this domain.
Deep learning models have been proposed in the time series context for classification <cit.>, clustering <cit.>, averaging <cit.>, representation learning <cit.>, adversarial attacks <cit.>,  
Even though deep learning approaches proven to be very powerful for TSC, they present a large amount of trainable parameters, which often leads to a long training time, inference time and storage usage.

For this reason, some works started to question the need of such a large complexity in deep learning models for TSC such as ROCKET and its variants <cit.>.
Like for images, deep learning also presents a large complexity, which limits the usage of the models on small devices such as mobile phones and robots <cit.>.
Furthermore, Large Language Models (LLM) also shown to be very effective <cit.>, and that their complexity can be decreased, while preserving performance <cit.>.

In this paper, we address the methodology of reducing the complexity of deep learning models, while preserving the performance of the TSC task.
We argue that a large complex model may not be necessary in order to perform well on the UCR archive.
However, simply removing layers and or parameters to reduce complexity may not guarantee to preserve the performance.
For this reason, the neural network architecture often requires additional techniques in order to balance between complexity and performance.
In this work, we borrow existing techniques that have been efficiently used in state-of-the-art architectures on time series data.
These techniques are multiplexing convolutions <cit.>, dilated convolutions <cit.>, and custom filters <cit.>.
By combining these three techniques with a modified version of a small non complex model, the Fully Convolution Network (FCN) <cit.>, we propose a new architecture, named Light Inception with boosTing tEchniques (LITE).
The proposed model uses only 2.34% of the number of parameters of the Inception model, while being competitive with state-of-the-art architectures.
For instance, Figure <ref> shows that on the FreezerSmallTrain dataset, the classification accuracy of LITE is much higher than other approaches with way less trainable parameters.
The reduction in number of parameters is
made possible thanks to the usage of DepthWise Separable Convolutions (DWSC) <cit.>.
The additional techniques used in this proposed architecture, multiplexing, dilated, and custom convolutions,

have the advantage of only slightly increasing the number of parameters by about 1,000.

To position the proposed architecture among the state-of-the-art, we compared not only the accuracy but also the training time and number of parameters.
A comparison of CO2 and Power consumption using CodeCarbon [https://codecarbon.io/] python library is also presented.


Assessing the utility of LITE in the case of both univariate and multivariate time series classification datasets is essential.
For this reason, we propose two versions: LITE for univariate time series and LITEMV for multivariate time series.
LITEMV differs form LITE only in the first layer, where we use DWSC instead of standard convolutions.
This is not done in the case of the first layer of LITE, when the input is univariate because it would lead to learn only one filter, which is not suitable for the learning task (more details on DWSC are given in Section <ref>.



Moreover, to address a real world application, we assess the performance of LITE in the case of human rehabilitation exercises, represented as multivariate time series.

For this task, the goal is to analyze the rehabilitation exercise performed by a patient and associate a score representing how good the exercise has been carried out.
We rely on the usage of the Kimore dataset <cit.>, a skeleton based human rehabilitation exercises dataset.
Each sample of this dataset is a

multivariate time series of skeleton poses changing through time.
These skeleton poses are extracted using Kinect-v2 cameras <cit.>.
In the original dataset, each skeleton sequence is associated with a continuous label for each sample representing a score between 0 and 100 given by an expert to evaluate the performance of the patient.
Since this is a regression problem, we re-orient it to a classification task by classifying whether or not the patient performed the exercise correctly by setting a threshold to divide the scores (threshold score is 50).
To understand the decision making process of our proposed deep learning model on this task, we also present a study of the interpretability of LITEMV using the Class Activation Map (CAM) <cit.>.


The main contributions of this work
are:


    
  * The LITE model is presented as a new architecture for TSC, with only 2.34% of the number of parameters of the Inception model; 
    
    
  * An adaptation of LITE to handle multivariate data (LITEMV);
    
    
  * Extensive experiments showing that LITE achieves state-of-the-art results 
    
    on the UCR archive
    
    ;
    
    
  * Extensive experiments showing that LITEMV achieves promising results on some datasets of the UEA archive;
    
    
  * A comparison based on the number of trainable parameters, number of FLOPS, training time, CO2 and Power consumption;
    
  * A deeper analysis presented as an ablation study to show the impact of each technique added to boost the proposed model;
    
    
  * The application of LITEMV in a real application of evaluating human rehabilitation exercises with interpretability analysis using CAMs.
    


In what follows, we present some related work in Section <ref>, discuss the details of our proposed architecture in Section <ref>, present some results compared to other approaches in Section <ref>, and conclude by drawing future work in Section <ref>.




§ BACKGROUND AND RELATED WORK

Time Series Classification (TSC) was widely investigated in the literature.
Some work addressed this problem using machine learning algorithms by comparing similarity metrics between the time series <cit.>, decisions based on random forest algorithm <cit.>,  
The problem of most of those algorithms is that they require huge amount of CPU time to perform their calculations, and can not be parallelized on a cluster of GPUs.
For these reasons, deep learning for TSC is being used in the recent years. 
Even though the performance of deep learning is better than machine learning algorithms, the number of parameters to be optimized may be very high. 
In what follows, we first define the problem at hand, then we present some works that tackled the TSC problem using machine and deep learning techniques.
Finally, we present some works that addressed the training time problem of deep learning models and the large number of parameters.



 §.§ Definitions

Let x be a 

multivariate

time series of length L, and let 𝒟={x_i,y_i}_i=0^N-1 be a dataset of N 

multivariate

time series x_i with their corresponding labels y_i. 
The goal of this work is to construct an algorithm to learn how to correctly classify each input time series to its corresponding label.



 §.§ Machine Learning for TSC

The basic approach to solve TSC tasks
is by using the Nearest Neighbour (NN) algorithm.
In order to use this algorithm, a specific similarity metric for time series data should be defined.
In <cit.>, the authors used the Dynamic Time Warping algorithm (DTW) to define a similarity metric for the NN algorithm.
However, this algorithm does not have the ability to extract features from the input samples. 
The work in <cit.> also used the same algorithm but with an upgraded version of DTW called shapeDTW that aligns a local neighborhood around each point.
The main limitation of the DTW algorithm and its variants is the time complexity, which is a function of the time series length, , 𝒪(L^2).



 §.§ Deep Learning for TSC

In this section, we present the
work done on TSC using deep learning approaches.
The simplest architecture is the Multi Layer Perceptron (MLP) proposed in <cit.> that uses fully connected layers and dropout operations. 
This architecture is limited given the fact that it ignores the temporal dependency in a time series.
The Fully Convolution Network (FCN) was also proposed in <cit.> that uses 1D convolution operations.
In this model, the backpropagation algorithm finds the best filters to extract features from the time series, and correctly classifies the samples. In this model, convolutions account for temporal dependencies in time series data, and they are also independent of the input time series length.
The authors in <cit.> also proposed the ResNet model, which uses the residual connections <cit.> to solve the vanishing gradient problem. 
A comparative study in <cit.> shows that using convolutions, especially ResNet, outperforms other models that use multi-scale transformation or pooling layers with convolutions <cit.>.
ResNet and FCN use Batch Normalization and ReLU activation instead of pooling operations after each convolution layer to avoid overfitting.
The state-of-the-art model in deep learning for TSC on the UCR archive <cit.> is InceptionTime <cit.>, where the authors adapted the original Inception model on images for time series data.
InceptionTime has the ability to detect multiple patterns of different length given to the multiplexing technique. This technique comes down to learning multiple convolution layers on the same input but with different characteristics.
It is important to note that InceptionTime is an ensemble of five Inception models each trained separately.


While FCN, ResNet have also been evaluated for multivariate data, some additional deep learning models have been proposed to address the task of TSC in the case of multivariate data <cit.>.


More recently, the Disjoint-CNN architecture <cit.>, considered a new methodology for applying convolutions on multivariate time series that separates the temporal and spatial patterns extraction into two steps.

The reference deep learning model for multivariate TSC on the UEA archive <cit.> is ConvTran <cit.>, a Transformer based architecture.
ConvTran uses a Disjoint-CNN based encoder followed by self-attention layers <cit.> with novel absolute and relative positional encoders.




 §.§ Reducing Model Complexity

Even though deep learning for TSC shown to be very effective, it still has some issues.
One of these issues we address in this work is the large number of parameters to be optimized, which increases the training time as well.
Recently, in <cit.> a new approach was proposed, called ROCKET, that also includes convolution operations,
but is way faster than InceptionTime. 
They proposed not to learn few filters with a backpropagation algorithm, but instead randomly generate a large number of filters with different characteristics. These characteristics include the length of the filter, the bias value, the dilation rate,  
It has been shown that on the UCR archive, no statistical significant difference can be found between InceptionTime and ROCKET.
The main advantage of ROCKET compared to InceptionTime is the training and inference time.
Some adaptations of ROCKET were proposed in order to boost its performance even more such as MiniROCKET <cit.> and MultiROCKET <cit.>.
Knowledge distillation <cit.> was also approached for the TSC model called FCN <cit.>. In this study, the authors proposed a smaller variant of FCN with a lower number of convolution layers and filters to learn.

Furthermore, the work in <cit.> proposed to hand-craft some custom convolution filters instead of randomly generate them. Those hand-crafted filters are constructed in a way to get activated on increasing and decreasing intervals as well as peaks in the time series.
By using these filters, the authors were able to construct a Hybrid FCN (H-FCN). Results on the UCR archive shown that H-FCN is statistically significantly better than FCN and is competitive with InceptionTime.
The H-FCN model uses the custom filters in parallel to the learned filters in the first layer.

Some work to optimize the complexity of large models were proposed in Computer Vision as well.
For instance, the authors in <cit.> proposed the usage of DWSC instead of standard ones.



The MobileNet architecture as proposed in <cit.> proven to be very competitive with state-of-the-art models on ImageNet <cit.> with way less complexity.

In Natural Language Processing, some works proposed the usage of Small Language Models (SLM) as a one-shot learning approach <cit.>. The authors showed that with way less parameters than GPT-3 <cit.>, their model can have no significant difference in performance.




§ PROPOSED APPROACH




 §.§ Convolutions for TSC

Multiple Convolution Neural Networks (CNNs) were proposed for the task of TSC and they all proved how they outperform other methods.
The Fully Convolution Network (FCN) is a simple three layered network, where each layer is composed of 1D convolutions followed by a batch normalization and a ReLU activation function.
As FCN is only composed of simple 1D convolution layers, its performance generally lags behind more advanced architectures using residual connections (ResNet) or Inception modules (InceptionTime).
In this paper, we present an adaptation of the FCN architecture that only has 2.34% of the number of parameters of the Inception model. 
Given this significant drop in the number of parameters, boosting techniques are used in order to preserve the performance of Inception.

First, we discuss about two ways of applying convolutions with less number of parameters, while preserving the performance. The first approach uses standard convolutions with BottleNecks (BN), and the second uses DWSC.



  §.§.§ Standard Convolutions with BottleNecks

Many approaches that use CNN based architectures suffer from the problem of high number of parameters such as ResNet <cit.>.
For this reason, the authors of InceptionTime <cit.> proposed to use a BottleNeck operation in order to reduce the number of parameters.
This BottleNeck operation is made of 1D convolutions with a unit kernel size. 

The following example shows the impact of this operation on reducing the number of parameters.
Suppose at a depth d in the network, the input number of channels is C_in. The following convolution layer of depth d+1 projects the input into a new space with a number of channels C_out using a kernel of size k. On the one hand, without a BottleNeck operation, the number of learned parameters is C_in*C_out*k. On the other hand, with a BottleNeck operation that uses C_bn filters of size 1, the number of learned parameters is C_in*C_bn*1+C_bn*C_out*k.
This operation reduces the number of parameters if and only if the following inequality is true:


    C_in*C_bn + C_bn*C_out*k < C_in*C_out*k ,


which indicates that the condition on the BottleNeck operation is:


    C_bn < C_in*C_out*k/C_in + C_out*k.


The goal of the BottleNeck operation is to learn the same number of filters in the output channels (C_out), while reducing, at the same time,

the intermediate learned filters between the input and output channels (C_in*C_out).



  §.§.§ DepthWise Separable Convolution DWSC

DWSC can be divided into two phases: DepthWise convolution (Phase 1), and PointWise convolution (Phase 2).
A visual representation of the DWSC operation is presented in Figure <ref>.

In standard convolutions, if the input sample of length L has C_in channels and the desired output is a space with C_out channels using a kernel of size k, then the number of learned parameters is C_in*C_out*k. The number of multiplications is L*Cout*C_in*k.



  
DepthWise convolution In this phase, if the convolution is done using a kernel of size k, then the number of learned filters is C_in, and the output number of channels will be C_in (such as the input). In other words, for each dimension of the input time series, one filter is learned.



  
PointWise convolution This phase projects the output of the DepthWise convolution into a space with a desired number of channels C_out. This is done by applying a standard convolution with C_out filters of kernel size 1 (a BottleNeck).

Hence, by combining these two phases, the number of learned parameters in a DWSC becomes C_in*k + C_in*C_out.

The following calculation finds the condition that the DWSC have less parameters to learn compared to the standard one:


    C_in*C_out*k_standard Conv   > C_in*k + C_in*C_out_separable Conv
    
            C_in*C_out*k    > C_in*(k+C_out)
    
            C_out*k    > k + C_out
    
            k*(C_out - 1)    > C_out
    
            k    > C_outC_out - 1k > 1.


This means that if the number of desired output channels is high enough (if C_out >= 3 the previous equation holds), then DWSC have less parameters to learn compared to the standard convolutions.

The number of multiplications performed in the DWSC is C_in*L*k + Cin*C_out*L*1. The following calculation finds the second condition for when DWSC have less multiplications to perform compared to standard convolutions:


    C_in*C_out*L*k_standard convs   > C_in*L*k + C_in*C_out*L*1_separable convs
    
            C_in*C_out*L*k    > C_in*L*(k + C_out)
    
            C_out*k    > k + C_out
    
            k(C_out - 1)    > C_out
    
            k    > C_outC_out-1k > 1.


This concludes that DWSC have less parameters to learn with less number of multiplications to perform compared to standard convolutions.
In this work, we present a comparison between the usage of DWSC or standard ones + BottleNecks.
Our results demonstrate that with the techniques added to boost DWSC, we can maintain performance, while significantly reducing the number of parameters to optimize.

After defining two techniques to use convolutions in a more optimized way concerning number of parameters and multiplications, some other techniques should be defined as well.
These techniques aim to minimize the impact of parameters reduction in convolutions operations explained above.



 §.§ Boosting Techniques

The following techniques are borrowed from the literature.



  
Multiplexing
Multiplex convolution was proposed in the architecture of Inception <cit.>. Its main idea is to learn in parallel different convolution layers of different kernel size.
A multiplexing example is shown in Figure <ref>.





  
Dilation
Dilated convolutions were not very explored for deep supervised learning on TSC but they were used in self-supervised models showing to be very effective <cit.>.
Dilation in convolutions filters defines the number of empty cells in the kernel. Suppose a kernel of length 3 has the following parameters k = [k_0,k_1,k_2], a dilation of rate 2 will produce the following kernel k = [k_0, skip, k_1, skip, k_2].
The skip parameter indicates that the convolution layer will not use the values of the input aligned with this index of the kernel.
A visualization of the dilation effect on convolution can be seen in Figure <ref>.
Dilation will help increasing the receptive field of a model without having to add deeper layers because the dilated kernel will find the deeper combinations in the same layer.





  
Custom Filters
Custom filters were proposed in <cit.>. The authors hand-crafted some kernels in order to detect specific patterns in the input time series. These filters were then added to Inception and results on the UCR archive have shown that such filters can help with the generalization and boost the performance. This is due to the fact that these filters are generic and fixed (not learned).
This allows the model to focus on learning new patterns harder to detect.



 §.§ Proposed architecture



  §.§.§ Light Inception with boosTing tEchniques (LITE)

In our proposed architecture, we reduce the number of parameters, while preserving performance. This is obtained by using DWSC and the previously explained boosting techniques.
First, custom filters are used in the first layer parallel to the first layer. Second, in this first layer, multiplexing convolution is used in order to detect different patterns of different characteristics (three parallel convolution layers). Third, the second and third layer present the usage of dilation in their kernels.
It is important to notice that for the first layer, standard convolutions are used instead of DWSC. This is due to the fact that the input time series is univariate and DWSC will learn only one filter.
A summary of the architecture is given in Figure <ref>.






  §.§.§ LITEMV: Adapting LITE for Multivariate Time Series Data

The first layer of the LITE architecture uses the standard convolution in the case of univariate data.
This is due to the fact that using DepthWise Separable Convolutions in the first layer means the model would only learn one filter and then re-scale the output with one learned value.
For this reason we kept the usage of Standard Convolutions in the first layer.
However, in the case of multivariate data, the model will learn a filter per channel and then learn how to combine in the PointWise Convolution step.
A second information that is important to note is that the custom filters in the first layer of LITE are basically Standard Convolutions.
This means in the case of multivariate input, the custom filter will be applied to each channel independently and the output are summed.
This is done for each of the used custom filters.
This can be an issue in the case of multivariate input given we suppose an equal importance per custom filter for all channels when summing the outputs.
In the multivariate version of LITE, we instead concatenate the outputs of all custom filters operations on all channels.
We refer in the rest of this work to the multivariate version of LITE as LITEMV.




  §.§.§ Ensemble

Ensemble learning is a technique of combining the prediction of multiple models in order to reduce the variance, and it has been shown to be very effective in the literature <cit.>. Applying an ensemble of multiple classification models is equivalent to find the average predicted distribution of all the models.
This average distribution is finally used for choosing the predicted class.
This motivated us to build an ensemble of multiple LITE models to form LITETime 

and LITEMVTime

(by adding the suffix Time following <cit.>). Moreover, the usage of an ensemble in the case of LITE is also motivated by its small architecture and the fact that it can boost less complex architectures even more.



§ EXPERIMENTAL EVALUATION




 §.§ Datasets

The UCR archive <cit.> is the largest directory for the TSC problem. It is publicly available <cit.>.
The archive contains 128 datasets of univariate TSC tasks.
Some tasks involves Electrocardiography (ECG) time series data and some are observations of Sensors,  

To evaluate the performance of the proposed architecture on more datasets, we detail in Section <ref> some experiments on multivariate time series data.
The usage of multivariate data is a bit different given the uniqueness of this data of having multiple variables changing through time.
We rely on the UEA Multivariate Time Series Classification (MTSC) archive <cit.>.
This archive consists of 30 different datasets ranging from medical data recording ECG signals to speech recognition.
The number of channels in this archive range from 2 to 1345.


Each dataset is split into a training and a testing set. The labels are available for all the samples. In order to train the model on a normalized dataset, we apply z-normalization over all the samples independently. This normalization technique reduces the time series samples to a zero mean and unit variance sequence.



 §.§ Implementation Details

Our results were obtained on the UCR archive using a GTX 1080 GPU with 8GB of VRAM.
In the experiments, we accounted for the training time, inference time (testing time), CO2 and Power consumption.
The model used for testing is the best model during training, chosen by monitoring the training loss. The Adam optimizer was used with a Reduce on Plateau learning rate decay method by monitoring the training loss. For the Adam optimizer we used the default set up of the Tensorflow Python model. The model is trained with a batch size of 64 for 1500 epochs.

For the hyper-parameters of the LITE architecture presented in Figure <ref>, the following setup was used:
N_i=6, variations in kernel sizes =[2,4,8,16,32,64]; N_d=6, variations in kernel sizes =[2,4,8,16,32,64]; N_p=5, variations in kernel sizes =[6,12,24,48,96]; N=32; K=40; D0=1 in order to start with no dilation and increase with depth (D1=2 and D2=4).

The same hyper-parameters were used in the case of LITEMV.

The source code is publicly available at .



 §.§ Results and Discussion



Below, we provide the comprehensive set of experimental results acquired during the course of this study.

First, we present an efficiency comparison between LITE and other architectures in the literature.
This study is done over the case of univariate data only because in the case of multivariate data the study will depend on each dataset and its number of channels.
Second, we present an ablation study to make sure that LITE utilizes all of its boosting techniques during training.
Third, we present the performance using the accuracy metric on both the UCR and UEA archives.
Finally, we evaluate the usage of LITEMVTime in a real world application for human motion rehabilitation on the Kimore dataset.




  §.§.§ Number of Parameters, FLOPS Training Time, Testing time, CO2 and Power Consumption

Table <ref> summarizes the number of parameters, the number of FLoating-point Operations Per Second (FLOPS), training time, inference time, CO2 and Power consumption over the 128 datasets of the UCR archive.
We show the number of trainable parameters of the architecture without the last classification Fully Connected layer because it depends on each dataset (number of classes). The rest of the information is summed over the 128 datasets of the UCR archive and averaged over five different runs.

First, the table shows that the smallest model in terms of number of parameters is the LITE with 9,814 parameters.
This is mainly due to the usage of DWSC instead of standard ones.
Compared to FCN ResNet and Inception, LITE has only 3.7% 1.95% and 2.34% of their number of parameters, respectively.
Second, the fastest model in the training phase is LITE, with a training time of 0.62 days.
LITE is 2.79, 3.08 and 2.71 time faster than FCN, ResNet and Inception, respectively. 
Third, LITE is the model that consumes the smallest amount of CO2 and Power, 0.1048 grams and 0.2468 Watts, respectively.
Compared to the other approaches, LITE presents the fastest and most ecologic model for TSC compared to FCN, ResNet and Inception.
We believe, given the factors explained above, that LITE can be used for the deployment of deep learning for TSC in small machine such as mobile phones.




  §.§.§ Accuracy Performance on the UCR Archive



In what follows, a one-vs-one comparison is 
presented between the models in order to show that LITE can preserve the performance of the more complex architectures.
This one-vs-one comparison comes down to a Win/Tie/Loss count on the 128 datasets of the UCR archive between two classifiers.
This comparison is visualized in Figures <ref> and <ref>. Each point in these plots represents one dataset of the UCR archive. The x-axis contains the accuracy value on the test set using classifier-x and the y-axis the ones using classifier-y.

In order to evaluate the significance of the Win/Tie/Loss comparison, a statistical Wilcoxon Signed Rank Test <cit.> is used. This test will return a statistical value, the P-Value, representing how significant the difference is between the two classifiers. If the P-Value is low, this would mean that the difference in performance between the two classifiers is statistically significant.
If the last condition is not true, it means that there are not enough examples (datasets) to find a statistical significant difference between the classifiers. This Wilcoxon test needs a P-Value threshold, usually in the literature a 0.05 (or 5%) threshold is used.

On the one hand, the results presented in Figure <ref> show that LITE beats FCN significantly (low P-Value), and is statistically not significant than ResNet (high P-Value).
The results compared to ResNet are impressive given the small complexity of LITE (almost 1.95% of ResNet's number of parameters). On the other hand, the comparison shows that LITE still is not significantly close to Inception.
To study more the reason why LITE performs not as good as Inception with a large margin (more than 10%), we presented some characteristics of those datasets in Table <ref>.
This table shows that some of the datasets have either long time series or small training set.
Firstly, this indicates that Inception is
better than LITE on long time series given its large receptive field (deeper architecture). Secondly, this points out that Inception generalizes better better in the case where the dataset presents a small training set so it can generalize better.

Given that Inception still beats LITE, an ensemble comparison shows the real performance of the proposed architecture.
This is due to the fact that LITE has way less parameters (2.34% of Inception's number of parameters), which can make it sensitive to a higher variance when training with different initialization.
Applying an ensemble removes this variance as explained before.

The comparison between LITETime with InceptionTime and ROCKET is presented in Figure <ref>. This comparison shows that, given the 128 datasets of the UCR archive, there are not enough datasets to find a statistical significance in the difference of performance with InceptionTime and ROCKET. We included ROCKET in the ensemble comparison with LITETime because the motivation in ROCKET is to replace the ensemble technique by using random filters instead of learning them starting with different initialization.

Those last results suggest that in order to get a good performance on the UCR archive, a large complex architecture with a high number of parameters is not always needed.



For a multi-classifier comparison, the average rank of each model is shown in a Critical Difference Diagram <cit.> (CD Diagram) based on the ranking classifiers given the average rank over multiple datasets, the two tailed Wilcoxon Signed-Rank Test with the Holm multiple test correction <cit.>.
To generate the CD Diagram, we used the publicly available code .
This diagram also presents connections between classifiers, when the difference in performance is not statistically different following the Wilcoxon Signed Rank Test. A CD Diagram is presented in Figure <ref> and shows that LITETime comes 3rd on the average rank between ROCKET and InceptionTime.
The diagram also shows that FCN performs statistically significantly worse than LITE on the average rank.
Furthermore, on the UCR archive, no statistical significance can be observed between ResNet and LITE.
This last comparison shows the real impact of this work where LITE has almost 1.95% (Table <ref>) of ResNet's number of parameters.
These conducted results show that on a large amount of cases, there is no need for a complex model with high number of parameters to achieve good performance.





Furthermore, a new Multi-Comparison Matrix (MCM) evaluation tool was proposed that is stable to the variation of the addition and removal of classifiers <cit.>.
The MCM is presented in Figures <ref> and <ref> to compare LITE and LITETime, respectively, to other approaches in the literature.
The MCM uses the average accuracy on the UCR as an ordering metric instead of the average rank.
As presented in the MCMs, LITE and LITETime perform better than FCN and ResNet on the average accuracy and is closer to the performance of InceptionTime, which is not significantly different than LITETime (high p-value).
MCM has an advantage over the usage of the CD Diagram of being stable with the addition and removal of classifiers, given that the average performance would not change in this scenario unlike the average rank.
Another advantage is not using a multiple test correction for the P-Value significance test.









 §.§ Ablation Study - LITE




  §.§.§ Impact of Additional Techniques

The proposed LITE architecture, uses multiple techniques in order to improve the performance.
In order to show the impact of each technique on the proposed architecture, an ablation study is presented in this section.
First, the LITE is stripped down from the three used techniques: dilation, multiplexing and custom filters.
Second, given that for the multiplexing convolutions performed in the first layer there are a total of three layers with n filters, the Striped-LITE learns a total of 3n filters for the first layer.
The rest of the architecture is the same using DWSC without dilation.
We then add each boosting technique separately on the stripped LITE model and evaluate its performance.
The results of this ablation study are visualized in Figure <ref> in the form of a Heat Map.
Each cell of the Heat Map contains the Win Tie Loss count when evaluating the test accuracy on the UCR archive.
The addition of the P-Value statistics is presented in each cell in order to assess the significance in difference of performance.
The P-Value is emphasized in bold when it is lower than the specified threshold (0.05).
The colors of the Heat Map represent the difference in the average accuracy.



Results show that adding custom filters in the first layer as well as using multiplexing convolution in the first layer significantly boosts the performance.
The colors of the Heat Map indicates that adding the custom filters has also a positive impact on the average accuracy though it can add some parameters.
This is not the case for the multiplexing convolutions. We believe that this small average impact (0.34% overall) is not as important as the positive one. This is due to the fact that multiplexing reduces the number of parameters and wins over the majority of the datasets significantly.
The addition of the dilated convolution is shown to not have a statistical significance on the performance (P-Value >0.05). However, the average difference in accuracy shows that most of the times, using the dilated convolutions can improve the results. Given that dilation does not add more parameters and on average it boosts the performance, we keep it in the LITE architecture.
This is due to the fact that dilation increases the receptive field, so for large datasets this can be a boosting feature. The reason why sometimes Dilation can have a negative effect is because some of the datasets in the UCR archive do not require a large receptive field. 
Altogether, the LITE model (with the boosting techniques) will have less parameters compared to the striped LITE, while preserving performance compared to state-of-the-art models.
The decrease in number of parameters when using all the boosting techniques together comes from the fact that multiplexing removes more parameters than the custom filters adds.
Lastly, Figure <ref> shows the average rank of the models, such as in the CD Diagram explained in Section <ref>. The average rank of the Add-Custom-Filters is the lowest, while the Striped-LITE has the highest rank. Therefore, the worst model between the four presented in the Heat Map is the LITE without any boosting techniques.



  §.§.§ Impact of DWSC

To show the effect of DWSC as well, we replace them by standard convolutions followed by a BottleNeck. To get a non-noisy comparison, we used ensembles. 
Note that the usage of the ensemble technique is necessary in this case given that by removing the DWSC, the difference in number of parameters becomes very high (LITE has almost 11% of the compared model's number of parameters (the compared model has around 85,000 parameters).
In Figure <ref>, the one-vs-one comparison between LITETime and LITETime with Standard Convolutions is presented. Results show that the usage of DWSC does not have an effect on the performance because the P-Value is high 0.4556.
This means that the difference in performance is not statistically significantly different with less parameters mainly due to the usage of the DWSC.






  §.§.§ Number of LITE Models in the Ensemble

Given that LITETime is an ensemble of multiple LITE models, we stick to using five LITE models in LITETime in order to be fairly comparable to InceptionTime (ensemble of five Inception models).
It has been experimentally shown in <cit.> that no significant difference of performance can be found on the UCR archive when more than five Inception models are considered in the ensemble InceptionTime.
However, in the case of LITE, given its small architecture, its more bound to produce more variance and be less robust than Inception.
For this reason, we believe that the number of LITE models can be higher in the ensemble LITETime.
To experimentally prove our hypothesis, we trained ten different LITE models on all the UCR archive.
Ensembles of 1,2,3,…,10 are constructed by averaging over all possible ensemble combinations.
For instance, if the goal is to construct LITETime-3 (ensemble of three LITE models), we constructed the ensemble of all possible combinations of three LITE models from the pool of ten trained models.
We present the results on the CD diagram in Figure <ref>.







It can be seen from Figure <ref> that in the case of LITE, LITETime-5 (ensemble of five LITE models) is not the limit that we can achieve, instead it is LITETime-7.
This is due to the fact that LITE is almost 42 times smaller than Inception.


The ability to achieve this comes with the advantage of enhancing accuracy whilst LITETime exhibits a considerably smaller complexity compared to InceptionTime.
LITETime-5 with five models has almost 2.34% of InceptionTime's trainable parameters but LITETime-7 increases this percentage only to 3.27.
This can be seen with an example on the Beef dataset of the UCR archive in Figure <ref>.
In this figure, we plot how the number of models in the case of LITETime and InceptionTime changes the performance on unseen data.










 §.§ Accuracy Performance on the UEA Archive

In Table <ref>, we present the accuracy performance of LITEMVTime, LITETime and five different competitors: ConvTran, InceptionTime, Disjoint-CNN, FCN and ResNet.
The accuracies of all five competitors are reported from the ConvTran paper <cit.>.
For the LITEMVTime model, we can see that when it wins over other models, a significant gap is observed between the accuracies.
This can be clearly seen as well in the MCM plot of LITEMVTime with the rest of the models in Figure <ref>.
It can be seen that LITEMVTime comes second on the average performance, outperforming Disjoint-CNN and InceptionTime.
LITEMVTime still does not win on more than seven datasets compared to ConvTran, but looking deeper into Table <ref>, it can be seen that LITEMVTime can win with a large margin of accuracy.
For instance, for the EigenWorms dataset, the highest accuracy is reported by ConvTran with a value of 59.34%, however LITEMVTime produces an accuracy of 93.89% and LITETime with an accuracy of 95.42%.














To study in more details the cases where LITEMVTime works better than ConvTran with a significant margin, we present in Figure <ref> the difference in performance between these two models in function of the training samples per class.
This study is done to make sure if there are any common information about the cases where LITEMVTime works much better than ConvTran.
From Figure <ref>, we can see the most significant differences in performance are in the case of three datasets: StandWalkJump, EigenWorms and EthanolConcentration.
It can be seen that these datasets do not have the same range in the number of training examples.
For instance, StandWalkJump is considered to have a small training set with 4 training examples per class and 12 training examples in total.
However, the EthanolConcentration dataset has 66 training examples per class and 261 training samples in total.







Another approach to study the reason why these datasets would work better than ConvTran when using LITEMVTime is to dig into the number of dimensions, taking advantage that this is the case of multivariate time series data.
In Figure <ref>, we present the same information as in Figure <ref> but in function of the number of dimensions of the MTS datasets.
It can be seen that the same three datasets have a small number of dimensions.
It can be also seen that ConvTran always wins when the number of dimensions increases.









 §.§ Experiments on Human Motion Rehabilitation

To assess the performance of LITEMVTime in a real life application, we consider the Human Rehabilitation domain.
For this domain, the data is extracted from patients into the form of 3D skeleton based sequences.
A human expert has to asses for each patient if they are doing a movement (rehabilitation exercise) correctly or not.
This is a very important application and is highly correlated with this work given that this type of data can be regarded as Multivariate Time Series.
For each sequence recorded, we have a specific number of joints in a three dimensional space. 
In other words, each sequence is a multivariate time series with 3*J channels where J is the number of human joints.



For this experiment, we used the Kimore dataset <cit.>.
This dataset contains recorded sequences in the form of videos of patients performing rehabilitation exercises, which are then transformed into numerical MTS using kinect v2 <cit.>.
This dataset contains both healthy and unhealthy subjects performing five different rehabilitation exercises.
A human expert then assesses the quality of the performed exercise by providing a score between 0 and 100, from bad to good respectively.
A visualization of the distribution of these scores for each exercise is presented in Figure <ref> in the case of healthy and unhealthy subjects.
It can be seen, for instance, that most cases of unhealthy subjects tend to have a low score and a high score in the case of healthy subjects.
Given that this is a regression dataset, we re-orient the task to learn a deep learning model to evaluate the performance of a subject regardless of being healthy or not.
Instead, the evaluation is done as following:

    
  * if the subject's score is less than 50, the exercise is considered to be badly performed;
    
  * if the subject's score is higher than 50, the exercise is considered to be performed well.

Some examples of the five exercises are presented in Figure <ref>.







Each recorded sequence has a different length, which should be fixed to one common length in order to use deep learning models.
We resampled [https://docs.scipy.org/doc/scipy] all the samples in the data into the average length, which is 748.
The sequences contain 18 human joints with each in a 3D space.
The dataset is made of 71 examples per exercise, where each sequence is a recording of one subject.
For this reason, we can randomly split for each exercise the dataset into a 80%-20% train test sets.
In order to make sure samples of each class exist in the train test split, we make sure to stratify when randomly splitting the train/test sets with the labels (good/bad).







All samples are z-normalized in order to have a zero mean and unit standard variation.
In Table <ref>, we present the accuracy performance for each of the five exercises using the three deep learning models from the literature: FCN, ResNet and InceptionTime, our proposed architecture LITEMVTime, and a baseline classifier 1-Nearest Neighbour-Dynamic Time Warping (1-NN-DTW) <cit.>. The experiments using 1-NN-DTW are conducted using aeon Python pacakge <cit.>.
It can be concluded from the table that LITEMVTime is the best performing model for this task compared to the competitors following both the average performance and average rank over all exercises.
This results show that a small deep learning model such as LITEMVTime can be used to classify if a patient is healthy or not given a recorded sequence of their exercise.









 §.§ Explainability: Class Activation Map

Having a good performing deep learning model is important. However it is important to also be able to understand the decision making process of a deep learning model.
This field has been significantly targeted in the last decade and in the domain of TSC for the last five years.
Class Activation Map (CAM) is an explainability technique for deep CNNs, which helps interpreting the decision of CNNs as if they were black box models.
It was first introduced in <cit.> for image datasets and got first adapted to time series data in <cit.>.
It is important to note that the usage of CAM explainability necessities the usage of a global representative layer before the softmax classification layer.
An example of such layers is the Global Average Pooling (GAP) used in FCN, ResNet, Inception, LITE and LITEMV.
The outcome of a CAM in the case of TSC is a univariate time series.
Each time stamp represents the importance of this time stamp in the input time series that lead to a specific decision making.



For the mathematical setup of CAM, we defined:

    
  * the output of the last convolution layer as O(t), which is a multivariate time series with M variables, M being the number of filters of this layer;
    In other words, O_m(t) is the output univariate time series of filter m, where m ∈{0,1,…,M-1};
    
  * w^c={w_0^c,w_1^c,…,w_M-1^c} as the weights vector connecting the GAP output to the neuron of the winning class (the class with the highest probability value)

The CAM output can be defined as follows:

    CAM(t) = ∑_m=0^M-1 w_m^c.O_m(t)

This is followed by a min-max normalization on the CAM output.
For two given time stamps, the one with the highest CAM score contributed more in the decision making of the black box deep learning model.



In this work, we use the CAM explainability technique to better understand the decision making of the LITEMV model on the human rehabilitation dataset Kimore.
We use five different examples from each exercise, and produce the CAM output using a LITEMV model trained to solve the classification task of each exercise (see Figure <ref>).
Given that in the case of human skeleton based data, the input is a multivariate time series, we suppose that the produced CAM values represent the temporal axis.
In other words, for each time stamp, one CAM score represents the contribution of the current time pose (skeleton) to the classification.







To study more the changes in the CAM scores depending on the state of correctly classifying a sample, we present in Figure <ref> two CAM explanations on two samples of the same exercise.
The first sample is correctly classified as class 1 (score > 50) and the second is incorrectly classified as class 1 as it should be class 0 (score < 50).
It can be seen that the first sample (top) has higher intensity in the CAM colors (so in the CAM scores), than the second sample (bottom).
This indicates that the important time stamps for correctly classifying as class 1 will have lower scores, when the ground truth is actually class 0.
This is due to the fact the CAM scores consider the weights of the winning class in the classification layer and not the weights of the ground truth.









 §.§ Limitations of LITE and LITEMV

LITE and LITEMV have low complexity when compared to other architectures; so, one potential limitation can occur when using  these methods for handling big data. For instance, LITE and LITEMV could fall short when a training set including millions of data is used. 
This shortcoming could be solved by increasing the number of filters operating in the DWSC layers: in fact, given the efficient way DWSC's apply convolutions, this would result in a slight increase in the cost.



A second limitation that affects all the architectures detailed in this work, is related to the length of the time series samples. 
This can be circumvented by increasing the CNN's Receptive Field (RF), which is the length of input visibility of the CNN at the last layer.

Let us consider a CNN with L convolution layers, of kernel size K_i and dilation rate d_i, where i ∈{0,1,…,L-1}.
The Receptive Field <cit.> of this model is computed as:


    RF = 1+∑_d_i*K_i.






This quantity varies from one CNN to another. For instance, in the case of FCN <cit.>, the RF is 14=1+7+4+2.
This RF is small compared to the length of the time series in the UCR archive.
In the case of ResNet, the RF increases to 40, and to 235 for Inception.
In the case of LITE and LITEMV, the RF is 114, which has been shown to be sufficient to achieve state-of-the-art performance on the UCR archive.
However, this RF value needs to be increased if a dataset includes a significantly longer time series compared to this value.
The RF value can be increased by either increasing the length of the filters, or by adding more layers to increase the model's depth. In usual CNN models, however, this is not an efficient approach given the fact it drastically increases the complexity of the network. This does not occurs for LITE and LITEMV, making them suitable for such a solution.




§ CONCLUSIONS

In this paper, we addressed the Time Series Classification problem by reducing the number of parameters compared to existing deep learning approaches for Time Series Classification, while preserving performance of InceptionTime.
We presented a new architecture for Time Series Classification, LITE, and evaluated its performance on the UCR archive. 
LITE has only 2.34% of InceptionTime's number of parameters.
This model is faster than the state-of-the-art in training and inference time. It consumes as well less CO2 and power, which is a topic we believe to be very important nowadays.
Results have illustrated that the usage of LITE allows us to achieve state-of-the-art results on the UCR archive.
Furthermore, the presented ablation study demonstrated the importance of the techniques used in LITE.

Furthermore, we also adapted LITE to handle multivariate time series data, LITEMV.
Experiments on the UEA multivariate TSC archive showed promising performance on some datasets compared to the state-of-the-art.
Finally, we showcased the utility of LITEMV in a real application, where human rehabilitation exercises are evaluated. In this context, we showed that LITEMVTime outperforms other models on the Kimore dataset.

We believe this work can represent a starting point for optimizing deep learning architectures in the time series domain. We believe that this study can address clustering, representation learning and generative models as well. In future work, we aim to tackle these others domains given
the impressive performance of LITE compared to ResNet and InceptionTime.

Acknowledgments
This work was supported by the ANR DELEGATION
project (grant ANR-21-CE23-0014) of the French
Agence Nationale de la Recherche. The authors
would like to acknowledge the High Performance
Computing Center of the University of Strasbourg
for supporting this work by providing scientific sup-
port and access to computing resources. Part of the
computing resources were funded by the Equipex
Equip@Meso project (Programme Investissements
d’Avenir) and the CPER Alsacalcul/Big Data. The
authors would also like to thank the creators and
providers of the UCR, UEA Archives and the Kimore dataset.




§ DECLARATIONS





  * Funding This work was supported by the ANR DELEGATION
project (grant ANR-21-CE23-0014) of the French
Agence Nationale de la Recherche.

  * Conflict of interest The authors certify that they have no conflict of
interest in the subject matter or materials discussed in this manuscript.




  * Availability of data and materials All of the datasets used in this work are publicly available.

  * Code availability The source code is available on this github repository: 

  * Authors' contributions Conceptualization: AIF; Methodology: AIF, MD, SB, JW and GF; Experiments: AIF, MD; Validation: MD, SB, JW and GF; Writing—original draft: AIF; Writing—review and editing: AIF, MD, SB, JW and GF; Funding acquisition: MD; all authors have read and agreed to the published version of the manuscript.




























































