# Exploring Different Time-series-Transformer (TST) Architectures: A Case
Study in Battery Life Prediction for Electric Vehicles (EVs)

Niranjan Sitapure∗
Dept. of Chemical Engineering
Texas A&M University
College Station, TX 77801
<EMAIL_ADDRESS>
&Atharva Kulkarni
Dept. of Mechanical Engineering
Texas A&M University
College Station, TX 77801
<EMAIL_ADDRESS>

###### Abstract

In the past few years, the development of battery technology for application
in electric vehicles (EVs) has received significant attention. Although a
major part of the efforts has been geared towards the development of new
battery materials and chemistries, another challenge faced by EVs is an
accurate prediction of key battery parameters (e.g., state-of-charge (SOC),
temperature, and others), which are essential in the construction of advanced
battery management systems (BMS). Despite the presence of a plethora of
battery models (e.g., equivalent circuit model, single-particle model, etc.),
all the parameters that affect battery parameters cannot be covered by these
models in a computationally tractable manner. Specifically, EV operation
includes various non-battery-related parameters (e.g., ambient temperature,
user-defined cabin temperature, elevation, regenerative braking, etc.) that
affect battery performance. Since the incorporation of these auxiliary
parameters in traditional battery models is difficult, a data-driven approach
is suggested to capture their effect on battery performance. More precisely,
given the emergence of time-series-transformers (TSTs) that harness the power
of multiheaded attention and parallelization-friendly architecture, typical
state-of-the-art (SOTA) TST models (e.g., encoder-only, and vanilla-TST) are
explored and compared against LSTM. Furthermore, novel and unseen TST
architectures (i.e., encoder TST + decoder LSTM, and a hybrid TST-LSTM) are
also developed and compared against the aforementioned models. Also, to
address the challenge of battery life prediction in EVs using these novel TST
architectures, a dataset comprising 72 unique driving trips performed in a BMW
i3 (60 Ah) is considered. Here the objective is to develop accurate TST models
that take in environmental data, battery data, vehicle driving data, and
heating circuit data to predict SOC and battery temperature for future time
steps (i.e., 1 min to 5 mins).

_K_ eywords Time-series-transformers (TST); LSTM; Battery Life Prediction; SOC
Prediction; Electric Vehicles (EVs); Battery Management System (BMS)

##  1 Introduction

In recent years, the focus on advancing battery technology for electric
vehicles (EVs) has intensified, reflecting the industry’s commitment to
enhancing EV performance and sustainability [1, 2, 3]. Extensive research and
development efforts have been dedicated to creating innovative battery
materials and chemistries, aiming to optimize the overall efficiency and
longevity of EV batteries [4, 5, 6]. Despite these advancements, a critical
challenge remains in accurately predicting key battery parameters, such as
state-of-charge (SOC) and temperature, which are pivotal in the design and
implementation of sophisticated battery management systems (BMS) [7, 8, 9].
While various battery models, such as the equivalent circuit model and single-
particle model, have been formulated, they often fall short of providing a
comprehensive representation of all factors that significantly influence
battery performance. [10, 11, 12]. These models struggle to strike the right
balance between computational efficiency and encompassing the multitude of
variables that impact EV battery behavior. Among these factors are non-
battery-related parameters, including ambient temperature, user-defined cabin
temperature, elevation changes, and regenerative braking during EV operation
[13, 14, 15]. These external factors can exert a noticeable influence on
battery performance, but their integration into traditional battery models
proves challenging due to the complexities involved. Thus, given the massive
advent of fast and accurate machine learning (ML) and artificial intelligence
(AI) tools, a data-driven approach toward real-time battery life prediction is
a priority for various EV manufacturers and academics. [16, 17, 18, 19].

Within the field of data-driven prediction of battery performance, there is a
platter of variegated approaches. For instance, Kwon and colleagues have
extensively explored various subspace identification techniques like sparse
identification of system dynamics (SINDy) and operable adaptive sparse
identification of systems (OASIS) [20, 21, 22]. Recently, the authors
showcased the ability of a two-timescale OASIS model to handle intra-cycle and
inter-cycle battery degradation to accurately predict SOC in lithium batteries
[23]. On the other hand, Braatz and colleagues developed an ML approach that
employs a feature-based approach, where linear or nonlinear transformations of
raw data are generated and used within a regularized linear framework called
the elastic net. The model predicts the logarithm of cycle life by using a
linear combination of selected features [18]. More recently, Kim and
colleagues developed a long-short-term-memory (LSTM)-based prediction of
remaining battery life using a multichannel array of battery parameters [24].
That being said, most of the above models, utilize various battery parameters
(e.g., voltage, current, temperature, etc.) to predict SOC. However, during EV
operation various environmental parameters (e.g., ambient temperature,
elevation change, etc.), vehicle driving data (e.g., regenerative braking,
velocity, acceleration, etc), and heating circuit conditions (e.g., cabin
temperature, air vent temperature, etc.) also affect battery SOC. Thus, an
adept ML model that can assign adaptive weights to these factors in addition
to battery parameters will provide an accurate prediction of real-time SOC
during the operation of an EV.

To this end, it is worthwhile to explore the emergence of transformer models
that are used to construct remarkable powerful tools like ChatGPT, Bard,
CodeGPT, and many others [25, 26, 27, 28, 29]. These models have showcased
their capabilities in a wide range of tasks, from generating human-like text
responses to understanding and generating programming code, mainly due to
their utilization of several key mechanisms: multiheaded attention, positional
encoding (PE), and a parallelization-friendly architecture. The attention
mechanism is a fundamental component of transformer models that enables the
models to effectively capture and weigh the relationships between different
parts of the input data [26]. By conducting scaled-dot product calculations
between various input tensors, the attention mechanism selectively assigns
higher attention scores to time-steps that exhibit significant process changes
or specific patterns, allowing the models to focus on relevant information
while processing data [28]. Also, PE plays a crucial role as these models do
not inherently encode the sequential nature of input data (unlike LSTM
models), positional encoding provides a systematic way to incorporate sequence
information [27]. This allows the models to understand the relative positions
of elements in the input sequence, preserving the temporal context and aiding
in the recognition of patterns across time. Finally, the parallelization-
friendly architecture enables the generation of very large transformer models
(i.e., model parameters ($N_{p}\in[1,1000]$ M) that process a humongous
dataset that has been compiled from a plethora of different sources, thereby
enabling a high degree of multidisciplinary transfer learning [30]. Although
the initial development of transformer models was restricted to large-
language-models (LLMs), there have been several recent advances in the
development of time-series-transformers (TSTs) that are applicable to a wide
variety of chemical, mechanical, and electrical engineering applications [31,
32]. Most notably, Kwon and Sitapure showcased CrystalGPT, which is a unified
digital twin for 20+ different sugar crystal systems, and it is approximately
10 times more accurate than other state-of-the-art (SOTA) ML methods [33].
Similarly, Pfister and colleagues have tested multi-horizon temporal fusion
transformers on various real-world datasets including power-consumption
predictions by residents, volatility in the stock market, and consumer
spending in retail stores [34]. Moreover, TSTs also utilize attention
mechanism and leverage process data (e.g., temperature, concentration, and
crystal size in the case of CrystalGPT) or market data (e.g., stock indices
for 50+ volatile stocks and 100+ indicators for big-cap companies, etc.) for
the current and preceding $W$ time-steps (with a window size of $W$) to
predict the features over next $H$ time-steps. This approach facilitates a
contextual understanding of both short-term and long-term changes in process
states, providing valuable insights into the overall process dynamics.

The above demonstrations indicate that TSTs can potentially be a promising
avenue for developing data-driven models for battery life predictions in EVs.
Specifically, typical TST architectures (e.g., encoder-only, and vanilla-TST)
were developed and compared with LSTM models, which are considered to be
incumbents for time-series predictions. Furthermore, novel and unseen hybrid-
TST architectures were also constructed. For instance, a hybrid-TST model with
an encoder part pertaining to a typical TST network is combined with a decoder
comprising of simply multiple layers of an LSTM network is developed.
Similarly, another hybrid-TST model that utilizes the same architecture as a
vanilla-TST, albeit with replacing the feed-forward network (FFN) with an LSTM
layer is developed. The rationale here is that the attention mechanism
empowers the TST model to adaptively weigh different features and learn their
underlying interdependencies while the LSTM is adept at time-series modeling,
and thereby combining these two aspects can potentially outperform existing
TST and LSTM models. To evaluate this hypothesis, we utilize a dataset
comprising 72 unique driving tests performed on a BMW i3 (60 Ah) [35]. This
dataset includes environmental data, battery data, vehicle driving data, and
heating circuit data. The primary objective is to create TST models that can
effectively predict SOC and battery temperature for future time-steps, ranging
from 1 minute to 5 minutes. Overall, the current work addresses two
challenges; (a) the development of fast and accurate TST-based models for
predicting battery performance in EVs, and (b) testing different TST
architectures, and gaining insights into the difference of the predictive
performance showcased by these different TST models.

##  2 Different TST Architectures

###  2.1 Working of Typical Encoder-Decoder Transformers

In general, a vanilla-transformer architecture for natural language processing
(NLP) tasks consists of multiple encoder/decoder blocks with identical sub-
layers and a globally pooled output layer [27, 32]. The input data undergoes
four transformations within an NLP-transformer. First, text inputs are
preprocessed through truncation or padding and tokenization [26]. Second,
transformer networks use positional encoding (PE) to handle sequential data
without recurrence. Third, a stack of encoder blocks employs multiheaded self-
attention to calculate contextualized embeddings. Fourth, a stack of decoder
blocks uses multiheaded cross-attention to generate human-like text by
focusing on high-value cross-attention scores [32]. This framework is known as
the teacher-forcing method in training transformer models.

Figure 1: Schematic illustration of architecture for (a) v-TST, and (b) hybrid
TST-LSTM models.

Similar to NLP-transformers, TSTs utilize multiple encoder-decoder blocks, a
multiheaded attention mechanism, and PE to process time-series dynamics
comprising of state information. Specifically, TST takes in an input tensor
(i.e., $[X_{t-W},X_{t-W+1}...X_{t}]$) and yields an output tensor with a
prediction horizon of $H$ and dimension $v$ (i.e.,
$[y_{t+1},y_{t+2},...y_{t+H}]$). After using PE, the input tensor is processed
through a scaled-dot product query-key-value ([Q,K,V]) approach to compute
attention scores as shown below [27, 31]:

$\displaystyle A_{P,n}=\sum_{i}^{k}\lambda_{n,i}\textbf{V}$ (1)
$\displaystyle\lambda_{n,i}=\frac{exp\left(\textbf{Q}^{T}\textbf{K}_{i}/\sqrt{D_{k}}\right)}{\sum_{j=1}^{k}exp\left(\textbf{Q}^{T}\textbf{K}_{j}/\sqrt{D_{k}}\right)}$
$\displaystyle\sum_{i=1}^{k}\lambda_{n,i}=1$

where $A_{P,n}$ is the attention value for head $n$ in encoder block $P$,
Q$\in\mathbb{R}^{D_{k}}$ are queries, K$\in\mathbb{R}^{D_{k}}$ stands for
keys, and V$\in\mathbb{R}^{D_{v}}$ are values. Here, $D_{k}$ and $D_{v}$ are
the dimensions of keys and values, respectively. Next, all the attention
scores are processed by an FFN that captures the nonlinearity between inputs
and outputs. This process is repeated in each encoder and decoder block.
Previous literature works to provide a detailed explanation of the intricate
working of TST [32, 36, 31].

###  2.2 Constructing Different TST Architectures

As mentioned earlier, different TST architectures have been tested in this
work. First, a straightforward encoder-only TST (enc-TST) is constructed as
shown in Figure 2a. This model comprises only encoder blocks with the self-
attention mechanism that are stacked on top of each other, which is followed
by a final linear layer to predict the model outputs. Second, a vanilla-TST
(v-TST) is constructed as shown in Figure 1a. In a v-TST, $N$ encoder blocks
are stacked on each other which utilized a self-attention mechanism, and this
is followed by $N$ decoder blocks that utilized a cross-attention mechanism.
The entire encoder-decoder apparatus is connected to a final linear layer that
predicts the system outputs. The above two models represent typical TST
architectures that have gained attention in the past 2 years and have been
demonstrated for a plethora of different applications [31]. As explained
before, these models leverage the combination of attention mechanism and
positional encoding to provide high predictive performance [33].

Figure 2: Schematic illustration of architecture for (a) enc-TST, and (b) enc-
TST-dec-LSTM models.

Modifications to the above-mentioned traditional TST architectures can be
envisioned that can potentially provide better predictive performance. For
instance, LSTM networks that utilize explicit sequential-based model
architecture are considered to be the gold standard for time-series
predictions. Thus, if the LSTM network can be integrated with already
impressive TST models, then potentially better predictive performance can be
yielded from these new architectures. To this end, a hybrid TST-LSTM
architecture is developed as a modification to the traditional v-TST as shown
in Figure 1b. Basically, the FFN in the v-TST comprises a linear network of
neurons. However, in a hybrid TST-LSTM model, the FFN is replaced by LSTM
layers. The rationale here is that the LSTM network, which utilizes an
explicit sequential approach making it well-suited for time-series modeling,
can augment the predictive capabilities of the v-TST. Another approach that
follows the same rationale is developing an encoder-TST and decoder-LSTM (enc-
TST-dec-LSTM) model. In this architecture, the entire set of decoder blocks is
substituted with subsequently stacked LSTM layers as shown in Figure 2b. The
rationale here is that the encoder blocks will utilize a self-attention
mechanism to adeptly understand the interdependencies between system states,
and then feed it to the LSTM-based decoder blocks that can track the temporal
evolution of system states. Finally, considering the aforementioned TST
architectures, four TST models were constructed, trained, and tested against a
traditional LSTM model, which is considered the current SOTA model for time-
series modeling. The architectural details for these models are mentioned in
Table 1. Although the model sizes (i.e., number of parameters) might vary
between the models, the hyperparameters are kept as similar as possible (e.g.,
number of encoder-decoder blocks, number of attention heads, and others).
Also, it is noteworthy to understand that having a large number of model
parameters can be useful for capturing more subtle interdependencies between
system states but also may require a large amount of training data to ensure
appropriate estimation of all model parameters during model training. Thus,
different TST models selected in this work also help us provide insight into
the above aspect related to the number of model parameters.

| enc-TST | enc-TST-dec-LSTM | v-TST | Hybrid TST-LSTM | LSTM
---|---|---|---|---|---
# Encoders | 4 | 4 | 4 | 4 | -
# Decoders | 4 | 4 | 4 | 4 | -
# Attention Heads | 8 | 8 | 8 | 8 | -
Inner Dimensions ($d_{model}$) | 128 | 128 | 128 | 128 | 128
# Neurons in FFN | 128 | 128 | 128 | 128 | -
# Layers | - | - | - | - | 4
# Parameters | 400K | 950K | 1M | 3M | 475K
Table 1: Architectural details for different TST models and the reference LSTM
model.

###  2.3 Data Preprocessing

As mentioned before, we utilized a dataset comprising 72 unique driving tests
performed on a BMW i3 (60 Ah), which includes information about the
environment, battery metrics, vehicle driving parameters, and heating circuit
data every 0.1 seconds. Although the original dataset has information from 45+
different sensors, only relevant 15 features were selected. For example, the
original dataset has sensor data for [Temperature Vent right, Temperature Vent
central right, Temperature Vent central left, Temperature Vent right] that was
averaged to create a new variable [Average Vent Temperature], which follows
very similar time-series dynamics. For instance, all the vent sensors show a
maximum deviation of 0.01%, thereby suggesting that the inclusion of all the
vent sensor data will not provide any new information to the ML models. This
process was repeated for several different temperature sensors to reduce the
number of redundant model features while retaining the information from the
original features.

Figure 3: Temporal evolution of driving characteristics for an arbitrarily
selected trip performed using the BMW i3. Figure 4: Temporal evolution of
battery characteristics for an arbitrarily selected trip performed using the
BMW i3.

Furthermore, since all the sensor data is sampled at 0.1 seconds, there is
considerable process noise, which can be detrimental to TST models. This is
because TST models utilize attention mechanism to find interdependencies and
trends in system data, and the inclusion of noisy data does not allow them to
efficiently find patterns in the input data, which can lead to poor predictive
performance. Thus, a Savitzky-Golay (Savgol) filter is utilized to denoise the
sensor data for providing smoother inputs to the TST models without losing key
process dynamics. Basically, The Savgol filter is a digital signal processing
technique used to denoise process data by effectively removing noise while
preserving important underlying trends and features [37]. It works by fitting
a polynomial to local data points within a user-defined window, allowing
engineers and analysts to identify and study the underlying patterns in the
data [38]. The filter reduces random fluctuations, maintains data integrity,
and handles irregular sampling, making it suitable for various process data
denoising needs. Its tunable parameters and computational efficiency make it
an attractive option for denoising large datasets commonly encountered in
process monitoring and control, offering a valuable tool to analyze and
interpret process behavior and trends.

Figure 5: Temporal evolution of sensor data related to the heating circuit for
an arbitrarily selected trip performed using the BMW i3.

The resulting features can then be classified into three categories (i.e.,
driving, battery, and heating characteristics). For instance, Figures 3-5
depict the temporal evolution of various system states for the above-mentioned
three categories after the sensor data were denoised using a Savgol filter of
order 2. Data for such 40+ trips were compiled into a large dataset with 4000
datapoints for training, 1000 datapoints for validation, and model testing.
This dataset was shuffled to avoid any sequential bias and was stored as a
PyTorch dataset in a pickle file. Further, each of the above models (Table 1)
was trained and tested using the aforementioned dataset to ensure a fair
comparison of their predictive abilities. Basically, various sensor data were
considered as model inputs for current and past $W$ time-steps, and the
battery SOC and temperature were considered as outputs for $H$ time-steps in
the future.

|  | LSTM | enc-TST | v-TST | TST-LSTM | enc-TST-dec-LSTM
---|---|---|---|---|---|---
| # Parameters | 475K | 400K | 1M | 3M | 950K
| Training | 20 | 32 | 11 | 18 | 51
Validation | 21 | 33 | 17 | 19 | 68
$W=12$ and $H=6$ | Test | 22 | 34 | 17 | 24 | 72
|  $R^{2}$ (Test) | 0.978 | 0.966 | 0.988 | 0.981 | 0.931
| Training | 17 | 20 | 15 | 16 | 42
Validation | 25 | 24 | 16 | 20 | 44
$W=30$ and $H=6$ | Test | 26 | 33 | 18 | 22 | 53
|  $R^{2}$ (Test) | 0.973 | 0.979 | 0.98 | 0.976 | 0.956
| Training | 15 | 19 | 9 | 17 | 25
Validation | 17 | 24 | 10 | 18 | 35
$W=50$ and $H=30$ | Test | 22 | 26 | 11 | 23 | 39
|  $R^{2}$ (Test) | 0.983 | 0.974 | 0.988 | 0.981 | 0.963
Table 2: Compilation of training, validation, and testing results for
different TST models and their comparison with LSTM model. The values in bold
indicate the best-performing model for that case.

##  3 Results and Discussion

###  3.1 Comparison of Different TST Architectures

Training, validation, and testing performance using the above dataset and the
various TST architectures are tabulated in Table 2. It can be seen that across
all the different cases for different window sizes ($W$) and prediction
horizons ($H$), the v-TST model outperforms all models consistently.
Surprisingly, the traditional LSTM model comes in a close second and shows
relatively good performance as compared to the rest of the TST models except
v-TST. Meanwhile, the encoder-only TST model (enc-TST) shows adequate
predictive performance. Interestingly, the rationale for developing hybrid
TST-LSTM models (i.e., TST-LSTM, and enc-TST-dec-LSTM) does not show promise
as indicated by the least accurate model (enc-TST-dec-LSTM) and an adequately
good model (TST-LSTM). Moreover, the following general trend can be observed
amongst the different TST models:

$\text{v-TST}>\text{LSTM}>\text{TST-LSTM}>\text{enc-TST}>\text{enc-TST-dec-
LSTM}$ (2)

It is worthwhile to suggest the possible reasons for the above trend in a
sequential manner. Firstly, it is important to understand the reason v-TST
shows great predictive capabilities. In a v-TST with $n$ attention heads, each
attention head compute attention scores for each encoder and decoder block,
which are then combined using the weights $c_{j}$ and processed through an FFN
to approximate the input/output relationship. The depth and width of the v-TST
enable it to capture complex representations of the input/output relationship
[33]. Next, the parallelized nature of multiple attention heads in the v-TST
allows it to break down a unified mapping function between inputs and outputs
into multiple subspaces. Each attention head attends to different
interdependencies among various system states when trained with input data for
multiple systems. This process results in not only individual constituents but
also shared subspace models between different system states. Basically, this
distributed internal framework is essential for maintaining high accuracy when
applying the v-TST model to a new prediction task. In this case, v-TST learns
from sensor data for 30+ different driving trips by a BMW i3. When v-TST is
tested for a new trip, it finds similarities between the current trip and
previously learned trips and seamlessly interpolates to accurately predict the
battery performance (i.e., SOC and battery temperature) [39]. Secondly, the
enc-TST shows relatively poor performance due to its limitations in capturing
the full input-output relationship. Basically, enc-TST lacks the decoder
component responsible for generating the output sequence, leading to the loss
of information and reduced adaptability for sequence-to-sequence mapping.
Thus, in enc-TST, only self-attention amongst the input tensor ($X_{enc}$) is
utilized, and its connectivity to the output tensor ($X_{dec}$) is missing. In
contrast, a v-TST allows bidirectional information flow and utilizes the
‘cross-attention mechanism’ to focus on relevant parts of the input sequence
($X_{enc}$) and connect it with the output sequence ($X_{dec}$) during model
training. A detailed explanation of the working of TSTs is available in the
literature [33, 31, 34]

Thirdly, it is interesting to ponder on the reasons for the abysmal
performance of the enc-TST-dec-LSTM model. The lack of predictive capabilities
can be attributed to some key factors. For instance, the hybrid architecture
of the enc-TST-dec-LSTM model introduces additional complexity and mismatch in
the symmetry of the architecture, which can hinder the model’s ability to
generalize well to new data. Further, the combination of encoder-TST and
decoder-LSTM may not facilitate smooth information flow throughout the model.
More importantly, there is no ‘cross-attention’ module in enc-TST-dec-LSTM
which disables the model from learning the continuity between input
($X_{enc}$) and output ($X_{dec}$) tensors during model training, which is an
essential aspect of training transformer models. That being said, the reason
for the adequate performance of the TST-LSTM model, wherein the FFN in a v-TST
is replaced by LSTM layers, is a bit more interesting. In addition to the
architectural mismatch, there can be three key reasons for an inferior
performance. Basically, the attention mechanism in TSTs is a powerful feature
that enables the model to focus on relevant parts of the input sequence while
generating the output sequence. Replacing the FFN with LSTM layers might lead
to the loss of this crucial attention mechanism, affecting the model’s ability
to handle complex sequences effectively. For instance, the encoder/decoder
part of TST utilizes PE to time-stamp the features in the input tensor using
sinusoidal transformations, which are then processed via the attention
mechanism. However, the LSTM is adept at handling time series in an explicit
manner, which is lost during the PE process, thereby negating the benefit of
incorporating an LSTM layer instead of FFN. Also, even though there is a
‘cross-attention’ between encoder and decoder blocks (similar to v-TST), the
discontinuity in the attention mechanism during the FFN module may lead to
poor training. Next, replacing the FFN with LSTM layers introduces a higher
number of parameters for the same amount of training data as evident in Table
1. This increased parameter count can lead to overfitting, especially if the
dataset is limited, and the model might struggle to generalize well on unseen
data. Furthermore, the hybrid model nature of TST-LSTM architecture might be
more challenging to train and tune due to the combination of two different
architectures.

Lastly, we explore why LSTM can perform better than enc-TST and other hybrid
TST-LSTM models. LSTMs are designed specifically for sequential data, and they
have a built-in ability to process sequences in a step-by-step manner,
retaining information from previous time steps through hidden states. LSTMs
utilize recurrent connections that allow them to remember past information and
update their hidden states accordingly [40]. Also, LSTMs recurrent connections
help in leveraging the limited data more effectively for learning patterns and
trends. In simpler terms, the LSTM network has a lesser number of model
parameters for the same amount of training data. Furthermore, the symmetry of
a standalone LSTM network as compared to the hybrid TST-LSTM networks might
allow for more effective training. That being said, these characteristics are
not enough for the LSTM model to supersede the performance of v-TST. As
mentioned before, v-TST can break down a unified mapping function between
inputs and outputs into more adaptable subspaces that can be dynamically
weighted to make better time-series predictions. On the contrary, LSTM
networks attempt to find a single unified function between inputs and outputs,
thereby being prone to slight overfitting and poor generalizability [33].

###  3.2 Effect of Window Size ($W$)

The other aspect in Table 2 is the performance difference for changing $W$. It
is evident that model performance for all the TST models and LSTM model
increases for larger $W$. This is not a surprising observation and has been
shown to be true across previous TST models [41]. Essentially, larger $W$
allows the model to capture longer contextual dependencies, enabling a better
understanding of the broader relationships between distant tokens. This
enhanced modeling of long sequences reduces positional bias and improves the
attention mechanism’s ability to capture long-range dependencies. The
increased capacity provided by larger windows allows for more comprehensive
feature representation, leading to improved performance. Moreover, a larger
window size also enables more continuity between the input and output tensors
(i.e., $X_{enc}$ and $X_{dec}$) during model training, thereby allowing the
TST model to better understand the interdependencies between system states.

Figure 6: Temporal evolution of key battery metrics (i.e., SOC and
temperature) as predicted by different TST models for $W=12$. Figure 7:
Temporal evolution of key battery metrics (i.e., SOC and temperature) as
predicted by different TST models for $W=30$

This is evident from Figures 6 and 7, which showcase the model predictions for
key battery parameters (i.e., SOC and temperature) for an arbitrarily selected
trip performed in BMW i3. First, it is evident that the v-TST model shows the
highest $R^{2}$ values among all the models for both cases. Also, it can be
seen that the temporal predictions by enc-TST-dec-LSTM models have a large
oscillating and lack accuracy as indicated in Table 2. Further, these
oscillations can be attributed to the highly asymmetrical and complex
architecture of enc-TST-dec-LSTM as explained in the previous section. Lastly,
the LSTM model performs adequately well as seen in Table 2, and although it
shows some oscillating behavior for $W=12$, the oscillations are minimized for
$W=30$. Again, this occurs due to the fact that for higher $W$, all the models
have access to longer input sequences allowing them to better gauge the
trajectory of different system states.

###### Remark 1

In Figures 6 and 7, only three models are depicted to avoid overcrowding of
results. Moreover, v-TST is picked as it shows the least testing error in
Table 2. While enc-TST-dec-LSTM is chosen as it shows the highest testing
errors. And the LSTM model is chosen as it is a baseline model that is the
current SOTA for time-series predictions.

###  3.3 Discussion

The above sections demonstrated the capability of different TST models for
achieving high predictive performance in complex time-series modeling tasks.
Building on the above models and some of the ones developed in the literature
[33, 31], TST-based time-series modeling can be applied to a plethora of
different applications. For example, another approach that can hugely benefit
from the incorporation of various TST models is the hybrid modeling of various
chemical, mechanical, and electrical systems. Basically, Hybrid models that
combine system-agnostic first principles with system-specific data-driven
parameters have been demonstrated for fermentation, fracking, and other
chemical processes [42, 43, 44, 45]. For instance, in industrial fermentation,
physics-based reactor models inclusive of mass and heat transfer can be
integrated with a system-specific deep neural network (DNN) that predicts the
fermentation kinetics to simulate the process. That said, DNNs can exhibit
limitations in accurately predicting complex time-varying parameters commonly
found in intricate chemical processes. Moreover, DNNs lack the capability to
leverage contextual information regarding the trajectory of process dynamics.
Empowered by multiheaded attention mechanisms and PE, TST models can
potentially perform better for hybrid modeling of intricate chemical and
mechanical engineering applications.

Another modeling paradigm wherein attention-based TST models can be utilized
is in multiscale models, which integrate physical laws at different time and
length scales to uncover mechanisms governing observable phenomena. For
instance, a considerable number of multiscale models have been developed in
the past for particulate systems, thin-film depositions, pulping processes,
biochemical processes, and others. [46, 47, 48, 49, 50, 51, 52, 53]. In these
cases, there are a number of process parameters (e.g., kinetic constants,
growth rates, product characteristics, etc.) that connect the different time
and length scales, and it can often be non-trivial to incorporate all these
interactions in a high-fidelity multiscale model. In such cases, TST models
can utilize an attention mechanism to emphasize the most dominating
interacting parameters to reduce model complexity. On the other hand, similar
attention-based TST models can be utilized in conjunction with a multiscale
model to reduce the plant-model mismatch in some of these models. For example,
an existing multiscale model might consider 1-2 interacting parameters (out of
10), thereby resulting in certain deviations from experimental/pilot/process
data. Thus, this multiscale model can be coupled with a TST that utilizes
available process data to correct these deviations leading to more accurate
yet computationally non-prohibitive multiscale models.

##  4 Conclusions

While significant efforts have been devoted to developing new battery
materials and chemistries, accurately predicting crucial battery parameters
(e.g., SOC and temperature), remains a challenge. Thus to explore SOTA data-
driven approaches for accurate prediction of battery characteristics, in this,
work, various TST architectures have been explored. Specifically, four
different TST models (i.e., enc-TST, v-TST, TST-LSTM, and enc-TST-dec-LSTM)
were trained and tested against a traditional LSTM model. A practical dataset
comprising 72 distinct driving trips performed in a BMW i3 (60 Ah), which
records sensor data for environmental conditions, driving, and battery
characteristics is utilized to predict SOC and battery temperature for $H$
future time-steps. The testing results show an interesting trend in model
performance (i.e., v-TST > LSTM > TST-LSTM > enc-TST > enc-TST-dec-LSTM). This
behavior is attributed to the fact that hybrid TST architectures like TST-LSTM
and enc-TST-dec-LSTM result in asymmetrical models that also lose critical
information from attention scores due to the incorporation of distinctly
different LSTM layers. That being said, all the models show better performance
(i.e., lower errors) for increasing $W$, as longer input sequences provide
more contextual information leading to better model training. Overall, the
present work showcases an interesting phenomenon, wherein more sophisticated
and complex hybrid TST-LSTM architectures underperform as compared to their
traditional TST and LSTM counterparts. Further, the above models demonstrate
the utilization of TST models to enhance the accuracy of battery
characteristics predictions, contributing to the advancement of battery
management systems in electric vehicles.

## References

  * [1] Aiping Wang, Sanket Kadam, Hong Li, Siqi Shi, and Yue Qi.  Review on modeling of the anode solid electrolyte interphase (SEI) for lithium-ion batteries.  npj Computational Materials, 4(1):1–26, March 2018.  Number: 1 Publisher: Nature Publishing Group.
  * [2] Naoki Nitta, Feixiang Wu, Jung Tae Lee, and Gleb Yushin.  Li-ion battery materials: present and future.  Materials Today, 18(5):252–264, 2015.
  * [3] Björn Nykvist and Måns Nilsson.  Rapidly falling costs of battery packs for electric vehicles.  Nature Climate Change, 5(4):329–332, 2015.
  * [4] Mihit H Parekh, Suyash Oka, Jodie Lutkenhaus, and Vilas G Pol.  Critical-point-dried, porous, and safer aramid nanofiber separator for high-performance durable lithium-ion batteries.  ACS Applied Materials & Interfaces, 14(25):29176–29187, 2022.
  * [5] Arumugam Manthiram.  A reflection on lithium-ion battery cathode chemistry.  Nature communications, 11(1):1550, 2020.
  * [6] Yanyan Zhao, Oliver Pohl, Anand I Bhatt, Gavin E Collis, Peter J Mahon, Thomas Rüther, and Anthony F Hollenkamp.  A review on battery market trends, second-life reuse, and recycling.  Sustainable Chemistry, 2(1):167–205, 2021.
  * [7] Niranjan Sitapure, Hyeonggeon Lee, Francisco Ospina-Acevedo, Perla B Balbuena, Sungwon Hwang, and Joseph S Kwon.  A computational approach to characterize formation of a passivation layer in lithium metal anodes.  AIChE Journal, e17073, 2020.
  * [8] Hyeonggeon Lee, Niranjan Sitapure, Sungwon Hwang, and Joseph Sang-Il Kwon.  Multiscale modeling of dendrite formation in lithium-ion batteries.  Computers & Chemical Engineering, page 107415, 2021.
  * [9] Andrea Pozzi, Marcello Torchio, Richard D Braatz, and Davide M Raimondo.  Optimal charging of an electric vehicle battery pack: A real-time sensitivity-based model predictive control approach.  Journal of Power Sources, 461:228133, 2020.
  * [10] Gyuyeong Hwang, Niranjan Sitapure, Jiyoung Moon, Hyeonggeon Lee, Sungwon Hwang, and Joseph Sang Kwon.  Model predictive control of lithium-ion batteries: Development of optimal charging profile for reduced intracycle capacity fade using an enhanced single particle model (SPM) with first-principled chemical/mechanical degradation mechanisms.  Chemical Engineering Journal, 435:134768, 2022.
  * [11] Marcello Torchio, Nicolas A. Wolff, Davide M. Raimondo, Lalo Magni, Ulrike Krewer, R. Bushan Gopaluni, Joel A. Paulson, and Richard D. Braatz.  Real-time model predictive control for the optimal charging of a lithium-ion battery.  In 2015 American Control Conference (ACC), pages 4536–4541, Chicago, IL, USA, July 2015. IEEE.
  * [12] Bharatkumar Suthar, Venkatasailanathan Ramadesigan, Paul W. C. Northrop, Bhushan Gopaluni, Shriram Santhanagopalan, Richard D. Braatz, and Venkat R. Subramanian.  Optimal control and state estimation of lithium-ion batteries using reformulated models.  In 2013 American Control Conference, pages 5350–5355, June 2013.  ISSN: 2378-5861.
  * [13] AR Dehghani-Sanij, E Tharumalingam, MB Dusseault, and R Fraser.  Study of energy storage systems and environmental challenges of batteries.  Renewable and Sustainable Energy Reviews, 104:192–208, 2019.
  * [14] Peter Keil and Andreas Jossen.  Aging of lithium-ion batteries in electric vehicles: Impact of regenerative braking.  World Electric Vehicle Journal, 7(1):41–51, 2015.
  * [15] Matthias Steinstraeter, Marcel Lewke, Johannes Buberger, Tobias Hentrich, and Markus Lienkamp.  Range extension via electrothermal recuperation.  World Electric Vehicle Journal, 11(2):41, 2020.
  * [16] Zhiyong Tian, Lai Tu, Chen Tian, Yi Wang, and Fan Zhang.  Understanding battery degradation phenomenon in real-life electric vehicle use based on big data.  In 2017 3rd International Conference on Big Data Computing and Communications (BIGCOM), pages 334–339. IEEE, 2017.
  * [17] Chade Lv, Xin Zhou, Lixiang Zhong, Chunshuang Yan, Madhavi Srinivasan, Zhi Wei Seh, Chuntai Liu, Hongge Pan, Shuzhou Li, and Yonggang Wen.  Machine learning: an advanced platform for materials development and state prediction in lithium-ion batteries.  Advanced Materials, 34(25):2101474, 2022.
  * [18] Kristen A Severson, Peter M Attia, Norman Jin, Nicholas Perkins, Benben Jiang, Zi Yang, Michael H Chen, Muratahan Aykol, Patrick K Herring, Dimitrios Fraggedakis, et al.  Data-driven prediction of battery cycle life before capacity degradation.  Nature Energy, 4(5):383–391, 2019.
  * [19] Man-Fai Ng, Jin Zhao, Qingyu Yan, Gareth J Conduit, and Zhi Wei Seh.  Predicting the state of charge and health of batteries using data-driven machine learning.  Nature Machine Intelligence, 2(3):161–170, 2020.
  * [20] Bhavana Bhadriraju, Abhinav Narasingam, and Joseph S Kwon.  Machine learning-based adaptive model identification of systems: Application to a chemical process.  Chemical Engineering Research and Design, 152:372–383, 2019.
  * [21] Bhavana Bhadriraju, Abhinav Narasingam, and Joseph Sang Kwon.  Machine learning-based adaptive model identification of systems: application to a chemical process.  Chemical Engineering Research and Design, 152:372–383, 2019.
  * [22] Bhavana Bhadriraju, Joseph Sang Kwon, and Faisal Khan.  OASIS-P: Operable adaptive sparse identification of systems for fault prognosis of chemical processes.  Journal of Process Control, 107:114–126, 2021.
  * [23] Bhavana Bhadriraju, Joseph Sang-Il Kwon, and Faisal Khan.  An adaptive data-driven approach for two-timescale dynamics prediction and remaining useful life estimation of li-ion batteries.  Computers & Chemical Engineering, page 108275, 2023.
  * [24] Kyungnam Park, Yohwan Choi, Won Jae Choi, Hee-Yeon Ryu, and Hongseok Kim.  Lstm-based battery remaining useful life prediction with multi-channel charging profiles.  IEEE Access, 8:20786–20798, 2020.
  * [25] OpenAI.  GPT-4 technical report.  2303.08774, 2023.
  * [26] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint, 1810.04805, 2018.
  * [27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems, 30, 2017.
  * [28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [29] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
  * [30] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-LM: Training multi-billion parameter language models using model parallelism.  arXiv preprint, 1909.08053, 2019.
  * [31] Niranjan Sitapure and Joseph Sang-Il Kwon.  Exploring the potential of time-series transformers for process modeling and control in chemical systems: an inevitable paradigm shift?  Chemical Engineering Research and Design, 194:461–477, 2023.
  * [32] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun.  Transformers in time series: A survey.  arXiv preprint, 2202.07125, 2022.
  * [33] Niranjan Sitapure and Joseph Sang Kwon.  CrystalGPT: Enhancing system-to-system transferability in crystallization prediction and control using time-series-transformers.  Computers & Chemical Engineering, 177:108339, 2023.
  * [34] Bryan Lim, Sercan Ö Arık, Nicolas Loeff, and Tomas Pfister.  Temporal fusion transformers for interpretable multi-horizon time series forecasting.  International Journal of Forecasting, 37(4):1748–1764, 2021.
  * [35] Matthias Steinstraeter, Johannes Buberger, and Dimitar Trifonov.  Battery and heating data in real driving cycles.  2020\.
  * [36] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu.  Are transformers effective for time series forecasting?  arXiv preprint, 2205:13504, 2022.
  * [37] Neal B Gallagher.  Savitzky-golay smoothing and differentiation filter.  Eigenvector Research Incorporated, 2020.
  * [38] Md Shahinoor Rahman, Liping Di, Ranjay Shrestha, G Yu Eugene, Li Lin, Lingjun Kang, and Meixia Deng.  Comparison of selected noise reduction techniques for modis daily ndvi: An empirical analysis on corn and soybean.  In 2016 Fifth International Conference on Agro-Geoinformatics (Agro-Geoinformatics), pages 1–5. IEEE, 2016.
  * [39] Daoquan Chen, Weicong Hong, and Xiuze Zhou.  Transformer network for remaining useful life prediction of lithium-ion batteries.  IEEE Access, 10:19621–19628, 2022.
  * [40] Niranjan Sitapure and Joseph S Kwon.  Require process control? LSTMc is all you need!  arXiv preprint, 2306:07510, 2023.
  * [41] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive language models beyond a fixed-length context.  arXiv preprint, 1901.02860, 2019.
  * [42] Mohammed Saad Faizan Bangi and Joseph S Kwon.  Deep hybrid modeling of chemical process: application to hydraulic fracturing.  Computers & Chemical Engineering, 134:106696, 2020.
  * [43] Mohammed Saad Faizan Bangi, Katy Kao, and Joseph Sang Kwon.  Physics-informed neural networks for hybrid modeling of lab-scale batch fermentation for $\beta$-carotene production using saccharomyces cerevisiae.  Chemical Engineering Research and Design, 179:415–423, 2022.
  * [44] Parth Shah, M Ziyan Sheriff, Mohammed Saad Faizan Bangi, Costas Kravaris, Joseph Sang Kwon, Chiranjivi Botre, and Junichi Hirota.  Deep neural network-based hybrid modeling and experimental validation for an industry-scale fermentation process: Identification of time-varying dependencies among parameters.  Chemical Engineering Journal, 441:135643, 2022.
  * [45] Dongheon Lee, Arul Jayaraman, and Joseph S Kwon.  Development of a hybrid model for a partially known intracellular signaling pathway through correction term estimation and neural network modeling.  PLoS Computational Biology, 16(12):e1008472, 2020.
  * [46] Niranjan Sitapure, Robert Epps, Milad Abolhasani, and Joseph S Kwon.  Multiscale modeling and optimal operation of millifluidic synthesis of perovskite quantum dots: towards size-controlled continuous manufacturing.  Chemical Engineering Journal, page 127905, 2020.
  * [47] Niranjan Sitapure, Robert W Epps, Milad Abolhasani, and Joseph S Kwon.  CFD-based computational studies of quantum dot size control in slug flow crystallizers: Handling slug-to-slug variation.  Industrial & Engineering Chemistry Research, 60(13):4930–4941, 2021.
  * [48] Niranjan Sitapure and Joseph Sang-Il Kwon.  Neural network-based model predictive control for thin-film chemical deposition of quantum dots using data from a multiscale simulation.  Chemical Engineering Research and Design, 183:595–607, 2022.
  * [49] Joseph S Kwon, Michael Nayhouse, Panagiotis D Christofides, and Gerassimos Orkoulas.  Modeling and control of shape distribution of protein crystal aggregates.  Chemical Engineering Science, 104:484–497, 2013.
  * [50] Joseph S Kwon, Michael Nayhouse, Gerassimos Orkoulas, and Panagiotis D Christofides.  Crystal shape and size control using a plug flow crystallization configuration.  Chemical Engineering Science, 119:30–39, 2014.
  * [51] Joseph S Kwon, Michael Nayhouse, Gerassimos Orkoulas, and Panagiotis D Christofides.  Enhancing the crystal production rate and reducing polydispersity in continuous protein crystallization.  Industrial & Engineering Chemistry Research, 53(40):15538–15548, 2014.
  * [52] Hyun-Kyu Choi and Joseph Sang Kwon.  Modeling and control of cell wall thickness in batch delignification.  Computers & Chemical Engineering, 128:512–523, 2019.
  * [53] Hyun-Kyu Choi and Joseph S Kwon.  Multiscale modeling and predictive control of cellulose accessibility in alkaline pretreatment for enhanced glucose yield.  Fuel, 280:118546, 2020.