# Tora: Trajectory-oriented Diffusion Transformer for Video Generation

Zhenghao Zhang1, Junchao Liao1, Menghao Li1, Long Qin1, Weizhi Wang1

###### Abstract

Recent advancements in Diffusion Transformer (DiT) have demonstrated
remarkable proficiency in producing high-quality video content. Nonetheless,
the potential of transformer-based diffusion models for effectively generating
videos with controllable motion remains an area of limited exploration. This
paper introduces Tora, the first trajectory-oriented DiT framework that
integrates textual, visual, and trajectory conditions concurrently for video
generation. Specifically, Tora consists of a Trajectory Extractor (TE), a
Spatial-Temporal DiT, and a Motion-guidance Fuser (MGF). The TE encodes
arbitrary trajectories into hierarchical spacetime motion patches with a 3D
video compression network. The MGF integrates the motion patches into the DiT
blocks to generate consistent videos following trajectories. Our design aligns
seamlessly with DiT’s scalability, allowing precise control of video content’s
dynamics with diverse durations, aspect ratios, and resolutions. Extensive
experiments demonstrate Tora’s excellence in achieving high motion fidelity,
while also meticulously simulating the movement of the physical world.

Figure 1:  More generated samples. Tora accommodates a wide array of visual
conditions, encompassing both single starting frames and combinations of
initial and final frames (as seen in the sixth row), and expertly handles
multiple trajectories to precisely manipulate multiple objects. It adeptly
facilitates video generation across diverse aspect ratios, resolutions, and
durations, thereby ensuring flexible and adaptive content creation. Check out
our project page for all video demos.

## Introduction

Diffusion models (Dhariwal and Nichol 2021; Ramesh et al. 2022) have
demonstrated their capability to generate diverse and high-quality images or
videos. Previously, video diffusion models (Ho et al. 2022b; Blattmann et al.
2023; Zhang et al. 2023a) predominantly employed U-Net architectures (Olaf
Ronneberger 2015), focusing primarily on synthesizing videos of limited
duration, typically around two seconds, and were constrained to fixed
resolutions and aspect ratios. Recently, Sora (Brooks et al. 2024), a text-to-
video generation model leveraging Diffusion Transformer (DiT) (Peebles and Xie
2023), has showcased video generation capabilities that significantly outstrip
current state-of-the-art methods. Sora excels not only in the production of
high-quality videos ranging from 10 to 60 seconds, but also distinguishes
itself through its capacity to handle diverse resolutions, various aspect
ratios, adherence to the laws of actual physics.

Video generation entails the creation of consistent motion within a sequence
of generated images, thereby emphasizing the importance of motion control.
Prior research endeavors, such as VideoComposer (Wang et al. 2023a) and
DragNUWA (Yin et al. 2023), have successfully implemented generalized motion
manipulation techniques. These methods typically involve extracting motion
vectors and optical flow data to direct the video generation models. Building
upon these advancements, MotionCtrl (Wang et al. 2023b) innovates by
separately managing camera movements and object motions, thereby expanding the
achievable motion ranges and patterns. Despite promising controllable motion
quality, these methods are limited to generating videos of up to 16 frames in
length and at a lower, fixed resolution. Moreover, limitations in the number
of frames can hinder the seamless portrayal of motion, particularly in
scenarios involving significant positional shifts. This often leads to issues
such as motion blur, appearance distortions, and unnatural movements, such as
parallel drifting, which deviate from real-world physical dynamics.

To address these challenges, we introduce Tora—a pioneering DiT model that
simultaneously integrates text, image, and trajectory data, ensuring
meticulous and versatile control over video content. Tora innovatively embeds
the provided trajectory into multiple patches, enhancing motion fluidity and
more closely adhering to the natural principles of object kinetics, leveraging
its remarkable scaling properties. Notably, our work adopts OpenSora (Zheng et
al. 2024), an open-source version of Sora, as the foundational DiT framework.
To facilitate motion control, we design two innovative modules: the Trajectory
Extractor (TE), which transforms arbitrary trajectories into hierarchical
spacetime motion patches, and the Motion-guidance Fuser (MGF), meticulously
crafted to integrate these motion patches within the stacked architecture of
DiT blocks. More specifically, TE initially converts positional displacements
along trajectories between successive frames into the RGB domain via flow
visualization techniques. These visualized displacements undergo Gaussian
filtering to mitigate scattered issues. Subsequently, a 3D Variational
Autoencoder (VAE) (Kingma and Welling 2013) encodes trajectory visualizations
into spacetime motion latents, which share the same latent space with video
patches. The motion latents are then decomposed into multiple levels of motion
conditions via a series of stacked lightweight modules. Our VAE architecture
is inspired by MAGVIT-v2 (Yu et al. 2023b) but simplified by foregoing
codebook dependencies and utilizing solely VAE reconstruction loss for
training. The MGF integrates adaptive normalization layers (Perez et al. 2018)
to infuse multi-level motion conditions into the corresponding DiT blocks. We
explored various adaptations of transformer blocks including adaptive layer
normalization, cross-attention, and extra channel connections to inject the
motion conditions. Among these, adaptive layer normalization emerged as the
most effective to generate consistent videos following trajectories.

To train Tora, annotated videos with captions and movement trajectories are
essential. We adapt OpenSora’s workflow to transform raw videos into high-
quality video-text pairs and leverage an optical flow estimator (Xu et al.
2023) for trajectory extraction. Significantly, we combine motion segmentation
results (Zhao et al. 2022) with flow scores to filter out instances that
primarily contain camera movement. This strategic approach improves our
ability to accurately follow the trajectory of specific objects within videos.
As a result, this careful selection process leads to the creation of a dataset
containing 630k high-quality video clips with consistent motion. With an
adapter-like strategy (Mou et al. 2024), we solely train the temporal blocks,
together with the TE and MGF. This strategy seamlessly integrates DiT’s
inherent generative knowledge with external motion signals.

The main contributions of our work are as follows:

  * •

We introduce Tora, the first trajectory-oriented DiT for video generation. As
illustrated in Figure 1, Tora seamlessly integrates a broad range of visual
and trajectory instructions, thereby proficiently enabling the creation of
motion-manipulable videos.

  * •

To align with the scalability of DiT, we design a novel trajectory extractor
and a motion-guidance fusion mechanism to obtain spacetime motion patches,
subsequently injecting these patches into DiT blocks. We ablate several
architecture choices and offer empirical baselines for future research in DiT-
based motion control.

  * •

Experiments demonstrate that Tora is capable of generating 720p resolution
videos with varying aspect ratios, extending up to 204 frames, all guided by
the specified trajectories. Furthermore, it demonstrates superiority in
simulating movements within the physical world.

## Related Work

### Diffusion models for Video Generation

Diffusion models have demonstrated an impressive capability to generate high-
quality video samples. Previous research (Ho et al. 2022b, a; Singer et al.
2022; Wu et al. 2023; Khachatryan et al. 2023; Zhang et al. 2023b) commonly
used video diffusion models (VDMs) that incorporated temporal convolutional
and attention layers into the pre-trained image diffusion models.
Subsequently, VideoCrafter (Chen et al. 2023) and SVD (Blattmann et al. 2023)
expand the application of video diffusion models to larger datasets, while
TF-T2V (Wang et al. 2024a) directly learn from extensive text-free videos.
Lumiere (Bar-Tal et al. 2024) introduces the concept of temporal downsampling
within the 3D U-Net framework. Nonetheless, these methods encounter
limitations in generating long videos, owing to the inherent constraints on
capacity and scalability within the UNet design. Conversely, DiT-based models
(Brooks et al. 2024; Zheng et al. 2024; Bao et al. 2024) can directly generate
videos extending up to tens of seconds. Sora (Brooks et al. 2024) converts
visual data into a unified representation, facilitating large-scale training
and enabling the generation of 1-minute high-definition video. Vidu (Bao et
al. 2024) is capable of generating both realistic and imaginative videos in
various aspect ratios and resolutions. For our study, we adopt OpenSora (Zheng
et al. 2024) as the foundational model, which is an open-source alternative to
Sora.

Figure 2:  Overview of the Tora Architecture. For achieving trajectory-
controlled DiT-based video generation, we introduce two novel modules: the
Trajectory Extractor and the Motion-guidance Fuser. The Trajectory Extractor
employs a 3D motion VAE to embed trajectory vectors into the same latent space
as video patches, effectively preserving motion information across consecutive
frames. Subsequently, it uses stacked convolutional layers to extract
hierarchical motion features. The Motion-guidance Fuser utilizes adaptive
normalization layers to seamlessly inject these multi-level motion conditions
into the corresponding DiT blocks, ensuring the generation of videos that
consistently follow the defined trajectories. Our method aligns with the
scalability of DiT, enabling the creation of high-resolution, motion-
controllable videos with prolonged durations.

### Motion control in Video Generation

To better control motion in generated video, a multitude of studies have
endeavored to introduce diverse motion signals in VDMs. Pioneering works like
MotionDirector (Zhao et al. 2023) and VMC (Jeong, Park, and Ye 2024) have
utilized reference videos to extract motion patterns applicable to diverse
video generations. VideoComposer (Wang et al. 2023a) expands upon this by
adopting depth maps, sketches, or motion vectors from references as
conditional inputs for motion control. Nonetheless, these methodologies are
limited to reproducing existing motion patterns. Conversely, approaches that
leverage trajectories or bounding boxes (Yin et al. 2023; Dai et al. 2023;
Wang et al. 2024b) in video generation promise greater adaptability and user
accessibility. DragNUWA (Yin et al. 2023) breaks new ground by integrating
trajectory-based conditioning into VDMs, facilitating complex camera and
object movements. AnimateAnything (Dai et al. 2023) employs motion masks for
precise control over the moving regions. TrailBlazer (Wan-Duo Kurt Ma 2023),
employs explicit attention mechanisms to maneuver generated objects along
precise trajectories. MotionCtrl (Wang et al. 2024b) facilitates more flexible
control, allowing separate adjustment of both camera movements and individual
object motions. However, all of them often yield noticeable artifacts in both
motion consistency and visual presentation when applied to longer video
sequences. In contrast, our method first integrates trajectories into DiT
architecture, specifically designed to accommodate scaling properties. This
enables motion fluidity and closer adherence to the physical world.

## Methodology

### Preliminary

Latent Video Diffusion Model (LVDM). The LVDM enhances the stable diffusion
model (Ramesh et al. 2022) by integrating a 3D U-Net, thereby empowering
efficient video data processing. This 3D U-Net design augments each spatial
convolution with an additional temporal convolution and follows each spatial
attention block with a corresponding temporal attention block. It is optimized
employing a noise-prediction objective function:

$l_{\epsilon}=||\epsilon-\epsilon_{\theta}(z_{t},t,c)||^{2}_{2},$ (1)

Here, $\epsilon_{\theta}(\cdot)$ signifies the 3D U-Net’s noise prediction
function. The condition $c$ is guided into the U-Net using cross-attention for
adjustment. Meanwhile, $z_{t}$ denotes the noisy hidden state, evolving like a
Markov chain that progressively adds Gaussian noise to the initial latent
state $z_{0}$:

$z_{t}=\sqrt{\bar{\alpha}_{t}}z_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I),$
(2)

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}(1-\beta_{t})$ and $\beta_{t}$ is a
coefficient that controls the noise strength in step $t$.

Diffusion Transformer (DiT). The DiT (Peebles and Xie 2023) constitutes a
seminal architectural innovation that synergistically combines the advantages
of diffusion models with the transformer architectures (Vaswani et al. 2017).
This integration is purposed to transcend the limitations of traditional
U-Net-based LDMs, augmenting their performance, versatility, and scalability.
While maintaining the overarching formulation congruent with established LDM
frameworks, the paradigm shift resides in the substitution of the U-Net with a
transformer architecture for the denoising function $\epsilon_{\theta}(\cdot)$
learning, thereby marking a pivotal advance in the realm of generative
modeling.

### Tora

Tora adopts OpenSora as the foundational model for its DiT architecture. To
facilitate precise and user-friendly motion control across diverse durations,
Tora introduces two novel motion-processing components: the Trajectory
Extractor (TE) and the Motion-guidance Fuser (MGF). These modules serve to
encode provided trajectories into multi-level spacetime motion patches,
meticulously integrating these patches within the stacked structure of DiT
blocks. An overview of Tora’s workflow is depicted in Figure 2.

Spatial-Temporal DiT. The ST-DiT architecture incorporates two distinct block
types: the Spatial DiT Block (S-DiT-B) and the Temporal DiT Block (T-DiT-B),
arranged in an alternating sequence. The S-DiT-B comprises two attention
layers, each performing Spatial Self-Attention (SSA) and Cross-Attention
sequentially, succeeded by a point-wise feed-forward layer that serves to
connect adjacent T-DiT-B block. Notably, the T-DiT-B modifies this schema
solely by substituting SSA with Temporal Self-Attention (TSA), preserving
architectural coherence. Within each block, the input, upon undergoing
normalization, is concatenated back to the block’s output via skip-
connections. By leveraging the ability to process variable-length sequences,
the denoising ST-DiT can handle videos of variable durations.

During processing, a video autoencoder (Yu et al. 2023a) is first employed to
diminish both spatial and temporal dimensions of videos. To elaborate, it
encodes the input video $X\in\mathbb{R}^{L\times H\times W\times 3}$ into
video latent $z_{0}\in\mathbb{R}^{l\times h\times w\times 4}$, where $L$
denotes the video length and $l=L/4,h=H/8,w=W/8$. $z_{0}$ is next
“patchified”, resulting in a sequence of input tokens $I\in\mathbb{R}^{l\times
s\times d}$. Here, $s=hw/p^{2}$ and $p$ denotes the patch size. $I$ is then
forwarded to the ST-DiT, which models these compressed representations. In
both SSA and TSA, standard Attention is performed using Query (Q), Key (K),
and Value (V) matrices:

$Q=W_{Q}\cdot I_{norm};K=W_{K}\cdot I_{norm};V=W_{V}\cdot I_{norm}$ (3)

Here, $I_{norm}$ represents the normalized $I$, $W_{Q},W_{K},W_{V}$ are
learnable projection matrices. The textual prompt is first embedded utilizing
a T5 text encoder. The cross-attention mechanism is employed between the
embedded prompt and the intermediate features from either the SSA or the TSA.

Trajectory Extractor. Trajectories have proven to be a more user-friendly
method for controlling the motion of generated videos. Specifically, given a
trajectory $traj=\left\\{(x_{i},y_{i})\right\\}_{i=0}^{L-1}$, where
$(x_{i},y_{i})$ denotes the spatial position $(x,y)$ at the $i$-$th$ frame the
trajectory passes through. Previous studies primarily encode the horizontal
offset $u(x_{i},y_{i})$ and the vertical offset $v(x_{i},y_{i})$ as the motion
condition for $(x_{i},y_{i})$:

$u(x_{i},y_{i})=x_{i+1}-x_{i};~{}v(x_{i},y_{i})=y_{i+1}-y_{i}.$ (4)

However, the DiT model employs a video autoencoder and a patchification
process to convert the video into video patches. Here, each patch is derived
across multiple frames, rendering it inappropriate to straightforwardly employ
frame-to-frame offsets. To address this, our TE converts the trajectory into
motion patches, which inhabit the same latent space as the video patches.
Particularly, we begin by transforming the $traj$ into a trajectory map
$g\in\mathbb{R}^{L\times H\times W\times 2}$, enhanced with a Gaussian Filter
to mitigate scatter. Notably, the first frame employs a fully-zero map.
Afterward, the trajectory map $g$ is translated into the RGB color space,
producing $g_{vis}\in\mathbb{R}^{L\times H\times W\times 3}$ through a flow
visualization technique. We use a 3D VAE to compress trajectory maps,
achieving an 8x spatial and 4x temporal reduction, aligning with OpenSora
framework. Our 3D VAE is based on the Magvit-v2 architecture, with spatial
compression initialized using the VAE of SDXL (Podell et al. 2023) to
accelerate convergence. We train the model using only reconstruction loss to
obtain the compact motion latent representation $g_{m}\in\mathbb{R}^{l\times
h\times w\times 4}$ from the $g_{vis}$.

To match the size of the video patches, we use the same patch size on $g_{m}$
and encode it using a series of convolutional layers, resulting in spacetime
motion patches $f\in\mathbb{R}^{l\times s\times d_{v}}$. Here $d_{v}$ is the
dimension of motion patches. The output of each convolutional layer is skip-
connected to the input of the next layer to extract multi-level motion
features:

$f_{i}=Conv^{i}~{}(f_{i-1})+f_{i-1}.$ (5)

where $f_{i}$ is the motion condition for $i$-$th$ ST-DiT block.

Figure 3:  Different designs of the Motion-guidance Fuser for incorporating
trajectory conditioning. Adaptive Norm demonstrates the best performance.

Motion-guidance Fuser. To incorporate DiT-based video generation with the
trajectory, we explore three variants of fusion architectures that inject
motion patches into each ST-DiT block. These designs are illustrated in Figure
3.

  * -

Extra channel connections. Denote $h_{i}\in\mathbb{R}^{l\times s\times d}$ as
the resultant output from the $i$-$th$ block of the ST-DiT. Following the
widespread use of concatenation in GAN-based LVDM, the motion patches are
simply concatenated with the previous hidden state $h_{i-1}$ along the channel
dimension. An additional convolution layer is then added to maintain the same
latent size:

$h_{i}=Conv~{}([h_{i-1},f_{i}])+h_{i-1}.$ (6)

  * -

Adaptive Norm layer. Inspired by the adaptive normalization layers employed in
GANs, we initially convert $f_{i}$ into scale $\gamma_{i}$ and shift
$\beta_{i}$ by adding two zero-initialized convolution layers into the ST-DiT
block. Subsequently, $\gamma_{i}$ and $\beta_{i}$ are integrated into $h_{i}$
through a straightforward linear projection:

$h_{i}=\gamma_{i}\cdot h_{i-1}+\beta_{i}+h_{i-1}.$ (7)

  * -

Cross-Attention layer. The ST-DiT block has been modified to include an
additional Cross-Attention layer following the SSA or TSA, with the motion
patches serving as the key and value to integrate with the hidden state h:

$h_{i}=CrossAttn~{}([h_{i-1},f_{i}])+h_{i-1}.$ (8)

We experiment with three types of fusion architectures, finding that the
adaptive norm demonstrates the best generation performance and compute
efficiency. For the remainder of the paper, MGF employs the adaptive norm
layer unless otherwise specified.

Method | FVD ($\downarrow$) | CLIPSIM ($\uparrow$) | TrajError ($\downarrow$)
---|---|---|---
16-frame | 64-frame | 128-frame | 16-frame | 64-frame | 128-frame | 16-frame | 64-frame | 128-frame
VideoComposer (Wang et al. 2023a) | 529 | 668 | 856 | 0.2335 | 0.2284 | 0.2236 | 15.11 | 29.14 | 58.76
DragNUWA (Yin et al. 2023) | 475 | 593 | 784 | 0.2385 | 0.2341 | 0.2305 | 10.04 | 17.33 | 41.25
AnimateAnything (Dai et al. 2023) | 487 | 602 | 775 | 0.2399 | 0.2342 | 0.2313 | 13.39 | 27.28 | 51.33
TrailBlazer (Wan-Duo Kurt Ma 2023) | 459 | 581 | 756 | 0.2403 | 0.2351 | 0.2322 | 11.68 | 19.47 | 44.10
MotionCtrl (Wang et al. 2024b) | 463 | 572 | 731 | 0.2412 | 0.2376 | 0.2331 | 9.42 | 16.46 | 38.39
Ours | 438 | 460 | 494 | 0.2447 | 0.2435 | 0.2418 | 7.23 | 8.45 | 11.72
Table 1: Quantitative comparisons with state-of-the-art motion-controllable
video generation models. As the number of generated frames increases, Tora
demonstrates a growing performance advantage over the UNet-based methods,
maintaining a high degree of stability in trajectory control.

### Training Strategy and Data Processing

To achieve fine-grained control while generating a video using arbitrary
trajectories, as well as text, images, or their composites, we introduce
several training strategies for different condition injections.

Motion condition training. Inspired by DragNUWA and MotionCtrl, we use a two-
stage training approach for trajectory learning. In the first stage, we
extract the dense optical flow (Xu et al. 2023) from the training video as the
trajectory, offering richer information to accelerate motion learning. During
the second stage, to adapt the model from complete optical flow to more user-
friendly trajectories, we select a random sample of 1 to $N$ object
trajectories from the optical flow according to the motion segmentaton results
and flow scores. To address the scattered issues of sparse trajectories, we
apply a Gaussian filter for refinement. Upon completion of the two-stage
training phase, Tora enables flexible motion control using arbitrary
trajectories.

Image condition training. We follow the mask strategy employed by OpenSora to
support visual conditioning. Specifically, we randomly unmask frames during
training, and the video patches of the unmasked frames are not subjected to
any noise. This enables our Tora model to seamlessly integrate text, images,
and trajectories into a unified model.

Data Processing. For Tora, it is crucial that the video clips in the training
dataset are annotated with both captions and movement trajectories (optical
flow). To fulfill this requirement, we employ a structured data processing
method. Initially, raw videos are segmented into shorter clips based on scene
detection. Subsequently, these clips are evaluated using aesthetic scores.
Given our criteria, static videos are excluded by retaining only those with
optical flow scores exceeding 3. To better obtain object trajectories, We
employ the results of motion segmentation and camera detection to remove
instances that primarily contain camera trajectories in dynamic scenes.
Additionally, dramatic camera or object motions in some videos can cause
significant optical flow deviations, interfering with trajectory training.
Consequently, we retain these videos with a probability of $(1-score/100)$.
For eligible videos, we generate captions using the PLLaVA model (Xu et al.
2024). To balance speed and performance, we utilize the 13B version.

## Experiments

### Experimental Setup

Implementation Details. We train Tora based on OpenSora v1.2 weights. The
resolutions of the training videos vary from 144p to 720p. The numbers of
frames range from 51 to 204. To balance the training FLOP and required memory
for different resolutions and numbers of frames per iteration, we adjust the
batch size accordingly from 1 to 25. We use Adam Optimizer (Kingma and Ba
2015) with a learning rate of $2\times 10^{-5}$ on 4 NVIDIA A100. We train
Tora with the dense optical flow for 2 epochs and fine-tune it with the sparse
optical flow for 1 epoch. The maximum number of sampling trajectories $N$ is
set to 16. The inference step and the guidance scale are set to 30 and 7.0,
respectively.

Dataset. Our training videos are sourced from three datasets: 1) Panda-70M
(Chen et al. 2024), from which we use the training-10M subset containing high-
quality videos; 2) Mixkit (Envato 2024); and 3) Interval Video. After applying
our data processing pipeline, we select approximately 630k eligible videos for
the training dataset. For inference, we curate 185 clips with diverse motion
trajectories and scenes, to serve as a new benchmark for evaluating the motion
controllability. Additional information about the construction of the
evaluation datasets is available in our supplementary materials.

Metrics. We leverage standard metrics such as Fréchet Video Distance (FVD)
(Unterthiner et al. 2018), and CLIP Similarity (CLIPSIM) (Wu et al. 2021) to
quantitatively evaluate video quality. For assessing motion controllability,
we leverage Trajectory Error (TrajError), which computes the average distance
between the generated and pre-defined trajectories. Human evaluation is also
introduced.

Figure 4:  Comparison of Trajectory Error across various resolutions and
durations. Unlike U-Net models, our method maintains motion control with a
gradual increase in error.

### Quantitative and Qualitative Results

We compare our method against prevalent motion-guided video generation
approaches. The evaluation is conducted under three settings: 16, 64, and 128
frames, all with a resolution of 512 $\times$ 512 for fair comparison. The
provided trajectories are clipped to accommodate different evaluated video
lengths. For most U-Net based methods, we employ sequenced inference, wherein
the last frame generated by the previous batch serves as the visual condition
for the current batch, aligning with their inference settings. As illustrated
in Table 1, under the 16-frame setting commonly employed by U-Net based
methods, MotionCtrl and DragNUWA exhibit better alignment with the provided
trajectories but are still weaker compared to our proposed Tora. As the frame
count increases, the U-Net based methods show significant deviations in
certain frames, with misalignment errors propagating and leading to
deformation, motion blur, or object disappearance in subsequent sequences. In
contrast, Tora demonstrates high robustness to varying frame numbers due to
the integration of the transformer’s scaling ability. The movement generated
by Tora is smoother and more aligned with the physical world. When evaluated
under the 128-frame test setting, Tora’s trajectory accuracy surpasses other
methods by a factor of 3 to 5, demonstrating its exceptional motion control
capabilities. In Figure 4, we provide an analysis of the Trajectory Error
across varying resolutions and durations. Unlike U-Net based models, which
exhibit significant trajectory errors over time, Tora shows only a gradual
increase in Trajectory Error as duration increases. This gradual increase in
error aligns with the decrease in video quality observed in the DiT model as
the duration extends. The results clearly indicate that our method maintains
effective trajectory control over longer durations.

Figure 5:  Qualitative Comparisons on Trajectory Control. All methods are capable of generating objects that follow the given trajectory. However, Tora not only adheres precisely to the specified trajectory but also produces smoother movement that conforms to the physics world. Please visit our supplemental materials for detailed video results.  Method | FVD | CLIPSIM | TrajError
---|---|---|---
Sampling Frame | 581 | 0.2304 | 27.61
Average Pooling | 558 | 0.2325 | 20.97
3D VAE | 513 | 0.2358 | 14.25
Table 2: Evaluations of the impact of different trajectory compression
methods. The 3D VAE yields superior results in motion-conditioned encoding.

Figure 5 presents a comparative analysis of our proposed method against
mainstream motion control techniques. In the first scenario, which involves
the co-movement of two men, all methods are able to generate relatively
accurate motion trajectories. However, our approach demonstrates superior
visual quality. This advantage is primarily due to the utilization of longer
sequence frames, which facilitate smoother motion trajectories and more
realistic background rendering. For instance, in our generated bicycle
scenario, the human legs exhibit realistic pedaling motions, whereas the
output from DragNUWA features legs that appear almost horizontally floating,
violating physical realism. Additionally, both DragNUWA and MotionCtrl suffer
from significant motion blur towards the end of their videos. Furthermore,
MotionCtrl introduces unintended camera movements during the riding sequence,
despite the absence of camera movement conditions. In another case, DragNUWA
shows severe deformation of the lantern as the provided trajectory rises and
falls continuously. Although MotionCtrl’s trajectory is relatively accurate,
the resulting video fails to match the expected depiction of two lanterns.
Overall, our method not only adheres closely to the provided trajectories but
also minimizes object deformation, thereby ensuring higher fidelity in motion
representation.

### Ablation study

In this section, we conduct serval ablation studies to analyze the effects of
our design choices. All models are evaluated using 480p resolution, a 16:9
aspect ratio, and 204 frames.

Method | FVD | CLIPSIM | TrajError
---|---|---|---
Extra Channel | 542 | 0.2329 | 21.07
Cross Attention | 526 | 0.2354 | 18.36
Adaptive Norm | 513 | 0.2358 | 14.25
Table 3: Different variants of motion fusion blocks employed in MGF. Adaptive
Norm works best.

Trajectory Compression. To incorporate the trajectory vector within the same
latent space as video patches, we investigate three distinct methods for
trajectory compression, as summarized in Table 2. The first method samples the
mid-frame as the keyframe for successive 4-frame intervals and employs patch-
unshuffle for spatial compression. Despite its simplicity, this method proves
suboptimal for motion control, primarily due to the potential flow estimation
errors when encountering rapid motions or occlusions. Additionally, the
magnified dissimilarity between patches induced by the chosen frame interval
exacerbates learning challenges. The second method utilizes average pooling to
gather successive frames. While this captures a general sense of movement, it
inadvertently sacrifices precision by homogenizing the trajectory’s direction
and magnitude, thereby diluting critical motion details. To preserve the
trajectory information between consecutive frames as much as possible, we
further employ a 3D VAE to extract the global context of successive trajectory
intervals. The trajectory data is visually translated into an RGB image format
to leverage existing 3D VAE weights. Extensive training on a large scale of
trajectory videos with this setup yields the most favorable outcomes,
underscoring the efficacy of our tailored 3D VAE approach in trajectory
compression.

Block design and integrated position of MGF. We train the three variant MFG
blocks as previously described, with the results presented in Table 3.
Notably, the adaptive norm block achieves lower FVD and Trajectory Error
compared to both cross-attention and extra channel conditioning methods, while
also exhibiting the highest computational efficiency. This superiority is
attributed to its capability for dynamic feature adaptation based on diverse
conditions without requiring strict alignment, a limitation often encountered
with cross-attention. Moreover, it ensures temporal consistency by modulating
the conditional information over time, which is crucial for injecting motion
cues. In contrast, channel concatenation may cause information congestion,
rendering motion signals less effective. During training, we observe that
initializing the normalization layer as the identity function is important for
achieving optimal performance. Additionally, we evaluate the integration
position of the MGF module within the Spatial DiT and Temporal DiT blocks. Our
findings reveal that embedding the MGF within the Temporal DiT block notably
enhances trajectory motion control, as evidenced by a substantial decrease in
Trajectory Error (from 23.39 to 14.25). This approach enhances the MGF’s
efficiency in interacting with temporal dynamics, thereby significantly
improving the fidelity of motion synthesis.

Training Strategies. We evaluate the effectiveness of the two-stage training
approach, with the results summarized in Table 4. Training exclusively with
dense optical flow proves ineffective as it fails to accurately capture the
intricate details of the provided sparse trajectories. On the other hand,
training solely with sparse trajectories offers limited information, making
the learning process more challenging. By initially training with dense flows
and subsequently fine-tuning with sparse trajectories, our model demonstrates
improved adaptability to various types of trajectory data. This method not
only enhances overall performance but also increases the model’s versatility
in handling diverse motion patterns.

Motion-guidance | FVD | CLIPSIM | TrajError
---|---|---|---
Dense Flow | 601 | 0.2307 | 39.34
Sparse Flow | 556 | 0.2334 | 24.73
Hybird | 513 | 0.2358 | 14.25
Table 4: Ablation of the type of training trajectories. “Hybrid” denotes the
proposed two-stage training strategy.

## Conclusion

This paper introduces Tora, the first trajectory-oriented Diffusion
Transformer framework for video generation that integrates text, image, and
trajectory conditions. Tora effectively encodes arbitrary trajectories into
spacetime motion patches, which align with the scaling properties of DiT,
thereby enabling more realistic simulations of physical world movements. By
employing a two-stage training process, Tora achieves motion-controllable
video generation across a wide range of durations, aspect ratios, and
resolutions. Remarkably, it can generate high-quality videos that adhere to
specified trajectories, producing up to 204 frames at 720p resolution. This
capability underscores Tora’s versatility and robustness in handling diverse
motion patterns while maintaining high visual fidelity. We hope our work
provides a strong baseline for future research in motion-guided Diffusion
Transformer methods.

## References

  * Bao et al. (2024) Bao, F.; Xiang, C.; Yue, G.; He, G.; Zhu, H.; Zheng, K.; Zhao, M.; Liu, S.; Wang, Y.; and Zhu, J. 2024.  Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models.  _arxiv_ , abs/2405.04233.
  * Bar-Tal et al. (2024) Bar-Tal, O.; Chefer, H.; Tov, O.; Herrmann, C.; Paiss, R.; Zada, S.; Ephrat, A.; Hur, J.; Li, Y.; Michaeli, T.; Wang, O.; Sun, D.; Dekel, T.; and Mosseri, I. 2024.  Lumiere: A Space-Time Diffusion Model for Video Generation.  _arxiv_ , abs/2401.12945.
  * Blattmann et al. (2023) Blattmann, A.; Dockhorn, T.; Kulal, S.; Mendelevitch, D.; Kilian, M.; Lorenz, D.; Levi, Y.; English, Z.; Voleti, V.; Letts, A.; Jampani, V.; and Rombach, R. 2023.  Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets.  _arXiv preprint arXiv:2311.15127_.
  * Brooks et al. (2024) Brooks, T.; Peebles, B.; Holmes, C.; DePue, W.; Guo, Y.; Jing, L.; Schnurr, D.; Taylor, J.; Luhman, T.; Luhman, E.; Ng, C.; Wang, R.; and Ramesh, A. 2024.  Video generation models as world simulators.
  * Chen et al. (2023) Chen, H.; Xia, M.; He, Y.-Y.; Zhang, Y.; Cun, X.; Yang, S.; Xing, J.; Liu, Y.; Chen, Q.; Wang, X.; Weng, C.-L.; and Shan, Y. 2023.  VideoCrafter1: Open Diffusion Models for High-Quality Video Generation.  _ArXiv_ , abs/2310.19512.
  * Chen et al. (2024) Chen, T.; Siarohin, A.; Menapace, W.; Deyneka, E.; Chao, H.; Jeon, B. E.; Fang, Y.; Lee, H.; Ren, J.; Yang, M.; and Tulyakov, S. 2024.  Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers.  _arxiv_ , abs/2402.19479.
  * Dai et al. (2023) Dai, Z.; Zhang, Z.; Yao, Y.; Qiu, B.; Zhu, S.; Qin, L.; and Wang, W. 2023.  Fine-Grained Open Domain Image Animation with Motion Guidance.  _arxiv_ , abs/2311.12886.
  * Dhariwal and Nichol (2021) Dhariwal, P.; and Nichol, A. 2021.  Diffusion Models Beat GANs on Image Synthesis.  _ArXiv_ , abs/2105.05233.
  * Envato (2024) Envato, E. 2024.  mixkit:Free assets for your next video project.
  * Ho et al. (2022a) Ho, J.; Chan, W.; Saharia, C.; Whang, J.; Gao, R.; Gritsenko, A.; Kingma, D. P.; Poole, B.; Norouzi, M.; Fleet, D. J.; et al. 2022a.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_.
  * Ho et al. (2022b) Ho, J.; Salimans, T.; Gritsenko, A.; Chan, W.; Norouzi, M.; and Fleet, D. J. 2022b.  Video diffusion models.
  * Jeong, Park, and Ye (2024) Jeong, H.; Park, G. Y.; and Ye, J. C. 2024.  VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models.  In _CVPR_.
  * Khachatryan et al. (2023) Khachatryan, L.; Movsisyan, A.; Tadevosyan, V.; Henschel, R.; Wang, Z.; Navasardyan, S.; and Shi, H. 2023.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  _arXiv preprint arXiv:2303.13439_.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  In Bengio, Y.; and LeCun, Y., eds., _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kingma and Welling (2013) Kingma, D. P.; and Welling, M. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Mou et al. (2024) Mou, C.; Wang, X.; Xie, L.; Wu, Y.; Zhang, J.; Qi, Z.; and Shan, Y. 2024.  T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models.  In _Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024_ , 4296–4304.
  * Olaf Ronneberger (2015) Olaf Ronneberger, T. B., Philipp Fischer. 2015.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , 234–241. Springer.
  * Peebles and Xie (2023) Peebles, W.; and Xie, S. 2023.  Scalable diffusion models with transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 4195–4205.
  * Perez et al. (2018) Perez, E.; Strub, F.; de Vries, H.; Dumoulin, V.; and Courville, A. C. 2018.  FiLM: Visual Reasoning with a General Conditioning Layer.  In _Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024_ , 3942–3951.
  * Podell et al. (2023) Podell, D.; English, Z.; Lacey, K.; Blattmann, A.; Dockhorn, T.; Müller, J.; Penna, J.; and Rombach, R. 2023.  SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis.  _arxiv_ , abs/2307.01952.
  * Ramesh et al. (2022) Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022.  Hierarchical text-conditional image generation with clip latents, 2022.  _URL https://arxiv. org/abs/2204.06125_ , 7.
  * Singer et al. (2022) Singer, U.; Polyak, A.; Hayes, T.; Yin, X.; An, J.; Zhang, S.; Hu, Q.; Yang, H.; Ashual, O.; Gafni, O.; et al. 2022.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_.
  * Unterthiner et al. (2018) Unterthiner, T.; van Steenkiste, S.; Kurach, K.; Marinier, R.; Michalski, M.; and Gelly, S. 2018.  Towards Accurate Generative Models of Video: A New Metric & Challenges.  _arXiv:1812.01717_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 5998–6008.
  * Wan-Duo Kurt Ma (2023) Wan-Duo Kurt Ma, W. B. K., J. P. Lewis. 2023.  TrailBlazer: Trajectory Control for Diffusion-Based Video Generation.  _arXiv preprint arXiv:2401.00896_.
  * Wang et al. (2023a) Wang, X.; Yuan, H.; Zhang, S.; Chen, D.; Wang, J.; Zhang, Y.; Shen, Y.; Zhao, D.; and Zhou, J. 2023a.  VideoComposer: Compositional Video Synthesis with Motion Controllability.  _arXiv preprint arXiv:2306.02018_.
  * Wang et al. (2024a) Wang, X.; Zhang, S.; Yuan, H.; Qing, Z.; Gong, B.; Zhang, Y.; Shen, Y.; Gao, C.; and Sang, N. 2024a.  A Recipe for Scaling up Text-to-Video Generation with Text-free Videos.
  * Wang et al. (2023b) Wang, Z.; Yuan, Z.; Wang, X.; Chen, T.; Xia, M.; Luo, P.; and Shan, Y. 2023b.  MotionCtrl: A Unified and Flexible Motion Controller for Video Generation.  In _arXiv preprint arXiv:2312.03641_.
  * Wang et al. (2024b) Wang, Z.; Yuan, Z.; Wang, X.; Chen, T.; Xia, M.; Luo, P.; and Shan, Y. 2024b.  MotionCtrl: A Unified and Flexible Motion Controller for Video Generation.  In _SIGGRAPH_.
  * Wu et al. (2021) Wu, C.; Huang, L.; Zhang, Q.; Li, B.; Ji, L.; Yang, F.; Sapiro, G.; and Duan, N. 2021.  Godiva: Generating open-domain videos from natural descriptions.  _arXiv preprint arXiv:2104.14806_.
  * Wu et al. (2023) Wu, J. Z.; Ge, Y.; Wang, X.; Lei, S. W.; Gu, Y.; Shi, Y.; Hsu, W.; Shan, Y.; Qie, X.; and Shou, M. Z. 2023.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 7623–7633.
  * Xu et al. (2023) Xu, H.; Zhang, J.; Cai, J.; Rezatofighi, H.; Yu, F.; Tao, D.; and Geiger, A. 2023.  Unifying Flow, Stereo and Depth Estimation.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 45(11): 13941–13958.
  * Xu et al. (2024) Xu, L.; Zhao, Y.; Zhou, D.; Lin, Z.; Ng, S.; and Feng, J. 2024.  PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning.  _arxiv_ , abs/2404.16994.
  * Yin et al. (2023) Yin, S.; Wu, C.; Liang, J.; Shi, J.; Li, H.; Ming, G.; and Duan, N. 2023.  Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.  _arXiv preprint arXiv:2308.08089_.
  * Yu et al. (2023a) Yu, L.; Cheng, Y.; Sohn, K.; Lezama, J.; Zhang, H.; Chang, H.; Hauptmann, A. G.; Yang, M.-H.; Hao, Y.; Essa, I.; et al. 2023a.  Magvit: Masked generative video transformer.
  * Yu et al. (2023b) Yu, L.; Lezama, J.; Gundavarapu, N. B.; Versari, L.; Sohn, K.; Minnen, D.; Cheng, Y.; Gupta, A.; Gu, X.; Hauptmann, A. G.; Gong, B.; Yang, M.; Essa, I.; Ross, D. A.; and Jiang, L. 2023b.  Language Model Beats Diffusion - Tokenizer is Key to Visual Generation.  _arXiv_ , abs/2310.05737.
  * Zhang et al. (2023a) Zhang, S.; Wang, J.; Zhang, Y.; Zhao, K.; Yuan, H.; Qin, Z.; Wang, X.; Zhao, D.; and Zhou, J. 2023a.  I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models.  _arXiv preprint arXiv:2311.04145_.
  * Zhang et al. (2023b) Zhang, Y.; Wei, Y.; Jiang, D.; Zhang, X.; Zuo, W.; and Tian, Q. 2023b.  ControlVideo: Training-free Controllable Text-to-Video Generation.  _arXiv preprint arXiv:2305.13077_.
  * Zhao et al. (2023) Zhao, R.; Gu, Y.; Wu, J. Z.; Zhang, D. J.; Liu, J.; Wu, W.; Keppo, J.; and Shou, M. Z. 2023.  MotionDirector: Motion Customization of Text-to-Video Diffusion Models.  _arxiv_ , abs/2310.08465.
  * Zhao et al. (2022) Zhao, W.; Liu, S.; Guo, H.; Wang, W.; and Liu, Y. 2022.  ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild.  In _Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII_ , volume 13692, 523–542.
  * Zheng et al. (2024) Zheng, Z.; Peng, X.; Yang, T.; Shen, C.; Li, S.; Liu, H.; Zhou, Y.; Li, T.; and You, Y. 2024.  Open-Sora: Democratizing Efficient Video Production for All.


# Churn Prediction via Multimodal Fusion Learning: Integrating Customer
Financial Literacy, Voice, and Behavioral Data

1st David Hason Rudd  Faculty of Engineering and IT
The University of Technology Sydney
Sydney, Australia
<EMAIL_ADDRESS>2nd Huan Huo  Faculty of Engineering and IT
The University of Technology Sydney
Sydney, Australia
<EMAIL_ADDRESS>3rd Md Rafiqul Islam  Information Systems (Data
Analytics)
Australian Institute of Higher Education (AIH)
Sydney, Australia
<EMAIL_ADDRESS>4rd Guandong Xu  Faculty of Engineering and IT
Advanced Analytics institute (AAi)
Sydney, Australia
<EMAIL_ADDRESS>
###### Abstract

In today’s competitive landscape, businesses grapple with customer retention.
Churn prediction models, although beneficial, often lack accuracy due to the
reliance on a single data source. The intricate nature of human behavior and
high-dimensional customer data further complicate these efforts. To address
these concerns, this paper proposes a multimodal fusion learning model for
identifying customer churn risk levels in financial service providers. Our
multimodal approach integrates customer sentiments, financial literacy (FL)
level, and financial behavioral data, enabling more accurate and bias-free
churn prediction models. The proposed FL model utilizes a SMOGN-COREG
supervised model to gauge customer FL levels from their financial data. The
baseline churn model applies an ensemble artificial neural network and
oversampling techniques to predict churn propensity in high-dimensional
financial data. We also incorporate a speech emotion recognition model
employing a pre-trained CNN-VGG16 to recognize customer emotions based on
pitch, energy, and tone. To integrate these diverse features while retaining
unique insights, we introduced late and hybrid fusion techniques that
complementary boost coordinated multimodal co-learning. Robust metrics were
utilized to evaluate the proposed multimodal fusion model and hence the
approach’s validity, including mean average precision and macro-averaged F1
score. Our novel approach demonstrates a marked improvement in churn
prediction, achieving a test accuracy of 91.2%, a Mean Average Precision (MAP)
score of 66, and a Macro-Averaged F1 score of 54 through the proposed hybrid
fusion learning technique compared with late fusion and baseline models.
Furthermore, the analysis demonstrates a positive correlation between negative
emotions, low FL scores, and high-risk customers.

###### Index Terms:

Churn prediction, multimodal learning, feature fusion, financial literacy,
speech emotion recognition, customer behavior

##  I Introduction

Understanding and predicting customer churn has become paramount for long-term
success in an increasingly competitive business landscape [1]. Customer churn,
or customer attrition, directly affects a company’s revenue and growth
potential, making it essential to be accurately predicted and prevented [2].
Furthermore, the human cognitive system is inherently susceptible to cognitive
biases, such as anchoring bias, the availability heuristic, and the bandwagon
effect [9]. These biases can significantly skew customer’s feeling and
decision-making towards products and services. Misguided customer interactions
or suboptimal presentations of complex financial products can trigger these
biases, prompting a focus on potential risks over service benefits. For
financial service providers, understanding these psychological factors is
crucial. It allows them to tailor product design, marketing strategies, and
customer relationship management (CRM) to mitigate negative biases and churn
rate. Therefore, predicting customer emotion and knowledge about products and
services is not just advantageous—it’s pivotal for businesses striving for
success in today’s highly competitive marketplace.

Customer historical data recorded by organizations is increasing
exponentially, and data analytics specialists contend that the traditional
customer churn analysis method, which heavily relied on transactional data and
customer demographics, is no longer a robust solution. A more comprehensive
approach is required to better deal with the heterogeneous data input in
various modalities.

The current study aims to predict customer churn propensity by incorporating
diverse modalities for the first time, to our best knowledge. The proposed
approach incorporates survey analysis regarding customer financial literacy
level, emotion detection from customer voices (CV), and CRM data to offer a
more comprehensive representation of churn risk.

The proposed approach utilizes multimodal machine learning (ML) methods for a
more holistic understanding of customer churn. This approach is characterized
by considering multiple facets of the customer experience, from interactions
with customer service to engagement with a product or service. Multimodality
in ML refers to the integration of two or more distinct inputs, recorded in
different media formats, within a single ML model. The unique aspect of these
inputs is that they cannot be unambiguously mapped into one another using an
algorithm [11]. We proposed a fusion learning to force similarities of these
inputs into a coordinated feature representation space. Subsequently, a
decision-level fusion is applied to rank churn risk.

###  I-A Sentiment analysis

Customer sentiment analysis can be divided into para linguistic, using CV
analysis, or linguistic or text based approaches. Recorded customer service
calls can be analyzed to understand customer satisfaction. Although sentiment
analysis using natural language processing (NLP) can be used to transcribe and
analyze these calls; meaning is often also conveyed by tone, pitch, and
loudness. Thus, humans provide and receive considerable meta-information
during a conversation apart from the text. This is an inherent problem for
text-based sentiment analysis in NLP, whereas non-linguistic emotion detection
through speech signal processing is somewhat more robust [3]. Customers
negative emotions are often good indicators of product dissatisfaction, and
these customers can be considered to have higher churn risk due to their
discernible discontent, warranting special attention. To decode emotion from
CV, we introduced a voice-based sentiment analysis model[3]. The central
innovation of our work lies in the development of a novel hybrid feature map
generator algorithm using harmonic and percussive components of Mel
Spectrogram that improves speech emotion recognition (SER).

###  I-B Financial literacy

Customer financial literacy can be a crucial factor to predict customer churn
within the financial services industry. Financially literate customers are
generally better equipped to comprehend the range of products and services
available, enabling them to better select the most suitable option(s) to meet
their requirements. This reduces dissatisfaction and subsequent churn
probability. In contrast, customers with low financial literacy could struggle
to understand the options, and hence make sub-optimal decisions regarding
financial products and services, leading to dissatisfaction and potential
migration to competing financial institutions perceived to offer superior
products. We introduced a SMOGN-COREG model to deal with unlabelled examples
via semi-supervised regression (SSR) methods [8]. We could significantly
contribute by enhancing prediction precision beyond relying on labeled data in
supervised methods.

###  I-C Multimodal modelling

The current study focused on analyzing synergistic relationships between
predictors in each unimodal context, with the objective is to enhance churn
prediction by utilizing a coordinated representation space that forces
similarity for three input data formats: finance network, CRM, and inbound
audio call data. We investigated the effectiveness for each previously
proposed unimodal. Given the identified challenges and opportunities, this
study introduces several significant contributions as below:

  * -

In a first-of-its-kind approach, we proposed a multimodal hybrid fusion model
that integrates distinct databases, including CRM, CV, and FL data. This
enables us to add an essential layer of understanding customer behavior for
churn prediction by discerning customer emotion from their vocal attributes
and measuring financial literacy level from their account performance and
survey data.

  * -

We built a specific coordinated feature representation space and translation
matrix that reflects data similarity meaningfully to capture interactions
between modalities, with the capability of adding future modalities like
textual features in the proposed framework.

  * -

Identifying significant correlations between negative emotions, low FL, and
increased churn risk.

Thus, the proposed multimodal churn model leveraged customer segmentation by
detecting negative emotions in CVs during interactions with call center
operators, subsequently tailoring the response to address specific customer
concerns and requirements, and gaining insights into potential dissatisfaction
or frustration causes.

##  II Recent works

Considerable research effort is evolving towards more sophisticated churn
prediction methods to better draw advantages from data growth. Hence,
multimodal churn modelling will continue to advance, providing ever more
refined strategies for churn prediction by encompassing various customer
experience facets for holistic customer churn understanding [14].

De Caigny explored the benefits from incorporating textual data into customer
churn prediction (CCP) models and highlighted superior performance from
convolutional neural networks (CNNs) compared with other text mining methods
for CCP, achieving 89.87% accuracy. Outcomes from their study will help guide
researchers regarding text mining for improved CCP [4].

Nhi NY and Liu proposed predicting client churn by analyzing unstructured
data, e.g. audio calls, in machine learning models. Their approach expanded
traditional methods, which tend to focus on structured data, such as
demographics and transactions. In contrast, their proposed model combined text
mining with a gradient boosting tree algorithm to improve the client churn
model predictive performance. This approach achieved significant predictive
accuracy improvement, increasing at least 5% across various customer datasets
[5].

Kimura highlighted benefits from combining boosting algorithms with hybrid
resampling methods for CP, which could significantly enhance prediction
effectiveness and practicality in both academic and industry settings. The
challenge from imbalanced datasets was explicitly discussed, and showed that
although hybrid resampling methods, including synthetic minority over-sampling
technique (SMOTE), SMOTE-ENN, and SMOTE Tomek-Links, offer significant
potential, they are yet to be widely adopted for CCP [6]. Ahn and Hwang
reviewed churn prediction methodologies, highlighting their chronological
progression and underscored the necessity for adaptable and flexible
approaches contingent on specific data formats. They also showed that very few
previous studies adopted a multimodal approach to address churn prediction
[7].

As delineated some recent work in Table I from our comprehensive literature
review, many studies primarily focus on using singular structured data
sources, such as CRM databases, for churn prediction tasks. Conversely, there
needs to be more research that employs multiple data sources for the same
purpose. To the best of our knowledge, our current investigation is the first
that incorporates three discrete input data sets for the churn prediction
problem.

TABLE I: Summary of the comparative studies – Recent Modalities Ref. | Input | Learning | Prediction | Industry
---|---|---|---|---
[19] | 1a | Unimodal | RF | Finance
[18] | 1 | Ensemble | GBT+k-medios | Telecom
[17] | 1 | Unimodal | RF | Telecom
[16] | 1 | Ensemble | DL+LSTM | Game
[15] | 1 | Ensemble | LSTM+HS | Game
[5] | 2b | Feature Fusiond | GBT | Finance
[4] | 2 | Feature Fusion | CNN+Logit | Finance
Ours | 3c | Hybrid Fusione | CNNs+DL | Finance
a1: Structured data (e.g., demographic, account, and CRM data)
b2: Structured + textual data (e.g., call log script and e-messages)
c3: Structured + voice + financial literacy (qualitive data)
dFeature Fusion: Multimodal feature fusion modelling (or early fusion)
eHybrid Fusion: Multimodal hybrid (early + late) fusion modelling

Therefore, the present study undertook a gap analysis and subsequently
proposed an suitable methodology for CCP. This provides significant
contributions by incorporating churn predictors, such as customer positive and
negative emotional feedback from audio calls, and customer financial literacy
level from survey data. The proposed synergistic hybrid feature fusion
techniques and subsequent studies building on that framework, will mitigate
bias for CCP.

##  III Methodology

Our methodology encompasses three modalities to analyze customer behavior:
baseline customer churn, SER, and the FL model, which are elaborated in the
subsequent sections.

###  III-A Customer financial literacy modelling

Considering substantial impact from inadequate FL on customer churn, our first
modality emphasizes FL in particular [8]. We developed a semi-supervised
regression model, (SMOGN-COREG) to measure customer financial literacy derived
from a large and unlabeled dataset. The proposed method boosts the learning
process by generating synthetic samples for minority class. The SMOGN approach
was applied during pre-processing and subsequently combined with non-
parametric multi learner semi-supervised regression to enhance model
performance, balancing the response variable distribution by increasing the
number of rare but important cases in the dataset [13]. The model output is FL
measurement in a range between 0 and 1.

###  III-B Emotion recognition modelling

The second unimodal leverages the voice based sentiment analysis to recognize
customer positive and negative emotions during their interaction with the call
center [3]. The model employs the Mel Spectrogram to provide insight into
voice tone, pitch, and tempo; extracting acoustic features and percussive and
harmonic components from the signal frequency domain [12].

We employed a pre-trained CNN to develop a framework decipher emotions from
CV, incorporating a feature map generator function to extract harmonic and
percussive components via a median filter applied to the signal spectrum’s
axes. Two feature vectors were subsequently created by averaging these
components, and computing the Mel spectrogram logarithm. Finally, we construct
a two-dimensional image feature map as input for the pre-trained CNN-VGG16
network to classify emotions into binary outcomes positive and negative
classes, represented as 0 and 1, respectively.

###  III-C Baseline churn model

The third unimodal evaluates customer churn risk using demographic and
financial behavior sourced from CRM platforms [10]. The proposed approach
integrates recursive feature elimination (RFE), SMOTE, and ANN with Deep
Learning (ANN DL) to construct the churn classifier. Thus, the proposed
framework predicts potential churn within the upcoming six months using data
from the prior 12-month window. A customer was deemed a ’churner’ if they
terminated their account within six-month prediction period, and the outcome
is encapsulated in a binary variable, 0 or 1 indicates active or terminated
account throughout the subsequent six months.

###  III-D Proposed multimodal fusion learning

The proposed multimodal is able to analyze customer behavior from a diverse
array of data sources highlighting complex interrelationships between
different customer behavior facets. The model is designed with different
components, including independent unimodal learning, feature representation
space, translation or mapping function, and hybrid fusion.

Figure 1, shows how gauge customer financial literacy level, customer emotion,
and churn propensity were employed to segment customers as low-risk (loyal or
non-churner), Mid-risk (potential churner), and high-risk (high probability
churner). For example, a high risk churn customer may exhibit low financial
literacy, negative sentiment towards the company’s offerings, and heightened
propensity to churn.

### Hybrid fusion:

The proposed model includes hybrid fusion (multi-level fusion) by
incorporating early and late fusion. Hence, hybrid fusion generates input data
for the baseline churn model by fusing prediction outcomes from FL and SER
models for the CRM dataset.

### Late fusion:

We established a prototype multimodal learning model with late fusion to
evaluate the proposed hybrid fusion method efficacy compared with other
methods. Early fusion data was not utilized as churn model input, with all
data sources directly provided to the modalities.

Figure 1: Proposed multimodal hybrid fusion learning method workflow to
integrate various modalities

###  III-E Feature representation space and translation

Extracting specific features associated with each modality presents
significant challenges due to inherent disparities in scales, units, and
distributions. Therefore, the proposed multi-modal model fusion these
heterogeneous features while maintaining their unique insights towards churn
modeling. We propose a specific coordinated representation space that reflects
data similarity meaningfully to capture interactions between modalities with
the ultimate goal to understand complementary customer behavior across various
touchpoints, such as call center interaction, FL survey, and historic
demographic data. The proposed coordinated representation space can be
expressed as

$\small f(x_{i})\sim c(x_{i})\sim v(x_{i})$ (1)

where $x_{i}$ is the modalities and functions $f$, $v$, and $c$ representing
independent unimodal learning networks, including FL, SER, and baseline churn
model into the coordinated representation space; and $\sim$ is similarity or
coordination in the projection into this space. We employ a translation
dictionary to map features extracted from each modality while preserving
similarity and meaningful patterns. The mapping matrix is represented through
several logical propositions defined by experts, each corresponding to
different condition(s) on the unimodal features. Each unimodal output data
structure is mapped into two level numeric nominal variables based on a
predetermined prediction confidence $P(\text{{pred}})$ threshold, defining a
coordinated representation space by logical proposition(s). Constant weights
are assigned to each matrix array $C_{i}$, $F_{i}$ and $V_{i}$ depending on
the unimodal feature weight in multimodal learning, which can be expressed as,

$\forall x_{i},P1(x_{i}):f(x_{i})<P(\text{{pred}})\implies F_{i}=1;\neg
P1(x_{i})\implies 0$ $\forall x_{i},P2(x_{i}):c(x_{i})\leq
P(\text{{pred}})\implies C_{i}=0;\neg P2(x_{i})\implies 2$
$\displaystyle\forall
x_{i},P3(x_{i}):v(x_{i})\in\\{\text{'Happiness'},\text{'Neutral'}\\}\implies
V_{i}=0;$ $\displaystyle\neg
P3(x_{i}):v(x_{i})\in\\{\text{'Sadness'},\text{'Anger'}\\}\implies V_{i}=1$

where $P1(x_{i})$, $P2(x_{i})$, and $P3(x_{i})$ denote propositions for each
unimodal feature, respectively. Estimating suitable weights for each feature
indicator is challenging. This study assigned weights to the indicators guided
by expert knowledge regarding churn modeling and superannuation fund business
intricacies. Therefore, implicit feature translation alignment, often
incorporated as an intermediate step in mapping data, is imperative for any
subsequent task.

###  III-F Hybrid fusion strategy

The proposed approach employed hybrid fusion to leverage complementary
knowledge from unimodal representations while ensuring logical coherence. This
incorporates model and decision-level fusion, creating a multi-level fusion
framework. We enforced constant fusion weight $C_{i}+F_{i}+V_{i}\geq 0$ and
$F_{i}=V_{i}$ by assuming fusion weights for FL and SER models had equal
influential contribution to enhance baseline model performance. Therefore, a
decision fusion $D_{i}$ was defined to integrate complementary information and
hence rank the churn risk as low, mid, or high. The proposed decision fusion
matrix $D_{i}$ can be expressed as

$\small D_{i}=\begin{bmatrix}I_{\text{{lowrisk}}}\\\ I_{\text{{midrisk}}}\\\
I_{\text{{highrisk}}}\end{bmatrix}=I*\begin{bmatrix}C_{1}+F_{1}+V_{1}\\\
C_{2}+F_{2}+V_{2}\\\ \vdots\\\ C_{i}+F_{i}+V_{i}\end{bmatrix}$ (2)

subject to fusion conditions

  * •

Low risk Churner:

$\small\begin{split}I_{\text{{lowrisk}}}=I(D_{i}=0)\cdot I(C_{i}=0)\cdot
I(F_{i}=0)\cdot I(V_{i}=0)\\\ +I(D_{i}=1)\cdot I(C_{i}=0)\cdot I(F_{i}=1)\cdot
I(V_{i}=0)\\\ +I(D_{i}=1)\cdot I(C_{i}=0)\cdot I(F_{i}=0)\cdot
I(V_{i}=1)\end{split}$

If $I_{\text{{lowrisk}}}=1$, customer is classified as low risk.

  * •

Mid risk churner:

$\small\begin{split}I_{\text{{midrisk}}}=I(D_{i}=2)\cdot I(C_{i}=2)\cdot
I(F_{i}=0)\cdot I(V_{i}=0)\\\ +I(D_{i}=2)\cdot I(C_{i}=0)\cdot I(F_{i}=1)\cdot
I(V_{i}=1)\\\ \end{split}$

If $I_{\text{{midrisk}}}=1$, customer is classified as mid risk.

  * •

High risk churner:

$\small\begin{split}I_{\text{{highrisk}}}=I(D_{i}=4)\cdot I(C_{i}=2)\cdot
I(F_{i}=1)\cdot I(V_{i}=1)\\\ +I(D_{i}=3)\cdot I(C_{i}=2)\cdot I(F_{i}=1)\cdot
I(V_{i}=0)\\\ +I(D_{i}=3)\cdot I(C_{i}=2)\cdot I(F_{i}=0)\cdot
I(V_{i}=1)\end{split}$

If $I_{\text{{highrisk}}}=1$, customer is classified as high risk.

Where $D_{i}\in\\{0,4\\}$ is a fusion output weight that embodies ranked risk
for each member $i$. $C_{i}$, represents the fundamental customer churn
understanding based on historical patterns; $F_{i}$ is quantitative financial
behavior understanding, i.e., customer FL; $V_{i}$ is a qualitative measure,
reflecting customer emotional state and satisfaction level derived from their
phone communication with call center; $I(\cdot)$ is an indicator function
where $I(\cdot)=1$ if the condition in the brackets is true and 0 otherwise.

Thus, the logical operators are designed such that exactly one of
$I_{\text{{lowrisk}}}$, $I_{\text{{midrisk}}}$, and $I_{\text{{highrisk}}}$
should = 1 for a given customer risk level query. This ensures each customer
is classified into exactly one risk category.

The synergistic integration occurs when these modalities fuse distinct
learnings to generate a multimodal co-learning allows a coordinated
representation of two modalities FL and SER influence complementary the
training of another. Thus, this co-learning strategy mitigates the limitations
of relying on a single source of information, thereby enhancing the model’s
robustness.

##  IV Experiments

###  IV-A Data Sources

We employed real-world data from multiple sources to evaluate the proposed
approach efficiency, including customer historic data from finance networks,
demographic data from CRM platforms, and voice data, as shown in Table II.

Customer voices were replaced with similar voice from standard emotion
databases to maintain privacy while preserving emotional content in the
recorded voice files from inbound calls. The proposed method built a CV
database leveraging the correlation between negative emotions and high risk
churn customers with low financial literacy, as well as the association
between positive emotions and low risk churn customers. The EMODB database, a
standardized resource for categorizing emotions based on voice recordings, was
utilized to label customer emotions [14]. The FL database was built by
combining financial network data and an online customer survey to measure FL
level. After initial data processing, training data was enhanced to (4154,
140). Table II shows the various data sources and their specification.

TABLE II: Databases Description Sub | Specification
---|---
Database | No. datasets | Sources | Attributes / Size
SDB1 | 1 | Financial Networks | 68 / 64K
SDB2 | 12 | CRM | 87 / 294m
SDB3 | 1 a | Audio | 4 / 110min. voice
aEMODB database

###  IV-B Evaluation criteria

The mean average precision (MAP) metric was used to objectively evaluate the
proposed method solving quality for churn ranking,

$\small\text{{MAP}}=\frac{1}{Q}\sum_{q=1}^{Q}\frac{1}{m_{q}}\sum_{k=1}^{n}P(k)\cdot
rel(k)$ (3)

where $Q$ is the total number of queries; $q$ is the current query, $m\\_q$ is
the relevant churn risk level for the $q$-th query; $P(k)$ is the precision at
cut-off $k$ in the list; $rel(k)$ is an indicator function such $rel(k)=1$ if
the churn risk at rank $k$ is a relevant document, and $rel(k)=0$ otherwise.
We calculated MAP by taking the mean of average precision (AP) across all risk
levels (low, mid, high) for queries or instances. MAP = 1 means the system can
find all the churn risk levels accurately. The AP was computed for an
individual customer risk rank query as follows.

  * -

Calculate the precision for each relevant risk rank retrieved, i.e., the
proportion of relevant retrieved risk ranks up to that point, using constant
weight for each risk rank.

  * -

Calculate the mean AP across all queries. This provides an overall measure for
system performance across a risk rank query set.

We also employed the macro-averaged F1 score (MA-F1), since the datasets were
typically considerably imbalanced. It was imperative to evaluate smaller class
performance with equivalent weighting to larger classes, and F1 score can be
calculated independently for each class,

$\small\text{Macro-Averaged F1 Score}=\frac{1}{N}\sum_{i=1}^{N}F1\text{
Score}_{i}$ (4)

where $F1-score_{i}$ indicates F1 for the $i_{th}$ class and N is the number
of classes.

###  IV-C Correlation analysis

Figure 2(a) shows the linear relationship between modalities and the proposed
multimodal model output. Thus, the proposed financial literacy and churn
unimodal exhibits approximately balanced influences on model performance;
whereas the SER unimodal exhibits strong positive correlation = 0.66,
confirming significant impact on the multimodal model performance.

Figure 2: a) Strong positive relationship between customer emotion recognition
and multimodal learning performance. b) Relation of customer emotion feedback
and financial literacy level with customer tenure.

Figure 2(b) shows weightings for low FL and negative emotions (large circles).
Thus, clients who renewed their accounts annually exhibited high FL and
positive feedback; whereas customers with negative feelings did not renew
their accounts or exited.

###  IV-D Validation and comparison

To validate the effectiveness of the proposed method, we investigated the
impact of modality integration on performance. Various evaluation metrics were
used, including test accuracy, recall, F1-score, AUC, MAP, and MA-F1. The
Figure 3 (a) shows the highest performance is achieved in hybrid fusion method
with contribution of all modalities (FL+Churn+SER) with test accuracy of
91.2%.

Moreover, as shown in Figure 3(a) FL has the greatest impact on performance
compared with SER. Figure 3(b) shows that the hybrid fusion approach improved
churn risk distribution by shifting customers from low into mid and high risk
classes.

Figure 3: (a) Multimodal model performance is significantly improved by
combining early and late fusion (i.e., hybrid fusion) considering all
features; (b) Risk levels for various multimodal fusion methods, more
customers identified as mid and high risk using the proposed hybrid fusion
method.

Figure 4 compares ROC curves for late and hybrid fusion. Higher AUC in Fig. 4
(b) confirms the advantage of integrating multiple modalities, leveraging
multi-level fusion benefits.

Figure 4: Hybrid fusion exhibits higher AUC than other methods considered.

Table III compares risk identification accuracy using MAP and MA F1 metrics.
Multimodal learning with hybrid fusion strategy achieved significantly
improved results; MAP score = 66 and MA-F1 = 54.

TABLE III: Metrics for the different fusion methods considered. Metricsa | Fusion Method
---|---
None | DF (without LF) | HF (LF+DF)
MAP % ± STD | 51 ± 0.8 | 65 ± 0.7 | 66 ± 0.1
MA F1 % ± STD | 47 ± 0.1 | 47 ± 1.1 | 54 ± 0.6
aA higher value implies superior result

Python code for our proposed framework with further result visualization is
available on our GitHub
repository111https://github.com/DavidHason/multimodal$\\_$churn$\\_$model.
This resource simplifies the reproduction and enhancement of our study’s
experimental results.

##  V Discussion

This study proposed a multimodal fusion learning model for predicting customer
churn. To the authors’ best knowledge, this for the first time an attempt to
integrate diverse modalities including customer’s voice (CV), financial
literacy (FL) survey, and CRM record data to predict churn risk at three
levels (low, mid, and high). These modalities provide distinct insights,
thereby contributing to improved churn prediction precision.

The FL model utilized a SMOGN-COREG supervised model to infer customer FL from
their financial behavior data. The baseline churn model, developed on a robust
SMOTE and ensemble ANN algorithm combination, accurately predicted churn
propensity in the context of massive and high dimensionality data. The
proposed SER model employed a pre-trained CNN-VGG16 model to discern customer
emotions from their vocal attributes, providing an additional layer to help
understand customer behavior. Our findings highlight significant correlations
between negative emotions, low FL, and increased churn risk. Furthermore,
comparative analysis underscored the superiority of the hybrid fusion
technique, achieving a MAP score of 66 and a test accuracy of 91.2%, thus
outperforming both non-fusion and late fusion approaches.

##  VI Conclusion and future works

Despite the promising results, the proposed approach has limitations; in
particular, the coordinated representation in the multimodal fusion method
could lose valuable intermodal information. Future study will explore a joint
representation approach, concatenating features from different modalities
before commencing learning, providing the model with a broader, richer data
set. This could enhance the learning process efficacy, particularly when
employing a powerful model such as Deep ANN.

The second area of potential future work involves extending the multimodal
approach by integrating textual features as a fourth modality. It can be
represented as $t(x_{i})$ a textual learning unimodal and embedded into
existing coordinated representation space which is expressed in equation (1).
This would broaden the model’s scope and enhance the multifaceted analyses of
customer behavior.

## Acknowledgment

This work is partially supported by the Australian Research Council under
grant number: DP22010371, LE220100078, DP200101374 and LP170100891

## References

  * [1] Chen, Z., & Fan, Z. (2011, June). Building comprehensible customer churn prediction models: a multiple kernel support vector machines approach. In ICSSSM11 (pp. 1–4). IEEE.
  * [2] Lemmens, A., & Gupta, S. (2020). Managing churn to maximize profits. Marketing Science, 39(5), 956–973.
  * [3] Author names removed
  * [4] De Caigny, A., Coussement, K., De Bock, K. W., & Lessmann, S. (2020). Incorporating textual information in customer churn prediction models based on a convolutional neural network. International Journal of Forecasting, 36(4), 1563-1578.
  * [5] Vo, N. N., Liu, S., Brownlow, J., Chu, C., Culbert, B., & Xu, G. (2018). Client churn prediction with call log analysis. In Database Systems for Advanced Applications: 23rd International Conference, DASFAA 2018, Gold Coast, QLD, Australia, May 21-24, 2018, Proceedings, Part II 23 (pp. 752-763). Springer International Publishing.
  * [6] Kimura, T. (2022). CUSTOMER CHURN PREDICTION WITH HYBRID RESAMPLING AND ENSEMBLE LEARNING. Journal of Management Information & Decision Sciences, 25(1) (pp. 1 – 23).
  * [7] Ahn, J., Hwang, J., Kim, D., Choi, H., & Kang, S. (2020). A survey on churn analysis in various business domains. IEEE Access, 8, 220816-220839.
  * [8] Author names removed
  * [9] Oechssler, J., Roider, A., & Schmitz, P. W. (2009). Cognitive abilities and behavioral biases. Journal of Economic Behavior & Organization, 72(1), 147-152.
  * [10] Author names removed
  * [11] Baltrušaitis, T., Ahuja, C., & Morency, L. P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2), 423-443.
  * [12] Chu, S., Narayanan, S., & Kuo, C. C. J. (2009). Environmental sound recognition with time–frequency audio features. IEEE Transactions on Audio, Speech, and Language Processing, 17(6), 1142-1158.
  * [13] Branco, P., Torgo, L., & Ribeiro, R. P. (2017, October). SMOGN: a pre-processing approach for imbalanced regression. In First international workshop on learning with imbalanced domains: Theory and applications (pp. 36-50).
  * [14] Gopal, P., & MohdNawi, N. B. (2021, December). A Survey on Customer Churn Prediction using Machine Learning and data mining Techniques in E-commerce. In 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (pp. 1-8). IEEE.
  * [15] Kristensen, J. T., & Burelli, P. (2019, August). Combining sequential and aggregated data for churn prediction in casual freemium games. In 2019 IEEE Conference on Games (CoG) (pp. 1-8). IEEE.
  * [16] Guitart, A., Chen, P. P., & Periáñez, Á. (2018). The winning solution to the IEEE CIG 2017 game data mining competition. Machine Learning and Knowledge Extraction, 1(1), 252-264.
  * [17] Ullah, I., Raza, B., Malik, A. K., Imran, M., Islam, S. U., & Kim, S. W. (2019). A churn prediction model using random forest: analysis of machine learning techniques for churn prediction and factor identification in telecom sector. IEEE access, 7, 60134-60149.
  * [18] Liu, R., Ali, S., Bilal, S. F., Sakhawat, Z., Imran, A., Almuhaimeed, A., … & Sun, G. (2022). An intelligent hybrid scheme for customer churn prediction integrating clustering and classification algorithms. Applied Sciences, 12(18), 9355.
  * [19] de Lima Lemos, R. A., Silva, T. C., & Tabak, B. M. (2022). Propension to customer churn in a financial institution: A machine learning approach. Neural Computing and Applications, 34(14), 11751-11768.