# Long-Term Typhoon Trajectory Prediction: A Physics-Conditioned Approach
Without Reanalysis Data

Young-Jae Park *1, Minseok Seo*2, Doyi Kim2, Hyeri Kim2, Sanghoon Choi3,
Beomkyu Choi2,
Jeongwon Ryu2, Sohee Son2, Hae-Gon Jeon1, Yeji Choi2
1 Gwangju Institute of Science and Technology, 2 SI Analytics, 3 The
University of Manchester
Y.-J. Park and M.-S. Seo provided equal contributions to this work.

###### Abstract

In the face of escalating climate changes, typhoon intensities and their
ensuing damage have surged. Accurate trajectory prediction is crucial for
effective damage control. Traditional physics-based models, while
comprehensive, are computationally intensive and rely heavily on the expertise
of forecasters. Contemporary data-driven methods often rely on reanalysis
data, which can be considered to be the closest to the true representation of
weather conditions. However, reanalysis data is not produced in real-time and
requires time for adjustment because prediction models are calibrated with
observational data. This reanalysis data, such as ERA5, falls short in
challenging real-world situations. Optimal preparedness necessitates
predictions at least 72 hours in advance, beyond the capabilities of standard
physics models. In response to these constraints, we present an approach that
harnesses real-time Unified Model (UM) data, sidestepping the limitations of
reanalysis data. Our model provides predictions at 6-hour intervals for up to
72 hours in advance and outperforms both state-of-the-art data-driven methods
and numerical weather prediction models. In line with our efforts to mitigate
adversities inflicted by typhoons, we release our preprocessed PHYSICS TRACK
dataset, which includes ERA5 reanalysis data, typhoon best-track, and UM
forecast data.

##  1 INTRODUCTION

As global warming accelerates, the intensity of typhoons is on the rise (Walsh
et al., 2016; IPCC, 2023). Accurate typhoon trajectory prediction is of
paramount importance to allow enough time for emergency management and to
organize evacuation efforts. This warning period is more critical in countries
that lack sufficient infrastructure to forecast typhoons. Many nations utilize
numerical weather prediction (NWP) models for typhoon trajectory forecasting
and base their typhoon preparedness measures on this forecasting information.
NWP models make predictions for atmospheric variables like geopotential
height, wind vectors, and temperature using atmospheric governing equations.
Forecasters employ the specific characteristics of typhoons in their NWP
output to infer the typhoon trajectory.

However, interpreting the outputs of NWP models depends on the expertise of
the forecasters, as well as the use of other specialized tracking algorithms,
such as the European Centre for Medium-Range Weather Forecasts (ECMWF) Tracker
(Bi et al., 2023; Lam et al., 2022). To address these issues, data-driven
typhoon trajectory prediction methods have been proposed (Huang et al., 2023;
Rüttgers et al., 2019). Additionally, methods to generate reanalysis data for
applying ECMWF Tracker (ECMWF, 2021) have also emerged (Pathak et al., 2022;
Espeholt et al., 2022; Lam et al., 2022; Nguyen et al., 2023). Yet, they
possess the following limitations: Weather forecasting typically uses the
ECMWF ReAnalysis-v5 (ERA5) (Dee et al., 2011) data for training and inference.
However, ERA5 data goes through post-correction processes, including data
assimilation based on NWP forecast data. As a result, although ERA5 data shows
high accuracy, it is not accessible until 3-5 days after the typhoon.

Consequently, they are unsuitable for real-time tasks like typhoon trajectory
prediction. In addition, existing data-driven real-time trajectory prediction
models (Huang et al., 2022) restrict their forecasts to less than 24 hours.
This considerably shorter prediction window, compared to the NWP models, makes
them incapable of planning effective damage control for typhoons.

For data-driven typhoon trajectory prediction, the dataset known as Best Track
(Knapp et al., 2010) has been a cornerstone. Documenting the intensity and
central points of typhoons, its records extend back to 1950. Correspondingly,
the ERA5 reanalysis data, which provides invaluable insights as an additional
physics-conditioned dataset, is accessible for the same time range. In
contrast, the Unified Model (UM) (Brown et al., 2012) dataset is produced in
near-real time, including geopotential height, wind vectors, and other
atmospheric variables using primitive equations such as the conservation of
momentum, mass, energy, and water mass. However, the UM dataset inherently has
several limitations.

: The UM data records only include data from 2010 to the present. Furthermore,
while the UM dataset adheres to physical formulas, it does not guarantee an
exact representation of real-world values and can have potential errors (Brown
et al., 2012). Figure 1 and Table 1 compare the UM dataset with ERA5. As
illustrated, the UM data significantly differs from ERA5 in several areas.
Meanwhile, as Table 1 reveals, while the UM data has only a 3-hour data
acquisition delay, it exhibits larger errors.

In this study, we address these issues by introducing a Long-Term Typhoon
Trajectory Prediction method (LT3P). LT3P primarily consists of two
components: a physics-conditioned encoder and a trajectory predictor. The
first component, the physics-conditioned encoder, encodes influential
variables representing the typhoon’s trajectory and intensity. In this initial
phase, we harness three atmospheric variables—geopotential height, zonal, and
meridional wind—across three pressure levels (700, 500, and 250 hPa) sourced
from the ERA5 dataset. After this, all network parameters are frozen. The next
step exclusively focuses on the bias corrector from UM data to ERA5. This
correction process is fine-tuned on UM’s geopotential height and wind vector
data from pressure levels 250, 500, and 700 — available since 2010 — combined
with the Best Track dataset. The second component, the trajectory predictor,
accepts the typhoon’s central coordinates as input and forecasts its
trajectories for the future 72 hours. Features from the physics-conditioned
encoder are cross-attended within this trajectory predictor, facilitating
precise physics-conditioned typhoon trajectory predictions.

Three primary contributions can be summarized:

  * •

We propose, for the first time, a real-time +72 hours typhoon trajectory
prediction model without reanalysis data.

  * •

We provide the preprocessed dataset PHYSICS TRACK, and training, evaluation,
and pretrained weights of LT3P.

  * •

In the +72 hours forecast, we achieved state-of-the-art results, outperforming
the NWP-based typhoon trajectory forecasting models by significant margins.

Figure 1: Visualization and comparison of the UM and ERA5 data at pressure
levels of 250, 500, and 700 hPa, along with a difference map of two datasets.
The UM forecast data is at a lead time of +72 hours, and the ERA5 data
corresponds to that forecasted time. This analysis covers the Western North
Pacific basin, with latitudes ranging from 0 to 59.75°N and longitudes from
100°E to 179.75°E. Table 1: Comparison of ERA5 and NWP datasets: Highlighting
the trade-off between real-time data availability and forecast accuracy. While
ERA5 boasts high accuracy, it lacks immediacy. In contrast, NWP provides real-
time data but may contain forecasting errors.

| Type | Data Availability | Real-time | Limitations | Forecast Accuracy
---|---|---|---|---|---
ERA5 | Reanalysis Dataset | 1950 - Present | No | Time delay for data availability | $\approx 95\%$
UM | NWP Model | Approx. 2010 - Present | Yes | Potential for model errors in forecasting | Low
IFS | NWP Model | 1974 - Present | Yes | Potential for model errors in forecasting | High
GFS | NWP Model | 2015 - Present | Yes | Potential for model errors in forecasting | Middle

##  2 RELATED WORK

Typhoon trajectory prediction and human trajectory prediction are closely
related in that they focus on modeling future trajectories. Unlike the
trajectories of general robots and vehicles (Jiang et al., 2023), typhoon
trajectories have high uncertainty because the direction and speed of movement
are not deterministic (Golchoubian et al., 2023). These characteristics are
similar to pedestrian trajectory prediction (Gu et al., 2022) in that the
direction and speed of pedestrian movement are not determined and its
uncertainty is high.

However, while human predictions focus on capturing social interactions
between agents, typhoon predictions consider overall atmospheric and
geoscientific variables. In this section, we explore a typhoon trajectory
prediction that takes into account the benefits of two distinct fields.

###  2.1 Trajectory Prediction

Trajectory prediction has primarily evolved from a focus on human trajectory
prediction (Alahi et al., 2016; Gupta et al., 2018; Huang et al., 2019;
Mangalam et al., 2020; Salzmann et al., 2020; Gu et al., 2022), as it has been
extensively applied to human-robot interaction systems (Mavrogiannis et al.,
2023).

SocialGAN (Gupta et al., 2018) employs a Generative Adversarial Network (GAN)
framework for predicting realistic future paths. By leveraging latent space
vectors, it is able to generate various outputs that capture the intricate
nuances of socially acceptable and multimodal human behaviors. PECNet
(Mangalam et al., 2020) offers a distinct conditional VAE-based approach by
transforming the latent spaces. It aims to strike a balance between the
fidelity and diversity of predicted samples through its truncation trick in
the latent space. Trajectron++ (Salzmann et al., 2020) provides a holistic
methodology, predicting the probability of a latent distribution during
inference. This robustness arises from its ability to adapt distribution
parameters based on external input, such as prior trajectory data. Graph
Neural Network (GNN)-based approaches (Huang et al., 2019; Mohamed et al.,
2020; Shi et al., 2021) have attracted interest in modeling and predicting
trajectories in scenarios with dense populations, such as pedestrian
movements. Recently, MID (Gu et al., 2022) proposes a high-performing
trajectory prediction method. This method enhances the conventional trajectory
prediction structure—which previously consisted only of long short-term memory
(LSTM), multi-layer perceptron (MLP), and GNN—by integrating Transformer and
diffusion processes.

However, when human trajectory prediction methods are applied to typhoon
trajectory prediction, the results typically show poor performance (Huang et
al., 2022; 2023). Unlike pedestrian scenes with multiple interacting entities,
typhoons usually appear singly or in pairs at most. This difference implies
that the detailed social modeling crucial for pedestrian predictions might be
less important in the case of typhoons. While GNN techniques excel in complex
interactive settings, their efficacy is reduced by the inherent nature of
typhoons.

###  2.2 Typhoon Trajectory Prediction

Forecasting the trajectory of typhoons is still a complex task, affected by
many variables including weather conditions, sea surface temperatures, and
topography (Emanuel, 2007). As studies have progressed, forecasting methods
have improved, moving from basic statistical models to more advanced
contemporary numerical approaches (Wang et al., 2015; Chen & Zhang, 2019).
Particularly, NWP systems, which leverage an abundance of meteorological data,
have emerged as fundamental tools for typhoon trajectory prediction. However,
the interpretation of NWP model outputs often requires the expertise of
meteorologists or relies on complex typhoon trajectory prediction algorithms.

On the other hand, data-driven approaches (Pathak et al., 2022; Espeholt et
al., 2022; Lam et al., 2022; Nguyen et al., 2023) demonstrate notable
performance in the fields of climate science, suggesting their potential
applicability in forecasting typhoon trajectories. Initial attempts integrate
Recurrent Neural Networks (RNNs) with specific data types but often lead to
unsatisfactory results (Alemany et al., 2019). Leveraging a broader range of
meteorological data has enhanced prediction accuracy, even though this
introduces challenges in data processing and encoding (Giffard-Roisin et al.,
2020). Notably, the recent leading approaches in typhoon trajectory prediction
emphasize multi-trajectory forecasts, enabled through GANs (Huang et al.,
2022; Dendorfer et al., 2021)

Despite these efforts, data-driven models still exhibit limitations,
particularly with respect to their long-term forecast performance (Chen et
al., 2020; Huang et al., 2023) and real-time. In this context, our work aims
to pioneer long-term predictions by integrating physical values from UM data,
which are outputs from a real-time-capable NWP model, with the power of data-
driven methods.

##  3 METHOD

The LT3P model takes the typhoon center coordinates
$C_{o}:=\\{c\\}_{i=1}^{t_{o}}=\\{x_{\text{lon}},y_{\text{lat}}\\}_{i=1}^{t_{o}}$,
where $t_{o}$ represent the input time sequences, while $x_{\text{lon}}$ and
$y_{\text{lat}}$ denote the longitude and latitude. Here, $i$ represents the
index in the time sequence. In LT3P, the inputs are the geopotential height
and wind vector from the UM, represented as $\Phi_{P}=\\{z_{p}\\}_{p\in
P,i=t_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}o}}+1}^{t_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}o}}+t_{f}}$
and $\vec{W}_{P}=\\{w_{p}\\}_{p\in
P,i=t_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}o}}+1}^{t_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}o}}+t_{f}}$,
respectively. Here, $P$ refers to the pressure levels and we use it as
$P=\\{250,500,700\\}\text{hPa}$. $t_{f}$ represent the output time sequences.
LT3P then outputs the future typhoon center coordinates
$C_{f}:=\\{u\\}_{i=t_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}o}}+1}^{t_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}o}}+t_{f}}=\\{x_{lon},y_{lat}\\}_{i=t_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}o}}+1}^{t_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}o}}+t_{f}}$.

Hence, LT3P comprises a physics-conditioned model which extracts the
representation from $\Phi$ and $\vec{W}$, and a trajectory predictor that
forecasts the typhoon trajectory. LT3P is optimized through a dual-branch
training strategy. During a pre-trained phase, the physics-conditioned model
is trained using ERA5 as in (Nguyen et al., 2023). Subsequently, in a bias
correction phase, the trajectory predictor is trained to produce outputs akin
to ERA5, even when UM data is used as the input.

###  3.1 Preliminaries

The NWP forecasting system primarily hinges on a core prediction formula for
the geopotential height $\Phi$ and the wind vector $\vec{W}$ based on
conservation laws:

$\frac{\partial}{\partial
t}\begin{bmatrix}\Phi,&\vec{W},&\rho,&T,&q\end{bmatrix}=\mathcal{F}\left(\vec{W},\Phi,\rho,T,q\right),$
(1)

where $\mathcal{F}$ refers to interactions among variables following the
conservation laws. The variables $\rho,T,$ and $q$ denote air density,
temperature, and specific humidity, respectively.

Conservation equations underpin the NWP’s forecasting. For instance, a
momentum conservation with the spatial gradient $\nabla$ is expressed by:

$\frac{\partial\vec{W}}{\partial t}=-\vec{W}\cdot\nabla\vec{W}\frac{\nabla
P}{\rho}+\vec{g},$ (2)

and a mass conservation is below:

$\frac{\partial\rho}{\partial t}+\nabla\cdot(\rho\vec{W})=0.$ (3)

Other conserved quantities include energy, related to geopotential height
$\Phi$ and temperature $T$, and water vapor content or specific humidity $q$.

Note on Accuracy and Time $t$. In the NWP system, a time $t$ indicates
progression. For longer predictions up to +72 hours, their accuracy depends on
various factors: initial conditions, physics parameterization schemes, spatial
resolution, and so on. While the conservation law captures large-scale
atmospheric dynamics effectively, it might not work on small-scale details due
to local conditions like microclimates, terrain variations, or transient
phenomena.

The Earth’s atmosphere is a chaotic system (Lorenz, 1963). Despite relying on
conservation laws, NWP models are inevitably prone to errors due to the
inherent imperfections in our understanding of physical processes and the true
state of the atmosphere. These discrepancies underscore the importance of
data-driven approaches, in spite of the potential error in the UM data.

###  3.2 Pre-Training on ERA5 Dataset

Figure 2: Overview of LT3P. In phase 1, the physics-conditioned model
$R(\cdot)$ is trained for the weather forecasting task to encode information
from $\Phi$, $\vec{W}_{u}$, and $\vec{W}_{v}$. Subsequently, in phase 2, using
UM as the input, the typhoon trajectory prediction model $D(\cdot)$ is trained
in conjunction with the bias-corrector $B(\cdot)$ to make accurate
predictions.

The ERA5 dataset, accumulated at 6-hour intervals from 1950 to the present,
requires a significant amount of time to train a model from scratch. To
address this challenge, we have adopted a strategy of constructing a
foundational model for typhoon trajectory prediction.

The physics-conditioned model of LT3P is trained for a weather forecasting
task, similar to (Nguyen et al., 2023), to effectively learn the
representations of the highly-related variables for typhoon trajectory
prediction in (Giffard-Roisin et al., 2020): $\Phi$, $\vec{W}_{u}$, and
$\vec{W}_{v}$.

As shown in Figure 2, the physics-conditioned model consists of an encoder-
transformer-decoder architecture. The encoder and decoder are built using 3D
convolutions for spatio-temporal modeling, and the transformer is also
constructed as a 3D transformer.

The objective of the physics-conditioned model is as follows:

$L_{pre\\_train}=\frac{1}{T\times V\times H\times
W}\sum_{i=1}^{T}\sum_{j=1}^{V}\sum_{k=1}^{H}\sum_{l=1}^{W}(\tilde{X}_{t_{f}}^{i,j,k,l}-X_{t_{f}}^{i,j,k,l})^{2},$
(4)

where $T$ is the final lead time, $V$ represents 9 variables, with
$\Phi,\vec{W}_{u}$, and $\vec{W}_{v}$ for each of the 3 pressure levels. $H$
and $W$ denote height and width, respectively. $\tilde{X}$ is the forecasting
value for ERA5 from time $t$ to $t+t_{f}$ in the physics-conditioned model,
given $V$ as input from the period $t-t_{f}$ to $t$. $X$ is its ground truth
(GT).

###  3.3 Bias correction & Typhoon Trajectory Prediction

Bias correction. Since ERA5 is reanalysis data, this limits the applicability
on real-time tasks, which makes us use the UM data. Hence, the typhoon
trajectory prediction model, LT3P, accepts two primary inputs: the
representation encoded by the $R(\cdot)$ —derived from the physics formula
presented in Section. 3.1 using $\Phi$ and $\vec{W}$—and the typhoon’s center
coordinates $C_{o}$. The model then produces the output $C_{f}$.

However, since our $R(\cdot)$ was trained with the ERA5 data, it may struggle
with extracting meaningful representations from the UM data due to their
domain gap. To solve this problem, LT3P uses the bias-corrector $B(\cdot)$ to
adapt the variables generated through UM to the variables of ERA5. The
$B(\cdot)$ has the following objective functions:

$L_{Bias\\_correction}=\frac{1}{T\times V\times H\times
W}\sum_{i=1}^{T}\sum_{j=1}^{V}\sum_{k=1}^{H}\sum_{l=1}^{W}(\hat{X}_{t_{f}}^{i,j,k,l}-X_{t_{f}}^{i,j,k,l})^{2},$
(5)

where $\hat{X}$ is the ERA5 prediction which is bias-corrected version of the
UM dataset.

Typhoon Trajectory Prediction. In the second phase, LT3P takes ERA5 data from
1950 to 2010 and UM data from 2010 to 2018 as inputs in order to perform
typhoon trajectory prediction. The main model of LT3P, which predicts the
center coordinates of the typhoon, generates the final trajectory by applying
a cross-attention between the 3D features from the $R(\cdot)$’s transformer
and the 1D features of the typhoon coordinates from the LT3P model’s encoder.
The cross attention enables LT3P to take fully advantage of both data-driven
and physics-based prediction. It also allows us to fuse these two features
regardless of the shape or length of the features. The cross attention is
defined as:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)\cdot
V,$

where $Q=W^{(i)}_{Q}\cdot\phi_{i}(R(\Phi,\vec{W}))$,
$K=W^{(i)}_{K}\cdot\tau_{\theta}(C_{p})$, and
$V=W^{(i)}_{V}\cdot\tau_{\theta}(C_{p})$. The $\theta$ represents the set of
parameters associated with the $D(\cdot)$. Here,
$\phi_{i}(z_{t})\in\mathbb{R}^{N\times d_{i}}$ denotes a (flattened)
intermediate representation of the transformer implementing $\theta$.
$W^{(i)}_{V}\in\mathbb{R}^{d\times d_{i}}$, $W^{(i)}_{Q}\in\mathbb{R}^{d\times
d_{\tau}}$, and $W^{(i)}_{K}\in\mathbb{R}^{d\times d_{\tau}}$ are learnable
projection matrices.

Therefore, in LT3P, the final objective function is defined as follows:

$L_{total}=L_{Bias\\_correction}+L_{trajectory}.$ (6)

Note that various trajectory prediction baselines such as GAN, CVAE, and
diffusion-based models can be applied, whose effectiveness will be validated
in section 4.3.

Table 2: Ensemble typhoon trajectory prediction results (FDE): Note that while
the original MGTCF study only includes 24-hour predictions, for comparison
purposes, we modified it to input 48 hours at 6-hour intervals and predict +72
hours at 6-hour intervals. Bold indicates the best performance, and underline
indicates the second-best performance. Note that Dataset refers to the dataset
used for inference.

Method | Real-time | Dataset | Distance (km)
---|---|---|---
BST | ERA5 | UM | 6 h | 12 h | 18 h | 24 h | 30 h | 36 h | 42 h | 48 h | 54 h | 60 h | 66 h | 72 h
SocialGAN (Gupta et al., 2018)-Ens | $\checkmark$ | $\checkmark$ | - | - | 33.16 | 63.25 | 93.33 | 131.31 | 176.71 | 226.24 | 288.78 | 353.79 | 431.03 | 519.93 | 611.67 | 716.90
STGAT (Huang et al., 2019)-Ens | $\checkmark$ | $\checkmark$ | - | - | 56.68 | 100.51 | 160.08 | 209.83 | 269.81 | 325.44 | 377.86 | 435.69 | 492.44 | 550.37 | 613.97 | 682.16
PECNet (Mangalam et al., 2020)-Ens | $\checkmark$ | $\checkmark$ | - | - | 41.61 | 56.74 | 82.26 | 112.28 | 147.96 | 185.92 | 232.79 | 279.29 | 330.84 | 388.25 | 445.61 | 518.46
MID (Gu et al., 2022)-Ens | $\checkmark$ | $\checkmark$ | - | - | 109.31 | 173.35 | 230.45 | 289.62 | 359.10 | 422.11 | 490.93 | 564.23 | 638.04 | 712.97 | 798.63 | 881.34
MMSTN (Huang et al., 2022)-Ens | $\checkmark$ | $\checkmark$ | - | - | 49.72 | 109.23 | 173.64 | 246.26 | 334.62 | 430.45 | 552.15 | 667.34 | 819.22 | 969.13 | 1126.26 | 1300.59
MGTCF (Huang et al., 2023)-Ens | $\times$ | $\checkmark$ | $\checkmark$ | - | 46.12 | 81.68 | 116.81 | 155.72 | 201.26 | 249.42 | 302.19 | 361.14 | 427.35 | 499.85 | 576.80 | 655.48
JTWC (Chen et al., 2023) | $\checkmark$ | - | - | - | - | - | - | 87.09 | - | - | - | 146.45 | - | - | - | 217.05
JMA-GEPS (Chen et al., 2023) | $\checkmark$ | - | - | - | - | 62.83 | - | 98.10 | - | 138.26 | - | 180.04 | - | 222.75 | - | 259.07
ECMWF-EPS (Chen et al., 2023) | $\checkmark$ | - | - | - | - | 57.75 | - | 78.87 | - | 108.28 | - | 138.97 | - | 176.02 | - | 210.71
NCEP-GEFS (Chen et al., 2023) | $\checkmark$ | - | - | - | - | 52.34 | - | 76.17 | - | 105.55 | - | 142.26 | - | 185.36 | - | 225.59
UKMO-EPS (Chen et al., 2023) | $\checkmark$ | - | - | - | - | 62.74 | - | 90.90 | - | 122.90 | - | 160.41 | - | 200.89 | - | 245.28
LT3P (UM Only)-Ens | $\checkmark$ | $\checkmark$ | - | $\checkmark$ | 42.20 | 58.83 | 88.39 | 114.84 | 147.39 | 189.03 | 220.41 | 255.57 | 320.47 | 334.88 | 370.64 | 390.92
LT3P (Bias-corrected UM)-Ens | $\checkmark$ | $\checkmark$ | - | $\checkmark$ | 6.30 | 19.45 | 22.83 | 39.81 | 50.29 | 60.76 | 65.38 | 70.01 | 83.98 | 100.88 | 110.36 | 143.03
LT3P (ERA5 Only)-Ens | $\times$ | $\checkmark$ | $\checkmark$ | - | 5.29 | 6.80 | 12.34 | 19.90 | 24.09 | 30.42 | 35.92 | 39.29 | 44.83 | 50.30 | 60.11 | 70.94

  * *

Note that best track data from MMSTN and MGTCF could not be obtained because
they use data from specific organizations.
Therefore, the experimental results of each of the two studies were evaluated
only in 2019.

##  4 Evaluations

In this section, we first describe the dataset and the pre-processing methods
used as well as an implementation detail of our LT3P. We then show
quantitative and qualitative evaluations and provide an extensive ablation
study to check the effectiveness of each component in LT3P.

###  4.1 Experimental Settings

Physics Track dataset. We collect the center coordinates of typhoons at 6-hour
intervals (00, 06, 12, 18) from 1950 to 2021. We use data corresponding to the
Western North Pacific basin, with latitudes ranging from 0 to 59.75°N and
longitudes from 100°E to 179.75°E, as typhoons overwhelmingly occur within
this region. (Stull, 2011) Any typhoon that dissipated within 48 hours is
excluded from the dataset. The training data uses years 1950-2018 and
comprises 1,334 typhoons, while the test dataset covers years 2019-2021 and
includes 90 typhoons. Note that for hyperparameter tuning, we train on data
from 1950 to 2016 and set the 2017 to 2018 data as the validation set.
Subsequently, we evaluate the final performance of model using the entire
dataset from 1950 to 2018.

In sync with the duration of the typhoon, we utilize variables in the ERA5
dataset which are closely associated with typhoons, notably the geopotential
height and wind vectors. Data for the pressure levels at 250, 500, and 700 hPa
are collected. For training purposes, the ERA5 dataset is employed from 1950
to 2018, and the data spanning 2019, 2020, and 2021 is set aside. Regarding
the real-time data from the UM, we compile data from the years 2010 to 2021,
ensuring the inclusion of the same variables and pressure levels as found in
the ERA5 dataset.

Note: All variables in ERA5 and UM are normalized using the formula
$(\text{variable}-\text{mean})/\text{std}$ because no specific minimum-maximum
range is predetermined for each variable.

For both the ERA5 and UM datasets, information is extracted to match the
conditions of the best track. Therefore, the input dimension for ERA5 and UM
is set to $(B\times T\times V\times H\times W)\rightarrow(B\times 12\times
9\times 240\times 320)$. Note that the resolution has been adjusted using bi-
linear interpolation.

Evaluation metrics. We compare our LT3P with global meteorological agencies,
including the Joint Typhoon Warning Center (JTWC), Japan Meteorological Agency
(JMA), European Centre for Medium-Range Weather Forecasts (ECMWF), National
Centers for Environmental Prediction (NCEP), and the United Kingdom
Meteorological Office (UKMO). Additionally, the Global Ensemble Forecast
System (GEPS) refers to the results of 20 separate generated forecasts (Zhou
et al., 2022) and then combines them into an ensemble average forecast, which
is to average the sampled forecasting results. Therefore, the data-driven
models are evaluated using the ensemble average method to benchmark their
performance compared to the ensemble average NWP models. Performance is
measured at 6-hour intervals, consistent with existing meteorological
agencies.

We use (1) Average Displacement Error (ADE) - average Euclidean distance
between a prediction and ground-truth trajectory; (2) Final Displacement Error
(FDE) - Euclidean distance between a prediction and ground-truth final
destination.

Table 3: Results of Stochastic Typhoon Trajectory Predictions (FDE): This
table presents results generated from 20 samples, consistent with the
conventional approach in NWP-based GEPS (Zhou et al., 2022), which employs 20
ensemble members for the final prediction.

Method | Distance (km)
---|---
6 h | 12 h | 18 h | 24 h | 30 h | 36 h | 42 h | 48 h | 54 h | 60 h | 66 h | 72 h
SocialGAN (Gupta et al., 2018) | 17.59 | 30.30 | 40.97 | 50.30 | 64.40 | 71.63 | 88.00 | 97.81 | 110.49 | 126.92 | 144.66 | 164.99
STGAT (Huang et al., 2019) | 32.81 | 55.24 | 86.86 | 110.93 | 147.86 | 180.03 | 204.79 | 233.53 | 259.93 | 277.09 | 298.62 | 322.54
PECNet (Mangalam et al., 2020) | 28.40 | 34.34 | 45.88 | 57.36 | 69.44 | 81.17 | 97.84 | 112.34 | 130.22 | 150.18 | 169.22 | 188.74
MID (Gu et al., 2022) | 24.54 | 42.66 | 56.52 | 73.41 | 96.04 | 116.86 | 136.44 | 160.55 | 184.09 | 205.78 | 230.09 | 256.26
MMSTN (Huang et al., 2022) | 19.46 | 42.44 | 67.38 | 93.59 | 128.97 | 162.90 | 193.29 | 232.58 | 269.17 | 309.55 | 348.10 | 384.57
MGTCF (Huang et al., 2023) | 20.84 | 37.59 | 52.45 | 67.29 | 83.09 | 104.13 | 128.90 | 156.98 | 186.62 | 218.18 | 249.22 | 280.90
LT3P (Bias-corrected UM) | 1.97 | 2.31 | 3.32 | 8.80 | 17.89 | 25.53 | 30.57 | 36.39 | 41.12 | 47.38 | 51.87 | 65.24
LT3P (ERA5 Only) | 0.66 | 1.12 | 2.26 | 2.34 | 21.93 | 26.36 | 26.99 | 41.39 | 40.24 | 49.63 | 52.97 | 65.95

  * *

* Note that best track data from MMSTN and MGTCF could not be obtained because they use data from specific organizations.
Therefore, the experimental results of each of the two studies were evaluated
only in 2019.

Implementation details. The physics-conditioned model $R(\cdot)$ is comprised
of an encoder, translator and decoder. The encoder and decoder are composed of
(Conv, LayerNorm, SiLU) (Elfwing et al., 2018)) and (Conv, LayerNorm, SiLU,
PixelShuffle), respectively. In the experiments, the encoder and decoder
consist of four blocks each. The translator is selected as the transformer
block and has a total of three blocks. The trajectory predictor $D(\cdot)$
follows GAN (Gupta et al., 2018), CVAE (Mangalam et al., 2020) and diffusion
(Gu et al., 2022), respectively, and the experimental results in Table 2 and
Table 3 are obtained from the diffusion-based predictor.

For training, we use a machine with 8 Nvidia-Quadro RTX 8000 GPUs. The batch
size is set to 128, and we use Adam optimizer with a learning rate of 0.001.
Additionally, an Exponential Moving Average with a momentum of 0.999 is
applied, and the training is conducted for 2,000 epochs. Note that we did not
employ data augmentation, as the geometric and intensity values in typhoon
forecasting are extremely sensitive. Applying data augmentation might
compromise the integrity of physical rules. For fair comparison, the data-
driven baseline models such as SocialGAN (Gupta et al., 2018), STGAT (Huang et
al., 2019), PECNet (Mangalam et al., 2020), MID (Gu et al., 2022), and MMSTN
(Huang et al., 2022) are trained and evaluated on the best track dataset,
while MGTCF (Huang et al., 2023) utilizes both the best track and ERA5
datasets.

###  4.2 Experimental Results

#### Quantitative results.

In Table 2, we compare the performance of LT3P with the state-of-the-art human
trajectory prediction methods as well as data-driven typhoon prediction
methods. The table also includes forecast results from several renowned
meteorological agencies, including JTWC, JMA, ECMWF, NCEP, GEFS, and UKMO,
which are based on the conservation laws in Section. 3.2. Our observations
indicate that LT3P (ERA5 Only) exhibits the most promising results, with LT3P
(Bias-corrected UM) trailing closely behind. It is important to highlight that
MID, which has achieved the best results among data-driven human trajectory
predictors, performs poorly in ensemble forecasting. This discrepancy arises
from the inherent goal of human trajectory prediction models. They have
developed to capture social relations between agents, and its probabilistic
distribution attempts to cover a wide and diverse distribution to represent
all feasible socially-acceptable paths and destination.

Furthermore, it is noteworthy that our model consistently outperforms the
numerical prediction models of the established operational forecast centers.
These experimental results demonstrate the potential of real-time NWP data to
serve as an alternative to reanalysis data in the field of real-time typhoon
trajectory prediction. Note that studies in (Bi et al., 2023; Lam et al.,
2022), which predict reanalysis data and then use the ECMWF-tracker for
typhoon trajectory prediction, are excluded in our comparisons because code
and weights are unavailable, and there are no access to the ECMWF-tracker
(ECMWF, 2021).

Table 3 shows the performance results from our stochastic trajectory
prediction and the existing approaches to data-driven trajectory prediction.
We report the results have the lowest error among 20 generated trajectories.
As revealed in Table 3, LT3P stands out as the state-of-the-art model for
stochastic trajectory prediction. We also observe significant performance
improvements in other data-driven trajectory prediction models. These
encouraging results can be attributed to the inherent strength of existing
data-driven methods, which are designed to generate a diverse and wide range
of trajectory predictions.

Figure 3: Qualitative analysis between LT3P and other trajectory prediction
baselines. Note that the typhoon trajectory provided by JMA exists at 24-hour
intervals, so the three coordinates have been linearly connected.
Additionally, the error (in km) of the final coordinate is also indicated. The
probability map is visualized using kernel density estimation (KDE).

#### Qualitative Results

Figure 3 presents the qualitative results for ensemble and stochastic
predictions. In Figure 3-(c), only the JMA model and our LT3P predict the
change in trajectory of typhoons like DANAS well. These results highlight the
importance of physics values in typhoon trajectory prediction. In Figure
3-(f), LT3P superiority is also confirmed as well. In addition, MID, using
only coordinates, outperforms the physics-based JMA. Such instances show the
promising potential of data-driven models. Additionally, in the case of
stochastic results, LT3P does not aim for a wide and diverse range of
predictions, but focuses on generating forecasts that closely approximate the
actual typhoon trajectory. These qualitative results are supporting evidence
for LT3P’s strong performance in ensemble forecasting, as reported in Table 2.

Figure 4 presents the differences in zonal wind field between UM 72-hour
forecast and ERA5 before and after the bias correction of UM. The figure shows
that the UM exhibits a significantly smaller bias after the bias-correction
phase than the original UM field. These show that the atmospheric conditions
across the entire region, including the trajectories of the typhoons, have
been well corrected in LT3P.

###  4.3 Ablation Study

To assess the efficacy of each component within the LT3P framework, a
systematic analysis is conducted: a baseline trained solely on the UM, a joint
training scheme trained on ERA5 from 1950-2009 and the UM from 2010-2018, an
extended method incorporating phase 1 pre-training and, the full LT3P pipeline
with the bias correction. The results are reported in Table 5. The empirical
findings in Table 5 suggest that all components, barring the UM Only training,
yield good results. In particular, the LT3P configuration, which leverages all
the components, achieves the best performance.

Table 5 presents the experimental results when applying well-known trajectory
prediction models such as GAN, CVAE, and diffusion-based methods to the LT3P
framework. In Table 5, the diffusion-based method achieves the better
performance than the others. Moreover, all the methods, including GAN, CVAE,
and diffusion, exhibit significant performance improvements when incorporating
UM data compared to using only coordinate data (refer to the prediction result
on 72 hours in Table 2). These findings indicate that the performance gains
can be attributed to the LT3P framework itself rather than to the backbone
models.

Figure 4: Zonal wind bias of the UM before and after bias correction with
respect to ERA5.

Table 4: Ablation study on the efficiency of each component of LT3P.

Component | Metric
---|---
UM | Joint Training | Pre-Training | Bias-correction | ADE/FDE (km)
✓ |  |  |  | 219.46 / 390.92
✓ | ✓ |  |  | 80.39 / 190.75
✓ | ✓ | ✓ |  | 85.62 / 198.11
✓ | ✓ | ✓ | ✓ |  65.25 / 143.03

Table 5: Ablation study on backbone models.

Backbone | Metric
---|---
ADE/FDE (km)
SocialGAN (Gupta et al., 2018) | 69.99 / 155.23
PECNet (Mangalam et al., 2020) | 69.81 / 152.07
MID (Gu et al., 2022) |  65.25 / 143.03

##  5 Conclusion

We propose the Long-Term Typhoon Trajectory Prediction (LT3P) for real-time
typhoon trajectory prediction without reanalysis data. To the best of our
knowledge, LT3P is the first data-driven typhoon trajectory prediction that
employs a real-time NWP dataset. Unlike methods that generate reanalysis data,
LT3P predicts the typhoon’s central coordinates without any need for an
additional forecaster and algorithms. In addition, our LT3P does not require
weather forecasting experts, making it easily accessible for various
institutions with limited meteorological infrastructure. Using extensive
evaluations, we confirm that our LT3P achieved state-of-the-art performance in
typhoon trajectory prediction. However, our model has only been applied to
typhoons and has not been applied to other regions of tropical cyclones. In
future work, we plan to apply it to all kinds of tropical cyclones. Lastly, we
hope to contribute to the field of climate AI by releasing our PHYSICS TRACK
dataset, training, test codes of LT3P, and pretrained weights to the public.

## References

  * Alahi et al. (2016) Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese.  Social lstm: Human trajectory prediction in crowded spaces.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2016.
  * Alemany et al. (2019) Sheila Alemany, Jonathan Beltran, Adrian Perez, and Sam Ganzfried.  Predicting hurricane trajectories using a recurrent neural network.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2019.
  * Bi et al. (2023) Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian.  Accurate medium-range global weather forecasting with 3d neural networks.  _Nature_ , pp. 1–6, 2023.
  * Brown et al. (2012) Andrew Brown, Sean Milton, Mike Cullen, Brian Golding, John Mitchell, and Ann Shelly.  Unified modeling and prediction of weather and climate: A 25-year journey.  _Bulletin of the American Meteorological Society_ , 93(12):1865–1877, 2012.
  * Chen et al. (2023) G. Chen, T. Li, M. Yang, and X. Zhang.  Evaluation of western north pacific typhoon track forecasts in global and regional models during the 2021 typhoon season.  _Atmosphere_ , 14(3):499, 2023.
  * Chen et al. (2020) Rui Chen, Weimin Zhang, and Xiang Wang.  Machine learning in tropical cyclone forecast modeling: A review.  _Atmosphere_ , 11(7):676, 2020.
  * Chen & Zhang (2019) Xingchao Chen and Fuqing Zhang.  Development of a convection-permitting air-sea-coupled ensemble data assimilation system for tropical cyclone prediction.  _Journal of Advances in Modeling Earth Systems_ , 11(11):3474–3496, 2019.
  * Dee et al. (2011) Dick P Dee, S M Uppala, Adrian J Simmons, Paul Berrisford, Paul Poli, Shinya Kobayashi, U Andrae, MA Balmaseda, G Balsamo, d P Bauer, et al.  The era-interim reanalysis: Configuration and performance of the data assimilation system.  _Quarterly Journal of the royal meteorological society_ , 137(656):553–597, 2011.
  * Dendorfer et al. (2021) Patrick Dendorfer, Sven Elflein, and Laura Leal-Taixé.  Mg-gan: A multi-generator model preventing out-of-distribution samples in pedestrian trajectory prediction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * ECMWF (2021) ECMWF.  Tropical cyclone activities at ecmwf.  https://www.ecmwf.int/sites/default/files/elibrary/2021/20228-tropical-cyclone-activities-ecmwf.pdf, 2021.
  * Elfwing et al. (2018) Stefan Elfwing, Eiji Uchibe, and Kenji Doya.  Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.  _Neural networks_ , 107:3–11, 2018.
  * Emanuel (2007) Kerry Emanuel.  Environmental factors affecting tropical cyclone power dissipation.  _Journal of Climate_ , 20(22):5497–5509, 2007.
  * Espeholt et al. (2022) Lasse Espeholt, Shreya Agrawal, Casper Sønderby, Manoj Kumar, Jonathan Heek, Carla Bromberg, Cenk Gazen, Rob Carver, Marcin Andrychowicz, Jason Hickey, et al.  Deep learning for twelve hour precipitation forecasts.  _Nature communications_ , 13(1):1–10, 2022.
  * Giffard-Roisin et al. (2020) Sophie Giffard-Roisin, Mo Yang, Guillaume Charpiat, Christina Kumler Bonfanti, Balázs Kégl, and Claire Monteleoni.  Tropical cyclone track forecasting using fused deep learning from aligned reanalysis data.  _Frontiers in big Data_ , pp. 1, 2020.
  * Golchoubian et al. (2023) Mahsa Golchoubian, Moojan Ghafurian, Kerstin Dautenhahn, and Nasser Lashgarian Azad.  Pedestrian trajectory prediction in pedestrian-vehicle mixed environments: A systematic review.  _IEEE Transactions on Intelligent Transportation Systems_ , 2023.
  * Gu et al. (2022) Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu.  Stochastic trajectory prediction via motion indeterminacy diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Gupta et al. (2018) Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi.  Social gan: Socially acceptable trajectories with generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.
  * Huang et al. (2022) Cheng Huang, Cong Bai, Sixian Chan, and Jinglin Zhang.  Mmstn: A multi-modal spatial-temporal network for tropical cyclone short-term prediction.  _Geophysical Research Letters_ , 49(4):e2021GL096898, 2022.
  * Huang et al. (2023) Cheng Huang, Cong Bai, Sixian Chan, Jinglin Zhang, and YuQuan Wu.  Mgtcf: Multi-generator tropical cyclone forecasting with heterogeneous meteorological data.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2023.
  * Huang et al. (2019) Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang.  Stgat: Modeling spatial-temporal interactions for human trajectory prediction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019.
  * IPCC (2023) IPCC.  Ar6 synthesis report: Climate change 2023.  https://www.ipcc.ch/report/ar6/syr/downloads/report/IPCC_AR6_SYR_FullVolume.pdf, 2023.
  * Jiang et al. (2023) Chiyu Jiang, Andre Cornman, Cheolho Park, Benjamin Sapp, Yin Zhou, Dragomir Anguelov, et al.  Motiondiffuser: Controllable multi-agent motion prediction using diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Knapp et al. (2010) Kenneth R Knapp, Michael C Kruk, David H Levinson, Howard J Diamond, and Charles J Neumann.  The international best track archive for climate stewardship (ibtracs) unifying tropical cyclone data.  _Bulletin of the American Meteorological Society_ , 91(3):363–376, 2010.
  * Lam et al. (2022) Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Alexander Pritzel, Suman Ravuri, Timo Ewalds, Ferran Alet, Zach Eaton-Rosen, et al.  Graphcast: Learning skillful medium-range global weather forecasting.  _arXiv preprint arXiv:2212.12794_ , 2022.
  * Lorenz (1963) Edward N Lorenz.  Deterministic nonperiodic flow.  _Journal of atmospheric sciences_ , 20(2):130–141, 1963.
  * Mangalam et al. (2020) Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon.  It is not the journey but the destination: Endpoint conditioned trajectory prediction.  In _Proceedings of European Conference on Computer Vision_. Springer, 2020.
  * Mavrogiannis et al. (2023) Christoforos Mavrogiannis, Francesca Baldini, Allan Wang, Dapeng Zhao, Pete Trautman, Aaron Steinfeld, and Jean Oh.  Core challenges of social robot navigation: A survey.  _ACM Transactions on Human-Robot Interaction_ , 12(3):1–39, 2023.
  * Mohamed et al. (2020) Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel.  Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Nguyen et al. (2023) Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover.  Climax: A foundation model for weather and climate.  In _International Conference on Learning Representations_ , 2023.
  * Pathak et al. (2022) Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al.  Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators.  _arXiv preprint arXiv:2202.11214_ , 2022.
  * Rüttgers et al. (2019) Mario Rüttgers, Sangseung Lee, Soohwan Jeon, and Donghyun You.  Prediction of a typhoon track using a generative adversarial network and satellite images.  _Scientific reports_ , 9(1):6057, 2019.
  * Salzmann et al. (2020) Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone.  Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data.  In _Proceedings of European Conference on Computer Vision_. Springer, 2020.
  * Shi et al. (2021) Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Mo Zhou, Zhenxing Niu, and Gang Hua.  Sgcn: Sparse graph convolution network for pedestrian trajectory prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Stull (2011) Roland Stull.  _Meteorology for Scientists and Engineers_.  Univ. of British Columbia, 3 edition, 2011.  ISBN 978-0-88865-178-5.
  * Walsh et al. (2016) Kevin JE Walsh, John L McBride, Philip J Klotzbach, Sethurathinam Balachandran, Suzana J Camargo, Greg Holland, Thomas R Knutson, James P Kossin, Tsz-cheung Lee, Adam Sobel, et al.  Tropical cyclones and climate change.  _Wiley Interdisciplinary Reviews: Climate Change_ , 7(1):65–89, 2016.
  * Wang et al. (2015) Yuqing Wang, Yunjie Rao, Zhe-Min Tan, and Daria Schönemann.  A statistical analysis of the effects of vertical wind shear on tropical cyclone intensity change over the western north pacific.  _Monthly Weather Review_ , 143(9):3434–3453, 2015.
  * Zhou et al. (2022) Xiaqiong Zhou, Yuejian Zhu, Dingchen Hou, Bing Fu, Wei Li, Hong Guan, Eric Sinsky, Walter Kolczynski, Xianwu Xue, Yan Luo, et al.  The development of the ncep global ensemble forecast system version 12.  _Weather and Forecasting_ , 37(6):1069–1084, 2022.


# Adversarial Stacked Auto-Encoders for Fair Representation Learning

Patrik Joslin Kenfack  Adil Mehmood Khan  Rasheed Hussain  S.M. Ahsan Kazmi

###### Abstract

Training machine learning models with the only accuracy as a final goal may
promote prejudices and discriminatory behaviors embedded in the data. One
solution is to learn latent representations that fulfill specific fairness
metrics. Different types of learning methods are employed to map data into the
fair representational space. The main purpose is to learn a latent
representation of data that scores well on a fairness metric while maintaining
the usability for the downstream task. In this paper, we propose a new fair
representation learning approach that leverages different levels of
representation of data to tighten the fairness bounds of the learned
representation. Our results show that stacking different auto-encoders and
enforcing fairness at different latent spaces result in an improvement of
fairness compared to other existing approaches.

Machine Learning, ICML

##  1 Introduction

Representation learning has made a significant mark in the field of Machine
Learning (ML) over the past decade. It has technologies that extract useful
information or features from data to improve the classification or predictive
performance of models, or even generate new synthetic realistic data. Several
applications for different kind of tasks have emerged such as, machine
translations (Baltrušaitis et al., 2018), anomalies detection (Rivera et al.,
2020), objects and actions recognition (Papageorgiou & Poggio, 2000), etc.

ML models are widely used in real life to make decisions that can affect
people’s lives, e.g., loan applicant, college admission, criminal justice,
hiring, etc. Models trained with biased data can lead to unfair decisions
(Mehrabi et al., 2019). In fact, these models mainly rely on human-generated
data to learn patterns that are then used to make predictions on the new
unseen data. However, real-world data are already tainted by prejudices and
unfair decisions (historical bias), which reflect the flaws of our society.
Historical bias is one origin of algorithmic bias. Another source of
algorithmic bias is the representation bias (Mehrabi et al., 2019). It arises
when certain groups of the population are underrepresented within the data.
For example, a facial recognition model trained with data containing
considerably more white faces than black faces will tend to be less accurate
when used on black faces. To this end, the algorithmic bias occurs when biases
in the data are learned by the model and therefore lead to unfair decisions
(Dwork et al., 2012; Kenfack et al., 2021; Hardt et al., 2016).

One approach to mitigate the impact of biases from the data is the fair
representation learning. With this technique, the input data is mapped into a
new representation which is enforced to satisfy a given fairness metric while
maintaining the utility of the representation as much as possible. The learned
representation can then be used for any downstream task such as classification
or data generation, with better chances of yielding fair results. Existing
works by Madras et al.; Edwards & Storkey used adversarial learning to enforce
the fairness of the representation with respect to statistical parity. They
used an auto-encoder as a generator whose aim is to learn a latent space such
that an adversary cannot predict the sensitive feature (gender, race, etc.)
from the learned latent representation. In Madras et al., the authors proposed
a learning objective for other fairness metrics such as equalized odds and
equal opportunity (section 2) with theoretical bounds of fairness.

This work builds on top of the previous works where we propose a fair
representation learning approach based on adversarial stacked auto-encoders.
However, our proposed approach leverages different level of representation of
the input data to tighten the fairness bounds of the learned representation.
In fact, the success of deep networks can be attributed to their ability to
exploit the unknown structure in the input distribution to discover useful
features at multiple levels. In this multi-level representations, the higher-
level learned features are defined in terms of lower-level features (Bengio et
al., 2013). For instance, Khan et al. Khan & Fraz showed that performing data
augmentation in the feature space (and at different level of representation),
can improve predictive performances of the neural network. Similarly, a
generative model proposed by Huang et al. Huang et al. leveraged different
level of representation to improve the quality of the generated images.
Applying fairness at a given level does not guarantee that information about
the sensitive attribute is removed, as it may not all be presented at the
given level.

In essence, we hypothesize that the above arguments may also be useful for
improving fairness, which was confirmed by our empirical results. Intuitively,
the main idea is to approach an optimal adversary via sequential learning in
which one adversary is used to enforce fairness on a low-level representation.
This low-level representation is then used as input for a higher-level
representation on which another adversary is trained to enforce fairness on
that representation by improving the previous adversary.

The reminder of the paper is organized as follow. In Sections 2 and 3, we
present related work and background, respectively. In Section 4, we introduce
our fair representation learning approach that tightens the fairness bounds.
In Section 5, we present empirical results which show the effectiveness of our
learned representation on several real-life datasets. In Section 6, we
conclude the paper.

##  2 Related Work

Pre-processing techniques are used to mitigate biases from the data by
enforcing a given fairness property while maintaining the utility of the
predictions. The objective of fair representation learning is to learn a
representation of the data that is most likely to produce fair results for
downstream tasks. In (Zemel et al., 2013), the authors presented the first
fair representation learning approach which removes dependencies on the
sensitive attribute by mapping input data to new points called prototypes.
Prior work in this direction focuses on statistical parity, equalized odds,
and equal opportunities.

The goal is to learn a representation that will remove all the dependencies in
regards to the sensitive attribute from the training data, while retaining as
much information as possible. In (Louizos et al., 2016), the authors proposed
the Variational Fair Auto-Encoder (VFAE), a variant of variational auto-
encoder that maps the input data into a latent space while discarding
information about the sensitive attributes from the data as much as possible.
Thus the sensitive attributes are treated as nuisance variable. To do this,
the authors (i) used a factorized prior $p(z)p(s)$ where $z$ is the latent
representation and $s$ is the sensitive attribute, and (ii) added a
regularization term to encourage the independence between $z$ and $s$ using
maximum mean discrepancy.

In (Edwards & Storkey, 2015), the authors proposed an approach to learn fair
representation using adversarial learning to enforce demographic parity.
Similarly, in Beutel et al., the authors explored the particular fairness
levels achieved by the algorithm from (Edwards & Storkey, 2015) and showed how
other fairness metrics can be achieve by varying the distribution of the
adversary’s input. Madras et al. Madras et al. extended the previous work by
proposing adversarial objectives that yield fair and transferable
representations that in turn admit fair classification outcomes. They provided
adversarial objective functions for each fairness metric that upper bounds the
unfairness of arbitrary downstream classifiers in the limit of adversarial
training.

In this work, we propose a new fair representation learning approach built
upon previous works, which aims to improve the fairness of models through
stacked adversarial learning. We enforce fairness at different levels of
representation in order to tighten the fairness bounds of the final
representation.

##  3 Background

In this section, we introduce fairness notions used throughout this paper and
concepts related to fair representation learning and adversarial learning.

###  3.1 Fairness

Consider a training data $\mathcal{D}=\\{X,Y,S\\}$, where
$x_{i}\in\mathbb{R}^{n}$ is the feature vector, $y_{i}\in\\{0,1\\}$ is the
label, and $S$ is the binary protected attribute (e.g., gender, race, etc.).
Learning a fair representation means mapping the input data $X$ into a new
representation $X^{\prime}$ such that $X^{\prime}$ will satisfy one of the
following fairness criteria:

  * •

Statistical parity: It is also known as Demography parity ($\Delta_{DP}$).
This fairness criteria promotes the independence between the predictor outcome
($\hat{Y}$ a function of $X^{\prime}$) and the sensitive attribute.
$\hat{Y}\bot\;S$, i.e., a predictor satisfies the statistical parity if
$P(\hat{Y}|S=0)=P(\hat{Y}|S=1)$ (Dwork et al., 2012). For example, a loan
approval system will achieve statistical parity if it does not deny loans to
men more often than to women. A drawback of this fairness criterion is that it
allows unqualified applicants to be selected as long as the acceptance rate is
the same for both groups. However, when the sensitive attribute correlates
with the target variable, a drop in accuracy can be observed.

  * •

Equalized Odds: In contrast to $\Delta_{DP}$, Equalized Odds ($EO$) promotes
the conditional independence between the prediction outcome and the sensitive
attribute given the class label ($\hat{Y}\bot S|Y$). A predictor outcome
$\hat{Y}$ trained with $X^{\prime}$ satisfies EO if
$P(\hat{Y}=y|S=0,Y=y)=P(\hat{Y}=y|S=1,Y=y),\forall y\in\\{0,1\\}$. In other
words the False Positive Rate (FPR) and the True Positive Rate (TPR) of groups
should be the same. One advantage of equalized odds is that it admits the
perfect model $\hat{Y}=Y$ (Hardt et al., 2016; Verma & Rubin, 2018).

  * •

Equal opportunity: Similar to EO, Equal opportunity (EOpp) only considers the
case where $Y=1$ ($\hat{Y}\bot S|Y=1$). A predictor outcome $\hat{Y}$
satisfies EOpp if $P(\hat{Y}=1|S=0,Y=1)=P(\hat{Y}=1|S=1,Y=1)$. In other words,
groups should have the same TPR.

It is worth noting that the predictor trained with fairness constraints are
less accurate than the ones trained without it (Kamishima et al., 2011). Thus
fairness comes at the expense of accuracy. A desired property is to provide
fair representation with lower fairness accuracy trade-off.

###  3.2 Adversarial Learning

Inspired by the game theory, adversarial learning consists of two neural
networks (generator and discriminator) trained in an adversarial manner. The
generator’s ($G$) goal is to fool the discriminator ($D$) by sampling as
realistic examples as possible such the discriminator –which the goal is to
distinguish between fake samples and real samples–, will not be able to make
the difference between examples $G(z)$ sampled from $G$ using the random noise
vector $z$ and real examples $x$. Thus, $G$ and $D$ play a min-max game with
value function $V(G,D)$:

$\begin{split}\begin{aligned}
\underset{G}{\mathrm{min}}\>\underset{D}{\mathrm{max}}\>V(D,G)=\;&\mathbb{E}_{x\sim
p_{data}(x)}[\mathrm{log}\>D(x)]\\\ &+\mathbb{E}_{z\sim
p_{z}}[1-\mathrm{log}\>D(G(z))]\end{aligned}\end{split}$ (1)

where $D$ seeks to maximize this quantity while $G$ seeks to minimize it.

##  4 Methods

Figure 1: Adversarial Stacked Auto-Encoders architecture

In this section, we describe the architecture of our proposed model and the
training procedure. Figure 1 presents an overview of the architecture and the
training process.

###  4.1 Model Architecture

Our main idea is to stack different Encoders ($E_{i}$), Decoders ($D_{i}$),
classifier $f_{i}$, and adversary ($h_{i}$), in order to get different levels
of representation of the input data. The intuition here is that, different
level of representation can exhibit different details of information from the
data. Enforcing fairness at a given level does not guarantee that fairness
bounds are tight enough, unless the adversary is an optimal one, which may not
be available in non-convex settings. Our goal is to approach this optimal
adversary in an incremental may.

At a each level $i$, we have different components: the learned representation
$z_{i}$ yielded by the encoder $E_{i}$, the corresponding decoded
representation $z^{\prime}_{i}$ produced by the decoder $D_{i}$, the adversary
network $f_{i}$ that enforces the fairness of that representation and the
predictor network $h_{i}$ that enforces the utility of the representation.
$z_{0}$ represents the input data $X$, and $z^{\prime}_{0}$ the final
reconstructed output ($X^{\prime}$). The overall loss at each level $i$ is
defined as the linear combination of three loss terms: the reconstruction loss
($\mathcal{L}^{rec}_{E_{i},D_{i}}$), the adversary loss
($\mathcal{L}^{adv}_{f_{i}}$) and the predictor loss
($\mathcal{L}^{adv}_{h_{i}}$):

$\mathcal{L}(G_{i},D_{i},f_{i},h_{i})=\alpha\mathcal{L}^{rec}_{E_{i},D_{i}}+\beta\mathcal{L}^{Adv}_{f_{i}}+\gamma\mathcal{L}^{Class}_{h_{i}}$
(2)

In the above equation, $\alpha$, $\beta$ and $\gamma$ are the weights
associated with each loss. Thus, $\mathcal{L}^{rec}_{E_{i},D_{i}}$ is the loss
of reconstructing the encoded representation $z_{i}$ by the decoder $D_{i}$.
For the reconstruction loss we use the Root Mean Squared Error (RMSE):
$\mathcal{L}^{rec}_{E_{i},D_{i}}=\frac{1}{|X|}||z^{\prime}_{i}-D_{i}(E_{i}(z_{i-1}))||^{2}_{2}$.
The adversarial loss is to enforce the representation to satisfy certain
fairness constraint. For instance, to satisfy statistical parity, the
adversary loss is defined as cross entropy loss:

$\mathcal{L}^{adv}_{f_{i}}=\frac{1}{|X|}\sum_{s,\hat{s}\in
S,\hat{S}}s\cdot\mathrm{log}(\hat{s}+(1-s)\cdot\mathrm{log}(1-\hat{s}))$ (3)

The adversary network at the level $i$ tries to minimize the loss of
predicting the sensitive attribute $S$ from the encoded representation
$z_{i}$, while the predictor and the generator (typically auto encoder) try to
maximize it. The losses of predictor and adversary can be defined as cross
entropy loss or using loss functions proposed in (Madras et al., 2018) to
satisfy equalized odds and equal opportunities. Thus at each level, we have
the following min-max problem:

$\underset{G_{i},D_{i},h_{i}}{\mathrm{min}}\;\underset{f_{i}}{\mathrm{max}}\;\mathcal{L}(G_{i},D_{i},f_{i},h_{i})$
(4)

To have a different representation at each level, we vary the dimension of
each latent space, from higher to lower dimensions ($|z_{i}|>|z_{i+1}|$).

###  4.2 Model training

At a given level $i$, we realize the classifier, auto-encoders and adversary
as neural networks and alternate gradient decent and ascent steps to optimize
their parameters according to 4. First the encoder-classifier-decoder takes a
gradient step to minimize $\mathcal{L}$ (Equation 4) while the adversary
$f_{i}$ is fixed, then $f_{i}$ takes a step to maximize $\mathcal{L}$ with
fixed auto encoder and classifier. We use a relaxation of adversary objectives
proposed by (Madras et al., 2018; Beutel et al., 2017), i.e., to achieve
Equalized Odds, in addition to the latent space $z$, we passed the class label
$Y$ to the adversary. To achieve Equal Opportunity, the loss function (Eq 4)
is computed only using samples where $Y=0$.

The training is performed sequentially, starting with an initial latent
representation $z_{1}$ trained using the input data. During the first
training, the adversary $f_{1}$ enforces fairness (typically $\Delta_{DP}$,
$\Delta_{EO}$, or $\Delta_{EOpp}$) of the lower level representation $z_{1}$.
Afterwards, a new latent space of lower dimension $z_{2}$ (higher level
representation) is stacked, and uses the pre-trained representation $z_{1}$ as
input.

The number of stacked layers on which the fairness constraints are imposed
depends on the depth of the neural network and are specified as a hyper
parameter. In the experiments, we used a Multi Layers Perceptron (MLP) network
for the encoder and decoder with one hidden layer. Initially, fairness is
applied on the hidden layer ($z_{1}$), then the output layer (latent space) is
stacked and used as the final representation ($z_{2}$). In the testing phase,
we get rid of all the decoders, adversaries, and classifiers. Only the
encoders are used to map the input data into the fair space.

##  5 Experiments

(a) German Dataset

(b) Adult Dataset

Figure 2: Accuracy-fairness trade-offs of classification tasks on the German
(first row) and Adult (second rows) datasets. Our learned representation
always lower bounds the fairness results of the representation learned by
vanilla approach (LAFR). It shows that the fairness bounds of our approach is
more tight. However, we can observe a slight decrease in accuracy compared to
other representations.

We present experiments on two standard real world datasets widely used for
fair classification as suitable benchmarks to compare the performance of
different machine learning methods: The Adult Income dataset (Asuncion &
Newman, 2007) has 48843 instances of demographic information of American
adults, described with 14 features. The target variable indicates whether
individual’s income is larger than 50K US dollars. The German credit dataset
(Jeff et al., ) has 1000 instances of bank account information represented by
20 features with the aim to classify bank account holders into credit class
good or bad. For both datasets, we use gender as the single protected
(sensitive) attribute. We demonstrate the effectiveness of our approach
compared to standard fair representation learning techniques. The standard
(vanilla) approach is fair representation learning with fairness applied only
to one latent space.

###  5.1 Fair Classification

Figure 2 shows the fairness results of the MLP trained with the representation
obtained by our approach compared to the representation produced by the
vanilla approach (Learning Adversarial Fair Representation $-$ LAFR) and MLP
trained with original input data (MLP-unfair). For the vanilla approach, we
used a network architecture with one hidden layer of 20 units, and latent
space of 8 units for Adult dataset, 15 hidden units and 8 output units for the
German dataset. We trained models with the same architecture using LAFR
approach and our approach with two levels of representation, i.e., we trained
an adversary on the hidden layer and then stacked the output layer and trained
another adversary on it. We used single-hidden-layer neural networks for each
of our classifier and adversary with 20 hidden units. The equation 4 is
optimized using Adam optimizer (Kingma & Ba, 2014) with learning rate of
$0.01$, a batch size of 64, trained for 150 epochs for Adult dataset and 1000
for the German credit. We run the experiment seven times with different values
of $\beta$ $(1,2,3,5,15)$, with $\alpha=0$ and $\gamma=1$.

Similar to the process used by Madras et al., we created a feed-forward model
which consisted of our frozen, adversarially-learned encoders followed by an
MLP with one hidden layer, with a loss function of cross entropy with no
fairness modifications. We reported the mean over all runs per $\beta$ and we
used a validation procedure to evaluate. The results shows that representation
produced by our model always lower bounds the fairness of standard approaches.
This shows that our approach provides tighter fairness bounds. However, since
the main objective of our work is to improve the fairness, a decrease in
accuracy is observed compared to the standard approach, which we attribute to
the trade-off between fairness and accuracy.

###  5.2 Classification on Downstream Tasks

Table 1: Comparison of $\Delta_{DP}$ on classification tasks using logistic
regression and random forest model on Adult and German datasets

Model | Unfair | LAFR | Ours
---|---|---|---
Adult
Logistic Regression |  0.53$\pm$ 0.008 |  0.51$\pm$ 0.009 |  0.21$\pm$ 0.004
Random Forest |  0.54$\pm$ 0.001 |  0.49$\pm$ 0.001 |  0.25$\pm$ 0.007
German
Logistic Regression |  0.36$\pm$ 0.08 |  0.33$\pm$ 0.09 |  0.08$\pm$ 0.04
Random Forest |  0.27$\pm$ 0.03 |  0.23$\pm$ 0.06 |  0.11$\pm$ 0.05

Learning fair representation is a model-agnostic approach to mitigate
unfairness, i.e., the learned representation can be used for any downstream
task and not only for neural network based models. We tested linear and non-
linear models on representation produced by our model and standard approach.
We trained the representation using the network architecture described in
previous section, without hyper-parameter tuning and using $\alpha=0$,
$\beta=1$, $\gamma=1$. We also trained models on the original dataset without
fairness constraints.

Table 1 shows $\Delta_{DP}$ reported from 5-fold cross validations on Adult
and German datasets. Results shows that the representation produced by our
model also provides better fairness performances when trained using classical
machine leaning algorithms such as Linear Regression and Random Forest. We
observed similar results for other fairness metrics (EO, EOpp).

##  6 Conclusion

In this paper, we showed that applying fairness at different levels of
representation improves the fairness performance of the learned
representation. In this regard, we proposed an adversarial stacked auto-
encoder architecture which expose different level of representation of the
input data, on which several adversary networks are trained sequentially to
tighten the fairness bounds of the final representation (lowest level
representation).

Our empirical results show that this approach outperforms standard adversarial
fair representation learning approach in terms of fairness. Intuitively, our
learning process lead to learning an optimal adversary in incremental way.
However, stabilizing adversarial training of fair representations remains an
important issue that we plan to address in our future work.

## References

  * Asuncion & Newman (2007) Asuncion, A. and Newman, D.  Uci machine learning repository, 2007.
  * Baltrušaitis et al. (2018) Baltrušaitis, T., Ahuja, C., and Morency, L.-P.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(2):423–443, 2018.
  * Bengio et al. (2013) Bengio, Y., Courville, A., and Vincent, P.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1798–1828, 2013.
  * Beutel et al. (2017) Beutel, A., Chen, J., Zhao, Z., and Chi, E. H.  Data decisions and theoretical implications when adversarially learning fair representations.  _arXiv preprint arXiv:1707.00075_ , 2017.
  * Dwork et al. (2012) Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R.  Fairness through awareness.  In _Proceedings of the 3rd innovations in theoretical computer science conference_ , pp. 214–226, 2012.
  * Edwards & Storkey (2015) Edwards, H. and Storkey, A.  Censoring representations with an adversary.  _arXiv preprint arXiv:1511.05897_ , 2015.
  * Hardt et al. (2016) Hardt, M., Price, E., and Srebro, N.  Equality of opportunity in supervised learning.  _arXiv preprint arXiv:1610.02413_ , 2016.
  * Huang et al. (2017) Huang, X., Li, Y., Poursaeed, O., Hopcroft, J., and Belongie, S.  Stacked generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5077–5086, 2017.
  * (9) Jeff, L., Surya, M., Lauren, K., and Julia, A.  How we analyzed the compas recidivism algorithm.
  * Kamishima et al. (2011) Kamishima, T., Akaho, S., and Sakuma, J.  Fairness-aware learning through regularization approach.  In _2011 IEEE 11th International Conference on Data Mining Workshops_ , pp. 643–650. IEEE, 2011.
  * Kenfack et al. (2021) Kenfack, P. J., Dmitrievich Arapov, D., Hussain, R., Kazmi, S., and Mehmood Khan, A.  On the fairness of generative adversarial networks (gans).  _arXiv e-prints_ , pp. arXiv–2103, 2021.
  * Khan & Fraz (2020) Khan, A. and Fraz, K.  Post-training iterative hierarchical data augmentation for deep networks.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Louizos et al. (2016) Louizos, C., Swersky, K., Li, Y., Welling, M., and Zemel, R.  The variational fair autoencoder.  In _International conference on learning representations_ , 2016.
  * Madras et al. (2018) Madras, D., Creager, E., Pitassi, T., and Zemel, R.  Learning adversarially fair and transferable representations.  In _International Conference on Machine Learning_ , pp. 3384–3393. PMLR, 2018.
  * Mehrabi et al. (2019) Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A.  A survey on bias and fairness in machine learning.  _arXiv preprint arXiv:1908.09635_ , 2019.
  * Papageorgiou & Poggio (2000) Papageorgiou, C. and Poggio, T.  A trainable system for object detection.  _International journal of computer vision_ , 38(1):15–33, 2000.
  * Rivera et al. (2020) Rivera, A. R., Khan, A., Bekkouch, I. E. I., and Sheikh, T. S.  Anomaly detection based on zero-shot outlier synthesis and hierarchical feature distillation.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2020\.
  * Verma & Rubin (2018) Verma, S. and Rubin, J.  Fairness definitions explained.  In _2018 ieee/acm international workshop on software fairness (fairware)_ , pp. 1–7. IEEE, 2018.
  * Zemel et al. (2013) Zemel, R., Wu, Y., Swersky, K., Pitassi, T., and Dwork, C.  Learning fair representations.  In _International conference on machine learning_ , pp. 325–333. PMLR, 2013.