# Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning and
Levels-of-Experts

Kun Wang, Hao Wu, Guibin Zhang, Junfeng Fang, Yuxuan Liang${}^{\textrm{{\char
0\relax}}}$, Yuankai Wu, Roger Zimmermann, and Yang Wang${}^{\textrm{{\char
0\relax}}}$ K Wang, H. Wu, J.F. fang and Y. Wang are all with University of
Science and Technology of China, Hefei, Anhui, P.R.China. Guibin Zhang is with
Tongji University and The Hong Kong University of Science and Technology
(Guangzhou), P.R.China. Yuxuan Liang is with INTR Thrust & DSA Thrust, The
Hong Kong University of Science and Technology (Guangzhou). Yuankai Wu is with
the College of Computer Science, Sichuan University. Roger Zimmermann is with
the University of Singapore, Singapore. ✉ Yang Wang and Yuxuan Liang are the
corresponding authors. Email<EMAIL_ADDRESS>&<EMAIL_ADDRESS>
###### Abstract

In this paper, we address the issue of modeling and estimating changes in the
state of the spatio-temporal dynamical systems based on a sequence of
observations like video frames. Traditional numerical simulation systems
depend largely on the initial settings and correctness of the constructed
partial differential equations (PDEs). Despite recent efforts yielding
significant success in discovering data-driven PDEs with neural networks, the
limitations posed by singular scenarios and the absence of local insights
prevent them from performing effectively in a broader real-world context. To
this end, this paper propose the universal expert module – that is, optical
flow estimation component, to capture the evolution laws of general physical
processes in a data-driven fashion. To enhance local insight, we painstakingly
design a finer-grained physical pipeline, since local characteristics may be
influenced by various internal contextual information, which may contradict
the macroscopic properties of the whole system. Further, we harness currently
popular neural discrete learning to unveil the underlying important features
in its latent space, this process better injects interpretability, which can
help us obtain a powerful prior over these discrete random variables. We
conduct extensive experiments and ablations to demonstrate that the proposed
framework achieves large performance margins, compared with the existing SOTA
baselines.

###### Index Terms:

Spatio-temporal dynamics, Neural discrete learning, Optical flow estimation.

##  1 Introduction

The spatio-temporal physical process is a way of describing a sustained
phenomenon involving all objects and matters in the physical world [1, 2].
These phenomena range from atoms to the earth, to the far reaches of galaxies
and span the entire timeline. Dynamical systems, primarily derived from first
principles [3, 4, 5, 6, 7], provide a well-established mathematical tool for
representing real-world physical processes [8]. They model the complex
interactions between variables that evolve over time and space, such as
Newton’s second law [9], the advection–diffusion equation [10, 11], and the
Navier-Stokes equations [12, 13], to name just a few. Formally, dynamical
systems involve partial differential equations for analyzing, predicting, and
understanding the evolution of the system’s state [14, 15, 16, 17, 18].

Traditional spatio-temporal dynamical systems utilize numerical simulations
for modeling, and the mathematical philosophies in these systems are universal
to have a wide range of applications in classical mechanical systems [19, 20,
21], electrical circuits [22, 23], turbulence [24, 25, 26], gas [27, 28] and
fire science [29, 30], and many other system that evolves with time.
Nonetheless, the systems encounter significant challenges when faced with two
conditions [31, 32]: (1) Uncertainty Principle. The laws governing complex
unfamiliar physical systems are difficult to describe or define intuitively in
terms of physical equations, and there are many unknown or partially known
equations in nonlinear dynamics [33, 34, 35]. (2) System equation complexity.
Understanding the dynamics of objectives requires solving PDEs [36, 37].
However, with the sharp increase of observable data, the complexity of the
system to solve PDEs increases exponentially, leading to delays in the system
[38, 39].

In brief, the performance of numerical simulation methods depends largely on
the initial settings and the designed algorithms. Furthermore, it is plagued
by the drawbacks of considerable computational resource consumption and the
incapacity to meet the demands of real-time prediction.

Figure 1: An example of dynamic evolution process of fire. Two fires (★) broke
out in a building and we showcase the global and local speed of fire spread.

Recently, Deep Learning (DL) methods have emerged as promising approaches for
modeling complex systems with the dilemmas mentioned earlier [40, 41, 42, 43,
44, 45], as DL provides an efficient optimization framework to automatically,
adaptively, and dynamically extract intrinsic patterns from continuous
physical processes. Unlike classic dynamic systems, DL frameworks often
sacrifice the explicit understandings of physical rules (aka., expert priors),
resorting to the large-scale observable data and capturing the implicit
patterns that serve as substitutes for physical laws. However, this comes at a
cost; discarding expert priors may lead to model collapse under limited data
and complex scenarios. Such knowledge of physics ensures that our predictions
remain within a testable range without anomalies. To this end, several
proposals have integrated dynamic physics laws into their frameworks for
facilitating predictions, also known as _Physics-Informed Neural Networks_
(PINN).

Research Gap. In general, a physical process is expressed as a sequence of
snapshots, and the modeling objective can be deemed as a Spatio-Temporal (ST)
prediction task [46, 47, 48]. The most existing PINNs achieve the
encapsulation of the fundamental physical insights by introducing PDEs into
the loss or likelihood functions [49, 50]. Once the specific expert component
(e.g., weather component) is pre-trained for a single scenario, it can be
invoked repeatedly with low computational cost. However, a single scenario
cannot satisfy various and general physics scenarios, which necessitates a
laborious dataset-specific design for prior selection. Worse still, PINNs are
commonly employed for addressing challenges related to the state estimation of
unknown spatiotemporal points [50]. When applied to prediction tasks involving
the future evolution of noisy physical processes, PINNs exhibit limitations in
terms of compatibility and performance.

Figure 2: Top. An overview of our framework. Bottom. Our proposal framework
exhibits exceptional improvements over existing models, demonstrating its
factuality and steerability.

Furthermore, off-the-shelf ST modules [51, 52, 35, 53], only focus on global
understanding, neglecting _local fidelity_ – local dynamics may differ from
global dynamics. For instance, Fig. 1 shows velocity slices of two successive
frames illustrating fire’s progression. Globally, the fire’s flow rate is
continuously increasing as it intensifies and expands. However, at a local
level, the convection caused by multiple indoor fire points can lead to some
number of areas experiencing lower flow rates due to mutual interference. We
argue that this phenomenon is prevalent, with local characteristics
manifesting in specific spatiotemporal scenes, while the global situation is
entirely opposite. It is of immense practical significance to pay close
attention to these local areas, especially in emergency situations such as
fires, where knowledge of low flow rates can better guide the crowd and assist
firefighters in dealing with the crisis more efficiently. To address the above
issue, it is necessary to develop custom models or modify existing ones to
better capture the local dynamics of the systems.

Towards this end, we introduce a general expert-level physics modeling scheme
that operates at multiple granularities with good interpretability, dubbed
PhysicNet. As depicted in Fig. 2, we commence by hierarchically observing
various receptive fields to globally and locally enhance the observed data.
Then we send the hierarchical observations to two pipelines; in the top-level
and bottom-level pipelines, we harness adaptive global/local encoders,
respectively, and automatically characterize the state of physical processes
from multiple viewpoints. We utilize a vector quantization (VQ) prior bank,
referred to as “codebank” to identify the most compact, comprehensive, and
discrete embedding space and priors, instead of allocating capacity towards
noise and imperceptible details. By combining VQ [54, 55] with the
representation capabilities of neural networks, our model can enhance its
expressive ability and improve its local insight and fidelity.

After an iterative process of transferring and transforming, we transmit the
feature map, which contains physical knowledge, to the global/local general
expert component (called GgEC or LgEC), enabling them to perceive physical
processes. These expert components are utilized to process data that conforms
to a continuous physical context process. Concretely, we introduce an optical
flow estimation constraint term to describe the direction and magnitude of
motion for each pixel in a sequence of observations, which ensures that its
process conforms to the evolution process of the physical situation. Our work
is simple yet effective, allowing for accurate predictions of local details.
We are confident that the insights gained from this study will inspire further
research on modeling physical processes and their potential applications. In
short, our contributions can be summarized as:

  * •

We present a data-driven neural network that integrates expert knowledge and
extracts crucial underlying patterns for accurate observations while
maintaining interpretability. Furthermore, by emphasizing the integration of
local characteristics into the model, we are proposing a new direction with
ongoing and future efforts in the field. Since our framework employs a data-
driven approach to capture crucial information from noisy datasets, it
mitigates the data limitations inherent in PINNs. Consequently, our approach
exhibits enhanced robustness in real-world scenarios, thereby showcasing
greater resilience.

  * •

Our paradigm can automatically discover physical evolution laws from high-
dimensional data by combining Vector discrete learning theory and neural
network representations. We refine and extract hidden state variables from the
network encoding, enabling powerful and interpretable dynamical system
prediction in discrete, sparse parsimonious dynamics.

  * •

Our optical flow estimation module effectively captures the spatio-temporal
data variations, ensuring that the model accurately grasps ST dynamics, which
in turn guarantees the stability of predictions.

  * •

Extensive experiments on four real-world datasets across various baselines to
demonstrate the effectiveness and efficiency of the proposed framework. We
highlight some shocking performances of our method on SEVIR-Infrared Satellite
imagery dataset in the bottom part of the Fig. 2.

##  2 Preliminary & Related work

Problem Formulation. Physical systems evolve continuously based on their
dynamics, which can be represented as a function with environment space $X$
and state space $S\subset X$: $X_{t+\Delta t}=\mathcal{F}\left(X_{t}\right)$,
$\mathcal{F}$ describes system state evolution from current to next time step.
We address the forecasting of physical dynamics as a spatiotemporal
forecasting problem, _i.e._ , mapping high-dimensional visual observations
$\left[\mathcal{X}_{t}\right]_{t=1}^{T},\mathcal{X}_{t}\in\mathbb{R}^{C\times
H\times W}$ to a relatively low-dimensional embedding $L_{s}$, which is then
projected onto a spatiotemporal sequence of $K$ future steps
$\left[\mathcal{X}_{T+t^{\prime}}\right]_{t^{\prime}=1}^{K},\mathcal{X}_{T+t^{\prime}}\in\mathbb{R}^{C\times
H\times W}$, where $H$ and $W$ denote the number of spatial grids with
$C$-dimensional observations on each grid.

Local and Global Insight. We introduce, for the first time, a combined global
and local spatiotemporal modeling approach that aims to simultaneously capture
both overall trends and localized intricacies. Drawing inspiration from
related work in contrastive learning [56], we define local insights as
coverage areas that are below 50% of the original region. Given the unknown
nature of crucial local regions, we adopt a random cropping method to extract
$\cal H$ distinct local perspectives. A larger value of $\cal H$ necessitates
greater computational resources, whereas a smaller $\cal H$ may result in the
loss of significant local areas.

Physics-Informed Neural Networks (PINN). Scientists explore the potential of
PINNs in modeling system dynamics by incorporating physics priors or logical
constraints into neural networks [57, 58, 59, 60, 61, 62, 63, 49, 64].
Physical control equations connect deep neural networks to numerical
differential equations, resulting in the development of generalized
architectures with specific functionalities [65, 66, 67] that reduce
complexity in simulating physical processes. Our research aims to model the
entire process of physical dynamics based on PINNs, effectively capturing non-
linear and dynamic relationships among system components [68, 69, 70, 71].
Unfortunately, PINNs are typically applied to idealized noise-free data for
addressing state estimation of unknown spatiotemporal points. They do not
exhibit significant expressive power when applied to our real-world
spatiotemporal prediction problems.

Optical flow estimation. Substantial advancements in optical flow estimation
have been achieved with the incorporation of deep learning [72, 73, 74].
Traditional methods relied on variational principles [75], optimizing flow
prediction through continuous optimization with various objective terms, which
has led to discrete optimization improvements for modeling optical flows [76].
More recent efforts have pivoted towards deep learning frameworks that employ
coarse-to-fine strategies and iterative refinements. Notably, the RAFT model
emerged as a pivotal development, introducing a convolutional GRU update
mechanism and achieving state-of-the-art performance [77]. Enhancements to
this model have been made by [78], who integrated a self-attention-style
global motion aggregation module to refine the handling of occlusions. In our
work, we first involve the optical flow in to ST prediction task to constrain
the prediction process to be stable and reliable.

Neural Discrete Learning. Recently, models which leverage discrete
representation through vector quantization have been proposed. Milestones such
as VQ-VAE [54] and VQ-VAE2 [79] show great promise in image, speech and video
generation, leading to numerous follow-up studies [80, 81, 82, 83]. Broadly
speaking, VQ-VAE consists of an encoder network, a quantization module, a
decoder network, and a loss function. The encoder network takes an input image
and encodes it into a low-dimensional latent representation. The quantization
module then maps this continuous latent representation to a discrete codebank
of embedding vectors. The decoder network takes the discrete code and decodes
it back into a reconstructed image. Our work is the first to incorporate
vector quantization encoding techniques into physical processes, for learning
important latent features in both global and local physical states, and aiding
in better priors for temporal processes.

Spatio-temporal prediction methods. Spatio-temporal data mining, a merger of
classical spatial data mining and temporal data mining, focuses on the
extraction of patterns, relationships, and knowledge from spatial and temporal
data. Generally, spatio-temporal prediction methods can be classified into
CNN-based approaches [84, 85, 86], RNN-based techniques [87, 88, 89, 90, 91,
52], and other models encompassing combinations [92, 93], as well as
transformer-based models [94, 51, 95]. Interestingly, while various existing
models built upon graph neural networks (GNNs) are available, their primary
focus revolves around graph data handling [96, 97, 98, 99, 100], which falls
outside the purview of our study.

Deep Learning for Dynamical Systems. DL has become a popular tool for
predicting dynamical system behavior. As an early study in this area, ConvLSTM
integrates CNNs with LSTM to jointly capture spatial and temporal information
in precipitation data [45]. Advanced models, such as PredRNN [101], E3D-LSTM
[102], and SimVP [103], further improve accuracy and robustness by devising
sophisticated architectures. [47] presents a recursive physical unit that
separates PDE dynamics from unknown complementary information to enhance
predictions using DL. Researchers have also investigated modeling system
dynamics from video data, proposing various methods for extracting physical
parameters from input data [35, 104, 105]. However, the generalization ability
of these models is limited by the requirement of known equations governing
system dynamics.

##  3 Methodology

Figure 3: The pipeline of our model. Conv2d: 2D convolutional layer. Deconv2d:
deconvolutional operator for upsamling. DC block: Deconvolutional block.

In this paper, we encapsulate physical process prediction as a spatiotemporal
forecasting task as [51] does. The pipeline of our PhysicNet is presented in
Fig. 3, whose input is the historical observations, i.e., a spatiotemporal
sequence
$\left[\mathcal{X}_{t}\right]_{t=1}^{T},\mathcal{X}_{t}\in\mathbb{R}^{C\times
H\times W}$ at the left bottom. We aim to parallelly predict the future $K$
steps of the spatiotemporal sequence. In the following parts, we will
delineate our model components by introducing Fig. 3 from left to right.

###  3.1 Random Cropping to Multi-Grained Observations

The observations used in this study, which include global and local views, are
encoded using similar forms (but with different kernel sizes). However, many
current models [103, 106, 101] mainly focus on global physical processes,
which can result in inherent limitations: global insight can compromise local
fidelity since globally significant information may not be important or may
not even appear in the local context. Consequently, such models may fail to
provide reliable understanding of specific instances. Going beyond the global
view, several (in this paper, we set ${\cal H}=3$ local views for the trade-
off between the accuracy and the efficiency, the left side of Fig 3) local
views of resolution ${32^{2}}$ covering only small areas (less than $50\%$) of
the original image. Similar to [56], we randomly select the crops in original
input, since we do not know at first which local observations have specific
physical processes. For ease of understanding, we characterize the global
observation contains $T$, $C$, $H$, and $W$ dimensions of the entire sequence
while local views with smaller height $H^{\prime}$ and width $W^{\prime}$. In
this fashion, we explicitly embody the local importance by inspecting the
local areas.

###  3.2 Adaptive Encoders Towards Global/Local Expressivity

After getting multi-grained observations, we employ adaptive encoders toward
acquiring embedding representations for information extraction. Let the tensor
${{\cal X}_{g}}\in{\mathbb{R}^{{\cal H}\times T\times H\times W}}$ denote
global observation and ${\cal X}_{l}\in\mathbb{R}^{{\cal{H}}\times T\times
C\times H^{{}^{\prime}}\times W^{{}^{\prime}}}$ represent the fine-grained
local views, adaptive local and global encoder stack $N_{e}$ blocks
(Conv2d+LayerNorm+Relu) for extracting spatial features, i.e., convoluting $C$
on $(H,W)$. The hidden feature of global embedding ($Z_{g}$) and local
embedding ($Z_{l}$) can be written as:

$\displaystyle\text{Global:}\;Z_{g}={\left\\{\operatorname{Relu}\left(\operatorname{LayerNorm}\left(\operatorname{Conv2d}\left(\mathcal{X}_{g}\right)\right)\right)\right\\}|}_{\times
N_{e}};$ (1)
$\displaystyle\text{Local:}\;Z_{l}={\left\\{\operatorname{Relu}\left(\operatorname{LayerNorm}\left(\operatorname{Conv2d}\left(\mathcal{X}_{l}\right)\right)\right)\right\\}|}_{\times
N_{e}},$

where the inputs are ${{\cal X}_{g}}$ / ${{\cal X}_{l}}$ and the output are
${Z_{g}}\in{\mathbb{R}^{{\hat{T}}\times{\hat{C}}\times{\hat{H}}\times{\hat{W}}}}$
and ${Z_{l}}\in{\mathbb{R}^{{\cal
H}\times{\hat{T}}\times{\hat{C}}\times{\hat{H}}^{{}^{\prime}}\times{\hat{W}}^{{}^{\prime}}}}$,
respectively. $(\cdot){|_{\times{N_{e}}}}$ indicates that the mapping blocks
have been applied $N_{e}$ times. In our work, we set the local encoder kernel
size to $3\times 3$ and the global encoder kernel size to $7\times 7$.

Figure 4: Fusion process of global and local pipelines.

Since there are multiple crops in the local pipeline, we conduct feedforward
process three times of local views but only one time of global view. After
characterizing the global view as a feature map before the decoder, we perform
local transformation three times. Finally, we conduct three times channel
fusion between global and local embeddings and then process the summation
operator. It’s worth emphasizing that during the forward propagation process,
we only calculate the global input once, and do not repeatedly calculate its
update. Once the global data is computed, we wait for the updated vector of
the local view to fuse, which significantly reduces computing power
expenditure (process is shown in Fig. 4, for ease of understanding, we omitted
the T dimension from the presentation).

###  3.3 Vector Quantization Prior bank

Recently, neural discrete representation learning [54, 55, 107, 108] has shown
great prominence in generative modeling of image [54, 79, 109], video [110,
80, 111] and audio [112, 113], to name just a few. Unlike continuous coding
models, neural discrete learning involves discrete codes with a prior
distribution. Drawing inspiration from this, we first involve this technology
in the realm of modeling physical processes, dubbed as vector quantization
prior bank or codebank, for highlighting the important features in latent
space rather than noise and imperceptible details.

Vector quantization prior bank. We define that the latent embedding space of
codebank $c=\left\\{c_{1},c_{2}\ldots
c_{\cal{O}}\right\\}\in\mathbb{R}^{\cal{O}\times D}$, where $\cal{O}$ is the
size of the discrete latent space (i.e., $\cal{O}$-way categorical) and $D$ is
the dimensionality of each quantized vector. As shown in Fig. 3, encoder
produces outputs $Z_{g}$ and $Z_{l}$, the discrete latent variables are
obtained through a nearest neighbor look-up in the shared embedding space as
demonstrated in Eq. 2. Here we define the posterior categorical distribution
${q_{i}}\left(Z\right)$ as the one-hot form:

$\small
q_{i}(Z)=\begin{cases}1,&i=\arg\mathop{\min}\limits_{j}\|Z-c_{j}\|_{2};\;\;Z=Z_{l}\;\;or\;\;Z_{g}\\\
0,&\text{otherwise}\end{cases}$ (2)

The input to general expert component is the corresponding latent vector
$c_{i}$ as shown in Eq. LABEL:vq2:

$\displaystyle{Z^{\left({VQ}\right)}}={c_{j}}\;\;\text{where}\;i=\arg\min_{j}\|Z-c_{j}\|_{2};$
(3)
$\displaystyle{Z^{\left({VQ}\right)}}=Z_{g}^{\left({VQ}\right)}\;or\;Z_{l}^{\left({VQ}\right)},$

where $Z_{g}^{\left({VQ}\right)}$ and $Z_{l}^{\left({VQ}\right)}$ are the
global and local quantized vectors from the codebank, respectively. It’s worth
emphasizing that there is no real gradient propagation here. To this end, we
adopt the reparameterization trick [54] and just copy gradient from expert
component input to encoder output. Afterward, we combine (sum) the outputs
from the codebank and the encoder to facilitate information fusion. We proceed
to consider the loss of our codebank part, here we follow [54] to give our
training objective as:
$\mathcal{L}^{(VQ)}=\left|\middle|\text{sg}[Z]-c\middle|\right|_{2}^{2}+\beta\left|\middle|Z-\text{sg}[c]|\right|_{2}^{2}$,
here sg denotes the stopgradient operator that is defined as identity at
forward computation time and has zero partial derivatives. Here, we have
removed the reconstruction loss as our output does not directly feed to the
decoder for reconstruction task, but rather is sent to the expert component.
Here existing two loss function ${\cal L}_{g}^{({VQ})}$ (global loss) and
${\cal L}_{l}^{({VQ})}$ (local loss), they share the same forms as shown
above. Since the physical process represents a sustained phenomenon that can
be better comprehended through previous observation, our code bank can learn
reliable priors and continuous physical events, rather than static ones.

Correlation with physical process. As illustrated above, our codebank can
acquire a dependable prior rather than a static one; physical processes are
ongoing phenomena that can be comprehended through past observations.
Additionally, our vector quantization strategy demonstrates proficiency in
identifying discrete events that define these physical processes.

###  3.4 Expert Component & Unified Decoder for Prediction

Optical flow field (also called motion field) refers to the pattern of
apparent motion of objects in a visual scene. An optical flow field is a two-
dimensional vector field that describes the direction and magnitude of motion
for each pixel in an image or sequence of images [72]. In our work, the local
and global expert modules process image sequences of varying shapes and
compute their corresponding optical flow field in natural physical processes
[114, 115, 116, 72]. Specifically, the local/global expert modules take the
embeddings with dimensions of
$({\hat{T},\hat{C},{{\hat{H}}^{\rm{{}^{\prime}}}},{{\hat{W}}^{\rm{{}^{\prime}}}}})$
and $({\hat{T},\hat{C},\hat{H},\hat{W}})$, respectively. The expert modules
process embedding sequences and output optical flow fields (Fig. 3).
Convolutional layers are utilized to extract features (local: $3\times 3$,
global: $7\times 7$), while deconvolution layers, also referred to as
transposed convolutional layers, are employed to upsample feature maps and
generate precise optical flow fields (also ensuring projecting onto the target
dimension). In some optical flow methods, deconvolution layers are referred to
as “optical flow decoders” [117, 118, 119].

In our implementation, the small area pixel motion field can capture more
local details, while the global motion field has a larger receptive field and
is capable of focusing on a wide pixel area. Expert components use CNN-based
frameworks 111During this process, several commonly-used CNN architectures,
such as U-Net, FlowNet, can be employed. These architectures comprise multiple
Conv2d and Deconv2d, as well as other types such as pooling, activation. for
extracting features and converting features into high-level semantic
representations for optical flow estimation. Generally, In general, the
optical flow estimation model takes the concatenation of two time-series
feature maps as input (assuming $(2c^{\prime},h^{\prime},w^{\prime})$
dimension) and outputs optical flow field $(u,v)$ in dimension
$(2,h_{of},w_{of})$ [119]. In this work, we first map the outputs of the
expert modules to a high-dimensional semantic representation of size
$({T,\tilde{C},\tilde{H},\tilde{W}})$ to mimic optical flow estimation. Then,
we also perform linear projection on the inputs before the expert modules,
mapping them to the same high-dimensional representation
$({T,\tilde{C},\tilde{H},\tilde{W}})$. For simplicity, we use convolutional
layer (Conv2d) for the linear projection, and more detailed information of
linear projection can be found in the appendix. We then match the outputs of
our analogical optical flow estimation and linear projection, and introduce an
additional optical flow estimation loss to enforce smoothness during the
matching phase:

$\footnotesize{{\cal
L}_{of}}=\mathop{\sum}\nolimits_{t}^{T}\mathop{\sum}\nolimits_{i,j}\left|{{I_{t}}\left({i,j}\right)-{I_{t+1}}\left({i+{u_{t}}\left({i,j}\right),j+{v_{t}}\left({i,j}\right)}\right)}\right|,$
(4)

where ${{\cal L}_{of}}$ denotes optical flow estimation loss, ${I_{t}}({i,j})$
stands for channel tensor of high-level semantics at $t$ point [45].
${u_{t}}\left({i,j}\right)$ and ${v_{t}}\left({i,j}\right)$ represent
displacement vectors from time $t$ to time $t+1$ point. Unlike traditional
models for optical flow estimation, our operations focus on high-level
semantics, and the $T$-dimension may not exhibit clear temporal
characteristics.

Correlation with physical process. Traditional ST frameworks do not have a
direct constraint on the smoothness of process. However, in this study, we
introduce supervision from the perspective of the motion vector field, which
can be considered a universal operator applicable to a wider range of physical
process tasks. By introducing the concept of the motion field, our model can
easily model the dynamical systems in a powerful and interpretable way.

###  3.5 Decoder

Fusing local and global representations. As there are multiple crops in the
local pipeline, we conduct feedforward process three times of local views but
only one time of global view. As shown in Fig. 3, we reduce the global
representation on the first dimension using a summation operation.
Subsequently, we execute channel fusion between global and local embeddings.

Decoder and optimization. In our model, we use only three deconvolutional
layers for the final decoding (with a DC block and $N_{d}$=3), which we find
to be both simpler and more efficient, without the need to introduce
additional parameters. We optimize our framework by adopting loss function
${\cal L}={{\cal L}_{of}}+{{\cal L}^{\left({VQ}\right)}}+{{\cal L}_{mse}}$,
where ${\cal L}_{of}$, ${\cal L}^{\left({VQ}\right)}$ and ${\cal L}_{mse}$
denote optical flow, codebank and MSE loss, respectively.

##  4 Experiments

In this section, we experimentally assess, analyze, and rationalize the impact
of our framework on three real-world physical datasets and a synthetic
dataset. We follow some previous endeavors [114, 47, 52, 120, 51, 35] to
unitize three metrics: 1) Structural Similarity Index Measure (SSIM): SSIM is
a metric for measuring the similarity between two images. 2) Peak Signal-to-
Noise Ratio (PSNR): PSNR is a measure of video or image quality that compares
the original signal to the compressed or transmitted signal. 3) MSE (Mean
Squared Error): MSE calculates the average of the squared differences between
the predicted and actual values. In our experiments, we consistently use the
previous 10 frames of observational data to predict the subsequent 10 frame
for base settings.

TABLE I: Statistics of the datasets used in the our paper. Dataset | Size | Seq. | Len. | Spatial Resolution
---|---|---|---|---
train | val | test | in | out | $H\times W$
Reaction–diffusion | 1500 | 500 | 500 | 10 | 10 | $128\times 128$
Fire | 1500 | 500 | 500 | 10 | 10 | $128\times 128$
SEVIR | 4500 | 500 | 1000 | 10 | 10 | $384\times 384$
MovingMNIST | 1500 | 500 | 500 | 10 | 10 | $128\times 128$

Datasets & Main Baselines. We conduct experiments on three spatio-temporal
physical datasets (Reaction–diffusion [35], Fire [35], SEVIR [121]) and a
synthetic dataset (MovingMNIST [88]), to verify the effectiveness of our
approach. Except for SEVIR (4500 images for training, 500, 1000 for validation
and testing), we chose 80%, 10%, and 10% of the raw data as the training,
validation, and test datasets, respectively. We compare our framework with
state-of-the-art physical constraint models (I, II, VI) and video
understanding methods (III, IV, V):

  * •

(I) Ad-fusion [114] demonstrates how to use physical principles to design a
neural network for complex prediction tasks, specifically for transport
problems that follow advection-diffusion principles.

  * •

(II) PhyDNet [47] has two branches and separates PDE dynamics from unknown
complementary information. It also introduces a new recurrent physical cell
for PDE-constrained prediction in latent space, inspired by data assimilation
techniques.

  * •

(III) PredRNN-V2 [52] possesses a zigzag memory flow that goes up and down all
layers, allowing visual dynamics to communicate and adding to the original
memory cell of LSTM.

  * •

(IV) MotionRNN [120] proposes a new RNN unit called MotionGRU, which can
transient variation and motion trend in a unified way.

  * •

(V) Earthformer [51] is a Transformer-based architecture, with a novel module
termed Cuboid Attention mechanism to efficiently learn spatiotemporal
representations.

  * •

(VI) MathRel [35] presents a principle to determine the probable number of
state variables in an observed system and their potential attributes.

Frameworks specifically for PDEs. To comprehensively validate the
generalization ability, we also conducted additional tests using some models
specifically designed for addressing continuous and PDE problems. These models
include Fourier Neural Operator (FNO) [122], Adaptive Fourier Neural Operator
(AFNO) [123], and TF-Net [60]. All three models are tailored to handle
continuous properties, especially PDEs.

Frameworks for video understanding. We also experimented with some video
understanding baselines, such as MIM [124] (CVPR 2019), CrevNet [106] (ICLR
2020) and SimVP [103] (CVPR 2022). Since our work can also be interpreted as a
video frame prediction task, we aim to compare these frameworks to demonstrate
the scalability of our algorithm.

TABLE II: Implementation details on four datasets. Dataset | learning rate | optimizer | $\beta$ | epoch
---|---|---|---|---
Reaction–diffusion | 0.01 | SGD | 0.99 | 1000
Fire | 0.01 | SGD | 0.99 | 1000
SEVIR | 0.01 | SGD | 0.99 | 20000
MovingMNIST | 0.01 | SGD | 0.99 | 500

Experimental settings. We train our framework and the state-of-the-art methods
(SOTAs) with the same experimental settings, including learning rate,
optimizer, etc. For ease of understanding, we summarize detail in Tab. II. In
our paper, we generate encoder configurations (ours-$S/B/L$) by adjusting
$N_{e}=2,4,8$. Concretely, small ($S$), base ($B$) and large ($L$) models have
$2$, $4$ and $8$ blocks, respectively. We also test the configuration of the
decoder, including the depth ($2,3,4,5,6$) and form of the blocks (DC, CL+DC,
CL+DC+R)222Here we abbreviate convolutional, deconvolutional and residual
layer as CL, DC and R, respectively.. However, we find that use three DC
blocks can yield the best performance. Specific experimental results will be
presented later in this paper. It is important to note that all results
discussed herein are based on 3 DC block and $256\times 64$ codebank size
without particular emphasis. All experiments are conducted on a server with 4
V100 GPUs using a batch size of 64.

TABLE III: Comparison with the state-of-the-art methods on MSE, SSIM and PSNR.
We report the mean and standard deviation of the results from three
experimental runs.

| Metrics | Ad-fusion | PhyDNet | PredRNN-V2 | MotionRNN | Earthformer | MathRel | PhysicNet (ours)
---|---|---|---|---|---|---|---|---
| Reaction
---
-diffusion
|  MSE ($\downarrow$)
---
SSIM ($\uparrow$)
PSNR ($\uparrow$)
| ${30.04_{\pm 0.45}}$
---
${0.866_{\pm 0.002}}$
${29.23_{\pm 0.23}}$
| ${9.77_{\pm 0.28}}$
---
${0.837_{\pm 0.039}}$
${27.25_{\pm 0.37}}$
| ${29.60_{\pm 0.39}}$
---
${0.913_{\pm 0.019}}$
${29.02_{\pm 0.02}}$
| ${6.69_{\pm 0.34}}$
---
${0.866_{\pm 0.024}}$
${28.97_{\pm 0.02}}$
| ${6.61_{\pm 0.17}}$
---
${0.864_{\pm 0.015}}$
${28.47_{\pm 0.04}}$
| ${6.44_{\pm 0.33}}$
---
${0.906_{\pm 0.027}}$
${33.72_{\pm 0.05}}$
| ${5.88_{\pm 0.01}}$
---
${0.920_{\pm 0.013}}$
${35.27_{\pm 0.02}}$
Fire |  | MSE
---
SSIM ($\uparrow$)
PSNR ($\uparrow$)
| ${7.78_{\pm 0.24}}$
---
${0.891_{\pm 0.017}}$
${23.58_{\pm 0.03}}$
| ${4.41_{\pm 0.11}}$
---
${0.870_{\pm 0.019}}$
${27.55_{\pm 0.04}}$
| ${7.21_{\pm 0.20}}$
---
${0.901_{\pm 0.009}}$
${29.05_{\pm 0.03}}$
| ${3.01_{\pm 0.16}}$
---
${0.903_{\pm 0.017}}$
${27.25_{\pm 0.04}}$
| ${2.19_{\pm 0.08}}$
---
${0.891_{\pm 0.007}}$
${29.03_{\pm 0.04}}$
| ${1.20_{\pm 0.04}}$
---
${0.917_{\pm 0.004}}$
${28.97_{\pm 0.03}}$
| ${0.98_{\pm 0.02}}$
---
${0.945_{\pm 0.009}}$
${29.92_{\pm 0.02}}$
| SEVIR
---
(Ours-$L$)
|  MSE/10 ($\downarrow$)
---
SSIM ($\uparrow$)
PSNR ($\uparrow$)
| ${10.54_{\pm 0.33}}$
---
${0.750_{\pm 0.018}}$
${27.55_{\pm 0.23}}$
| ${8.19_{\pm 0.34}}$
---
${0.870_{\pm 0.013}}$
${29.23_{\pm 0.37}}$
| ${6.73_{\pm 0.26}}$
---
${0.891_{\pm 0.008}}$
${27.25_{\pm 0.20}}$
| ${7.98_{\pm 0.32}}$
---
${0.870_{\pm 0.007}}$
${29.01_{\pm 0.16}}$
| ${5.82_{\pm 0.21}}$
---
${0.901_{\pm 0.005}}$
${28.97_{\pm 0.09}}$
| ${6.63_{\pm 0.26}}$
---
${0.902_{\pm 0.004}}$
${29.25_{\pm 0.20}}$
| ${4.84_{\pm 0.17}}$
---
${0.911_{\pm 0.005}}$
${33.72_{\pm 0.07}}$

  * 1

We take SSIM as the most representative index.

###  4.1 Performance on Real-world Datasets

TABLE IV: The effect of the encoder and codebank size on SSIM. w/o denotes the remove of the codebank | codebank size | Encoder size
---|---|---
|  Ours-$S$ |  Ours-$B$ |  Ours-$L$
| Reaction
---
-diffusion
| w/o
---
$128\times 32$
$256\times 64$
$512\times 128$
| $0.765$
---
$0.877$
$0.901$
$0.867$
| $0.765$
---
$0.784$
$0.920$
$0.892$
| $0.741$
---
$0.817$
$0.917$
$0.817$
Fire |  | w/o
---
$128\times 32$
$256\times 64$
$512\times 128$
| $0.877$
---
$0.923$
$0.927$
$0.892$
| $0.896$
---
$0.941$
$0.945$
$0.878$
| $0.831$
---
$0.935$
$0.939$
$0.893$
| SEVIR
---
| w/o
---
$128\times 32$
$256\times 64$
$512\times 128$
| $0.732$
---
$0.805$
$0.867$
$0.855$
| $0.741$
---
$0.831$
$0.901$
$0.881$
| $0.753$
---
$0.857$
$0.911$
$0.894$

Model comparison. Tab. III shows the results on the three physical datasets,
where we adopt the base model on Reaction-diffusion and Fire datasets, and the
large model on SEVIR. Primarily, our framework consistently surpasses the
previous SOTAs by substantial performance margins across all physical
datasets, e.g., performs 0.81$\sim$4.47 better than MathRel across three
datasets on PSNR, and achieves 0.007$\sim$0.028 improvements on SSIM. For
simplicity, we use SSIM as the major metric for further discussion. From Tab.
III, we also observe that while some physical methods may exhibit good
performances on certain data, their performance may not be satisfactory in
other scenarios. In particular, MathRel shows prominent performances on SEVIR
and Reaction-diffusion, however, it encounters $\sim$0.1 deterioration on SSIM
over the video understanding method PredRNN-v2.

Figure 5: Effects of decoder forms and the decoder depth on SSIM. Datapoints
are annotated with SSIM of the model and their size is proportional to areas.

Scalability study. We further evaluate the scalability of our framework,
including different encoder sizes (Ours-$S/B/L$), decoder forms, and codebank
sizes. We list our observations (Obs) as follows. Obs.1: Tab. IV demonstrates
that the size of the codebank should not be too large. We believe that an
excessive dimension may increase the burden on the model, leading to the
learning of noise. Obs.2: encoder sizes may need to vary based on the features
of different datasets. As the spatial resolution of SEVIR dataset is
384$\times$384 (others are 128$\times$128) and this dataset has the largest
data volume, it performs the best with our-$L$ design. In contrast, using
smaller versions (i.e., our-$S/B$) performs better on the other two datasets.
These facts validate that our encoder is highly scalable, enabling it to be
customized for specific datasets, thereby significantly enhancing its real-
world performance. Obs.3: Compare with encoder size, improved performance can
be attained by utilizing a moderate number of DC blocks, rather than an
excessive amount of them. As shown in Fig. 5, under three blocks, CL+DC and
CL+DC+R suffer $\sim 0.2$ deterioration on SSIM with only DC settings (about
0.9 vs. 0.7). To strike a balance between computational resources and
performance, we have opted for a solution that involves 3 DC blocks. See
decoder detailed forms in Fig. 6.

Figure 6:  The decoer forms of DC, CL+DC, CL+DC+R.  TABLE V: 3-run results on MovingMNIST. The mean and standard deviation are reported. Metrics | Ad-fusion | PhyDNet | PredRNN-V2 | MotionRNN | Earthformer | MathRel |  Ours-$B$ |  Ours-$L$
---|---|---|---|---|---|---|---|---
PSNR | ${26.27_{\pm 0.31}}$ | ${29.99_{\pm 0.29}}$ | ${29.17_{\pm 0.26}}$ | ${30.19_{\pm 0.23}}$ | ${31.33_{\pm 0.21}}$ | ${37.17_{\pm 0.27}}$ | ${38.38_{\pm 0.17}}$ | ${37.46_{\pm 0.28}}$
SSIM | ${0.750_{\pm 0.04}}$ | ${0.869_{\pm 0.05}}$ | ${0.901_{\pm 0.06}}$ | ${0.891_{\pm 0.05}}$ | ${0.917_{\pm 0.07}}$ | ${0.937_{\pm 0.04}}$ | ${0.948_{\pm 0.03}}$ | ${0.921_{\pm 0.01}}$

Visualization. Fig. 7 depicts the visualization results of our framework on
real-world infrared satellite imagery dataset SEVIR, from which we can easily
find that our model not only maintains the corresponding change
characteristics globally, but also preserves local fidelity remarkably well.
This indicates that our model is capable of capturing both global and local
spatiotemporal patterns in the data, making it a promising solution for a wide
range of applications that require accurate and efficient modeling of complex
spatiotemporal dynamics.

Figure 7: Visualization on SEVIR.

###  4.2 Results on MovingMNIST

In this section, we provide more results to better examine our framework, we
follow Earthformer [51] to use the public MovingMNIST dataset, and our target
is to predict the future 10 frames for each sequence conditioned on the first
10 frames.

Model comparison. The dynamics of the synthetic MovingMNIST dataset, in which
the digits move independently with constant speed, are oversimplified when
compared to the dynamics of complex physical systems. From Tab. V, we can find
that our frameworks (ours-$B/L$) are capable of learning long-range
interactions among digits and accurately predicting their future motion
trajectories. Furthermore, our model leverages global and local insights to
more precisely predict the positions of the digits (achiving $\sim 1.21/0.011$
improvements on PSNR and SSIM, respectively). We further showcase the
visualization examples of our framework with SOTA in Fig. 8. Firstly, it
excels in capturing structural information, outperforming existing models in
local processing. Secondly, the feature map generated by the vector
quantization prior bank effectively represents the original input information
with both global and local fidelity.

Figure 8: Visualization on the synthetic dataset (MovingMNIST).

###  4.3 Ablation study

While our proposal is quite effective, we believe there still exists
unapprehended questions. We are eager to know: ${{\cal Q}_{1}}:$ What is the
role of local modules in enhancing local insights? ${{\cal Q}_{2}}:$ What
roles do the general expert components play? ${{\cal Q}_{3}}:$ How the ${{\cal
L}_{of}}$ influence the performance?

Answer of ${{\cal Q}_{1}}$ and ${{\cal Q}_{2}}$. As illustrated in Tab. VI and
VII, the model’s performance experiences varying degrees of decline when the
local pipeline and general expert components are removed. These results
demonstrate the impressive performance of our model, which clearly benefits
from the inclusion of both the local pipeline and general expert components.
By leveraging these complementary sources of information, our model achieves
superior accuracy and robustness across a range of tasks and domains, making
it a highly promising solution for real-world applications.

TABLE VI: Ablations of local-level pipeline. | Top (Local) level pipeline (SSIM)
---|---
Reaction-diffusion ($B$) |  ${0.920_{\pm 0.013}}$ ($\checkmark$) |  ${0.911_{\pm 0.021}}$ ($\times$)
Fire ($B$) |  ${0.945_{\pm 0.009}}$ ($\checkmark$) |  ${0.935_{\pm 0.011}}$($\times$)
SEVIR ($L$) |  ${0.911_{\pm 0.005}}$ ($\checkmark$) |  ${0.897_{\pm 0.007}}$ ($\times$)
MovingMNIST ($B$) |  ${0.921_{\pm 0.005}}$ ($\checkmark$) |  ${0.918_{\pm 0.006}}$ ($\times$)
TABLE VII: Implementation details on four datasets. | Ablations of general expert components.
---|---
Reaction-diffusion ($B$) |  ${0.920_{\pm 0.013}}$ ($\checkmark$) |  ${0.897_{\pm 0.031}}$ ($\times$)
Fire ($B$) |  ${0.945_{\pm 0.009}}$ ($\checkmark$) |  ${0.917_{\pm 0.021}}$ ($\times$)
SEVIR ($L$) |  ${0.911_{\pm 0.005}}$ ($\checkmark$) |  ${0.893_{\pm 0.011}}$ ($\times$)
MovingMNIST ($B$) |  ${0.921_{\pm 0.005}}$ ($\checkmark$) |  ${0.903_{\pm 0.012}}$ ($\times$)

Answer of ${{\cal Q}_{3}}$. As illustrated in Tab VIII, we choose SOTAs
compare with our model (without (w/o) ${{\cal L}_{of}}$) to verify the
influence of optical flow estimation module. The results demonstrate a clear
decline in both SSIM and PSNR values when the optical flow estimation module
is removed. The reduction in these key metrics signifies a degradation in the
model’s ability to maintain structural integrity and signal quality. These
findings reaffirm our decision to incorporate this module.

TABLE VIII: 3-run results on SEVIR. The mean and standard deviation are reported. Metrics | PredRNN-V2 | Earthformer | MathRel |  Ours-$L$ |  Ours-$L$ w/o ${{\cal L}_{of}}$
---|---|---|---|---|---
PSNR | ${29.01_{\pm 0.16}}$ | ${28.97_{\pm 0.09}}$ | ${29.25_{\pm 0.20}}$ | ${33.72_{\pm 0.07}}$ | ${30.14_{\pm 0.09}}$
SSIM | ${0.891_{\pm 0.08}}$ | ${0.901_{\pm 0.005}}$ | ${0.902_{\pm 0.04}}$ | ${0.911_{\pm 0.005}}$ | ${0.875_{\pm 0.011}}$

###  4.4 Compare with video understanding frameworks

We further explored various video understanding benchmarks, including MIM
[124] (CVPR 2019), CrevNet [106] (ICLR 2020), and SimVP [103] (CVPR 2022).
Here we choose SEVIR and MovingMNIST as benchmarks. As depicted in Tab. IX, on
the MovingMNIST dataset, SimVP, introduced at CVPR 2022, yielded slightly
better results than MIM and CrevNet, with a PSNR of 38.02 and an SSIM score of
0.939. Notably, our method, denoted as Ours-$B$, surpassed all the above,
reaching a PSNR of 38.01 and an SSIM score of 0.945. On SEVIR, our method,
designated as Ours-$L$, outperformed the mainstream models by achieving a
significant PSNR of 33.72 and an SSIM score of 0.911. These results not only
underscore the advanced performance of our proposed framework but also
highlight its superiority over existing mainstream methodologies in video
understanding.

TABLE IX: Comparison among different and additional video understanding baselines across MovingMNIST and SEVIR datasets. To simplify comprehension, we run three times experiments and report the results of SSIM and PSNR. Dataset | Method | Conference | PSNR | SSIM
---|---|---|---|---
MovingMNIST | MIM | CVPR 2019 | 35.98 | 0.909
CrevNet | ICLR 2020 | 37.69 | 0.934
SimVP | CVPR 2022 | 38.02 | 0.939
Ours-$B$ | – | 38.01 | 0.945
SEVIR | MIM | CVPR 2019 | 29.99 | 0.854
CrevNet | ICLR 2020 | 31.45 | 0.895
SimVP | CVPR 2022 | 31.88 | 0.897
Ours-$L$ | – | 33.72 | 0.911

###  4.5 Additional experiments on SEVIR with Fourier-based Neural Network

In this part, we turn to quantificationally analyze our model generalizability
compare with specialized models [125, 123], which specifically designed for
addressing continuous and PDE problems. These models include Fourier Neural
Operator (FNO) [122], Adaptive Fourier Neural Operator (AFNO) [123], and TF-
Net [60]. All three models leverage the properties of the Fourier transform.
The Fourier transform is a technique that converts functions from the time
domain (or spatial domain) to the frequency domain, enabling these models to
process problems in the frequency domain and utilize certain mathematical
properties of the Fourier transform. We showcase the results in Tab X.

TABLE X: Comparison among different and additional Fourier-based network baselines across SEVIR. To simplify comprehension, we run three times experiments and report the results of SSIM and PSNR. Dataset | Method | MSE | SSIM | PSNR
---|---|---|---|---
SEVIR | AFNO | ${15.87_{\pm 0.89}}$ | ${0.521_{\pm 0.009}}$ | ${23.43_{\pm 0.18}}$
TF-Net | ${8.47_{\pm 0.33}}$ | ${0.774_{\pm 0.011}}$ | ${30.12_{\pm 0.23}}$
FNO | ${12.21_{\pm 0.75}}$ | ${0.583_{\pm 0.012}}$ | ${26.43_{\pm 0.13}}$
PhysicNet ($L$) | ${4.84_{\pm 0.17}}$ | ${0.911_{\pm 0.005}}$ | ${33.72_{\pm 0.07}}$

Based on the comparison presented in Table X, we can observe that our proposed
model, PhysicNet ($L$), consistently outperforms other networks across all
metrics on the SEVIR dataset. The Mean Square Error (MSE) achieved by
PhysicNet is significantly lower, indicating a superior predictive accuracy.
Additionally, PhysicNet’s Structural Similarity Index Measure (SSIM) is the
highest among the models, indicating that it captures the structural detail in
the prediction more effectively. Similarly, PhysicNet also delivers the
highest Peak Signal-to-Noise Ratio (PSNR), reflecting its superior ability to
maintain signal quality amidst potential noise.

The results demonstrate the exceptional generalization capability of PhysicNet
in comparison to other specialized models specifically designed for continuous
and PDE problems. While Fourier-based methods, like AFNO, FNO, and TF-Net,
leverage the mathematical properties of Fourier transforms to work in the
frequency domain, PhysicNet appears to be more efficient and accurate in
capturing and predicting complex patterns, possibly due to its design and
learning algorithm. In summary, PhysicNet represents a substantial step
forward in addressing complex continuous and PDE problems, offering a
promising approach for future research and practical applications.

###  4.6 Visual Comparison of Model Errors

In this section, we further delve into an analysis through visualization to
assess our model’s capability in capturing fine-grained details. We showcase
the visual errors of our model and compare them with those of current
mainstream frameworks. The weather dataset, SEVIR, is utilized as a benchmark
given its high-resolution pixel data and rich dimensional information.

Figure 9: Visualization on real-world infrared satellite imagery dataset
(SEVIR). Figure 10: Visualization on real-world infrared satellite imagery
dataset (SEVIR).

From the visualization results, it is readily evident that our model exhibits
a smaller prediction error compared to MathRel. Furthermore, our approach
handles fine details more adeptly. As observed in Fig 9 and 10, although
MathRel already achieves commendable prediction outcomes, it still falls short
in capturing local details, especially at the outline information. In
contrast, PhysicNet adeptly addresses this limitation, which corroborates our
hypothesis of joint global and local modeling.

##  5 Complexity analysis

To better validate the efficiency of the PhysicNet algorithm, we include
efficiency comparisons in the manuscript, which aid in a deeper understanding
of our algorithm’s reliability and effectiveness. As shown in Tab XI, we
observe significant variations in memory usage, computational complexity
(FLOPs), the number of parameters (in millions), and the training time per
epoch. The experimental analysis demonstrates that our PhysicNet model is an
effective algorithm among the compared models, achieving a good balance
between resource usage and computational performance. It utilizes less memory
and has lower computational complexity than the most demanding model, MIM,
while maintaining a competitive training time. This efficiency suggests that
PhysicNet is an advantageous choice for tasks requiring a judicious balance of
speed and computational resources.

TABLE XI: Comparison of different models in terms of memory usage, computational complexity (FLOPs), number of parameters (Params), and training time per epoch. Model | Memory (MB) | FLOPs (G) | Params (M) | Training time
---|---|---|---|---
FNO | 8.41 | 12.31 | 7.27 | 32s / epoch
MIM | 2331.34 | 179.21 | 38.44 | 154s / epoch
PredRNN-V2 | 1721.22 | 117.33 | 23.92 | 126s / epoch
SimVP | 421.17 | 17.24 | 46.82 | 65s / epoch
PhysicNet | 378.87 | 15.86 | 35.41 | 58s / epoch

##  6 Conclusion & Future work

In conclusion, our model proposes a vector quantization-based discrete
learning technique, combined with an expert module that captures the
underlying patterns of motion in order to model physical processes. By
leveraging these key components, our model achieves superior performance on a
variety of challenging tasks, highlighting the effectiveness and robustness of
our approach. We believe that our model has the potential to significantly
advance the field of machine learning and enable new breakthroughs in our
understanding of complex physical systems. We emphasize that the work
presented in this paper is primarily foundational in ST context.

Although our work has shown promising results on real-world physical datasets,
there are still some issues related to interpretability that need to be
further addressed. In particular, we plan to explore the use of manifold
learning techniques in our model and conduct more experiments on physical data
in other fields, which could help to inject greater interpretability into our
results. By leveraging these methods and datasets, we hope to better
understand the underlying factors that drive the performance of our model and
gain new insights into the physical systems.

##  7 Acknowledgment

This paper is partially supported by the National Natural Science Foundation
of China (No.62072427, No.12227901), the Project of Stable Support for Youth
Team in Basic Research Field, CAS (No.YSBR-005), Academic Leaders Cultivation
Program, USTC.

## References

  * [1] P. Benacerraf, “Mathematical truth,” _The Journal of Philosophy_ , vol. 70, no. 19, pp. 661–679, 1973.
  * [2] A. Newell, “Physical symbol systems,” _Cognitive science_ , vol. 4, no. 2, pp. 135–183, 1980.
  * [3] R. W. Pryor, _Multiphysics modeling using COMSOL®: a first principles approach_. Jones & Bartlett Publishers, 2009.
  * [4] D. A. Roberts, S. Yaida, and B. Hanin, _The principles of deep learning theory_. Cambridge University Press Cambridge, MA, USA, 2022.
  * [5] M. Bürkle, U. Perera, F. Gimbert, H. Nakamura, M. Kawata, and Y. Asai, “Deep-learning approach to first-principles transport simulations,” _Physical Review Letters_ , vol. 126, no. 17, p. 177701, 2021.
  * [6] Y. Yu, K. H. R. Chan, C. You, C. Song, and Y. Ma, “Learning diverse and discriminative representations via the principle of maximal coding rate reduction,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 9422–9434, 2020.
  * [7] K. H. R. Chan, Y. Yu, C. You, H. Qi, J. Wright, and Y. Ma, “Redunet: A white-box deep network from the principle of maximizing rate reduction,” _The Journal of Machine Learning Research_ , vol. 23, no. 1, pp. 4907–5009, 2022.
  * [8] P. W. Anderson, “More is different: broken symmetry and the nature of the hierarchical structure of science.” _Science_ , vol. 177, no. 4047, pp. 393–396, 1972.
  * [9] S. Pierson, “Corpore cadente…: Historians discuss newton’s second law,” _Perspectives on Science_ , vol. 1, no. 4, pp. 627–658, 1993.
  * [10] M. Sharan, A. K. Yadav, M. Singh, P. Agarwal, and S. Nigam, “A mathematical model for the dispersion of air pollutants in low wind conditions,” _Atmospheric Environment_ , vol. 30, no. 8, pp. 1209–1220, 1996.
  * [11] B. A. Egan and J. R. Mahoney, “Numerical modeling of advection and diffusion of urban area source pollutants,” _Journal of Applied Meteorology and Climatology_ , vol. 11, no. 2, pp. 312–322, 1972.
  * [12] X. Shan, X.-F. Yuan, and H. Chen, “Kinetic theory representation of hydrodynamics: a way beyond the navier–stokes equation,” _Journal of Fluid Mechanics_ , vol. 550, pp. 413–441, 2006.
  * [13] M. F. McCracken, “Artificial neural networks in fluid dynamics: A novel approach to the navier-stokes equations,” in _Proceedings of the Practice and Experience on Advanced Research Computing_ , 2018, pp. 1–4.
  * [14] D. Arrowsmith and C. M. Place, _Dynamical systems: differential equations, maps, and chaotic behaviour_. CRC Press, 1992, vol. 5.
  * [15] M. W. Hirsch, “The dynamical systems approach to differential equations,” _Bulletin of the American mathematical society_ , vol. 11, no. 1, pp. 1–64, 1984.
  * [16] M. W. Hirsch, S. Smale, and R. L. Devaney, _Differential equations, dynamical systems, and an introduction to chaos_. Academic press, 2012.
  * [17] T. C. Sideris, _Ordinary differential equations and dynamical systems_. Springer, 2013, vol. 2.
  * [18] F. Verhulst, _Nonlinear differential equations and dynamical systems_. Springer Science & Business Media, 2006.
  * [19] S. Wiggins, S. Wiggins, and M. Golubitsky, _Introduction to applied nonlinear dynamical systems and chaos_. Springer, 2003, vol. 2, no. 3.
  * [20] J. K. Hale and H. Koçak, _Dynamics and bifurcations_. Springer Science & Business Media, 2012, vol. 3.
  * [21] J. Humar, _Dynamics of structures_. CRC press, 2012.
  * [22] L. Wang, Q. Zhou, and S. Jin, “Physics-guided deep learning for power system state estimation,” _Journal of Modern Power Systems and Clean Energy_ , vol. 8, no. 4, pp. 607–615, 2020.
  * [23] V. Harish and A. Kumar, “A review on modeling and simulation of building energy systems,” _Renewable and sustainable energy reviews_ , vol. 56, pp. 1272–1292, 2016.
  * [24] P. Moin and K. Mahesh, “Direct numerical simulation: a tool in turbulence research,” _Annual review of fluid mechanics_ , vol. 30, no. 1, pp. 539–578, 1998.
  * [25] R. S. Rogallo and P. Moin, “Numerical simulation of turbulent flows,” _Annual review of fluid mechanics_ , vol. 16, no. 1, pp. 99–137, 1984.
  * [26] S. A. Orszag and G. Patterson Jr, “Numerical simulation of three-dimensional homogeneous isotropic turbulence,” _Physical review letters_ , vol. 28, no. 2, p. 76, 1972.
  * [27] J. Sanyal, S. Vásquez, S. Roy, and M. Dudukovic, “Numerical simulation of gas–liquid dynamics in cylindrical bubble column reactors,” _Chemical Engineering Science_ , vol. 54, no. 21, pp. 5071–5083, 1999.
  * [28] M. A. van der Hoef, M. van Sint Annaland, N. Deen, and J. Kuipers, “Numerical simulation of dense gas-solid fluidized beds: a multiscale modeling strategy,” _Annu. Rev. Fluid Mech._ , vol. 40, pp. 47–70, 2008.
  * [29] P. Yang, X. Tan, and W. Xin, “Experimental study and numerical simulation for a storehouse fire accident,” _Building and Environment_ , vol. 46, no. 7, pp. 1445–1459, 2011.
  * [30] T. Ma and J. Quintiere, “Numerical simulation of axi-symmetric fire plumes: accuracy and limitations,” _Fire Safety Journal_ , vol. 38, no. 5, pp. 467–492, 2003.
  * [31] J. Bakarji, K. Champion, J. N. Kutz, and S. L. Brunton, “Discovering governing equations from partial measurements with deep delay autoencoders,” _arXiv preprint arXiv:2201.05136_ , 2022.
  * [32] X. He, Q. He, and J.-S. Chen, “Deep autoencoders for physics-constrained data-driven nonlinear materials modeling,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 385, p. 114034, 2021.
  * [33] R. H. Cannon, _Dynamics of physical systems_. Courier Corporation, 2003.
  * [34] B. P. Bezruchko and D. A. Smirnov, _Extracting knowledge from time series: An introduction to nonlinear empirical modeling_. Springer Science & Business Media, 2010.
  * [35] B. Chen, K. Huang, S. Raghupathi, I. Chandratreya, Q. Du, and H. Lipson, “Automated discovery of fundamental variables hidden in experimental data,” _Nature Computational Science_ , vol. 2, no. 7, pp. 433–442, 2022.
  * [36] S. K. Godunov and I. Bohachevsky, “Finite difference method for numerical computation of discontinuous solutions of the equations of fluid dynamics,” _Matematičeskij sbornik_ , vol. 47, no. 3, pp. 271–306, 1959.
  * [37] M. Bär, R. Hegger, and H. Kantz, “Fitting partial differential equations to space-time dynamics,” _Physical Review E_ , vol. 59, no. 1, p. 337, 1999\.
  * [38] S. Yanchuk and G. Giacomelli, “Spatio-temporal phenomena in complex systems with time delays,” _Journal of Physics A: Mathematical and Theoretical_ , vol. 50, no. 10, p. 103001, 2017.
  * [39] C. Baker, G. Bocharov, and C. Paul, “Mathematical modelling of the interleukin–2 t–cell system: a comparative study of approaches based on ordinary and delay differential equation,” _Computational and Mathematical Methods in Medicine_ , vol. 1, no. 2, pp. 117–128, 1997.
  * [40] A. Graves and A. Graves, “Long short-term memory,” _Supervised sequence labelling with recurrent neural networks_ , pp. 37–45, 2012.
  * [41] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [42] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2017.
  * [43] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” _IEEE signal processing magazine_ , vol. 35, no. 1, pp. 53–65, 2018.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [45] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [46] E. de Bezenac, A. Pajot, and P. Gallinari, “Deep learning for physical processes: Incorporating prior scientific knowledge,” in _International Conference on Learning Representations_.
  * [47] V. L. Guen and N. Thome, “Disentangling physical dynamics from unknown factors for unsupervised video prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 474–11 484.
  * [48] S. Seo, C. Meng, and Y. Liu, “Physics-aware difference graph networks for sparsely-observed dynamics,” in _International conference on learning representations_ , 2020.
  * [49] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” _Journal of Computational physics_ , vol. 378, pp. 686–707, 2019.
  * [50] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang, “Physics-informed machine learning,” _Nature Reviews Physics_ , vol. 3, no. 6, pp. 422–440, 2021.
  * [51] Z. Gao, X. Shi, H. Wang, Y. Zhu, Y. B. Wang, M. Li, and D.-Y. Yeung, “Earthformer: Exploring space-time transformers for earth system forecasting,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 25 390–25 403, 2022.
  * [52] Y. Wang, H. Wu, J. Zhang, Z. Gao, J. Wang, S. Y. Philip, and M. Long, “Predrnn: A recurrent neural network for spatiotemporal predictive learning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 2, pp. 2208–2225, 2022.
  * [53] T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez, and P. W. Battaglia, “Learning mesh-based simulation with graph networks,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=roNqYL0_XP
  * [54] A. Van Den Oord, O. Vinyals _et al._ , “Neural discrete representation learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [55] V. Fortuin, M. Hüser, F. Locatello, H. Strathmann, and G. Rätsch, “Som-vae: Interpretable discrete representation learning on time series,” _arXiv preprint arXiv:1806.02199_ , 2018.
  * [56] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 9650–9660.
  * [57] R. Stewart and S. Ermon, “Label-free supervision of neural networks with physics and domain knowledge,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.
  * [58] N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti, “Visual interaction networks: Learning a physics simulator from video,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [59] K. Kashinath, M. Mustafa, A. Albert, J. Wu, C. Jiang, S. Esmaeilzadeh, K. Azizzadenesheli, R. Wang, A. Chattopadhyay, A. Singh _et al._ , “Physics-informed machine learning: case studies for weather and climate modelling,” _Philosophical Transactions of the Royal Society A_ , vol. 379, no. 2194, p. 20200093, 2021.
  * [60] R. Wang, K. Kashinath, M. Mustafa, A. Albert, and R. Yu, “Towards physics-informed deep learning for turbulent flow prediction,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 1457–1466.
  * [61] T. Xue, A. Beatson, S. Adriaenssens, and R. Adams, “Amortized finite element analysis for fast pde-constrained optimization,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 10 638–10 647.
  * [62] A. Sanchez-Gonzalez, N. Heess, J. T. Springenberg, J. Merel, M. Riedmiller, R. Hadsell, and P. Battaglia, “Graph networks as learnable physics engines for inference and control,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 4470–4479.
  * [63] J. Wu, E. Lu, P. Kohli, B. Freeman, and J. Tenenbaum, “Learning to see physics via visual de-animation,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [64] L. Lu, P. Jin, and G. E. Karniadakis, “Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators,” _arXiv preprint arXiv:1910.03193_ , 2019.
  * [65] J. Sirignano and K. Spiliopoulos, “Dgm: A deep learning algorithm for solving partial differential equations,” _Journal of computational physics_ , vol. 375, pp. 1339–1364, 2018.
  * [66] J. Han, A. Jentzen, and W. E, “Solving high-dimensional partial differential equations using deep learning,” _Proceedings of the National Academy of Sciences_ , vol. 115, no. 34, pp. 8505–8510, 2018.
  * [67] X. Meng, Z. Li, D. Zhang, and G. E. Karniadakis, “Ppinn: Parareal physics-informed neural network for time-dependent pdes,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 370, p. 113250, 2020.
  * [68] X.-Y. Liu, H. Sun, and J.-X. Wang, “Predicting parametric spatiotemporal dynamics by multi-resolution pde structure-preserved deep learning,” _arXiv preprint arXiv:2205.03990_ , 2022.
  * [69] C. Rao, H. Sun, and Y. Liu, “Hard encoding of physics for learning spatiotemporal dynamics,” _arXiv preprint arXiv:2105.00557_ , 2021.
  * [70] C. Rao, P. Ren, Y. Liu, and H. Sun, “Discovering nonlinear pdes from scarce data with physics-encoded learning,” _arXiv preprint arXiv:2201.12354_ , 2022\.
  * [71] X. Huang, Z. Ye, H. Liu, S. Ji, Z. Wang, K. Yang, Y. Li, M. Wang, H. Chu, F. Yu _et al._ , “Meta-auto-decoder for solving parametric partial differential equations,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 426–23 438, 2022.
  * [72] B. K. Horn and B. G. Schunck, “Determining optical flow,” _Artificial intelligence_ , vol. 17, no. 1-3, pp. 185–203, 1981.
  * [73] C. Zach, T. Pock, and H. Bischof, “A duality based approach for realtime tv-l 1 optical flow,” in _Pattern Recognition: 29th DAGM Symposium, Heidelberg, Germany, September 12-14, 2007. Proceedings 29_. Springer, 2007, pp. 214–223.
  * [74] T. Brox and J. Malik, “Large displacement optical flow: descriptor matching in variational motion estimation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 33, no. 3, pp. 500–513, 2010.
  * [75] M. Menze, C. Heipke, and A. Geiger, “Discrete optimization for optical flow,” in _Pattern Recognition: 37th German Conference, GCPR 2015, Aachen, Germany, October 7-10, 2015, Proceedings 37_. Springer, 2015, pp. 16–28.
  * [76] Q. Chen and V. Koltun, “Full flow: Optical flow estimation by global optimization over regular grids,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4706–4714.
  * [77] Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_. Springer, 2020, pp. 402–419.
  * [78] S. Jiang, D. Campbell, Y. Lu, H. Li, and R. Hartley, “Learning to estimate hidden motions with global motion aggregation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9772–9781.
  * [79] A. Razavi, A. Van den Oord, and O. Vinyals, “Generating diverse high-fidelity images with vq-vae-2,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [80] J. Walker, A. Razavi, and A. v. d. Oord, “Predicting video with vqvae,” _arXiv preprint arXiv:2103.01950_ , 2021.
  * [81] M. J. Bocus, X. Wang, R. Piechocki _et al._ , “Streamlining multimodal data fusion in wireless communication and sensor networks,” _arXiv preprint arXiv:2302.12636_ , 2023.
  * [82] X. Liu, T. Iqbal, J. Zhao, Q. Huang, M. D. Plumbley, and W. Wang, “Conditional sound generation using neural discrete time-frequency representation learning,” in _2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP)_. IEEE, 2021, pp. 1–6.
  * [83] T. Zhao, K. Lee, and M. Eskenazi, “Unsupervised discrete sentence representation learning for interpretable neural dialog generation,” _arXiv preprint arXiv:1804.08069_ , 2018.
  * [84] J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh, “Action-conditional video prediction using deep networks in atari games,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [85] M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” _arXiv preprint arXiv:1511.05440_ , 2015.
  * [86] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz, “Mocogan: Decomposing motion and content for video generation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1526–1535.
  * [87] M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra, “Video (language) modeling: a baseline for generative models of natural videos,” _arXiv preprint arXiv:1412.6604_ , 2014.
  * [88] N. Srivastava, E. Mansimov, and R. Salakhudinov, “Unsupervised learning of video representations using lstms,” in _International conference on machine learning_. PMLR, 2015, pp. 843–852.
  * [89] R. Villegas, J. Yang, Y. Zou, S. Sohn, X. Lin, and H. Lee, “Learning to generate long-term future via hierarchical prediction,” in _international conference on machine learning_. PMLR, 2017, pp. 3560–3569.
  * [90] R. Villegas, D. Erhan, H. Lee _et al._ , “Hierarchical long-term video prediction without supervision,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 6038–6046.
  * [91] T. Kim, S. Ahn, and Y. Bengio, “Variational temporal abstraction,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [92] D. Weissenborn, O. Täckström, and J. Uszkoreit, “Scaling autoregressive video models,” _arXiv preprint arXiv:1906.02634_ , 2019.
  * [93] M. Kumar, M. Babaeizadeh, D. Erhan, C. Finn, S. Levine, L. Dinh, and D. Kingma, “Videoflow: A flow-based generative model for video,” _arXiv preprint arXiv:1903.01434_ , vol. 2, no. 5, p. 3, 2019.
  * [94] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [95] C. Bai, F. Sun, J. Zhang, Y. Song, and S. Chen, “Rainformer: Features extraction balanced network for radar-based precipitation nowcasting,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 19, pp. 1–5, 2022.
  * [96] J. Sun, J. Zhang, Q. Li, X. Yi, Y. Liang, and Y. Zheng, “Predicting citywide crowd flows in irregular regions using multi-view graph convolutional networks,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 34, no. 5, pp. 2348–2359, 2020.
  * [97] S. Wang, J. Cao, and S. Y. Philip, “Deep learning for spatio-temporal data mining: A survey,” _IEEE transactions on knowledge and data engineering_ , vol. 34, no. 8, pp. 3681–3700, 2020.
  * [98] R. Jiang, D. Yin, Z. Wang, Y. Wang, J. Deng, H. Liu, Z. Cai, J. Deng, X. Song, and R. Shibasaki, “Dl-traff: Survey and benchmark of deep learning models for urban traffic prediction,” in _Proceedings of the 30th ACM international conference on information & knowledge management_, 2021, pp. 4515–4525.
  * [99] X. Jiang, P. Ji, and S. Li, “Censnet: Convolution with edge-node switching in graph neural networks.” in _IJCAI_ , 2019, pp. 2656–2662.
  * [100] K. Wang, Z. Zhou, X. Wang, P. Wang, Q. Fang, and Y. Wang, “A2djp: A two graph-based component fused learning framework for urban anomaly distribution and duration joint-prediction,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [101] Y. Wang, M. Long, J. Wang, Z. Gao, and P. S. Yu, “Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [102] Y. Wang, L. Jiang, M.-H. Yang, L.-J. Li, M. Long, and L. Fei-Fei, “Eidetic 3d lstm: A model for video prediction and beyond,” in _International conference on learning representations_.
  * [103] Z. Gao, C. Tan, L. Wu, and S. Z. Li, “Simvp: Simpler yet better video prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3170–3180.
  * [104] T.-Y. Yang, J. Rosca, K. Narasimhan, and P. J. Ramadge, “Learning physics constrained dynamics using autoencoders,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 17 157–17 172, 2022.
  * [105] M. Jaques, M. Burke, and T. Hospedales, “Physics-as-inverse-graphics: Unsupervised physical parameter estimation from video,” _arXiv preprint arXiv:1905.11169_ , 2019.
  * [106] W. Yu, Y. Lu, S. Easterbrook, and S. Fidler, “Efficient and information-preserving future frame prediction and beyond,” 2020.
  * [107] A. Pervez and E. Gavves, “Stability regularization for discrete representation learning,” in _International Conference on Learning Representations_ , 2021\.
  * [108] A. H. Liu, S. Jin, C.-I. J. Lai, A. Rouditchenko, A. Oliva, and J. Glass, “Cross-modal discrete representation learning,” _arXiv preprint arXiv:2106.05438_ , 2021.
  * [109] J. Peng, D. Liu, S. Xu, and H. Li, “Generating diverse structure for image inpainting with hierarchical vq-vae,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 775–10 784.
  * [110] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas, “Videogpt: Video generation using vq-vae and transformers,” _arXiv preprint arXiv:2104.10157_ , 2021\.
  * [111] Y. Zhang, W. Yan, P. Abbeel, and A. Srinivas, “Videogen: generative modeling of videos using vq-vae and transformers,” 2020.
  * [112] C. Gârbacea, A. van den Oord, Y. Li, F. S. Lim, A. Luebs, O. Vinyals, and T. C. Walters, “Low bit-rate speech coding with vq-vae and a wavenet decoder,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 735–739.
  * [113] A. Tjandra, S. Sakti, and S. Nakamura, “Transformer vq-vae for unsupervised unit discovery and speech synthesis: Zerospeech 2020 challenge,” _arXiv preprint arXiv:2005.11676_ , 2020.
  * [114] E. De Bézenac, A. Pajot, and P. Gallinari, “Deep learning for physical processes: Incorporating prior scientific knowledge,” _Journal of Statistical Mechanics: Theory and Experiment_ , vol. 2019, no. 12, p. 124009, 2019\.
  * [115] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” in _Computer Vision-ECCV 2004: 8th European Conference on Computer Vision, Prague, Czech Republic, May 11-14, 2004. Proceedings, Part IV 8_. Springer, 2004, pp. 25–36.
  * [116] A. Bruhn, J. Weickert, and C. Schnörr, “Lucas/kanade meets horn/schunck: Combining local and global optic flow methods,” _International journal of computer vision_ , vol. 61, pp. 211–231, 2005.
  * [117] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2758–2766.
  * [118] P. Fischer, A. Dosovitskiy, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. Van der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” _arXiv preprint arXiv:1504.06852_ , 2015.
  * [119] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, “Flownet 2.0: Evolution of optical flow estimation with deep networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2462–2470.
  * [120] H. Wu, Z. Yao, J. Wang, and M. Long, “Motionrnn: A flexible model for video prediction with spacetime-varying motions,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 15 435–15 444.
  * [121] M. Veillette, S. Samsi, and C. Mattioli, “Sevir: A storm event imagery dataset for deep learning applications in radar and satellite meteorology,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 22 009–22 019, 2020.
  * [122] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, “Fourier neural operator for parametric partial differential equations,” _arXiv preprint arXiv:2010.08895_ , 2020.
  * [123] J. Guibas, M. Mardani, Z. Li, A. Tao, A. Anandkumar, and B. Catanzaro, “Adaptive fourier neural operators: Efficient token mixers for transformers,” _arXiv preprint arXiv:2111.13587_ , 2021.
  * [124] Y. Wang, J. Zhang, H. Zhu, M. Long, J. Wang, and P. S. Yu, “Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 9154–9162.
  * [125] H. Wu, W. Xion, F. Xu, X. Luo, C. Chen, X.-S. Hua, and H. Wang, “Pastnet: Introducing physical inductive biases for spatio-temporal video prediction,” _arXiv preprint arXiv:2305.11421_ , 2023.

|  Kun Wang is currently a Ph.D. candidate of University of Science and
Technology of China (USTC). He has published papers on top conferences and
journals such as ICLR and TKDE. His primary research interests are Sparse
Neural Networks, ML4Science and Graph Neural Networks.
---|---
|  Hao Wu is a master’s student who is currently undergoing joint training by
the Department of Computer Science and Technology at the University of Science
and Technology of China (USTC) and the Machine Learning Platform Department at
Tencent TEG. His research interests encompass various areas, including spatio-
temporal data mining, modeling of physical dynamical systems, and meta-
learning, among others.
---|---
|  Guibin Zhang is currently an undergraduate in the Department of Computer
Science and Technology, College of Electronic and Information Engineering,
major in Data Science, Tongji University, Shanghai, China. His research
interest include data mining, graph representation learning and knowledge
modeling.
---|---
|  Junfeng Fang is currently a Ph.D.candidate of University of Science and
Technology of China(USTC). His primary research interests are Trustable AI,
Graph Neural Network and Diffusion Model.
---|---
|  Yuxuan Liang (Member, IEEE) is currently an Assistant Professor at
Intelligent Transportation Thrust, also affiliated with Data Science and
Analytics Thrust, Hong Kong University of Science and Technology (Guangzhou).
He is working on the research, development, and innovation of spatio-temporal
data mining and AI, with a broad range of applications in smart cities. Prior
to that, he completed his PhD study at NUS. He published over 40 peer-reviewed
papers in refereed journals and conferences, such as TKDE, AI Journal, TMC,
KDD, WWW, NeurIPS, and ICLR. He was recognized as 1 out of 10 most innovative
and impactful PhD students focusing on data science in Singapore by Singapore
Data Science Consortium.
---|---
|  Yuankai Wu (Member, IEEE) received the Ph.D. degree from the Beijing
Institute of Technology (BIT), Beijing, China, in 2019. He is a professor at
the College of Computer Science, Sichuan University, China. Prior to joining
Sichuan University in March 2022, he was an IVADO postdoc researcher with the
Department of Civil Engineering, McGill University. His research interests
include spatiotemporal data analysis, intelligent transportation systems, and
intelligent decision-making.
---|---
|  Roger Zimmermann (Senior Member, IEEE) received the M.S. and Ph.D. degrees
from the University of Southern California (USC), in 1994 and 1998,
respectively. He is a Professor with the Department of Computer Science,
National University of Singapore (NUS). He is also the Deputy Director with
the Smart Systems Institute (SSI) and a Key Investigator with the Grab-NUS AI
Lab. He has coauthored a book, seven patents, and more than 300 conference
publications, journal articles, and book chapters. His research interests
include streaming media architectures, multimedia networking, applications of
machine/deep learning, and spatial data analytics. He is currently an
Associate Editor for IEEE MultiMedia, Transactions on Multimedia Computing,
Communications, and Applications (TOMM) (ACM), Multimedia Tools and
Applications (MTAP) (Springer), and the IEEE Open Journal of the
Communications Society (OJ-COMS). He is a distinguished member of the ACM and
a senior member of the IEEE.
---|---
|  Yang Wang is now an associate professor at USTC. He got his Ph.D. degree at
University of Science and Technology of China in 2007, under supervision of
Professor Liusheng Huang. He also worked as a postdoc at USTC with Professor
Liusheng Huang. His research interest mainly includes wireless (sensor)
networks, distributed systems, data mining, and machine learning.
---|---