# Strategy and Skill Learning for
Physics-based Table Tennis Animation

Jiashun Wang<EMAIL_ADDRESS>Carnegie Mellon UniversityUSA ,  Jessica
Hodgins<EMAIL_ADDRESS>Carnegie Mellon University
and The AI InstituteUSA and  Jungdam Won<EMAIL_ADDRESS>Seoul National
UniversitySouth Korea

(2024)

###### Abstract.

Recent advancements in physics-based character animation leverage deep
learning to generate agile and natural motion, enabling characters to execute
movements such as backflips, boxing, and tennis. However, reproducing the
selection and use of diverse motor skills in dynamic environments to solve
complex tasks, as humans do, still remains a challenge. We present a strategy
and skill learning approach for physics-based table tennis animation. Our
method addresses the issue of mode collapse, where the characters do not fully
utilize the motor skills they need to perform to execute complex tasks. More
specifically, we demonstrate a hierarchical control system for diversified
skill learning and a strategy learning framework for effective decision-
making. We showcase the efficacy of our method through comparative analysis
with state-of-the-art methods, demonstrating its capabilities in executing
various skills for table tennis. Our strategy learning framework is validated
through both agent-agent interaction and human-agent interaction in Virtual
Reality, handling both competitive and cooperative tasks.

Character Animation, Physics-based Characters, Deep Reinforcement Learning,
Multi-character Interaction, Virtual Reality, Table Tennis

††submissionid: 558††journalyear: 2024††copyright: rightsretained††conference:
Special Interest Group on Computer Graphics and Interactive Techniques
Conference Conference Papers ’24; July 27-August 1, 2024; Denver, CO,
USA††booktitle: Special Interest Group on Computer Graphics and Interactive
Techniques Conference Conference Papers ’24 (SIGGRAPH Conference Papers ’24),
July 27-August 1, 2024, Denver, CO, USA††doi: 10.1145/3641519.3657437††isbn:
979-8-4007-0525-0/24/07††ccs: Computing methodologies Physical
simulation††ccs: Computing methodologies Motion Processing Figure 1. On the
left, two physically-simulated agents engage in a competitive match,
controlled by our strategy and skill controllers. The center panels show the
five learned skills: Forehand Drive, Push and Smash, Backhand Drive and Push,
showcasing the skill controller’s ability to execute a diverse set of skills.
On the right, a human interacts with the simulated agent through VR.

##  1\. Introduction

The integration of deep learning into physics-based character animation has
led to significant advancements in generating agile and natural motion,
enhancing the lifelike quality of characters in complex environments. To
increase the versatility of these characters, it is essential to ensure that
their skills can be reused in environments or conditions that may not
precisely match their training data. To achieve this goal, recent approaches
have focused on learning reusable skill embeddings. These approaches are
typically trained in two stages. Initially, characters learn various skill
embeddings by imitating reference motions. Then, in the task training stage,
they apply these skills to accomplish diverse tasks. These approaches have
demonstrated remarkable success in generating natural motion in various
environments.

However, when the differences between the skills are subtle, these approaches
often suffer from mode collapse during the task training phase. Specifically,
although agents (a.k.a. characters) can learn various skills during the
imitation stage, they tend to use a limited set of skills for the downstream
tasks, neglecting the diversity of their learned skills in the imitation
stage. Thus, mode collapse restricts the agents’ potential in scenarios that
require a diverse set of skills. Mode collapse also restricts exploration
during RL training, resulting in sub-optimal task performance.

Another relatively unexplored topic relates to the decision strategy of
agents, particularly their ability to dynamically formulate decision
strategies that encompass skill selection and associated skill goals in
response to task demands. Most previous studies either have not required a
diverse skill set or have relied on a human user to manually determine skills
for the agents. Agents have generally not been equipped with the capability to
employ different strategies to adapt to complex and dynamic environments.

Our research introduces a learning approach to enhance both the skill and
strategic decision-making capabilities of physically simulated agents. First,
we develop a hierarchical skill controller that enables agents to utilize
different table tennis skills and transition among them rapidly. This
controller effectively addresses mode collapse during task training. Second,
we develop a method for strategy learning, enabling agents to explicitly
select and utilize specific skills for different types of interaction, whether
competitive or cooperative. An overview of the results is in Figure 1.

We demonstrate the effectiveness of our approach through two interaction
environments: a table tennis match played between two simulated agents and a
match between a human and a simulated agent in virtual reality (VR). In the
agent-agent environment, the agents demonstrate improved skill diversity and
decision strategy in simulated table tennis matches compared to results
predicted by the previous techniques. In the human-agent interaction
environment, we evaluate both cooperative and competitive scenarios in real-
time interactions between humans and agents. These environments not only
validate our approach but also provide platforms for future research into
complex agent behaviors and human-agent dynamics. Code and data for this paper
are at https://jiashunwang.github.io/PhysicsPingPong/.

We summarize the contributions of this paper as follows:

  * •

A hierarchical skill controller that empowers physically simulated agents to
explicitly perform various skills, enabling rapid skill transitions. An
interaction learning framework designed to create a decision strategy allows
agents to continually learn and adapt, meeting the demands of competition or
cooperation in dynamic environments with other agents and with humans.

  * •

Novel results demonstrating our learning framework’s capacity to generate
intelligent decisions and natural motions for table tennis in two scenarios:
agent-agent interactions in a simulated environment and human-agent
interactions in a VR environment. The agent-agent environment is a platform
for developing and testing competitive and cooperative algorithms while the VR
environment allows natural human-agent interactions.

##  2\. Related Work

We review the closest related work in physics-based character animation with
reusable skills and multi-character animation. We review studies on
transitions among skills as we develop a method for skill selection and
transition. We further discuss relevant research in human-agent interaction in
VR.

###  2.1. Physics-based Character Animation

Incorporating physical laws into character animation allows for the
development of controllers that generate more realistic behaviors (Hodgins et
al., 1995; Laszlo et al., 1996). Optimization techniques, such as trajectory
optimization (de Lasa et al., 2010; Yin et al., 2008; Mordatch et al., 2012)
and sampling-based methods (Liu et al., 2010, 2016) have been widely explored.
Recently, deep reinforcement learning (DRL) has been shown to substantially
enhance control capabilities (Liu and Hodgins, 2017; Peng et al., 2017). Due
to its flexibility and ease of use, DRL methods eliminate the need for
designing complex objective functions while delivering outstanding results and
have attracted significant research interest as a result.

Data-driven methods have become prevalent in physics-based character animation
studies since a DRL-based method was introduced by Peng et al. (2018). The
idea has been extended for handling larger datasets (Bergamin et al., 2019;
Won et al., 2020) and for allowing recombination of existing state transitions
(Peng et al., 2021). Recently, much attention has been paid to reusable motor
skills. The idea is to learn a latent space of reference motions and then to
reuse the learnt space for downstream tasks. Various latent models have been
studied such as encoder-decoders with autoregression (Merel et al., 2019; Won
et al., 2021), spherical embedding (Peng et al., 2022; Tessler et al., 2023;
Dou et al., 2023), conditional variational autoencoder (VAE) (Won et al.,
2022; Yao et al., 2022), and vector-quantized VAE (Zhu et al., 2023). Some
researchers have also proposed part-wise models to maximize reusability of
reference motions (Bae et al., 2023; Xu et al., 2023).

Our system is designed for table tennis games, involving two players (i.e.,
agents). Two or more agents have been created primarily with kinematic
approaches (Shum et al., 2008, 2012; Liu et al., 2006; Kwon et al., 2008;
Wampler et al., 2010). There exist two recent approaches (Won et al., 2021;
Zhu et al., 2023) demonstrating examples of physically simulated boxing. Zhang
et al. (2023) build a system to learn tennis skills from broadcast videos and
produce rallies with a mirrored opponent. In their approach, kinematics-based
motion generation is utilized first, followed by physics-based tracking,
relying on residual forces and extra arm control for successful strikes. Skill
and target selection are not learned but rather performed manually or randomly
to create a scene including two players. In contrast, our method learns not
only agile and precise motor control to strike the ball but also strategies to
select skills and targets based on the movement of the opponent and the ball.

###  2.2. Transition of skills

Option-based methods (Sutton et al., 1999; Jain et al., 2021; Bagaria and
Konidaris, 2020; Klissarov et al., 2017; Konidaris and Barto, 2009) represent
skills as options, which are sequentially constructed, with each option’s
execution in the chain enabling the agent to execute the subsequent option.
Lee et al. (2019) propose learning additional transition policies to connect
primitive skills and introduce proximity predictors, which yield rewards based
on proximity suitable for initial states for the next skill. One challenge of
transitioning between different skills to chain long-horizon tasks is
addressed by terminal state regularization (Lee et al., 2021). Behavior Trees
are also a common method for planning the transition between different states
(Marzinotto et al., 2014; French et al., 2019; Cheng et al., 2023). These
methods achieve skill transitions by ensuring that the terminal state of the
previous stage is close to the initial state of the next stage. While these
methods work well for tasks that are not time-sensitive, table tennis, which
involves high-speed movements and rapid responses, poses a challenge as
players do not always hit the ball from a well-defined initial state.

###  2.3. Human-agent interaction

Research has focused on human sports training within VR (Liu et al., 2020;
Pastel et al., 2023). However, these studies often lack a physically simulated
opponent. There are commercial games that allow people to interact with an
agent in VR for sports activities, such as boxing, golf, and badminton. Eleven
Table Tennis (2016) is a VR-based table tennis game similar to the one we have
constructed, which enables a human to play with an agent. However, this agent
is not simulated with full-body dynamics, rather it is simulated with only a
floating head and a floating paddle. Advances in GPU-accelerated simulation
and our control algorithm, enable us to create a physically-simulated agent
with full-body dynamics that can play in real-time with humans. Another
relevant area involves enhancing the agent’s capabilities with human-in-the-
loop methodologies (Li et al., 2022; Brenneis et al., 2021; Seo et al., 2023;
Wang et al., 2023) using extended reality. Our work differs from previous
studies by bringing humans and agents into a unified environment allowing
bidirectional physical interaction, where they can cooperate and compete.

Figure 2. An overview of our method. Strategy action includes the skill
command and ball’s target landing location. Skill action includes the target
joint angles for PD controllers, blended from the outputs of imitation
policies.

##  3\. Method overview

We propose a hierarchical approach that includes a strategy-level controller
and a skill-level controller. The strategy-level controller takes the states
of the agent, opponent, and ball as inputs, and outputs a strategy action,
which includes the skill to use and the target landing location for the ball.
Meanwhile, the skill-level controller takes the states of the agent and ball,
along with the strategy action as inputs, and then generates a skill action,
which includes the target joint angles for PD controllers. An overview of our
method is in Figure 2 and Figure 3 shows the architecture of our method.

Figure 3. The architecture of our method. We train the skill-level controller
through the stages of imitation policies, ball control policies, and finally,
the mixer policy. We train the strategy-level controller after the skill-level
controller is ready and its weight is frozen. $\otimes\oplus$ stands for the
weighted sum in Equation 8.

##  4\. Skill-level Controller

Three stages are required to train our skill-level controller. Initially, we
train imitation policies using the motion capture data. Then the ball control
policy for each skill is learned, which enables the agent to hit back balls
using the corresponding imitation policy. Finally, we learn a policy that
enables the agent to perform various skills sequentially while making
plausible transitions among them. We call this policy the mixer policy. Once
the skill-level controller is trained, the agent can proficiently and
continuously execute various skills, sending balls to diverse target
locations.

###  4.1. Imitation Policy

We first categorize the motion capture dataset into five subsets corresponding
to each skill. This subdivision allows us to train the skill-specific
imitation policies. We also utilize all the data to train a universal
imitation policy. The imitation policy is represented as
$\pi^{i}(a^{i}|s,z^{i})$, where $i\in\\{1,2,3,4,5,u\\}$, $1\sim 5$ are indices
of different skills and $u$ is the index of the universal imitation policy.
$z^{i}$ is a latent variable sampled from a hyper-sphere distribution, and $s$
is the agent’s state. The goal of the imitation policy is to output an action
$a^{i}$ that leads to simulated motions similar to the reference motions.
Thus, each skill-specific imitation policy generates motions similar to its
corresponding reference motion in each skill subset, while the universal
imitation policy generates motions encompassing the entire motion capture
dataset. When solving specific tasks in the later stage, using a single
universal imitation policy trained with a variety of motions often leads to
the mode collapse problem. The agent does not explore various available skills
enough; instead, it repeats very limited skills, and the task performance
remains sub-optimal. Our controller design is inspired by mixture-of-experts
and mitigates this problem. Each imitation policy $\pi^{i}(a^{i}|s,z^{i})$ is
built by the adversarial framework ASE (Peng et al., 2022), where the policy
is updated so that it tricks a motion discriminator $D^{i}$. The transitions
$d_{M^{i}}(s,s^{\prime})$ existing in the motion capture dataset are used as
positive samples while the transitions $d_{\pi^{i}}(s,s^{\prime})$ generated
from the policy $\pi^{i}$ are used as negative samples. The discriminator is
trained by minimizing:

$\displaystyle\mathop{\min}_{D^{i}}$
$\displaystyle-\mathbb{E}_{d_{M^{i}}(s,s^{\prime})}\log(D^{i}(s,s^{\prime}))-\mathbb{E}_{d_{\pi^{i}}(s,s^{\prime})}\log(1-D^{i}(s,s^{\prime}))$
(1)
$\displaystyle+\lambda_{gp}\mathbb{E}_{d_{M^{i}}(s,s^{\prime})}\left\|\nabla_{\phi}D^{i}(\phi)\big{|}_{\phi=(s,s^{\prime})}\right\|^{2},$

where the last term is a gradient penalty regularization with a constant
factor $\lambda_{gp}$. We train encoders $q^{i}$ to encourage correspondence
between the transition $(s,s^{\prime})$ and the latent variable $z^{i}$. The
encoder is modeled as a von Mises-Fisher distribution and it is trained by
maximizing its log-likelihood:

(2)
$\begin{split}\mathop{\max}_{q^{i}}\mathbb{E}_{p(z^{i})}\mathbb{E}_{d^{\pi^{i}}(s,s^{\prime}|z^{i})}[\log
q^{i}(z^{i}|s,s^{\prime})],\\\
q^{i}(z^{i}|s,s^{\prime})=\frac{1}{Z}exp(\mu_{q^{i}}(s,s^{\prime})^{T}z^{i})\end{split}$

where $\mu_{q^{i}}(s,s^{\prime})$ is the mean of the distribution, and Z is a
normalization constant. Given a discriminator $D^{i}$, the reward to train
$\pi^{i}$ is defined as:

(3) $r_{t}=-\log(1-D^{i}(s_{t},s_{t+1}))+\beta\log
q^{i}(z^{i}_{t}|s_{t},s_{t+1}).$

where $\beta$ is the relative weight. Additionally, a diversity term is
included to encourage different latent variables to represent distinct
motions. Bringing everything together, the objective of $\pi^{i}$ becomes:

(4) $\displaystyle\mathop{\max}_{\pi^{i}}\ $
$\displaystyle\mathbb{E}_{p(Z)}\mathbb{E}_{p(\tau|\pi^{i},Z)}[\sum_{t=0}^{T}\gamma^{t}(r_{t})]$
$\displaystyle-\lambda_{D^{i}}\mathbb{E}_{d^{\pi^{i}}(s)}\mathbb{E}_{z^{i}_{1},z^{i}_{2}\sim
p(z^{i})}\left[\left(\frac{D_{KL}(\pi^{i}(\cdot|s,z^{i}_{1}),\pi^{i}(\cdot|s,z^{i}_{2}))}{0.5(1-z^{i}_{1}z^{i}_{2})}-1\right)^{2}\right],$

where $D_{KL}(\cdot||\cdot)$ measures the KL-divergence between two
distributions, $z^{i}_{1}$ and $z^{i}_{2}$ refer two different latent
variables, $\gamma$ is the discount factor and $\lambda$ is a constant used to
balance the weight.

###  4.2. Ball Control Policy

Once the agent can imitate each skill $i\in\\{1,2,3,4,5\\}$, we train ball
control policies $\omega^{i}(z^{i}|s,b,y)$ to enable the agent to hit and move
a ball launched from a random location to the desired location, where $s$
denotes the state of the agent, $b$ represents the state of the ball and $y$
is the target landing location for the ball. The task reward $r$ is a
composite of three terms: the paddle reward $r_{p}$, the ball reward $r_{b}$,
and the style reward $r_{s}$,

(5) $r(t)=w_{p}r_{p}(t)+w_{b}r_{b}(t)+w_{s}r_{s}(t),$

where $w_{p}$, $w_{b}$, and $w_{s}$ are the relative weights. The paddle
reward $r_{p}$ encourages the agent to position the paddle close to the ball.
The reward is defined as:

(6) $r_{p}(t)=\begin{cases}&\exp(-4||x_{p}(t)-x_{b}(t)||^{2}),\text{ if
}C_{bp}(t)=0,\\\ &0,\text{ otherwise},\end{cases}$

where $x_{p}(t)$ and $x_{b}(t)$ represent the positions of paddle and ball,
respectively, $C_{bp}(t)$ is a binary variable representing contact states.
$C_{bp}(t)=0$ means the ball has not contacted the paddle until time $t$,
while $C_{bp}(t)=1$ indicates the ball has contacted the paddle at time $t$ or
contacted previously before time $t$. It will be reset to $0$ whenever the
next ball is launched. The ball reward $r_{b}$ is given by:

(7) $r_{b}(t)=\begin{cases}&1+\exp(-4||x_{c}(t)-x_{t}(t)||^{2}),\\\
&\qquad\text{ if }C_{bp}(t)=1\text{ and }C_{bt}(t)=0,\\\ &0,\text{
otherwise},\end{cases}$

where $x_{t}(t)$ is the target landing location of the ball, $x_{c}(t)$
represents the anticipated landing location on the table, calculated using
Newton’s equation of motion for the point mass (i.e., quadratic trajectory),
with its state corresponding to the current position and velocity of the ball.
$C_{bt}(t)$ is a binary variable checking the contact history between the ball
and the table, which is updated similarly to $C_{bp}(t)$. The agent receives
the maximum reward when it successfully hits the ball and the ball moves
toward the target location. We also apply the style reward,
$r_{s}=-\log(1-D^{i}(s_{t},s_{t+1}))$ in the task training similarly to ASE
(Peng et al., 2022), where $D^{i}$ is the discriminator learned during the
previous stage.

###  4.3. Mixer Policy

While our agent can play table tennis using the ball control policies with the
corresponding imitation policies, its capability is limited to repeating a
single skill. Simply transitioning from one controller to another during play
often leads to failure due to a mismatch between the end state of one skill to
the start state of the next. To create plausible transitions among the
different skills, we learn a mixer policy $\omega^{m}(z^{m}|s,b,\delta,y)$,
which takes the agent state $s$, the ball state $b$, and the strategy action
$(\delta,y)$ as input, where $\delta$ is a one-hot vector determining the
skill to use and $y$ is the target ball landing location, then generates the
latent variable for the universal imitation policy $\pi^{u}$ and a set of
blending weights $\varphi$, mixing the skill actions in a joint-wise manner.
In other words, $\varphi$ determines which policy the agent relies on among
the transition and five different skills. The target joint angles for PD
controllers are computed as

(8)
$\begin{split}a&=\varphi\odot\pi^{u}(\cdot|s,z^{u})+(1-\varphi)\odot\sum_{i=1}^{5}\delta_{i}\pi^{i}(\cdot|s,z^{i})\end{split}$

where $\delta=(\delta_{1},\delta_{2},\delta_{3},\delta_{4},\delta_{5})$ is a
one-hot vector indicating the skill selected. While training the mixer policy,
the agent is asked to perform the ball control task with randomly launched
balls, randomly selected skills, and random target locations. The same rewards
used for learning ball control policies are employed, and the weights of all
other policies remain frozen.

##  5\. Strategy-level Controller

The strategy-level controller is developed by iterative behavior cloning
inspired by (Oh et al., 2018). More specifically, we first collect interaction
data by randomly sampling strategy actions during agent-agent play or human-
agent interactions with VR. This data is then used to update the strategy-
level controller, and we repeat this process by collecting new interaction
data with the latest strategy-level controller. When collecting interaction
data, there are two options: competition and cooperation. To train a
competitive strategy, we choose data that results in victories, whereas in a
cooperative strategy, we choose sequences where the opponent successfully
catches the ball.

A strategy-level controller produces a skill index and a target landing
location repeatedly so that they satisfy the requirements of different
applications. More specifically, the strategy-level controller $f$ takes the
strategy observation $o=(s,\tilde{s},b)$ as input where $s$, $\tilde{s}$, and
$b$ are the agent state, the opponent state, and the ball state, respectively,
then outputs the strategy action $c=(\delta,y)$, where $\delta$ is a one-hot
vector determining the skill to use, and $y$ is the target landing location of
the ball. The strategy action is updated when the ball starts moving from the
opponent to the agent.

Input: Number of iterations $N$, interaction environment $Env$.

Output: Updated policy $f$.

$f\leftarrow\mathrm{Random~{}initialization}$ ;

for  _$i\leftarrow 1$ to $N$_ do

$\\{(o^{\mathrm{expert}}_{k},c^{\mathrm{expert}}_{k})\\}^{K}_{k=1}\leftarrow$
Interact($Env$, $f$);

Apply stochastic gradient descent to update $f$ using Equation 9

end for

ALGORITHM 1 Strategy learning

To effectively learn a strategy-level controller, we adopt a behavior cloning
approach with iterative refinement, aiming to learn strategies from available
expert demonstrations
$\\{(o^{\mathrm{expert}}_{k},c^{\mathrm{expert}}_{k})\\}^{K}_{k=1}$ (see
Algorithm 1). As a structure of the controller, we utilize a Conditional
Variational Autoencoder (CVAE) to model the stochastic nature inherent in
sports gameplay. During training, the CVAE encoder takes $o$ and $c$ as inputs
and generates the mean $\mu$ and variance $\sigma^{2}$ of the posterior
Gaussian distribution $Q(u|\mu,\sigma^{2})$. We then sample a latent variable
$u$ from this distribution and concatenate it with observation $o$ as input
for the decoder, which reconstructs the action $c^{\prime}$. The training loss
is defined as:

(9)
$\sum^{K}_{k=1}||c^{\mathrm{expert}}_{k}-c^{\prime}_{k}||+\beta_{KL}D_{KL}(Q(u|\mu_{k},\sigma^{2}_{k})||\mathcal{N}(0,I)),$

where $D_{KL}(\cdot||\cdot)$ measures the KL divergence between the two
distributions and $\beta_{KL}$ is the relative weight. During inference the
decoder is utilized solely, it takes a randomly sampled latent variable $u$
and the observation $o$, and then generates the strategy action that guides
the agent to perform a corresponding skill. If the opponent successfully
returns the ball, this process repeats. We collect expert demonstrations from
two different interaction environments ($Env$ in Algorithm 1). The details of
each environment will be explained in Section 6.

##  6\. Interaction Environment

We introduce the agent-agent and human-agent interaction environments that we
build to validate the strategy learning approach.

The agent-agent interaction environment is an environment where two virtual
agents play table tennis with each other (Figure 1 left column). We name one
agent as our agent and the other as the opponent. In the process of learning a
strategy-level controller for our agent, the opponent uses a fixed heuristic
strategy-level controller while the controller for our agent is updated
iteratively. More specifically, we let our agent the opponent play with each
other using their own strategy-level controllers, collect those
demonstrations, and then use them to update our agent’s controller. If our
goal is to learn a competitive strategy, that can beat the opponent, we
selectively use demonstrations leading to wins. On the other hand, we use
demonstrations where the opponent successfully returns the ball when aiming to
learn a cooperative strategy. In our system, we utilize two types of heuristic
strategy-level controllers: a random strategy and a video strategy. The random
strategy selects skills and target landing locations randomly from a uniform
distribution. The video strategy is constructed by using broadcast videos. We
extract expert demonstrations
$\\{(o^{\mathrm{video}}_{k},c^{\mathrm{video}}_{k})\\}^{K}_{k=1}$ from
existing broadcast videos (20 minutes in total). Subsequently, we train a CVAE
using the behavior cloning method.

The human-agent interaction environment allows a human user to play with a
virtual agent. In our system, the user interacts with an agent by using a VR
device, including a head-mount display and a hand controller ((Figure 1 right
column)). The VR interface operates through Unity while the physics-based
simulation runs on Isaac Gym (Makoviychuk et al., 2021). To enable the
simulated agent to interact with a human user, we physically simulate the
user’s paddle, with its position and orientation controlled via signals from
the VR interface. Specifically, for paddle control, we use the VR hand
controller’s Cartesian pose $q_{\mathrm{user}}$ and the simulated paddle pose
$q_{\mathrm{sim}}$ to calculate the target velocity
$\dot{q}_{\mathrm{target}}=(q_{\mathrm{user}}-q_{\mathrm{sim}})/\Delta t$,
where $\Delta t$ is the simulation step. We use this target velocity as an
input to the velocity controller provided by the simulator. For visualization,
Unity takes the state of the simulated agent, user’s paddle, and ball as
inputs and renders them using visualization assets. This implementation
significantly reduces the amount of information exchange compared to a
previous study that sent stereo images (Seo et al., 2023), enabling real-time
interaction and gameplay. By considering a human user as the opponent, the
strategy-level controller of the agent can be built through the same pipeline
used for the agent-agent interaction environment.

Figure 4. Comparison with other methods with four skill commands. ASE and CASE
may use wrong skills as shown in the red box. ET may terminate earlier to
return to a preparation pose, as shown in the yellow boxes. Figure 5.
Transition results with only using forehand and backhand drive controllers.
Both controllers are trained with random initialized configurations from the
motion capture data. As shown in the red boxes, the agent attempts to use
another forehand drive before the next ball is launched, which prevents it
from switching back to a backhand drive in time.

##  7\. Experiments

We evaluate the skill-level controller based on motion quality and task
performance. We assess the strategy-level controller by examining its
effectiveness in agent-agent and human-agent interaction environments, with
the demands of both competition and cooperation scenarios.

###  7.1. Skill evaluation

We evaluate the skill performance from motion quality and task performance.
The evaluation of motion quality measures the naturalness of generated motions
when given the desired skill command and whether the agent performs the
correct skill. The evaluation of task performance measures the overall
proficiency in playing table tennis. We compare our method with two state-of-
the-art methods, ASE (Peng et al., 2022) and CASE (Dou et al., 2023), as well
as an explicit transition model (ET) which is a variant of our method with the
mixer policy $\omega^{m}$ removed from our model. We train an explicit
controller to handle skill transitions by taking over the control when the
ball passes the net until it is returned to the agent. The controller is also
built using the ball control-imitation architecture. The key difference
between our approach and ET is that ours provides continuous action blending
with the selected skill’s action at every time step, whereas ET does not.

####  7.1.1. Motion quality

We design three metrics to evaluate the motion quality, particularly to
evaluate the naturalness and mode collapse. The first metric is Discriminator
Score, which measures how similar the current strike motion is to the
reference motion of $i$-th target skill. Because we have five skills, we train
a discriminator $D^{i}_{test}$ for each skill and utilize the following
equation to calculate the score:

(10) $\text{Discriminator Score
i}=\frac{1}{T}\sum_{t=0}^{T-1}-\log(1-D^{i}_{test}(s_{t},s_{t+1})),$

where $T$ is the length of a single strike motion. The details of training
$D^{i}_{test}$ will be introduced in Appendix D. The second metric is Skill
Accuracy, to measure whether the agent performs the correct skill given the
target skill command. Specifically, given a motion sequence, we first classify
it by taking the index of the discriminator which provides the highest value.
Then, we compare it with the target skill command to calculate accuracy. The
third metric, Diversity Score, is designed to test whether motions for drive
and push commands are distinctive enough. In table tennis, motions within each
skill category (e.g., forehand drive vs. forehand push) might exhibit subtle
differences, even though their roles in gameplay can be significantly
distinct. Diversity Score measures the capability of distinguishing motions
that are visually similar. It is calculated by

(11) $\vspace{-0.05in}\text{Diversity
Score}=\frac{1}{2N^{2}}\sum_{i\in\\{1,3\\}}\sum_{m=1}^{N}\sum_{n=1}^{N}||s^{i}_{m}-s^{i+1}_{n}||,$

where $s^{i}$ is the state that the agent hits the ball under skill command
$i$. Specifically $i\in\\{1,3\\}$ stands for forehand drive and backhand drive
and $i+1\in\\{2,4\\}$ stands for forehand push and backhand push respectively.
$N$ is the total number of hits for each skill command. We only take into
account the moment when the agent’s paddle makes contact with the ball to
calculate this score. The first two metrics evaluate a general skill mode
collapse problem, for example, using forehand motions when being asked to use
backhand. The third metric is specifically designed to measure if the agent
has the ability to accurately perform drive and push skills.

The evaluation results are reported in Table 1, where the values are computed
with 10k balls randomly launched toward the agent equipped with the respective
skill controller. For the Discriminator Score, our method significantly
surpasses ASE and CASE, and achieves 15.6% higher score than ET. These results
prove our method generates motions that are the most similar to the reference
target skill. As shown in the Skill Accuracy results, our method uses the
correct skills to hit the ball in most cases (0.76 in Table 1). While ASE and
CASE only use the correct skill with an accuracy of 0.38 and 0.47. In
Diversity Score, our method achieves 30.7%, 32.3%, and 9.4% higher scores than
ASE, CASE, and ET respectively. We also show a qualitative comparison in
Figure 4. We find ASE and CASE often use forehand skills when asked to use
backhand skills, or vice-versa, as shown in the red boxes in Figure 4. And we
can’t observe any forehand smash skill. Even when the correct skills are used,
the naturalness remains insufficient. ET often does not complete the skills;
instead, the skills are terminated earlier to return to a preparation pose, as
shown in the yellow box in Figure 4. ASE and CASE often overlook skill
commands, tending to use relatively fewer skills. This error occurs because,
during the task training, these methods fall into mode collapse, making it
challenging to effectively explore various skills. In contrast, our approach
leverages an idea of the mixture-of-experts approach to avoid this problem. We
further test the use of individual skill controllers without any design for
transitions. Each skill controller is trained with randomly initialized
configurations sampled from the motion capture data. As shown in Figure 5,
after executing a forehand drive, the agent attempts another forehand drive
before the next ball is launched—a typical behavior for single-skill
controllers. This unnecessary movement prevents it from switching to a
backhand drive in time, ultimately causing a missed shot.

####  7.1.2. Task performance.

To assess the task performance of the skill controller, we evaluate two
aspects: sustainability and accuracy. Sustainability is determined by the
average number of successful continuous returns, while accuracy is measured by
the average distance in meters between the target landing location and the
actual contact location on the table. Besides testing on the training
distribution, we collect some ball tracking data with faster ball trajectories
from a match between high-ranking players and evaluate whether each method can
perform well with the testset of the ball tracking data. We report the
evaluation results in Table 2. The numbers in parentheses are the results of
the fine-tuning experiments. Our method can achieve the largest number of
average hits and the second-best accuracy. Although ET can achieve higher
accuracy, it is not sustainable, especially for more challenging balls. It
only achieves an average of 3.66 hits because it often lacks time to respond
to the next ball due to the explicit transition design.

Table 1. Comparisons on Discriminator Score, Skill Accuracy, and Diversity Score.  | ASE | CASE | ET | Ours
---|---|---|---|---
Discriminator Score | 1.62 | 2.28 | 4.95 | 5.72
Skill Accuracy | 0.38 | 0.47 | 0.69 | 0.76
Diversity Score | 6.13 | 6.05 | 7.32 | 8.01
Table 2. Task performance evaluation. Our method can achieve the longest average hits and the second best accuracy. | ASE | CASE | ET | Ours
---|---|---|---|---
Avg Hits | 9.54 (5.94) | 8.79 (5.28 | 6.55 (3.66) | 10.93 (6.28)
Avg error | 0.28 (0.33) | 0.35 (0.39) | 0.25 (0.28) | 0.26 (0.31)

####  7.1.3. Blending weights of the mixer policy.

We test the agent with different skills to hit the ball and visualize the
average blending weights $\varphi$ of the shoulder, elbow, and wrist joints in
Figure 8. We can observe that the weights of the mixer policy are usually
lowest at the moment the paddle contacts the ball, and higher before and after
transitions between different skills. It indicates a reliance on the pre-
trained ball control policy during ball strikes, and on the mixer policy
during transitions.

Figure 6. Skill command distribution of our method and RL. Figure 7. Target
landing locations of our method, RL and Human.

###  7.2. Evaluation for agent-agent interaction

We evaluate the performance of learned strategies in the agent-agent
interaction environment under both competition and cooperation settings. The
competition strategy aims to develop an agent that achieves a higher winning
rate than the opponent. The cooperation strategy develops an agent that can
play gently with the opponent to increase the length of rallies. As a
baseline, we learn a strategy policy via reinforcement learning (RL). Please
refer to Appendix E for details on training the RL baseline. Our method and
the RL baseline are compared by having them play with two types of opponents:
the random strategy and the video strategy opponent introduced in Section 6.
Each evaluation is computed over 10k points. Table 3 shows the winning rate
and the average rounds for the competition and cooperation settings. Our
strategy learning algorithm can achieve higher winning rates for the
competition setting and can maintain longer rallies for the cooperation
setting for both opponents.

In Figure 6 and 7, we visualize the histogram of skill commands from the
strategy policies, and the target landing locations. In Figure 7, we also
provide ball landing locations captured from real players during competitive
matches. We observe that our method has a more similar distribution of landing
locations to humans than RL. RL converges to less diverse skill commands and
it only hits to a small region of the table. In contrast, our method utilizes
various skills and target locations throughout the gameplay. We also include
qualitative gameplay visualizations in Figure 9. We further let RL and our
method compete with each other, and report the winning rates in Table 4. Each
method has two strategy policies trained with two opponents, therefore, we
have four matches in total. Because RL falls into a local minimum and overfits
to a specific opponent, our method achieves a much higher winning rate.

###  7.3. Evaluation of Human-agent interaction

Before learning strategies for the human-agent interaction environment, we
finetune the skill-level controller using the play data of a human user
interacting with the agent equipped with the original skill-level controller.
The finetuning is required because of the domain gap between what the
simulated agent has experienced and the styles of a real human user in the VR
environment. After finetuning the skill controller, strategies are learned by
following similar procedures as the agent-agent interaction environment. For
training a competition strategy, we use demonstrations that result in the
agent winning points, thus presenting more challenging returns for the human
opponent. In contrast, for training a cooperative strategy, we use
demonstrations where the human can maintain rallies, emphasizing easier ball
returns for the human. These demonstrations serve as expert demonstrations in
Algorithm 1. We report the winning rate of the agent and the average hits
between the user and the agent in Table 5. When playing with the initial
policies, the agent can achieve a winning rate of 64% and a rally with 4.04
hits on average. After two iterations of refinement of the competition
strategy, the agent can achieve a winning rate of 78% , and the average number
of hits drops to 3.75. For the cooperative strategy, the winning rate drops to
58%, and the user can achieve a rally with an average of 5.34 hits. These
results demonstrate that our strategy learning algorithm is also effective for
the human-agent interaction environment. We provide screenshots of real-time
human-agent gameplay video in Figure 10.

Table 3. Strategy evaluation. We report the winning rates for the competition setting and average rounds for the cooperation setting. | Competition | Cooperation
---|---|---
| RL | Ours | RL | Ours
Random op | 0.641 | 0.687 | 14.9 | 16.4
Video op | 0.637 | 0.681 | 15.6 | 18.2
Table 4. Winning rates between our method and RL. The opponent in parentheses is the opponent during training of the strategy policy.  | Ours (random op) | Ours (video op)
---|---|---
RL (random op) | 0.45 vs 0.55 | 0.47 vs 0.53
RL (video op) | 0.42 vs 0.58 | 0.42 vs 0.58
Table 5. Evaluation of human-agent interaction.  | Initial policies | Competition | Cooperation
---|---|---|---
Winning rate | 0.64 | 0.78 | 0.58
Avg hits | 4.04 | 3.75 | 5.34

##  8\. Discussion and Conclusion

Although our method produces agents that play competitively and more
naturally, it still has several limitations. First, although building
individual policies for each skill and combining them via the mixer policy
clearly improves the generated motion quality and task performance, our model
would not scale well to a dataset including hundreds of different skills.
Developing a hybrid model that combines our approach with a model learnable
from unlabeled motions to achieve both high motion quality and scalability
would be an interesting future research topic. Second, because our method is
data-driven, the captured motion quality significantly affects the final
motion quality. For example, the player tends to use large arm motions, and
this motion style appears in our results as well. However, in matches, using
less arm motion could be a way to conserve energy, and concealed movements can
also confuse the opponent. Lastly, although we employ a rigid-body simulation
for every component, including the ball, player, and table, where the ball can
spin as well, air resistance is modeled using only damping based on velocity,
rather than incorporating the Magnus effect, which bends the ball trajectory
due to air pressure differences. This omission could impact the realism of our
animations and the final strategies our system learned.

In this paper, we introduce a learning approach for physics-based table tennis
animation. We develop a hierarchical controller structure, which overcomes the
mode collapse problem that appears frequently in reusable latent-based models.
Our approach not only improves overall motion quality but also enables us to
learn effective decision strategies for two types of environments: agent-agent
and human-agent interactions.

###### Acknowledgements.

This work was partially completed during Jiashun Wang’s internship at The AI
Institute. Jungdam Won was partially supported by Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government(MSIT) [NO.2021-0-01343-004, Artificial Intelligence Graduate
School Program (Seoul National University)] and ICT(Institute of Computer
Technology) at Seoul National University. We would like to thank Murphy
Wonsick for helping to build the VR system and Melanie Danver for rendering
the results.

## References

  * (1)
  * Bae et al. (2023) Jinseok Bae, Jungdam Won, Donggeun Lim, Cheol-Hui Min, and Young Min Kim. 2023.  PMP: Learning to Physically Interact with Environments using Part-wise Motion Priors. In _ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 2023_. ACM, 64:1–64:10.  https://doi.org/10.1145/3588432.3591487
  * Bagaria and Konidaris (2020) Akhil Bagaria and George Konidaris. 2020.  Option Discovery using Deep Skill Chaining. In _8th International Conference on Learning Representations, ICLR 2020_. OpenReview.net.  https://openreview.net/forum?id=B1gqipNYwH
  * Bergamin et al. (2019) Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. 2019.  DReCon: data-driven responsive control of physics-based characters.  _ACM Trans. Graph._ 38, 6 (2019), 206:1–206:11.  https://doi.org/10.1145/3355089.3356536
  * Brenneis et al. (2021) Dylan J. A. Brenneis, Adam S. R. Parker, Michael Bradley Johanson, Andrew Butcher, Elnaz Davoodi, Leslie Acker, Matthew M. Botvinick, Joseph Modayil, Adam White, and Patrick M. Pilarski. 2021.  Assessing Human Interaction in Virtual Reality With Continually Learning Prediction Agents Based on Reinforcement Learning Algorithms: A Pilot Study.  _arXiv preprint arXiv:2112.07774_ (2021).  https://arxiv.org/abs/2112.07774
  * Cheng et al. (2023) Xuxin Cheng, Ashish Kumar, and Deepak Pathak. 2023.  Legs as Manipulator: Pushing Quadrupedal Agility Beyond Locomotion. In _IEEE International Conference on Robotics and Automation, ICRA 2023_. IEEE, 5106–5112.  https://doi.org/10.1109/ICRA48891.2023.10161470
  * de Lasa et al. (2010) Martin de Lasa, Igor Mordatch, and Aaron Hertzmann. 2010.  Feature-based locomotion controllers.  _ACM Trans. Graph._ 29, 4 (2010), 131:1–131:10.  https://doi.org/10.1145/1778765.1781157
  * Dou et al. (2023) Zhiyang Dou, Xuelin Chen, Qingnan Fan, Taku Komura, and Wenping Wang. 2023.  C$\cdot$ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters. In _SIGGRAPH Asia 2023 Conference Papers, SA 2023_. ACM, 2:1–2:11.  https://doi.org/10.1145/3610548.3618205
  * French et al. (2019) Kevin French, Shiyu Wu, Tianyang Pan, Zheming Zhou, and Odest Chadwicke Jenkins. 2019.  Learning Behavior Trees From Demonstration. In _International Conference on Robotics and Automation, ICRA 2019_. IEEE, 7791–7797.  https://doi.org/10.1109/ICRA.2019.8794104
  * Hodgins et al. (1995) Jessica K. Hodgins, Wayne L. Wooten, David C. Brogan, and James F. O’Brien. 1995.  Animating Human Athletics. In _Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1995_. ACM, 71–78.  https://doi.org/10.1145/218380.218414
  * Jain et al. (2021) Arushi Jain, Khimya Khetarpal, and Doina Precup. 2021.  Safe option-critic: learning safety in the option-critic architecture.  _Knowl. Eng. Rev._ 36 (2021), e4.  https://doi.org/10.1017/S0269888921000035
  * Klissarov et al. (2017) Martin Klissarov, Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017.  Learnings Options End-to-End for Continuous Action Tasks.  _arXiv preprint arXiv:1712.00004_ (2017).  http://arxiv.org/abs/1712.00004
  * Konidaris and Barto (2009) George Dimitri Konidaris and Andrew G. Barto. 2009.  Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining. In _Advances in Neural Information Processing Systems 22_. 1015–1023.  https://proceedings.neurips.cc/paper/2009/hash/e0cf1f47118daebc5b16269099ad7347-Abstract.html
  * Kwon et al. (2008) Taesoo Kwon, Young-Sang Cho, Sang Il Park, and Sung Yong Shin. 2008.  Two-Character Motion Analysis and Synthesis.  _IEEE Trans. Vis. Comput. Graph._ 14, 3 (2008), 707–720.  https://doi.org/10.1109/TVCG.2008.22
  * Laszlo et al. (1996) Joseph Laszlo, Michiel van de Panne, and Eugene Fiume. 1996.  Limit Cycle Control and Its Application to the Animation of Balancing and Walking. In _Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996_. ACM, 155–162.  https://doi.org/10.1145/237170.237231
  * Lee et al. (2021) Youngwoon Lee, Joseph J. Lim, Anima Anandkumar, and Yuke Zhu. 2021.  Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization. In _Conference on Robot Learning, 2021_ _(Proceedings of Machine Learning Research, Vol. 164)_. PMLR, 406–416.  https://proceedings.mlr.press/v164/lee22a.html
  * Lee et al. (2019) Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward S. Hu, and Joseph J. Lim. 2019.  Composing Complex Skills by Learning Transition Policies. In _7th International Conference on Learning Representations, ICLR 2019_. OpenReview.net.  https://openreview.net/forum?id=rygrBhC5tQ
  * Li et al. (2022) Chengxi Li, Pai Zheng, Shufei Li, Yat Ming Pang, and Carman K. M. Lee. 2022.  AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop.  _Robotics Comput. Integr. Manuf._ 76 (2022), 102321.  https://doi.org/10.1016/J.RCIM.2022.102321
  * Liu et al. (2006) C. Karen Liu, Aaron Hertzmann, and Zoran Popovic. 2006.  Composition of complex optimal multi-character motions. In _Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2006_. Eurographics Association, 215–222.  https://doi.org/10.2312/SCA/SCA06/215-222
  * Liu et al. (2020) Huimin Liu, Zhiquan Wang, Christos Mousas, and Dominic Kao. 2020.  Virtual Reality Racket Sports: Virtual Drills for Exercise and Training. In _2020 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2020_. IEEE, 566–576.  https://doi.org/10.1109/ISMAR50242.2020.00084
  * Liu and Hodgins (2017) Libin Liu and Jessica K. Hodgins. 2017.  Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning.  _ACM Trans. Graph._ 36, 3 (2017), 29:1–29:14.  https://doi.org/10.1145/3083723
  * Liu et al. (2016) Libin Liu, Michiel van de Panne, and KangKang Yin. 2016.  Guided Learning of Control Graphs for Physics-Based Characters.  _ACM Trans. Graph._ 35, 3 (2016), 29:1–29:14.  https://doi.org/10.1145/2893476
  * Liu et al. (2010) Libin Liu, KangKang Yin, Michiel van de Panne, Tianjia Shao, and Weiwei Xu. 2010.  Sampling-based contact-rich motion control.  _ACM Trans. Graph._ 29, 4 (2010), 128:1–128:10.  https://doi.org/10.1145/1778765.1778865
  * Makoviychuk et al. (2021) Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. 2021.  Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.  https://openreview.net/forum?id=fgFBtYgJQX_
  * Marzinotto et al. (2014) Alejandro Marzinotto, Michele Colledanchise, Christian Smith, and Petter Ögren. 2014.  Towards a unified behavior trees framework for robot control. In _2014 IEEE International Conference on Robotics and Automation, ICRA 2014_. IEEE, 5420–5427.  https://doi.org/10.1109/ICRA.2014.6907656
  * Merel et al. (2019) Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. 2019.  Neural Probabilistic Motor Primitives for Humanoid Control. In _7th International Conference on Learning Representations, ICLR 2019_. OpenReview.net.  https://openreview.net/forum?id=BJl6TjRcY7
  * Mordatch et al. (2012) Igor Mordatch, Emanuel Todorov, and Zoran Popovic. 2012.  Discovery of complex behaviors through contact-invariant optimization.  _ACM Trans. Graph._ 31, 4 (2012), 43:1–43:8.  https://doi.org/10.1145/2185520.2185539
  * Oh et al. (2018) Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. 2018.  Self-Imitation Learning. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018_ _(Proceedings of Machine Learning Research, Vol. 80)_. PMLR, 3875–3884.  http://proceedings.mlr.press/v80/oh18b.html
  * Pastel et al. (2023) Stefan Pastel, Katharina Petri, C. H. Chen, Ana Milena Wiegand Cáceres, M. Stirnatis, C. Nübel, Lasse Schlotter, and Kerstin Witte. 2023.  Training in virtual reality enables learning of a complex sports movement.  _Virtual Real._ 27, 2 (2023), 523–540.  https://doi.org/10.1007/S10055-022-00679-7
  * Peng et al. (2018) Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018.  DeepMimic: example-guided deep reinforcement learning of physics-based character skills.  _ACM Trans. Graph._ 37, 4 (2018), 143.  https://doi.org/10.1145/3197517.3201311
  * Peng et al. (2017) Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel van de Panne. 2017.  DeepLoco: dynamic locomotion skills using hierarchical deep reinforcement learning.  _ACM Trans. Graph._ 36, 4 (2017), 41:1–41:13.  https://doi.org/10.1145/3072959.3073602
  * Peng et al. (2022) Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. 2022.  ASE: large-scale reusable adversarial skill embeddings for physically simulated characters.  _ACM Trans. Graph._ 41, 4 (2022), 94:1–94:17.  https://doi.org/10.1145/3528223.3530110
  * Peng et al. (2021) Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. 2021.  AMP: adversarial motion priors for stylized physics-based character control.  _ACM Trans. Graph._ 40, 4 (2021), 144:1–144:20.  https://doi.org/10.1145/3450626.3459670
  * Seo et al. (2023) Mingyo Seo, Steve Han, Kyutae Sim, SeungHyeon Bang, Carlos Gonzalez, Luis Sentis, and Yuke Zhu. 2023.  Deep Imitation Learning for Humanoid Loco-manipulation Through Human Teleoperation. In _22nd IEEE-RAS International Conference on Humanoid Robots, Humanoids 2023_. IEEE, 1–8.  https://doi.org/10.1109/HUMANOIDS57100.2023.10375203
  * Shum et al. (2008) Hubert P. H. Shum, Taku Komura, Masashi Shiraishi, and Shuntaro Yamazaki. 2008.  Interaction patches for multi-character animation.  _ACM Trans. Graph._ 27, 5 (2008), 114.  https://doi.org/10.1145/1409060.1409067
  * Shum et al. (2012) Hubert Pak Ho Shum, Taku Komura, and Shuntaro Yamazaki. 2012.  Simulating Multiple Character Interactions with Collaborative and Adversarial Goals.  _IEEE Trans. Vis. Comput. Graph._ 18, 5 (2012), 741–752.  https://doi.org/10.1109/TVCG.2010.257
  * Sutton et al. (1999) Richard S. Sutton, Doina Precup, and Satinder Singh. 1999.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning.  _Artif. Intell._ 112, 1-2 (1999), 181–211.  https://doi.org/10.1016/S0004-3702(99)00052-1
  * Tennis (2016) Eleven Table Tennis. 2016.  https://elevenvr.com
  * Tessler et al. (2023) Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. 2023.  CALM: Conditional Adversarial Latent Models for Directable Virtual Characters. In _ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 2023_. ACM, 37:1–37:9.  https://doi.org/10.1145/3588432.3591541
  * Wampler et al. (2010) Kevin Wampler, Erik Andersen, Evan Herbst, Yongjoon Lee, and Zoran Popovic. 2010.  Character animation in two-player adversarial games.  _ACM Trans. Graph._ 29, 3 (2010), 26:1–26:13.  https://doi.org/10.1145/1805964.1805970
  * Wang et al. (2023) Chao Wang, Anna Belardinelli, Stephan Hasler, Theodoros Stouraitis, Daniel Tanneberg, and Michael Gienger. 2023.  Explainable Human-Robot Training and Cooperation with Augmented Reality. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA 2023_. ACM, 449:1–449:5.  https://doi.org/10.1145/3544549.3583889
  * Won et al. (2020) Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. 2020.  A scalable approach to control diverse behaviors for physically simulated characters.  _ACM Trans. Graph._ 39, 4 (2020), 33.  https://doi.org/10.1145/3386569.3392381
  * Won et al. (2021) Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. 2021.  Control strategies for physically simulated characters performing two-player competitive sports.  _ACM Trans. Graph._ 40, 4 (2021), 146:1–146:11.  https://doi.org/10.1145/3450626.3459761
  * Won et al. (2022) Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. 2022.  Physics-based character controllers using conditional VAEs.  _ACM Trans. Graph._ 41, 4 (2022), 96:1–96:12.  https://doi.org/10.1145/3528223.3530067
  * Xu et al. (2023) Pei Xu, Xiumin Shang, Victor B. Zordan, and Ioannis Karamouzas. 2023.  Composite Motion Learning with Task Control.  _ACM Trans. Graph._ 42, 4 (2023), 93:1–93:16.  https://doi.org/10.1145/3592447
  * Yao et al. (2022) Heyuan Yao, Zhenhua Song, Baoquan Chen, and Libin Liu. 2022.  ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters.  _ACM Trans. Graph._ 41, 6 (2022), 183:1–183:16.  https://doi.org/10.1145/3550454.3555434
  * Yin et al. (2008) KangKang Yin, Stelian Coros, Philippe Beaudoin, and Michiel van de Panne. 2008.  Continuation methods for adapting simulated skills.  _ACM Trans. Graph._ 27, 3 (2008), 81.  https://doi.org/10.1145/1360612.1360680
  * Zhang et al. (2023) Haotian Zhang, Ye Yuan, Viktor Makoviychuk, Yunrong Guo, Sanja Fidler, Xue Bin Peng, and Kayvon Fatahalian. 2023.  Learning Physically Simulated Tennis Skills from Broadcast Videos.  _ACM Trans. Graph._ 42, 4 (2023), 95:1–95:14.  https://doi.org/10.1145/3592408
  * Zhu et al. (2023) Qingxu Zhu, He Zhang, Mengting Lan, and Lei Han. 2023.  Neural Categorical Priors for Physics-Based Character Control.  _ACM Trans. Graph._ 42, 6 (2023), 178:1–178:16.  https://doi.org/10.1145/3618397

Figure 8. Visualization of the average blending weights $\varphi$ of the
shoulder, elbow, and wrist joints. The weights of the mixer policy are usually
lowest when the paddle contacts the ball, and higher before and after
transitions between different skills. Figure 9. Agent-agent gameplay. Blue
agent is applying our strategy-level controller. The red dot is the target. We
demonstrate four skills; the forehand smash is less obvious because the
opponent does not deliver high and slow shots. Figure 10. Human-agent
interaction screenshots. A human controls a simulated paddle and the agent is
simulated and controlled by our method.


# Testing theories of the glass transition with the same liquid, but many
kinetic rules

Cristina Gavazzoni<EMAIL_ADDRESS>Instituto de Física, Universidade
Federal do Rio Grande do Sul, Caixa Postal 15051, CEP 91501-970, Porto Alegre,
Rio Grande do Sul, Brazil  Carolina Brito<EMAIL_ADDRESS>Instituto
de Física, Universidade Federal do Rio Grande do Sul, Caixa Postal 15051, CEP
91501-970, Porto Alegre, Rio Grande do Sul, Brazil  Matthieu Wyart
<EMAIL_ADDRESS>Institute of Physics, Ecole Polytechnique Federale de
Lausanne, 729 BSP UNIL, 1015, Lausanne, Switzerland

###### Abstract

We study the glass transition by exploring a broad class of kinetic rules that
can significantly modify the normal dynamics of super-cooled liquids, while
maintaining thermal equilibrium. Beyond the usual dynamics of liquids, this
class includes dynamics in which a fraction $(1-f_{R})$ of the particles can
perform pairwise exchange or ‘swap moves’, while a fraction $f_{P}$ of the
particles can only move along restricted directions. We find that (i) the
location of the glass transition varies greatly but smoothly as $f_{P}$ and
$f_{R}$ change and (ii) it is governed by a linear combination of $f_{P}$ and
$f_{R}$. (iii) Dynamical heterogeneities (DH) are not governed by the static
structure of the material, their magnitude correlates instead with the
relaxation time. (iv) We show that a recent theory for temporal growth of DH
based on thermal avalanches holds quantitatively throughout the
$(f_{R},f_{P})$ diagram. These observations are negative items for some
existing theories of the glass transition, particularly those reliant on
growing thermodynamic order or locally favored structure, and open new avenues
to test other approaches, as we illustrate.

Understanding why liquids glass formers cease to flow near their glass
transition $T_{g}$ remains a challenge. At that point, the relaxation time
$\tau_{\alpha}$ beyond which stress relaxes is of order of minutes, which is
fifteen decades larger than at high temperatures. From $\tau_{\alpha}$, the
activation energy $E_{a}$ can be defined as
$\tau_{\alpha}=t_{0}\exp(E_{a}/T)$, where $t_{0}$ is a microscopic time scale
and $T$ is the temperature (in the units of the Boltzmann constant). In
liquids called fragile, $E_{a}$ can increase five-fold or more under cooling
Anderson (1995); Ediger  _et al._ (1996); Debenedetti and Stillinger (2001);
Berthier and Biroli (2011). As the dynamics slows down, it also becomes more
and more heterogeneous, corresponding to a growing length scale $\xi$ Kob  _et
al._ (1997); Yamamoto and Onuki (1998); Dalle-Ferrier  _et al._ (2007);
Karmakar  _et al._ (2014). Contrasting theories seek to explain these two
facts. In the first class of views, including Adam-Gibbs Adam and Gibbs (1965)
and Random First Order Theory (RFOT) Kirkpatrick  _et al._ (1989); Lubchenko
and Wolynes (2007); Biroli and Bouchaud (2012), the increase of activation
energy stems from the emergence of some order on a growing length $\xi$, that
must be destroyed by cooperative motion on that scale to relax the material.
$E_{a}$ can then be expressed in terms of purely thermodynamic quantities,
independently of the kinetic rules governing the dynamics. Some real space
approaches associate such a growing order to locally favored structures
Patrick Royall  _et al._ (2008); Kawasaki  _et al._ (2007). A second viewpoint
seeks to capture the mechanism of dynamical facilitation, whereby the
relaxation of a given region speeds up the relaxation of regions nearby.
Kinetically constrained models Ritort and Sollich (2003); Berthier  _et al._
(2011), such as the East model, capture this effect and suggest a scenario
Garrahan and Chandler (2002); Garrahan  _et al._ (2007); Hedges  _et al._
(2009) in which thermodynamics plays almost no role, but dynamics is
heterogeneous and the growth of activation energy stems from non-local
rearrangements taking place over $\xi$. At odds with these two views, a third
approach that includes free volume Turnbull and Cohen (1961) or elastic Dyre
(2006); Kapteijns  _et al._ (2021); Lerbinger  _et al._ (2022); Ciamarra  _et
al._ (2023) models, assumes that the activation energy is not controlled by a
growing length scale. Instead, it is governed by the energy cost of elementary
rearrangements of a few particles jumping over a barrier. The elastic coupling
between rearrangements Lemaître (2014); Chowdhury  _et al._ (2016); Tong  _et
al._ (2020); Wu  _et al._ (2015); Maier  _et al._ (2017); Steffen  _et al._
(2022); Flenner and Szamel (2015); Klochko  _et al._ (2022); Chacko  _et al._
(2021) leads to a correlated dynamics Ozawa and Biroli (2023) that can be
described in terms of avalanches of activated events Tahaei  _et al._ (2023).

Figure 1: Diagram indicating the number of degrees of freedom for spatial
dimension $d=2$ (a) and $d=3$ (c) as a function of the fraction $f_{R}$ of
particles that cannot swap and the fraction $f_{P}$ of particles whose
translation motion is restricted. Panel (b) shows the particles positions at
various time points within some time interval for $d=2$. Particles that can
move freely in all direction are shown in red, and particles that are
restricted to linear motion appear in blue. For $d=3$, this constraint is more
gentle, as particles can still move on planes as sketched in (c).

Molecular dynamics simulations of models of super-cooled liquids have been
extremely informative to characterize the glass transition Kob (1999), yet
distinct views on this phenomenon have been hard to definitely contrast Biroli
and Bouchaud (2012). Our present goal is to show that for some popular models
of liquids, a very broad class of kinetic rules can be considered, which can
continuously (and very significantly) speed up or slow-down the normal
dynamics, while preserving thermal equilibrium. Although these rules would be
hard to implement in actual experiments, they are equivalent to dynamics with
purely local rules, and as such theories of the glass transition should apply
to them. This approach thus opens an avenue to test more stringently theories
of glassy dynamics. Specifically, our work builds on ‘swap’ Monte Carlo
algorithms. In these algorithms, pairs of particles can exchange positions, in
addition to their usual translation moves Briano and Glandt (1984); Grigera
and Parisi (2001); Fernández  _et al._ (2007); Gutiérrez  _et al._ (2015);
Ninarello  _et al._ (2017). For continuously polydisperse systems, these
algorithms can speed up the dynamics by 15 orders of magnitude or more
Ninarello  _et al._ (2017), and can change the glass transition temperature
$T_{g}$ by up to a factor of two. It allows one to explore glasses with a
stability similar to that reached in experiments.

Our central result is to introduce a family of kinetic rules, where a fraction
$f_{R}$ of the particles cannot swap, and a fraction $f_{P}$ of the particles
can only move along randomly-chosen hyperplanes. We provide systematic
measurements of the dynamics in the $(f_{R},f_{P})$ diagram in two and three
dimensions, that includes the normal dynamics $(1,0)$ as well as swap $(0,0)$.
Overall, our observations are negative items for theories based on a growing
thermodynamic order. We discuss how devoted studies could be used in these
models to test alternative views of the glass transition.

Changing continuously the kinetic rules of liquids: Swap moves lead to a
considerable speed up of the dynamics Briano and Glandt (1984); Gutiérrez  _et
al._ (2015); Ninarello  _et al._ (2017). Importantly, despite its apparent
non-local character, swap dynamics can be conceived as a purely local
dynamics. Following Brito  _et al._ (2018); Briano and Glandt (1984); Hagh
_et al._ (2022), swap is equivalent to considering identical particles endowed
with an additional ‘breathing’ degree of freedom, allowing them to change
their size according to some chemical potential $\mu(R)$. Indeed, letting
pairs of particles exchange is equivalent, in the thermodynamic limit, to
letting individual particles exchange with a bath of particles of all possible
sizes $R$. $\mu(R)$ is then chosen to obtain the desired polydispersity, which
is continuous for continuously poly-disperse particles Brito  _et al._ (2018);
Briano and Glandt (1984). Adding such a degree of freedom per particle
dramatically softens the energy landscape Szamel (2019); Ikeda  _et al._
(2019); Brito  _et al._ (2018); Wyart and Cates (2017) while preserving
thermodynamic and structural properties. It also affects the dynamics:
following the center of the particles, and considering that a swap move
corresponds to a change of the size of two particles but not of their
position, leads to the following observation. Dynamical correlations grow
under cooling as for the normal dynamics, but the correlation length starts
growing at a much smaller temperature Ninarello  _et al._ (2017). To study
more systematically such effects and their consequences, following Gopinath
_et al._ (2022) we vary the parameter $f_{R}\in[0,1]$, characterizing the
number of particles that cannot swap. Unlike Gopinath  _et al._ (2022), we do
not consider that particles that swap positions perform a jump, as it leads to
very different dynamics.

Following this logic, we propose to add even more kinetic constraints by
restricting the motion of a fraction $f_{P}\in[0,1]$ of the particles, as
illustrated in Fig.1. Each such particle is forbidden to move along one random
direction associated to it, and for an infinite system they would be each
confined along a random hyperplane. Overall, the number of degrees of freedom
for a system of $N$ particles is then $N(d-f_{P}+1-f_{R})$. Note that for the
periodic boundary condition we consider below, this dynamics is ergodic as
such hyperplanes visit the neighborhood of any point with probability one, if
their orientation is randomly chosen. For dynamics that satisfy detailed
balance like ours, it ensures that thermal equilibrium will eventually be
reached. Thus structural and thermodynamic properties are only governed by
$\phi$, independently of the choices of kinetic rules embodied in
$(f_{R},f_{P})$. Note that other procedures were proposed to reduce the number
of degrees of freedom, such as pinning particles starting from an equilibrated
system as proposed e.g. in Cammarota and Biroli (2012). Yet in that case
ergodicity is obviously broken once pinned particles are chosen, and the
dynamical properties of the system are not translation-invariant anymore.

In the present work, we specifically perform Monte Carlo simulations of
systems with $N$ continuously polydisperse hard spheres particles of packing
fraction $\phi$ in a regular box of linear size $L$, with periodic boundary
conditions. For hard particles, instead of temperature, $\phi$ is the good
controlled parameter. The choice of polydispersity together with other
numerical details are shown in the Appendix A. Figure 1-(a,c) shows a
schematic diagram of the different dynamics we explore in $d=2$ and $d=3$
respectively, and indicate in color the associated number of degrees of
freedom. Figure1-(b) illustrates an example of the particles trajectories at a
short time for $d=2$, revealing which particles are restricted to linear
motion, and which ones are not.

Figure 2: $\tau_{\alpha}/\tau_{\alpha}^{*}$ as function of $\phi$ for
different values of $f_{R}$ and $f_{P}$ for $d=2$ (a) and $d=3$ (e). From
these curves, $\phi_{G}$ as a function of $f_{R}$ and $f_{P}$ is extracted, as
indicated in color for $d=2$ (b) and $d=3$ (f). The discs mark the simulation
data, from which extrapolations are made. (c,g) show $\phi_{G}$ as a function
of the number of degrees of freedom per particle $d+1-f_{R}-f_{P}$ and (d,h)
show that $\phi_{G}$ can be collapsed as a function of an effective number of
constraints $x=f_{P}+C(d)f_{R}$, with $C(2)=0.45$, and $C(3)=0.70$. The normal
dynamics is marked as a red square in these plots.

Dependence of the glass transition packing fraction $\phi_{G}$ with kinetic
rules: For $d=3$, our choice of polydispersity and Monte-Carlo algorithm
follows closely previous works Berthier  _et al._ (2016); Ninarello  _et al._
(2017), where it was shown that swap can speed up the dynamics by 15 decades
or more. Here we observe a giant speed up of swap in our $d=2$ system as well,
as documented in Appendix B. We consider the dynamics on the full phase
diagram $(f_{R},f_{P})$. Fig.2-(a,e) show $\tau_{\alpha}$ vs $\phi$ for such
dynamics as the parameters $(f_{R},f_{P})$ are varied, both for $d=2$ and
$d=3$. Fig.2-(b,f) represents the corresponding value of $\phi_{G}$ in color,
as extrapolated from the different measurements made as indicated by circles.
The most remarkable results are that (i) $\phi_{G}$ varies very significantly,
and continuously throughout the phase diagram. In particular, there is no
evidence for a region surrounding the normal dynamics where kinetic
constraints would not matter and $\phi_{G}$ would plateau. The normal dynamics
can be made continuously faster or slower. (ii) Observing these two diagrams,
it is apparent that most of the variation of $\phi_{G}$ is captured by a
linear combination of $f_{R}$ and $f_{P}$.This hypothesis is tested in
Fig.2-(d,h), where it is shown that $\phi_{G}(f_{R},f_{P})=\phi_{G}(x)$, where
$x$ can be thought of as an effective number of constraints
$x=f_{P}+C(d)f_{R}$, where the coefficient $C(d)\leqslant 1$ characterize the
relative effect of breathing degrees of freedom v.s. translational ones. (iii)
qualitative observations are independent of the spatial dimension $d$.

Figure 3:  (a) Maximum $\chi_{4}^{*}$ of the dynamic susceptibility
$\chi_{4}(t)$ for $d=3$ as a function of $\phi$ as $f_{R}$ as is varied
indicated in legend, and $f_{P}=0$. (b)
$\chi_{4}^{*}/\chi_{4}{{}^{*}}(\phi=0.4)$ as a function of $\phi/\phi_{G}$.
Comparing (a) and (b) reveals that $\phi/\phi_{G}$ is a much better predictor
of DH than $\phi$. (c) Coarsening length $l_{c}$ for $d=2$ v.s.  $\ln(t)$ as
$\phi$ is varied, for different values of $f_{R}$ indicated in legend (d) for
$f_{P}=0$ and $t\leq\tau_{\alpha}/3$. (d) $l_{c}$ collapses when axis are
rescaled as predicted by Eq.1. The full black line is the theoretical
prediction $l_{c}=a\sqrt{\chi_{4}^{*}}[\ln(\tau_{\alpha}/t)]^{-1}$ with
$a=9.5$.

Dynamical heterogeneities are not governed by structural properties: Various
theoretical approaches propose that dynamical heterogeneities are controlled
by equilibrated structural properties, such as the extension of locally-
favored structure Patrick Royall  _et al._ (2008); Kawasaki  _et al._ (2007)
or a point to set length Lubchenko and Wolynes (2007); Biroli and Bouchaud
(2012) entering RFOT (although additional effects such as facilitation can be
added to this theory to increase further dynamical correlations, see e.g.
Biroli and Bouchaud (2023) for a recent discussion). In our set-up, these
properties depend only on $\phi$, and not on the values of parameters
$f_{R},f_{P}$. Here instead we find that for the relaxation times we can probe
and the system we consider, the dynamical susceptibility $\chi_{4}$
characterizing the magnitude of dynamical heterogeneities are not governed by
$\phi$ only, but instead depend strongly on $f_{R},f_{P}$ as illustrated in
Fig.3(a). Much less variation in $\chi_{4}$ is found when plotted as a
function of $\phi/\phi_{G}$ as shown in Fig.3(b).

These results indicate that heterogeneities are not controlled by a length
scale that would directly appear in the structure. Indeed, dynamical
correlations are small for the swap dynamics at significant packing fraction
for which the normal dynamics is already very correlated Ninarello  _et al._
(2017). Thus, although locally favored structure may affect dynamical
heterogeneities in specific systems (such as two-dimensional systems of discs
that can display hexatic order Kawasaki  _et al._ (2007)), it does not appear
to be the case for continuously polydisperse particles, at least in the range
of time scale that can be probed numerically.

The temporal evolution of DH follows that of thermal avalanches: Starting from
a reference equilibrated configuration, coloring rearranging regions reveals a
coarsening phenomenon Chandler and Garrahan (2010); Scalliet  _et al._ (2022)
and a growing length scale $l_{c}(t)$, as shown in Fig.3. Quantitatively we
define $l_{c}(t)=\langle l^{3}\rangle_{t}/\langle l^{2}\rangle_{t}$ where the
average $\langle\rangle_{t}$ is made on all clusters of connected relaxed
particles at time $t$, and $l$ is the square root of the number of particles
involved in a cluster. In Tahaei  _et al._ (2023), a description of this
coarsening based on facilitation via thermal avalanches of local activated
rearrangements was proposed. It was refined and also argued to apply to creep
flows in elastic manifolds in de Geus  _et al._ (2024), and implies for
$t\ll\tau_{\alpha}$ and $l_{c}$ much smaller than the system size:

$l_{c}\sim\xi\left[\ln(\tau_{\alpha}/t)\right]^{-1/(\tilde{\sigma}\tilde{d}_{f})}$
(1)

where $\xi$ is the correlation length, $\tilde{\sigma}$ some exponent and
$\tilde{d}_{f}$ is the fractal dimension of the clusters. In this view,
$\xi^{\tilde{d}_{f}}\sim\chi_{4}^{*}$ which characterizes the typical cluster
volume. For $d=2$, $\tilde{d}_{f}\approx 2$ and $\tilde{\sigma}\approx 1/2$
Tahaei  _et al._ (2023), leading to
$l_{c}\sim\sqrt{\chi_{4}^{*}}\left[\ln(\tau_{\alpha}/t)\right]^{-1}$. As shown
in Fig.3(d), this prediction works remarkably well, independently of $f_{R}$.
It supports that thermal avalanches cause DH independently of the kinetic
rules chosen.

Discussion: Swap Monte Carlo algorithms can be restricted to local moves with
no significant effects on the dynamics Ninarello  _et al._ (2017). More
fundamentally, they are equivalent to a purely local kinetic rule where
particles can adapt their radii Briano and Glandt (1984); Brito  _et al._
(2018). For theories based on a growing order on some length $\xi_{coop}$ such
as RFOT, or approaches based on locally favored structures, barriers are
cooperative and can be expressed in terms of thermodynamic quantities alone.
They should be present for any local kinetic rules Wyart and Cates (2017),
including those studied here. Thus in these approaches, the core mechanism
slowing down the dynamics near the glass transition should not depend on the
choice of $(f_{R},f_{P})$. Authors in Berthier  _et al._ (2019) acknowledged
that swap and normal dynamics should asymptotically relax at the same pace
according to RFOT, but countered that pre-asymptotic corrections (not
currently described within this theory) may cause the observed difference.

The main difficulty with this view is that the dynamics is very different as
the parameters $(f_{R},f_{P})$ change: as shown in Fig.5, these dynamics do
not become equivalent even after a slowing down of 15 decades accessible
experimentally. To have predictive power, RFOT or theories based on
thermodynamic quantities should specify a value for the parameters
$(f_{R},f_{P})$ for which they apply. However, currently they don’t. Our
observation that $\phi_{G}$ continuously depends on $(f_{R},f_{P})$, and does
not plateau to some constant value around the normal dynamics $(1,0)$, shows
that the normal dynamics is just one among many other dynamics. This point
underlines the lack of predictive power of RFOT or related theories - at least
for the systems of continuously polydisperse particles studied here.

By contrast, for theories based on kinetic constraints or on local barriers
(known to depend on kinetic rules Brito  _et al._ (2018)), the fact that
$\phi_{G}$ should continuously vary with the amount of constraints is evident.
The normal dynamics is slower simply because it is a kinetically constrained
version of swap dynamics.

Conclusion: RFOT is a mean-field theory of the glass transition, which has
shown undeniable successes. It is exact in infinite dimension Charbonneau  _et
al._ (2014); Maimbourg  _et al._ (2016), correctly captures aspects of the
thermodynamics of super-cooled liquids Biroli  _et al._ (2008), and presents a
dynamical transition Kurchan and Laloux (1996); Biroli  _et al._ (2006);
Lubchenko and Wolynes (2007) akin to mode coupling theory, that describes some
aspects of liquid dynamics at intermediate temperatures Götze (2008). Yet our
results support that its description of activation near the glass transition
does not apply for the continuously polydisperse systems studied here.
Although our conclusions are restricted to these specific systems where swap
is so performant, these models are known to capture the key facts associated
with the glass transition Scalliet  _et al._ (2022).

The model we introduced, with its very large variation of dynamics with
different kinetic rules, offers new opportunities to test different theories
of the glass transition, extending previous observations that were only
considered for a single kinetic rule. As an illustration, we showed that the
time evolution of dynamical heterogeneities follow a description of
facilitation based on thermal avalanches of local rearrangements Tahaei  _et
al._ (2023) independently of the kinetic rules chosen, emphasizing the
robustness of this theory. Other debated issues that this model would help
resolve include what observable predicts best the regions which will flow
first, the so-called dynamical propensity, for which many candidates were
proposed Widmer-Cooper  _et al._ (2004); Lerbinger  _et al._ (2022); Alkemade
_et al._ (2022); Schoenholz  _et al._ (2017). Likewise, numerical tests put
forward to test theories of the slowing down of the dynamics (i.e. how the
activation energy $E_{a}$ depends on temperature or packing fraction) can now
be made much more stringent. The list includes kinetically constrained or
lattice gas models Toninelli  _et al._ (2006), that can be tested by analyzing
irreversible events Keys  _et al._ (2011). Furthermore, the notion that glassy
dynamics correspond to local rearrangements was supported by the measurement
of the density of state of local barriers Ciamarra  _et al._ (2023); and these
barriers were argued to be governed alternatively by global Dyre (2006) or
local Kapteijns  _et al._ (2021) elasticity, or by the amplitude of
vibrational motion Puosi and Leporini (2012). Varying $(f_{R},f_{P})$ in these
measurements will indicate which viewpoint is most likely correct. Overall, we
have added an axis to the glass transition problem by varying continuously
kinetic rules, affecting strongly observations and giving a new handle to
decide which theory of the glass transition actually applies.

###### Acknowledgements.

We thank the Simons collaboration for discussions, in particular L. Berthier,
G. Biroli, M. Ozawa and C. Scalliet. We thank J.P. Bouchaud, T. deGeus, W. Ji,
M. Muller, M. Pica Ciamarra, M. Popovic, A. Tahei and A. Rosso for
discussions, and J. Kurchan for exchanges at the beginning of this project. MW
acknowledges sup- port from the Simons Foundation Grant (No. 454953 Matthieu
Wyart) and from the SNSF under Grant No. 200021-165509. C.B. and C.G. thank
the Brazilian agency CAPES and CNPq for the financial support..

## References

  * Anderson (1995) D. Anderson, Science 267, 1618 (1995).
  * Ediger  _et al._ (1996) M. D. Ediger, C. A. Angell, and S. R. Nagel, The journal of physical chemistry 100, 13200 (1996).
  * Debenedetti and Stillinger (2001) P. Debenedetti and F. Stillinger, Nature 410, 259 (2001).
  * Berthier and Biroli (2011) L. Berthier and G. Biroli, Reviews of modern physics 83, 587 (2011).
  * Kob  _et al._ (1997) W. Kob, C. Donati, S. J. Plimpton, P. H. Poole, and S. C. Glotzer, Physical review letters 79, 2827 (1997).
  * Yamamoto and Onuki (1998) R. Yamamoto and A. Onuki, Physical Review E 58, 3515 (1998).
  * Dalle-Ferrier  _et al._ (2007) C. Dalle-Ferrier, C. Thibierge, C. Alba-Simionesco, L. Berthier, G. Biroli, J.-P. Bouchaud, F. Ladieu, D. L’Hôte, and G. Tarjus, Phys. Rev. E 76, 041510 (2007).
  * Karmakar  _et al._ (2014) S. Karmakar, C. Dasgupta, and S. Sastry, Annu. Rev. Condens. Matter Phys. 5, 255 (2014).
  * Adam and Gibbs (1965) G. Adam and J. H. Gibbs, The journal of chemical physics 43, 139 (1965).
  * Kirkpatrick  _et al._ (1989) T. R. Kirkpatrick, D. Thirumalai, and P. G. Wolynes, Physical Review A 40, 1045 (1989).
  * Lubchenko and Wolynes (2007) V. Lubchenko and P. G. Wolynes, Annu. Rev. Phys. Chem. 58, 235 (2007).
  * Biroli and Bouchaud (2012) G. Biroli and J.-P. Bouchaud, Structural Glasses and Supercooled Liquids: Theory, Experiment, and Applications , 31 (2012).
  * Patrick Royall  _et al._ (2008) C. Patrick Royall, S. R. Williams, T. Ohtsuka, and H. Tanaka, Nature materials 7, 556 (2008).
  * Kawasaki  _et al._ (2007) T. Kawasaki, T. Araki, and H. Tanaka, Physical review letters 99, 215701 (2007).
  * Ritort and Sollich (2003) F. Ritort and P. Sollich, Advances in physics 52, 219 (2003).
  * Berthier  _et al._ (2011) L. Berthier, G. Biroli, J.-P. Bouchaud, L. Cipelletti, and W. van Saarloos,  _Dynamical heterogeneities in glasses, colloids, and granular media_ , Vol. 150 (OUP Oxford, 2011).
  * Garrahan and Chandler (2002) J. P. Garrahan and D. Chandler, Physical review letters 89, 035704 (2002).
  * Garrahan  _et al._ (2007) J. P. Garrahan, R. L. Jack, V. Lecomte, E. Pitard, K. van Duijvendijk, and F. van Wijland, Physical review letters 98, 195702 (2007).
  * Hedges  _et al._ (2009) L. Hedges, R. Jack, J. Garrahan, and D. Chandler, Science 323, 1309 (2009).
  * Turnbull and Cohen (1961) D. Turnbull and M. H. Cohen, The Journal of chemical physics 34, 120 (1961).
  * Dyre (2006) J. Dyre, Rev. Mod. Phys. 78, 953 (2006).
  * Kapteijns  _et al._ (2021) G. Kapteijns, D. Richard, E. Bouchbinder, T. B. Schrøder, J. C. Dyre, and E. Lerner, The Journal of Chemical Physics 155, 74502 (2021).
  * Lerbinger  _et al._ (2022) M. Lerbinger, A. Barbot, D. Vandembroucq, and S. Patinet, Physical Review Letters 129, 195501 (2022).
  * Ciamarra  _et al._ (2023) M. P. Ciamarra, W. Ji, and M. Wyart, “The energy cost of local rearrangements, not cooperative effects, makes glasses solid,”  (2023).
  * Lemaître (2014) A. Lemaître, Phys. Rev. Lett. 113, 245702 (2014).
  * Chowdhury  _et al._ (2016) S. Chowdhury, S. Abraham, T. Hudson, and P. Harrowell, The Journal of chemical physics 144, 124508 (2016).
  * Tong  _et al._ (2020) H. Tong, S. Sengupta, and H. Tanaka, Nature communications 11, 1 (2020).
  * Wu  _et al._ (2015) B. Wu, T. Iwashita, and T. Egami, Physical Review E 91, 032301 (2015).
  * Maier  _et al._ (2017) M. Maier, A. Zippelius, and M. Fuchs, Physical review letters 119, 265701 (2017).
  * Steffen  _et al._ (2022) D. Steffen, L. Schneider, M. Müller, and J. Rottler, The Journal of Chemical Physics 157, 064501 (2022).
  * Flenner and Szamel (2015) E. Flenner and G. Szamel, Physical Review Letters 114, 025501 (2015).
  * Klochko  _et al._ (2022) L. Klochko, J. Baschnagel, J. Wittmer, H. Meyer, O. Benzerara, and A. Semenov, The Journal of Chemical Physics 156, 164505 (2022).
  * Chacko  _et al._ (2021) R. Chacko, F. Landes, G. Biroli, O. Dauchot, A. Liu, and D. Reichman, arXiv preprint 2103.01852 (2021), 10.48550/arXiv.2103.01852.
  * Ozawa and Biroli (2023) M. Ozawa and G. Biroli, Physical Review Letters 130, 138201 (2023).
  * Tahaei  _et al._ (2023) A. Tahaei, G. Biroli, M. Ozawa, M. Popović, and M. Wyart, arXiv preprint arXiv:2305.00219 (2023).
  * Kob (1999) W. Kob, Journal of Physics: Condensed Matter 11, R85 (1999).
  * Briano and Glandt (1984) J. Briano and E. Glandt, J. Chem. Phys. 80, 3336 (1984).
  * Grigera and Parisi (2001) T. Grigera and G. Parisi, Phys. Rev. E 63, 045102 (2001).
  * Fernández  _et al._ (2007) L. Fernández, V. Martín-Mayor, and P. Verrocchio, Physical review letters 98, 085702 (2007).
  * Gutiérrez  _et al._ (2015) R. Gutiérrez, S. Karmakar, Y. Pollack, and I. Procaccia, Europhys. Lett. 111, 56009 (2015).
  * Ninarello  _et al._ (2017) A. Ninarello, L. Berthier, and D. Coslovich, Phys. Rev. X 7, 021039 (2017).
  * Brito  _et al._ (2018) C. Brito, E. Lerner, and M. Wyart, Phys. Rev. X 8, 031050 (2018).
  * Hagh  _et al._ (2022) V. F. Hagh, S. R. Nagel, A. J. Liu, M. L. Manning, and E. I. Corwin, Proceedings of the National Academy of Sciences 119, e2117622119 (2022).
  * Szamel (2019) G. Szamel, Journal of Statistical Mechanics: Theory and Experiment 2019, 104016 (2019).
  * Ikeda  _et al._ (2019) H. Ikeda, P. Urbani, and F. Zamponi, Journal of Physics A: Mathematical and Theoretical 52, 344001 (2019).
  * Wyart and Cates (2017) M. Wyart and M. Cates, Phys. Rev. Lett. 119, 195501 (2017).
  * Gopinath  _et al._ (2022) G. Gopinath, C.-S. Lee, X.-Y. Gao, X.-D. An, C.-H. Chan, C.-T. Yip, H.-Y. Deng, and C.-H. Lam, Physical Review Letters 129, 168002 (2022).
  * Cammarota and Biroli (2012) C. Cammarota and G. Biroli, Proceedings of the National Academy of Sciences 109, 8850 (2012).
  * Berthier  _et al._ (2016) L. Berthier, D. Coslovich, A. Ninarello, and M. Ozawa, Physical review letters 116, 238002 (2016).
  * Biroli and Bouchaud (2023) G. Biroli and J.-P. Bouchaud, Comptes Rendus. Physique 24, 1 (2023).
  * Chandler and Garrahan (2010) D. Chandler and J. P. Garrahan, Annual review of physical chemistry 61, 191 (2010).
  * Scalliet  _et al._ (2022) C. Scalliet, B. Guiselin, and L. Berthier, Physical Review X 12, 041028 (2022).
  * de Geus  _et al._ (2024) T. W. de Geus, A. Rosso, and M. Wyart, arXiv preprint arXiv:2401.09830 (2024).
  * Berthier  _et al._ (2019) L. Berthier, G. Biroli, J.-P. Bouchaud, and G. Tarjus, J. Chem. Phys. 150, 094501 (2019).
  * Charbonneau  _et al._ (2014) P. Charbonneau, J. Kurchan, G. Parisi, P. Urbani, and F. Zamponi, J. Stat. Mech. 2014, 10009 (2014).
  * Maimbourg  _et al._ (2016) T. Maimbourg, J. Kurchan, and F. Zamponi, Phys. Rev. Lett. 116, 015902 (2016).
  * Biroli  _et al._ (2008) G. Biroli, J.-P. Bouchaud, A. Cavagna, T. Grigera, and P. Verrocchio, Nat. Phys. 4, 771 (2008).
  * Kurchan and Laloux (1996) J. Kurchan and L. Laloux, J. Phys. A 29, 1929 (1996).
  * Biroli  _et al._ (2006) G. Biroli, J.-P. Bouchaud, K. Miyazaki, and D. Reichman, Phys. Rev. Lett. 97, 195701 (2006).
  * Götze (2008) W. Götze,  _Complex dynamics of glass-forming liquids: A mode-coupling theory_ , Vol. 143 (OUP Oxford, 2008).
  * Widmer-Cooper  _et al._ (2004) A. Widmer-Cooper, P. Harrowell, and H. Fynewever, Physical review letters 93, 135701 (2004).
  * Alkemade  _et al._ (2022) R. M. Alkemade, E. Boattini, L. Filion, and F. Smallenburg, The Journal of Chemical Physics 156, 204503 (2022).
  * Schoenholz  _et al._ (2017) S. S. Schoenholz, E. D. Cubuk, E. Kaxiras, and A. J. Liu, Proceedings of the National Academy of Sciences 114, 263 (2017).
  * Toninelli  _et al._ (2006) C. Toninelli, G. Biroli, and D. S. Fisher, Physical review letters 96, 035702 (2006).
  * Keys  _et al._ (2011) A. S. Keys, L. O. Hedges, J. P. Garrahan, S. C. Glotzer, and D. Chandler, Physical Review X 1, 021013 (2011).
  * Puosi and Leporini (2012) F. Puosi and D. Leporini, The Journal of chemical physics 136, 041104 (2012).
  * Berthier  _et al._ (2012) L. Berthier, G. Biroli, D. Coslovich, W. Kob, and C. Toninelli, Physical Review E 86, 031502 (2012).

##  Appendix A Numerical Details

We use for system sizes $N_{d=2}=484$ and $N_{d=3}=512$, which allow to
perform extensive simulations as $\phi$, $f_{P}$ and $f_{R}$ are varied, while
having small finite-size effects for the dynamical range considered Berthier
_et al._ (2012). Our Monte Carlo algorithm follows previous choices Ninarello
_et al._ (2017): it involves displacements and swap moves, where the latter
are attempted with a probability of $20\%$. The magnitude $\delta l$ of
translation moves is chosen such that the acceptance ratio is $75\%$.

To achieve rapidly equilibration for any choice of $(\phi,f_{R},f_{P})$, we
first use our fastest Monte Carlo, corresponding to $(\phi,f_{R}=0,f_{P}=0)$.
Then our algorithm is run with the desired values of $(f_{R},f_{P})$ for
$10^{9}$ Monte Carlo steps. To check that equilibration was reached, we
compare relevant observables (such as correlation functions) in the first and
second half of the run, and test for consistency.

The magnitude of the effect of the SWAP in speeding up the dynamics is
affected by the width of the particle radii distribution. The degree of
polydispersity is quantified as in Ninarello  _et al._ (2017) by some quantity
$\delta=\sqrt{\langle\sigma^{2}\rangle-\langle\sigma\rangle^{2}}/\langle\sigma\rangle$,
where $\langle\sigma\rangle$ is the average diameter of the packing which our
unit length. In this work we use packings with sizes distribution shown in
Fig. 4, which have polidispersity $19\%$ and $23\%$ for $d=2$ and $d=3$
respectively.

Figure 4: Initial radius distribution for the (a) 2D system and (b) 3D system.

##  Appendix B The effect of the continuous swap

In order to characterize the dynamics of the system for $d=3$ we consider the
usual self-scattering correlation function
$F_{s}(k,t)=\left\langle\frac{1}{N}\sum_{i}e^{i\mathbf{k}\cdotp[\mathbf{r}_{j}(t)-\mathbf{r}_{j}(0)]}\right\rangle$,
where $\mathbf{r}_{j}(t)$ is the position of the particle $j$ at time $t$ and
the wave vector $\mathbf{k}$ satisfies
$|\mathbf{k}|=2\pi/\langle\sigma\rangle$. For $d=2$, this definition is not
well-suited (long-wavelength vibrational modes bring $F_{s}(k,t)$ to zero for
large $t$ and $N$, even in a crystal). This problem is fixed as is usually
done by considering the relative motion of particle with respect to their
neighbors. It can be achieved by introducing the correlation
$C(t)=\frac{1}{N}\sum_{i}C^{i}(t)$, with
$C^{i}(t)=\frac{1}{n_{i}(t)}\sum_{\langle
ij\rangle}W(1-\frac{|(\mathbf{r}_{j}(t)-\mathbf{r}_{j}(t_{0}))-(\mathbf{r}_{i}(t)-\mathbf{r}_{i}(t_{0}))|}{\langle
R\rangle})$ where $\langle R\rangle$ is the average radius of the particles,
$n_{i}(t)$ is the number of neighbors of the particle $i$ at time $t$\-
defined as all particles $j$ for which
$|\mathbf{r}_{j}(t)-\mathbf{r}_{i}(t)|<2.4\langle R\rangle$, an estimate of
the first shell of neighbors, and $W(x)$ is the Heaviside function. To extract
a relaxation time $\tau_{\alpha}$, $F_{s}(k,t)$ and $C(t)$ are fitted with an
stretched exponential function $f(t)=\exp(-(t/\tau_{\alpha})^{\beta})$, where
$\tau_{\alpha}$ is the relaxation time. The glass packing fraction
$\phi_{G}(f_{R},f_{P})$ is then defined such that
$\tau_{\alpha}=\tau_{\alpha}^{*}\equiv 10^{7}$ Monte Carlo steps per particle.
The associated stretch exponents $\beta_{G}$ are reported in the Appendix as a
function of $(f_{R},f_{P})$.

Speed up: For $d=3$, our choice of polydispersity and Monte-Carlo algorithm
follows closely previous works Berthier  _et al._ (2016); Ninarello  _et al._
(2017), where it was shown that swap can speed up the dynamics by 15 decades
or more. Here we observe a giant speed up in our $d=2$ system as well, that
continuously builds up as $f_{R}$ decreases toward the swap case $f_{R}=0$
starting from the normal dynamics $f_{R}=1$. Fig.5, in Appendix B, shows the
relaxation times $\tau_{\alpha}$ – extracted from correlation functions as
recalled in Appendix C – as a function of the packing fraction $\phi$ for
different values of $f_{R}$. It is notable that $\tau_{\alpha}$ depends very
significantly on $f_{R}$, but that this dependence is continuous. Finally, the
speed-up of swap can be approximately extrapolated to the case where the
normal dynamics would reach experimental time scales (i.e. would increase by
15 decades). The range of the corresponding $\phi_{G}^{exp}$ can be estimated
by fitting the curve $\tau_{\alpha}(\phi)$ of the normal dynamics
$f_{R}=1,f_{P}=0$ with plausible functional forms for $\tau_{\alpha}(\phi)$.
The functional forms used to fit $\tau_{\alpha}(\phi)$ and estimate
$\phi_{G}^{exp}$ are the Vogel-Fulcher-Tamman
$\tau_{\alpha}\sim\exp\left(\frac{A}{\phi_{VFT}-\phi}\right)$ or a form with
non-singular activation energy
$\tau_{\alpha}=\tau^{\prime}_{\infty}\exp\left(A^{\prime}(\phi_{c}-\phi)^{2}\right)$.
In agreement with previous such inferences Berthier  _et al._ (2016);
Ninarello  _et al._ (2017), we obtain that the swap dynamics $f_{R}=1$ has
only slowed-down by 3 to 6 decades at $\phi_{G}^{exp}$: the speed up conferred
by swap is very high.

Figure 5: $\tau_{\alpha}/\tau_{\alpha}^{*}$ as a function of $\phi$ for a two-
dimensional system for different values of $f_{R}$. The dotted lines shows the
values of $\phi_{VFT}$ and $\phi_{P}$ obtained by fitting the normal dynamics
simulation ($f_{R}=1$). These fits are shown the insert panel.

##  Appendix C Stretched exponents

Figures 6-(a,b) show $\beta_{G}$ in color as a function of $f_{R}$ and $f_{P}$
for 2D and 3D system respectively where $\beta_{G}$ is the value of $\beta$
obtained at $\phi_{G}$. These values were obtained through the fitting of
function $F_{s}(k,t)$ and $C(t)$ with an stretched exponential function
$f(t)=exp(-(t/\tau_{\alpha})^{\beta})$, where $\tau_{\alpha}$ is the
relaxation time. In both cases $\beta_{G}$ decreases as the system becomes
more restricted.

Figure 6: $\beta_{G}$ as a function of $f_{R}$ and $f_{P}$ for $d=2$ (a) and
$d=3$ (b). The color-code represents the value of $\beta_{G}$.

##  Appendix D Measure of $l_{c}(t)$

To define a coarsening length $l_{c}(t)$, we first define relaxed particles.
Particle $i$ is a relaxed particle if $C^{i}(t)\leq 0.5$. Recall that
$C^{i}(t)$ represents the proportion of particles $j$ that remain neighbors of
particle $i$ after a time $t$. Then we consider that two relaxed particles $i$
and $j$ belong to the same cluster if
$|\mathbf{r}_{j}(t)-\mathbf{r}_{i}(t)|<2.4\langle R\rangle$. An example of a
growing length scale is presented in Fig.(7) where relaxed particles are shown
in red.

Figure 7: Spatio-temporal evolution of the structural relaxation in a 2d,
$f_{R}=0,f_{P}=0$ and $\phi=0.79$, for which $\tau_{\alpha}=2.6\times 10^{5}$.
Red color indicates relaxed particles. From left to right, the snapshots are
taken at times $\tau_{\alpha}/10$ ($l_{c}\approx 2.7$), $\tau_{\alpha}/5$
($l_{c}\approx 4.5$) and $\tau_{\alpha}/4$ ($l_{c}\approx 5.7$).