# SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and
Benchmark

Zhengdi Yu1,2, Shaoli Huang2 , Yongkang Cheng2, Tolga Birdal1
1 Imperial College London
2 Tencent AI Lab
<EMAIL_ADDRESS>{shaolihuang<EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding author.

###### Abstract

In this paper, we present SignAvatars, the first large-scale multi-prompt 3D
sign language (SL) motion dataset designed to bridge the communication gap for
hearing-impaired individuals. While there has been an exponentially growing
number of research regarding digital communication, the majority of existing
communication technologies primarily cater to spoken or written languages,
instead of SL, the essential communication method for hearing-impaired
communities. Existing SL datasets, dictionaries, and sign language production
(SLP) methods are typically limited to 2D as the annotating 3D models and
avatars for SL is usually an entirely manual and labor-intensive process
conducted by SL experts, often resulting in unnatural avatars. In response to
these challenges, we compile and curate the SignAvatars dataset, which
comprises 70,000 videos from 153 signers, totaling 8.34 million frames,
covering both isolated signs and continuous, co-articulated signs, with
multiple prompts including HamNoSys, spoken language, and words. To yield 3D
holistic annotations, including meshes and biomechanically-valid poses of
body, hands, and face, as well as 2D and 3D keypoints, we introduce an
automated annotation pipeline operating on our large corpus of SL videos.
SignAvatars facilitates various tasks such as 3D sign language recognition
(SLR) and the novel 3D SL production (SLP) from diverse inputs like text
scripts, individual words, and HamNoSys notation. Hence, to evaluate the
potential of SignAvatars, we further propose a unified benchmark of 3D SL
holistic motion production. We believe that this work is a significant step
forward towards bringing the digital world to the hearing-impaired
communities. Our project page is https://signavatars.github.io/.

##  1 Introduction

According to the World Health Organization, there are 466 million people
suffering from hearing disability (Davis & Hoffman, 2019). Among them, there
are over 70 million who communicate via sign languages (SLs) resulting in more
than 300 different SLs across different hearing-impaired communities (The
World Federation, ). While the field of (spoken) natural language processing
(NLP) and language assisted computer vision (CV) are well explored, this is
not the case for the alternate and important communicative tool of SL, and
accurate generative models of holistic 3D avatars as well as dictionaries are
highly desired for efficient learning (Naert et al., 2020).

We argue that the lack of large scale, targeted SL datasets is an important
reasons for this gap putting a barrier in front of downstream tasks such as
digital simultaneous SL translators. On one hand, existing SL datasets and
dictionaries (Duarte et al., 2021; Albanie et al., 2021; Camgoz et al., 2018;
Hanke et al., 2020; Huang et al., 2018) are typically limited to 2D videos or
2D keypoints annotations, which are insufficient for learners (Lee et al.,
2023) as different signs could appear to be the same in 2D domain due to
_depth ambiguity_. On the other hand, while parametric holistic models exist
for human bodies (Pavlakos et al., 2019) or bodies & faces (Yi et al., 2023),
there is no unified, large-scale, multi-prompt 3D holistic motion dataset with
accurate hand mesh annotations, which are crucial for SL. The reason for this
is that the creation of 3D avatar annotation for SL is a labor-intensive,
entirely manual process conducted by SL experts and the results are often
unnatural (Aliwy & Ahmed, 2021).

Figure 1: Overview of SignAvatars, the first public large-scale multi-prompt
3D sign language holistic motion dataset. (upper row) We introduce a generic
method to automatically annotate a large corpus of video data. (lower row) We
propose a 3D SLP benchmark to produce plausible 3D holistic mesh motion and
provide a neural architecture as well as baselines tailored for this novel
task.

To address this challenge, in this paper, we first introduce the SignAvatars
dataset, synergizing various data sources from public datasets to continuous
online videos with mixed-prompt annotations including HamNoSys, spoken
language, and word. Overall, we compile $70K$ videos from $153$ signers
amounting to $8.34M$ frames. Unlike (Forte et al., 2023), our dataset is not
limited to isolated signs, i.e. single sign per video, where HamNoSys-
annotations are present, but includes continuous and co-articulated signs. To
augment our dataset with 3D full-body annotations, including 3D body, hand and
face meshes as well as 2D & 3D keypoints, we design an automated and generic
annotation pipeline, in which we perform a multi-objective optimization over
3D poses and shapes of face, hands and body. Our optimizer considers the
temporal information of the motion and respects the biomechanical constraints
in order to produce accurate hand poses, even in presence of complex,
interacting hand gestures. Apart from meshes and SMPL-X models, we also
provide a _hand-only_ subset with MANO annotations.

SignAvatars enables multitude of tasks such as 3D sign language recognition
(SLR) or the novel 3D sign language production (SLP) from text scripts,
individual words, and HamNoSys notation. To address this challenge and
accommodate diverse forms of semantic input, we propose a novel approach
utilizing a semantic Variational Autoencoder (VQVAE) (Van Den Oord et al.,
2017) that effectively maps these varied inputs to discrete code indices. This
_parallel linguistic feature generator_ is fused with a discrete motion
encoder within an auto-regressive model to generate sequences of code indices
derived from these semantic representations, strengthening the text-motion
correlation. Consequently, our method can efficiently generate sign motion
from an extensive array of textual inputs, enhancing its versatility and
adaptability to various forms of semantic information. We will demonstrate in
Sec 5 that building such reliance and correlation between the low-level
discrete representations leads to accurate, natural and sign-motion consistent
SL production compared to direct regression from a high-level CLIP feature.

To quantitatively & qualitatively evaluate the potential of SignAvatars, we
introduce a new benchmark and present the first results for 3D SL holistic
mesh motion production from multiple prompts including HamNoSys, spoken
language, and word. On this benchmark. We assess the performance of our Sign-
VQVAE against the baselines we introduce, where we show a relative improvement
of $200\%$. Though, none of these models can truly match the desired accuracy,
confirming the timeliness and the importance of SignAvatars.

To summarize, our contributions are:

  * •

We introduce SignAvatars, the first large-scale multi-prompt 3D holistic
motion SL dataset, containing diverse forms of semantic input.

  * •

To provide accurate annotations for SignAvatars, in the form of expressive 3D
avatar meshes, we introduce a multi-objective optimization capable of dealing
with the complex interacting hands scenarios, while respecting the
biomechanical hand constraints. We initialize this fitting procedure by a
novel multi-stage, hierarchical process.

  * •

We provide a new 3D sign language production (SLP) benchmark for SignAvatars,
considering multiple prompts and full-body meshes.

  * •

We further develop a VQVAE-based strong 3D SLP network significantly
outperforming the baselines, which are also introduced as part of our work.

We believe SignAvatars is a significant stepping stone towards bringing the 3D
digital world and 3D SL applications to the hearing-impaired communities, by
fostering future research in 3D SL understanding.

##  2 Related Work

#### 3D holistic mesh reconstruction (for SL)

Recovering holistic 3D human body avatars from RGB videos and parsing them
into parametric forms like SMPL-X (Pavlakos et al., 2019) or Adam (Joo et al.,
2018) is a well explored area (Yi et al., 2023; Pavlakos et al., 2019; Lin et
al., 2023). For example, Arctic (Fan et al., 2023) introduces a full-body
dataset annotated by SMPL-X, for 3D object manipulation. Hasson et al. (2019)
provide a hand-object constellations datasets with MANO annotations. However,
such expressive parametric models have rarely been applied to the SL domain.
Kratimenos et al. (2021) use off-the-shelf methods to estimate a holistic 3D
mesh on existing dataset (Theodorakis et al., 2014) but cannot deal with the
challenging occlusions and interactions, making them unsuitable for complex,
real scenarios. The latest concurrent work (Forte et al., 2023) can
reconstruct 3D holistic mesh when the signs are isolated and the pre-defined
interpolation and linguistic rules are provided, e.g. such as HamNoSys. As
such, it can not be generalized to multiple-sign videos or videos without
isolated-sign-specific HamNoSys annotation. Overall, the literature lacks a
robust method handling continuous and co-articulated SL videos with complex
hand interactions.

#### SL datasets

While there have been many well-organized continuous 2D SL motion datasets
(Duarte et al., 2021; Albanie et al., 2021; Camgoz et al., 2018; Hanke et al.,
2020; Huang et al., 2018), the only existing 3D SL motion dataset with 3D
holistic mesh annotation is in (Forte et al., 2023). As mentioned, this rather
small dataset only includes a single sign per video onyl with HamNoSys-
prompts. In contrast, SignAvatars provides a multi-prompt 3D SL holistic
motion dataset with continuous and co-articulated signs and fine-grained hand
mesh annotations.

#### SL applications

Arkushin et al. (2023) can generate 2D motion sequences from HamNoSys.
Saunders et al. (2020) and Saunders et al. (2021) are able to generate 3D
keypoint sequences relying on glosses. The avatar approaches are often hand-
crafted and produce robotic and unnatural movements. Apart from them, there
are also early avatar approaches (Ebling & Glauert, 2016; Efthimiou et al.,
2010; Bangham et al., 2000; Zwitserlood et al., 2004; Gibet et al., 2016) with
a pre-defined protocol and character. To the best of our knowledge, we present
the first large-scale 3D holistic SL motion dataset, SignAvatars. Built upon
the dataset, we also introduce the novel task and benchmark of 3D sign
language production, through different prompts (language, word, HamNoSys).

Data | Video | Frame | Duration (hours) | co-articulated | Pose Annotation | Signer
---|---|---|---|---|---|---
RWTH-Phoenix-2014T (Camgoz et al., 2018) | 8.25K | 0.94M | 11 | C | - | 9
DGS Corpus (Hanke et al., 2020) | - | - | 50 | C | 2D keypoints | 327
BSL Corpus (Schembri et al., 2013) | - | - | 125 | C | - | 249
MS-ASL (Joze & Koller, 2018) | 25K | - | 25 | I | - | 222
WL-ASL (Li et al., 2020) | 21K | 1.39M | 14 | I | 2D keypoints | 119
How2Sign (Duarte et al., 2021) | 34K | 5.7M | 79 | C | 2D keypoints, depth* | 11
CSL-Daily (Huang et al., 2018) | 21K | - | 23 | C | 2D keypoints, depth | 10
SIGNUM (Von Agris et al., 2008) | 33K | - | 55 | C | - | 25
AUTSL (Sincan & Keles, 2020) | 38K | - | 21 | I | depth | 43
Forte et al. (2023) | 0.05K | 4K | - | I | body mesh vertices* | -
SignAvatars (Ours) | 70K | 8.34M | 117 | Both | SMPL-X, MANO, 2D&3D keypoints | 153

Table 1: Modalities of publicly available sign language datasets. C, I
represent isolated and co-articulated (continuous) separately. * means the
annotation has not been released yet. To the best of our knowledge, our
dataset is the first publicly available 3D SL holistic continuous motion
dataset with whole-body and hand mesh annotations with the most parallel
modalities.

##  3 SignAvatars Dataset

#### Overview

SignAvatars is a holistic motion dataset composed of $96K$ video clips having
$13.7M$ frames in total, containing body, hand and face motions as summarized
in Sec 3.1. We compile SignAvatars by synergizing various data sources from
public datasets to online videos and form seven subsets, whose distribution is
reported in Sec 3.1. Since the individual subsets do not naturally contain
expressive 3D whole-body motion labels and 2D keypoints, we introduce a
unified automatic annotation framework providing rich 3D holistic parametric
SMPL-X annotations along with MANO subsets for hands. Overall, we provide
$117$ hours of $70K$ video clips with $8.34M$ frames of motion data with
accurate expressive holistic 3D mesh as motion annotations.

###  3.1 Dataset Characteristics

#### Expressive motion representation

To fill in the gaps of previous 2D-only SL data. Our expressive 3D holistic
body annotation consists of face, hands, and body, which is achieved by
adopting SMPL-X (Pavlakos et al., 2019). It uses standard vertex-based linear
blend skinning with learned corrective blend shapes and has N = 10475 vertices
and K = 67 joints. For time interval $[1:t]$,
$V_{1:T}=(v_{1},...,v_{t}),J_{1:T}=(j_{1},...,j_{t}),\theta_{1:T}=(\theta_{1},...,\theta_{t})$,
represent mesh vertices, 3d joints, and poses in 6D representation (Zhu et
al., 2019). Here the pose $\theta_{t}$ includes the body pose
$\theta_{t}^{b}\in R^{23\times 6}$ with global orientation and the hand pose
$\theta_{t}^{h}\in R^{30\times 6}$. Moreover, $\theta_{t}^{f}\in R^{6}$ and
$\phi$ represents the yaw pose and facial expressions respectively. For each
of the sequences, we use an optimized consistent shape parameter
$\tilde{\beta}$ as there is no signer change in each clip. Overall, a motion
state ${M}_{t}$ is represented as:
$M_{t}=({\theta_{t}^{b},\theta_{t}^{h},\theta_{t}^{f},\phi,\tilde{\beta}})$.
Moreover, as shown in Tab. 1, our dataset also provides a hand motion subset
by replacing the parametric representation from SMPL-X to MANO (Romero et al.,
2022): $M^{h}_{t}=(\theta_{t}^{h},\tilde{\beta})$, where $h$ is the _handed-
ness_ and $\tilde{\beta}$ is also an optimized consistent shape parameter.

#### Sign language notation

Similar to spoken languages, sign languages have special structures with a set
of linguistic rules (Blaisel, 1997) (e.g. grammar, lexicons). Unlike spoken
languages, they have no standard written forms. Moreover, there are over 300
different sign languages across the world, with hearing-impaired people who do
not know any SL. Hence, having only a single type of annotation is
insufficient in practice. To enable more generic applications for different
users, we provide more modalities in the SignAvatars dataset. Our SL
annotations can be categorized into four common types: HamNoSys, spoken
language, word, and gloss, which can be used for a variety of downstream
applications such as SLP and SLR.

Figure 2: Distribution of subsets. The number of frames for each clip in
different subsets. PJM, LSF, DGS and GSL are gathered in one group.

Data | Video | Frame | Type | Signer
---|---|---|---|---
Word | 21K | 1.39M | W | 119
PJM | 2.6K | 0.21M | H | 2
DGS | 1.9K | 0.12M | H | 8
GRSL | 0.8K | 0.06M | H | 2
LSF | 0.4K | 0.03M | H | 2
ASL | 34K | 5.7M | S | 11
GSL | 8.3K | 0.83M | S, G | 9
Ours | 70K | 8.34M | S, H, W, G | 153

Table 2: Statistics of sub-datasets. W, H, S, G represent word, HamNoSys,
spoken language and gloss separately.

#### Data sources

As shown in Sec 3.1, SignAvatars leverages our unified automatic annotation
framework to collect SL motion sequences in divrse modalities from various
different sources. Specifically, for co-articulated SL datasets like How2Sign
(Duarte et al., 2021) and How2 (Sanabria et al., 2018) with American Sign
Language (ASL) transcriptions, we collect sentence-level clips from the Green
Screen studio subset with multi-view frames, resulting in $35K$ clips for the
ASL subset. For GSL subset, we mostly gathered data from the publicly
available PHOENIX14T dataset (Camgoz et al., 2018) following the official
split to have $8.25K$ video clips. For isolated-sign SL videos with HamNoSys
annotations, we collect $5.8K$ clips from Polish SL corpus (Linde-Usiekniewicz
et al., 2014) for PJM, and German Sign Language (DGS), Greek Sign Language
(GRSL) and French Sign Language (LSF) from DGS Corpus (Prillwitz et al., 2008)
and Dicta-Sign (Matthes et al., 2012). we gathered $21K$ clips from word-level
sources such as WLASL (Li et al., 2020) to curate the isolated-sign videos
without HamNoSys annotation. We provide further details in our appendix.

###  3.2 Automatic Holistic Annotation

To efficiently auto-label the SL videos with motion data given only RGB online
videos, we design an automatic 3D SL annotation pipeline that is not limited
to isolated signs with HamNoSys annotations as in (Forte et al., 2023). To
ensure motion stability and 3D shape accuracy, while maintaining efficiency
during holistic 3D mesh recovery from SL videos, we propose an iterative
fitting algorithm minimizing an objective heavily regularized both
holistically and by _biomechanical hand constraints_ (Spurr et al., 2020):

$\begin{split}E(\theta,\beta,\phi)=\lambda_{J}L_{J}+\lambda_{\theta}L_{\theta}+\lambda_{\alpha}L_{\alpha}+\lambda_{\beta}L_{\beta}+\lambda_{\mathrm{smooth}}L_{\mathrm{smooth}}+\lambda_{\mathrm{angle}}L_{\mathrm{angle}}+L_{\mathrm{bio}}\end{split}$
(1)

where $\theta$ is the full set of optimizable pose parameters, and $\phi$ is
the facial expression. $L_{J}$ represents the 2D joint re-projection loss,
which optimizes the difference between joints extracted from the SMPL-X model,
projected into the image, with joints predicted with ViTPose (Xu et al., 2022)
and MediaPipe (Kartynnik et al., 2019). $L_{\theta}$ is the pose prior term
following SMPLify-X (Pavlakos et al., 2019). Moreover, $L_{\alpha}$ is a prior
penalizing extreme bending only for elbows and knees and $L_{\beta}$ is the
shape prior term. In addition, $L_{\mathrm{smooth}}$, $L_{\mathrm{angle}}$ and
$L_{\mathrm{bio}}$ are the smooth-regularization loss, angle loss and
biomechanical constraints, separately. Finally, each $\lambda$ denotes the
influence weight of each loss term. Please refer to the appendix for more
details. In what follows, we describe in detail our regularizers.

Holistic regularization. In terms of reasonable regularization terms, to
reduce the jittery results caused by the noisy 2D detected keypoints, we first
introduce a smooth term to avoid jitter results for body and hand motion
poses, which is defined as:

$L_{\mathrm{smooth}}=\sum_{t}(||\hat{\theta}^{b}_{1:T}||_{2}+||\tilde{\theta}_{1:T}^{h}||_{2}+||\theta^{h}_{2:T}-\theta^{h}_{1:T-1}||_{2}+||\theta^{b}_{2:T}-\theta^{b}_{1:T-1}||_{2})$
(2)

where $\hat{\theta}^{b}_{1:T}\in R^{Nxj_{b}x3}$ is the selected subset of pose
parameters from $\theta_{1:T}^{b}\in R^{NxJx3}$, and N is the frame number of
the video. $\tilde{\theta}^{h}\in R^{Nxj_{h}}$ is the selected subset of hand
parameters from $\theta_{1:T}^{b}\in R^{NxJ*3}$. The $j_{b}$ and $j_{h}$ is
the selected body joint number and hand parameter number, respectively.
Moreover, this could prevent implausible poses along the bone direction such
as twist angle.

After that, we add an angle limit prior term to penalize the hand pose lies
outside the plausible range:

$L_{\mathrm{angle}}=\sum_{t}(\mathcal{I}(||\theta^{h}_{1:T}||_{2};\theta_{\mathrm{min}}^{h},\theta_{\mathrm{max}}^{h})+\mathcal{I}(||\theta^{b}_{1:T}||_{2};\theta_{\mathrm{min}}^{b},\theta_{\mathrm{max}}^{b}))$
(3)

where $\mathcal{I}$ is the interval loss penalizing the outliers,
$\theta_{\mathrm{min}}^{h,b},\theta_{\mathrm{max}}^{h,b}$ is the pre-defined
interval, $\theta^{h},\theta^{b}$ is the selected subset of holistic poses.
Finally, the signer in each video clip will not change, so we can use an
optimized consistent shape parameters $\beta$ to represent the holistic body
shape. Specifically, our fitting procedure is split into five stages, where we
will optimize the shape for the first three stages of optimization to derive
the mean shape and freeze the shape in the following stages.

Figure 3: Overview of our automatic annotation pipeline. Given an RGB image
sequence as input for hierarchical initialization, it is followed by
optimization with temporal smoothing and biomechanical constraints. Finally,
it outputs the final results in a motion sequence of SMPL-X parameters.

#### Biomechanical hand constraints

Hand pose estimation from monocular RGB images is challenging due to fast
movements, interaction, frequent occlusion and confusion. To further improve
the hand motion quality and eliminate implausible hand pose. We apply
biomechanical constraints to our hands, which contain three terms: (i)
$L_{\mathrm{bl}}$ for bone length, (ii) $L_{\mathrm{palm}}$ for palmar region
optimization, and (iii) $L_{\mathrm{ja}}$ for joint angle priors.
Specifically, $L_{\mathrm{bio}}$ is defined as the weighted sum of:

$\centering\begin{split}&L_{\mathrm{bl}}=\sum_{i}\mathcal{I}(||b^{i}_{1:T}||_{2};b_{min}^{i},b_{max}^{i}),\quad
L_{\mathrm{ja}}=\sum_{i}D(\alpha_{1:T}^{i},H^{i})\\\
&L_{\mathrm{palm}}=\sum_{i}(\mathcal{I}(||c^{i}_{1:T}||_{2};c_{min}^{i},c_{max}^{i})+\mathcal{I}(||d^{i}_{1:T}||_{2};d_{\mathrm{min}}^{i},d_{\mathrm{max}}^{i})),\end{split}\@add@centering$
(4)

where $\mathcal{I}$ is the interval loss penalizing the outliers, $b_{i}$ is
the bone length of $i$-th finger bone and the optimization constraints the
whole sequence $[1:T]$. After that, we further constraint the curvature and
angular distance for the 4 root bones of palmar structures by penalizing the
outliers of curvature range $c^{i}_{max},c^{i}_{min}$ and angular distance
range $d_{\mathrm{max}}^{i},d_{\mathrm{min}}^{i}$. Inspired by Spurr et al.
(2020), we also apply constraints to the sequence of joint angles
$\alpha_{1:T}^{i}=(\alpha_{1:T}^{f},\alpha_{1:T}^{a})$ by approximating the
convex hull on $(\alpha^{f},\alpha^{a})$ plane with point set $H^{i}$ and
minimizing their distance $D$, where $(\alpha^{f},\alpha^{a})$ is the flexion
and abduction angles. The biomechanical loss is then computed as the weighted
sum of them:
$L_{\mathrm{bio}}=\lambda_{\mathrm{bl}}L_{\mathrm{bl}}+\lambda_{\mathrm{palm}}L_{\mathrm{palm}}+\lambda_{\mathrm{ja}}L_{\mathrm{ja}}$.
We refer the reader to our appendix for more details.

#### Hierarchical initialization

Given an RGB image sequence, we initialize the holistic SMPL-X parameters from
OSX (Lin et al., 2023). Though, due to the frequent occlusion and hand
interactions, OSX is not always sufficient for a good initialization.
Therefore, we further fuse OSX with ACR (Yu et al., 2023), PARE (Kocabas et
al., 2021) to improve stability under occlusion and truncation. For 2D
holistic keypoints initialization, we first train a whole-body 2D pose
estimation model on COCO-wholeBody (Jin et al., 2020) based on ViTPose (Xu et
al., 2022) and subsequently incorporated with MediaPipe (Kartynnik et al.,
2019) by fusing and feeding through a confidence-guided filter.

##  4 Applications: SignVAE

Our SignAvatars dataset enables the first applications to generate high-
quality and natural 3D sign language holistic motion along with 3D meshes from
both isolated and continuous SL prompts. To this end, motivated by the fact
that the text prompts are highly correlated and aligned with the motion
sequence, our method consists of a two-stage process designed to enhance the
understanding of varied inputs by focusing on both semantic and motion
aspects. In the first stage, we develop two codebooks - a shared semantic
codebook and a motion codebook - by employing two Vector Quantized Variational
Auto-Encoders (VQ-VAE). This allows us to map diverse inputs to their
corresponding semantic code indices and link motion elements to motion code
indices. In the second stage, we utilize an auto-regressive model to generate
motion code indices based on the previously determined semantic code indices.
This integrated approach ensures a coherent and logical understanding of the
input data, effectively capturing both the semantic and motion-related
information.

Figure 4: Overview of our 3D SLP network. Our method consists of a two-stage
process. We first create semantic and motion codebooks using two VQ-VAEs,
mapping inputs to their respective code indices. Then, we employ an auto-
regressive model to generate motion code indices based on semantic code
indices, ensuring a coherent understanding of the data.

#### SL motion generation

To produce stable and natural holistic poses, instead of directly mapping
prompts to motion, we leverage the generative model VQ-VAE as our SL motion
generator. As illustrated in Fig. 4, our SL motion VQVAE consists of an
autoencoder structure and a learnable codebook $Z_{m}$, which contains $I$
codes $Z_{m}=\\{z_{i}\\}_{i=1}^{I}$ with $z_{i}\in R^{d_{z}}$. We first encode
the given 3D SL motion sequence
$M_{1:T}=({\theta_{T}^{b},\theta_{T}^{h},\theta_{T}^{f},\phi})$, where $T$ is
the motion length, into a latent feature
$F^{m}_{1:(T/w)}=(f_{1}^{m},...,f_{1:(T/w)}^{m})\in R^{d_{z}}$, where $w=4$ is
used as the downsampling rate for the window size. Subsequently, we quantize
the latent feature embedding by searching for the nearest neighbour code in
the codebook $Z_{m}$. For the $j-th$ feature, the quantization code is found
by: $\hat{f_{j}^{m}}=\mathop{\arg\max}\limits_{z_{i}\in
Z}||f_{j}^{m}-z_{i}||_{2}$. Finally, the quantized latent features are fed
into decoders for reconstruction. In terms of the training of the SL motion
generator, we apply the standard optimization scheme with $L_{motion_{v}q}$:

$L_{m-vq}=L_{\mathrm{recon}}(M_{1:T},\hat{M}_{1:T})+||\mathrm{sg}[F_{1:T}^{m}]-\hat{F_{1:T}^{m}}||_{2}+\beta||F_{1:T}^{m}-sg[\hat{F_{1:T}^{m}}]||_{2}$
(5)

where $L_{\mathrm{recon}}$ is the MSE loss and $\beta$ is a hyper-parameter.
$\mathrm{sg}$ is the detach operation to stop the gradient. We provide more
details regarding the network architecture and training in our appendix.

#### Prompt feature extraction for parallel linguistic feature generation.

In terms of the linguistic condition $c$ from input prompt, typical motion
generation tasks usually leverage LLM to produce linguistic prior for
efficient learning. In our task of spoken language and word-level annotation,
we leverage CLIP (Radford et al., 2021) as our prompt encoder to obtain the
text embedding $E^{l}$. However, this does not extend to all the other SL
annotations we desire. As a remedy, to enable applications with different
prompts such as HamNoSys, instead of relying on the existing pre-trained CLIP,
we define a new prompt encoder for embedding. After quantizing the prompt
(e.g. HamNoSys glyph) into tokens with length $s$, we use an embedding layer
to produce the linguistic feature
$\hat{E^{l}_{1:s}}=(\hat{e}^{l}_{1},...,\hat{e}^{l}_{s})$ with same dimension
$d_{l}$ as the text embeddings of CLIP (Radford et al., 2021). For simplicity,
we use ”text” to represent all different input prompts. Subsequently,
motivated by the fact that the text prompts are highly correlated and aligned
with the motion sequence, we propose a _parallel linguistic feature generator_
(PLFG) coupled with the SL motion generator. In particular, we leverage a
similar quantization process using the codebook $Z_{l}$ and training scheme as
in the SL motion generator to yield linguistic features:

$L_{l-vq}=L_{\mathrm{recon}}(E^{l}_{1:s},\hat{E}^{l}_{1:s})+||sg[F^{l}_{1:s}]-\hat{F^{l}_{1:s}}||_{2}+\beta||F^{l}_{1:s}-sg[\hat{F^{l}_{1:s}}]||_{2}$
(6)

where $F^{l}_{1:s}$ is the latent feature after encoding the initial
linguistic feature. $\hat{F^{l}_{1:s}}$ is the quantized linguistic feature
after applying $\hat{f_{j}^{l}}=\mathop{\arg\max}\limits_{z_{i}\in
Z_{l}}||f_{j}^{l}-z_{i}||_{2}$ to $F^{l}_{1:s}$.

#### Sign-motion cross modelling and production

After training the VQVAE-based SL motion generator, we can map any motion
sequence $M_{1:T}$ to a sequence of indices
$X=[x_{1},...,x_{T/w},x_{\mathrm{EOS}}]$ through the motion encoder and
quantization, where $end$ is a learnable end token representing the $stop$
signal. After training both the SL motion generator and the linguistic feature
generator, our network will be jointly optimized in a parallel manner.
Specifically, we fuse the linguistic feature embedding $E_{l}$ and the
codebook index vectors of $Z_{l}$ to formulate the final condition for our
autoregressive code index generator. The objective for training the code index
generator can be seen as an autoregressive next-index prediction task,
represented with a $\mathrm{cross-entropy}$ loss between the likelihood of the
full predicted code index sequence and the real ones as
$L_{\mathrm{SLP}}=\mathbb{E}_{X\sim p(X)}[-\log{p}(X|c)]$.

Lastly, with the quantized motion representation, we generate the codebook
vectors in a temporal autoregressive manner and predict the distribution of
the next codebook indices given an input linguistic prompt as linguistic
condition $c$. After mapping the codebook indices $X$ to the quantized motion
representation $\hat{F}_{1:(T/w)}^{m}$, we are able to decode and produce the
final 3D holistic motion with mesh representations $M_{1:T}$.

##  5 Experimental Evaluation

With SignAvatars dataset, we have enabled more 3D applications for sign
language communities, especially 3D SL motion production. We now showcase the
effectiveness and contribution of SignAvatars on the benchmark and application
introduced in Sec 4. Note that, these are also thee first benchmark results
for 3D holistic SL motion production yielding mesh representations.

Figure 5: Qualitative results of 3D holistic SLP from different prompts (left
row: spoken language, top right: HamNoSys, bottom right: word). Within each
sample, the first two rows are the input prompts and the generated results.
The last row is the corresponding video clip from our dataset.

#### Evaluation metrics

To fully assess the quality of our motion generation, we evaluate the holistic
motion as well as the arm motion111The lower body is not evaluated in our
experiments as it is not related to the SL motion. Based on an evaluation
model trained following prior arts in motion generation (Tevet et al., 2022;
Zhang et al., 2023b), we use the scores and metrics of FID, Diversity,
Multimodality (MM), MM-Dist, MR-precision, whose details are provided in Sec
B.2. Unfortunately, there is no de-facto standard for evaluating 3D SLP in the
literature. While Lee et al. (2023) is capable of back-translating 3D SL
motion by treating it as a classification, it is tailored only for word-level
back-translation. While BLEU and ROUGE are commonly used in the back-
translation evaluation (Saunders et al., 2020; 2021), they are not generic for
other types of annotations such as HamNoSys or glosses. Since the generated
motion might differ in length from the real motion, absolute metrics like
MPJPE would also be unsuited. Inspired by (Arkushin et al., 2023; Huang et
al., 2021), we propose a new MR-Precision for motion retrieval as well as DTW-
MJE (Dynamic Time Warping - Mean Joint Error) (Kruskal, 1983) with standard
SMPL-X keypoint set without lower body, for evaluating the performance of our
method as well as the baselines.

#### Subsets & training settings.

Specifically, we report results on three representative subsets: (i) the
complete set of $ASL$ for spoken language (corresponding to $language$ in
Table 3), (ii) the $word$ subset with 300 vocabularies, (iii) combined subset
of DGS, LSF, PJM, and GRSL for HamNoSys. For training, we follow the official
splits for (i) and (ii). For (iii), we leverage a four-fold strategy where we
train on three of them and test on the other, repeated four times to have the
final results.

#### Qualitative analysis

Fig. 5 shows examples of our 3D holistic body motion generation results. As
observed, our method can generate plausible and accurate holistic 3D motion
from different prompts while containing some diversity enriching the
production results. We show further comparison against the state-of-the-arts
in Appendix.

#### Quantitative analysis.

We present detailed quantitative results in Tab. 3. It can be seen that the 3D
SLP with word-level prompts can achieve the best performance reaching the
quality of real motions. Learning from spoken languages is a naturally harder
task and we invite the community to develop stronger methods to produce 3D SLP
from spoken languages. To further evaluate the sign accuracy and effect of
body movement, we report separate results for individual arms (e.g.
$"Gesture"$), with slight improvements in FID and MR-Precision. However, it
will also degenerate the text-motion consistency (R-Precision and MM-dist) due
to the absence of body-relative hand position.

Since there is no work in the literature that can generate 3D holistic SL
motion with mesh representation from any of the linguistic sources (e.g.
spoken language, HamNoSys, gloss, …), we introduce a new baseline, where we
modify the latest HamNoSys-based SLP work, Ham2Pose (Arkushin et al., 2023)
(corresponding to _Ham2Pose-3d_ in Tab. 4), to take our linguistic feature as
input and to output SMPL-X standard keypoints. We train this network on the
same split as our SignVAE and use DTW-MJE for evaluation. We also regress the
keypoints from our holistic representation $M_{1:T}$ on third subset (iii). It
shows the potential that leveraging our SignAvatars dataset can easily enable
more 3D approaches and significantly improve the existing SLP applications by
simple adaptation compared to the original Ham2Pose. The results are reported
on the HamNoSys _holistic_ set for comparison as illustrated in Tab. 4. While
our method drastically improves over the baseline, the result is far from
ideal, motivating the need for better models for this new task.

Data Type | R-Precision ($\uparrow$) | FID ($\downarrow$) | Diversity ($\rightarrow$) | MM ($\rightarrow$) | MM-dist ($\downarrow$) | MR-Precision ($\uparrow$)
---|---|---|---|---|---|---
top 1 | top 3 | top 5 | top 1 | top 3 | top 5
Real motion | Language | 0.398±.005 | 0.612±.007 | 0.709±.008 | 0.017±.153 | 9.565±.075 | - | 2.655±.057 | - | - | -
HamNoSys | 0.455±.002 | 0.689±.006 | 0.795±.004 | 0.007±.062 | 8.754±.028 | - | 2.113±.023 | - | - | -
Word-300 | 0.499±.003 | 0.811±.002 | 0.865±.003 | 0.006±.054 | 8.656±.035 | - | 1.855±.019 | - | - | -
Holistic | Language | 0.375±.007 | 0.535±.008 | 0.661±.0059 | 1.238±.389 | 11.56±.101 | 1.129±.107 | 3.156±.067 | 0.441±.007 | 0.675±.007 | 0.731±.009
HamNoSys | 0.429±.004 | 0.657±.005 | 0.756±.002 | 0.884±.035 | 9.451±.087 | 0.941±.056 | 2.651±.027 | 0.552±.002 | 0.745±.010 | 0.813±.034
Word-300 | 0.475±.002 | 0.731±.003 | 0.815±.005 | 0.756±.021 | 8.956±.091 | 0.815±.059 | 2.101±.024 | 0.615±.005 | 0.797±.006 | 0.875±.002
Gesture | Language | 0.341±.008 | 0.491±.009 | 0.671±.010 | 0.975±.315 | 10.08±.121 | 1.156±.135 | 3.491±.089 | 0.475±.011 | 0.689±003 | 0.751±.004
HamNoSys | 0.435±.005 | 0.649±.004 | 0.745±.006 | 0.851±.033 | 8.944±.097 | 0.913±.036 | 2.876±.015 | 0.581±.004 | 0.736±.006 | 0.825±.008
Word-300 | 0.465±.001 | 0.711±.003 | 0.818±.003 | 0.715±.016 | 8.235±.055 | 0.801±.021 | 2.339±.027 | 0.593±.006 | 0.814±.005 | 0.901±.006

Table 3: Quantitative evaluation results for the 3D holistic SL motion
generation. Real motion is the sampled motions from the original holistic
motion annotation in the dataset. Holistic represents the results for
generated holistic motion. Gesture stands for the evaluation conducted on two
arms.

Method | DTW-MJE Rank ($\uparrow$)
---|---
top 1 | top 3 | top 5
Ham2Pose* | 0.092±.031 | 0.197±.029 | 0.354±.032
Ham2Pose-3d | 0.253±.036 | 0.369±.039 | 0.511±.035
SignVAE (Ours) | 0.516±.039 | 0.694±.041 | 0.786±.035

Table 4: Comparison with state-of-the-art SLP method from HamNoSys. *
represents using only 2D information.

Method | R-Precision ($\uparrow$) | MM-dist ($\downarrow$)
---|---|---
top 1 | top 3 | top 5
Ham2Pose-3d | 0.291±.006 | 0.386±.005 | 0.535±.005 | 3.875±.086
SignDiffuse | 0.285±.003 | 0.415±.005 | 0.654±.003 | 3.866±.054
SignVAE (Base) | 0.385±.008 | 0.613±.009 | 0.745±.007 | 3.056±.108
SignVAE (Ours) | 0.429±.009 | 0.657±.008 | 0.756±.008 | 2.651±.119

Table 5: Quantitative ablation study of SignVAE on HamNoSys $holistic$ subset
for comparison with prior arts.

#### Ablation study

To study the contribution of different components of our method, we have
modified MDM (Tevet et al., 2022) as our backbone to take our linguistic
feature as input. This corresponds to _SignDiffuse_ in Tab. 5. For the study
of our unique text-sign cross-modeling module, our baseline, _SignVAE (Base)_
, replaces the PLFG with a canonical pre-trained CLIP feature as input to the
encoder. As shown in Tab. 5, our joint scheme utilizing the PLFG can
significantly improve the prompt-motion consistency, resulting in an increase
in R-precision and MM-dist. Moreover, our VQVAE backbone quantizing the motion
representation into a motion codebook, enables interaction with the linguistic
feature codebook, leading to significant improvements in prompt-motion
correspondences and outperforms other baselines built with our linguistic
feature generator (SignDiffuse, Ham2Pose-3d) and generates more text-motion
consistent results.

##  6 Conclusion

We introduced SignAvatars, the first large-scale 3D holistic SL motion dataset
with expressive 3D human and hand mesh annotations, provided by our automatic
annotation pipeline. SignAvatars enables a variety of application potentials
for hearing-impaired communities. Built upon our dataset, we propose the first
3D sign language production approach to generate natural holistic mesh motion
sequences from SL prompts. We also introduce the first benchmark results for
this new task, 3D holistic SL motion production from diverse SL prompts. Our
evaluations on this benchmark clearly show the advantage of our new VQVAE-
based model over the baselines, we develop.

Limitations and Future Work: Having the first benchmark we proposed, there is
still a lack of in-depth investigation of other 3D techniques for 3D SL motion
generation. Especially, due to the lack of a sophisticated existing generic 3D
back-translation methods, the evaluation may not fully showcase the
superiority of our dataset and the proposed method. We leave this for a future
study. Moreover, combining 3D SLT and SLP to formulate a multi-modal generic
SL framework will be one of the future works. Developing a large sign language
model with more properties and applications in AR/VR will significantly
benefit the hearing-impaired society around the world.

## Ethics Statement

Our work is driven by a dedication to the advancement of knowledge and the
betterment of society. As an important step towards bringing our 3D digital
world to the hear-impaired communities, most importantly, our first publicly
available large-scale 3D holistic SL motion dataset will significantly boost
the development and novel research ideas of 3D SL applications especially 3D
SLP and 3D SLT, which can be used for SL understanding and simultaneous SL
virtual signer. Furthermore, our dataset is not only capable of 3D SL
research, but it also possesses a good potential for motion capture areas to
gain more accurate hand pose, as well as many 3D applications in AR/VR,
animation, and human-computer interaction (HCI). Our dataset will follow the
license of the collection source and will not present any negative social
impact.

All of our experiments were either run on publicly available datasets or on
data that is synthetically generated. While these datasets contained human
annotations, they have been licensed appropriately. We included these licenses
in our appendix. No human or animal subjects have been involved at any stage
of the work we were involved. Trained on this data, our models are designed to
enhance the understanding and production of sign languages, without causing
harm or perpetuating unjust biases, unless provided in the datasets. While we
do not foresee any issue with methodological bias, we have not analyzed the
inherent biases of our algorithm and there might be implications in
applications demanding utmost fairness.

We aptly acknowledge the contributions of researchers whose work laid the
foundation for our own. Proper citations and credit are given to previous
studies and authors. All authors declare that there are no conflicts of
interest that could compromise the impartiality and objectivity of this
research. All authors have reviewed and approved the final manuscript before
submission.

## Reproducibility Statement

We are committed to transparency in research and for this reason will make our
implementation publicly available upon publication. To demonstrate our
dedication, we have submitted all source code as part of the appendix.

## References

  * Albanie et al. (2021) Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, and Andrew Zisserman.  In _BOBSL: BBC-Oxford British Sign Language Dataset_ , 2021.
  * Aliwy & Ahmed (2021) Ahmed H Aliwy and A Alethary Ahmed.  Development of arabic sign language dictionary using 3d avatar technologies.  _Indonesian Journal of Electrical Engineering and Computer Science_ , 21(1):609–616, 2021.
  * Arkushin et al. (2023) Rotem Shalev Arkushin, Amit Moryossef, and Ohad Fried.  Ham2pose: Animating sign language notation into pose sequences.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 21046–21056, 2023.
  * Bangham et al. (2000) J Andrew Bangham, SJ Cox, Ralph Elliott, John RW Glauert, Ian Marshall, Sanja Rankov, and Mark Wells.  Virtual signing: Capture, animation, storage and transmission-an overview of the visicast project.  In _IEE Seminar on speech and language processing for disabled and elderly people (Ref. No. 2000/025)_ , pp. 6–1. IET, 2000.
  * Blaisel (1997) Xavier Blaisel.  David f. armstrong, william c. stokoe, sherman e. wilcox, gesture and the nature of language, cambridge, cambridge university press, 1995, x+ 260 p., bibliogr., index.  _Anthropologie et Sociétés_ , 21(1):135–137, 1997.
  * Camgoz et al. (2018) Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden.  Neural sign language translation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 7784–7793, 2018.
  * Davis & Hoffman (2019) Adrian C Davis and Howard J Hoffman.  Hearing loss: rising prevalence and impact.  _Bulletin of the World Health Organization_ , 97(10):646, 2019.
  * Duarte et al. (2021) Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto.  How2sign: a large-scale multimodal dataset for continuous american sign language.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 2735–2744, 2021.
  * Ebling & Glauert (2016) Sarah Ebling and John Glauert.  Building a swiss german sign language avatar with jasigning and evaluating it among the deaf community.  _Universal Access in the Information Society_ , 15:577–587, 2016.
  * Efthimiou et al. (2010) Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, John Glauert, Richard Bowden, Annelies Braffort, Christophe Collet, Petros Maragos, and François Goudenove.  Dicta-sign: sign language recognition, generation and modelling with application in deaf communication.  In _sign-lang@ LREC 2010_ , pp. 80–83. European Language Resources Association (ELRA), 2010.
  * Fan et al. (2023) Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges.  ARCTIC: A dataset for dexterous bimanual hand-object manipulation.  In _Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Feng et al. (2021) Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black.  Collaborative regression of expressive bodies using moderation.  In _International Conference on 3D Vision (3DV)_ , 2021.
  * Forte et al. (2023) Maria-Paola Forte, Peter Kulits, Chun-Hao P Huang, Vasileios Choutas, Dimitrios Tzionas, Katherine J Kuchenbecker, and Michael J Black.  Reconstructing signing avatars from video using linguistic priors.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12791–12801, 2023.
  * Gibet et al. (2016) Sylvie Gibet, François Lefebvre-Albaret, Ludovic Hamon, Rémi Brun, and Ahmed Turki.  Interactive editing in french sign language dedicated to virtual signers: Requirements and challenges.  _Universal Access in the Information Society_ , 15:525–539, 2016.
  * Hanke et al. (2020) Thomas Hanke, Marc Schulder, Reiner Konrad, and Elena Jahn.  Extending the public dgs corpus in size and depth.  In _sign-lang@ LREC 2020_ , pp. 75–82. European Language Resources Association (ELRA), 2020.
  * Hasson et al. (2019) Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid.  Learning joint reconstruction of hands and manipulated objects.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 11807–11816, 2019.
  * Huang et al. (2018) Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, and Weiping Li.  Video-based sign language recognition without temporal segmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Huang et al. (2021) Wencan Huang, Wenwen Pan, Zhou Zhao, and Qi Tian.  Towards fast and high-quality sign language production.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pp. 3172–3181, 2021.
  * Jin et al. (2020) Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo.  Whole-body human pose estimation in the wild.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020.
  * Joo et al. (2018) Hanbyul Joo, Tomas Simon, and Yaser Sheikh.  Total capture: A 3d deformation model for tracking faces, hands, and bodies.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8320–8329, 2018.
  * Joze & Koller (2018) Hamid Reza Vaezi Joze and Oscar Koller.  Ms-asl: A large-scale data set and benchmark for understanding american sign language.  _arXiv preprint arXiv:1812.01053_ , 2018.
  * Kartynnik et al. (2019) Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grundmann.  Real-time facial surface geometry from monocular video on mobile gpus.  _arXiv preprint arXiv:1907.06724_ , 2019.
  * Kocabas et al. (2021) Muhammed Kocabas, Chun-Hao P Huang, Otmar Hilliges, and Michael J Black.  Pare: Part attention regressor for 3d human body estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 11127–11137, 2021.
  * Kratimenos et al. (2021) Agelos Kratimenos, Georgios Pavlakos, and Petros Maragos.  Independent sign language recognition with 3d body, hands, and face reconstruction.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4270–4274. IEEE, 2021.
  * Kruskal (1983) Joseph B Kruskal.  An overview of sequence comparison: Time warps, string edits, and macromolecules.  _SIAM review_ , 25(2):201–237, 1983.
  * Lee et al. (2023) Taeryung Lee, Yeonguk Oh, and Kyoung Mu Lee.  Human part-wise 3d motion context learning for sign language recognition.  _arXiv preprint arXiv:2308.09305_ , 2023.
  * Li et al. (2020) Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li.  Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pp. 1459–1469, 2020.
  * Lin et al. (2023) Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li.  One-stage 3d whole-body mesh recovery with component aware transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Linde-Usiekniewicz et al. (2014) Jadwiga Linde-Usiekniewicz, Małgorzata Czajkowska-Kisil, Joanna Łacheta, and Paweł Rutkowski.  A corpus-based dictionary of polish sign language (pjm).  In _Proceedings of the XVI EURALEX International Congress: The user in focus_ , pp. 365–376, 2014.
  * Matthes et al. (2012) Silke Matthes, Thomas Hanke, Anja Regen, Jakob Storz, Satu Worseck, Eleni Efthimiou, Athanasia-Lida Dimou, Annelies Braffort, John Glauert, and Eva Safar.  Dicta-sign-building a multilingual sign language corpus.  In _5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon. Satellite Workshop to the eighth International Conference on Language Resources and Evaluation (LREC-2012)_ , 2012.
  * Naert et al. (2020) Lucie Naert, Caroline Larboulette, and Sylvie Gibet.  A survey on the animation of signing avatars: From sign representation to utterance synthesis.  _Computers & Graphics_, 92:76–98, 2020.
  * Nocedal & Wright (1999) Jorge Nocedal and Stephen J Wright.  _Numerical optimization_.  Springer, 1999.
  * Pavlakos et al. (2019) Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Prillwitz et al. (2008) Siegmund Prillwitz, Thomas Hanke, Susanne König, Reiner Konrad, Gabriele Langer, and Arvid Schwarz.  Dgs corpus project–development of a corpus based electronic dictionary german sign language/german.  In _sign-lang@ LREC 2008_ , pp. 159–164. European Language Resources Association (ELRA), 2008.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Romero et al. (2022) Javier Romero, Dimitrios Tzionas, and Michael J Black.  Embodied hands: Modeling and capturing hands and bodies together.  _arXiv preprint arXiv:2201.02610_ , 2022.
  * Sanabria et al. (2018) Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze.  How2: a large-scale dataset for multimodal language understanding.  _arXiv preprint arXiv:1811.00347_ , 2018.
  * Saunders et al. (2020) Ben Saunders, Necati Cihan Camgoz, and Richard Bowden.  Progressive transformers for end-to-end sign language production.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , pp. 687–705. Springer, 2020.
  * Saunders et al. (2021) Ben Saunders, Necati Cihan Camgoz, and Richard Bowden.  Mixed signals: Sign language production via a mixture of motion primitives.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 1919–1929, 2021.
  * Schembri et al. (2013) Adam Schembri, Jordan Fenlon, Ramas Rentelis, Sally Reynolds, and Kearsy Cormier.  Building the british sign language corpus.  _LaNguagE DocumENtatIoN & coNSErvatIoN_, 2013.
  * Sincan & Keles (2020) Ozge Mercanoglu Sincan and Hacer Yalim Keles.  Autsl: A large scale multi-modal turkish sign language dataset and baseline methods.  _IEEE Access_ , 8:181340–181355, 2020.
  * Spurr et al. (2020) Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz.  Weakly supervised 3d hand pose estimation via biomechanical constraints.  In _European conference on computer vision_ , pp. 211–228. Springer, 2020.
  * Tevet et al. (2022) Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ , 2022.
  * (44) of the Deaf The World Federation.  Who we are.  URL http://wfdeaf.org/who-we-are/.
  * Theodorakis et al. (2014) Stavros Theodorakis, Vassilis Pitsikalis, and Petros Maragos.  Dynamic–static unsupervised sequentiality, statistical subunits and lexicon for sign language recognition.  _Image and Vision Computing_ , 32(8):533–549, 2014.
  * Van Den Oord et al. (2017) Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Von Agris et al. (2008) Ulrich Von Agris, Moritz Knorr, and Karl-Friedrich Kraiss.  The significance of facial features for automatic sign language recognition.  In _2008 8th IEEE international conference on automatic face & gesture recognition_, pp. 1–6. IEEE, 2008.
  * Xu et al. (2022) Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao.  Vitpose: Simple vision transformer baselines for human pose estimation.  _Advances in Neural Information Processing Systems_ , 35:38571–38584, 2022.
  * Yi et al. (2023) Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black.  Generating holistic 3d human motion from speech.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 469–480, 2023.
  * Yu et al. (2023) Zhengdi Yu, Shaoli Huang, Fang Chen, Toby P. Breckon, and Jue Wang.  Acr: Attention collaboration-based regressor for arbitrary two-hand reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2023.
  * Zhang et al. (2023a) Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu.  Pymaf-x: Towards well-aligned full-body model regression from monocular images.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023a.
  * Zhang et al. (2023b) Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen.  T2m-gpt: Generating human motion from textual descriptions with discrete representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023b.
  * Zhu et al. (2019) Hao Zhu, Xinxin Zuo, Sen Wang, Xun Cao, and Ruigang Yang.  Detailed human shape estimation from a single image by hierarchical mesh deformation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 4491–4500, 2019.
  * Zwitserlood et al. (2004) Inge Zwitserlood, Margriet Verlinden, Johan Ros, Sanny Van Der Schoot, and T Netherlands.  Synthetic signing for the deaf: Esign.  In _Proceedings of the conference and workshop on assistive technologies for vision and hearing impairment (CVHI)_ , 2004.

##  Appendix A Appendix A: More Experimental Results

Figure 6: More examples of SignAvatars, $ASL$ subset. We have different shapes
of annotations presenting the accurate body and hand estimation. Figure 7:
More examples of SignAvatars, $HamNoSys$ subset. We have different shapes of
annotations presenting the accurate body and hand estimation. Figure 8: More
examples of SignAvatars, $word$ subset. We have different shapes of
annotations presenting the accurate body and hand estimation. Figure 9:
Comparison of 3D holistic body reconstruction. The results from PIXIE (Feng et
al., 2021), PyMAF-X (Zhang et al., 2023a), and ours.

###  A.1 Visualization of SignAvatars Dataset

In this section, we provide further details of our SignAvatars dataset and
present more visualization of our data in Fig. 6, 7 and 8. We also provide a
demo video in the supplementary materials.

###  A.2 Visualization of SignVAE generation

Furthermore, We share snapshot examples produced from our SignVAE,
demonstrating the application potential for 3D sign language production in our
demo video. For each sample, the first two rows are the input linguistic
prompts and the generated results. The last row is the corresponding video
clip from our dataset.

##  Appendix B More details for experiments and evaluation

#### Optimization strategy of automatic annotation pipeline

During optimization, we utilize an iterative five-stage fitting procedure to
minimize the objective function and use Adam optimizer with 1e-2 as the
learning rate. Moreover, a good initialization can significantly boost the
fitting speed of our annotation pipeline. At the same time, a well-pixel-
aligned body pose will also help the reconstruction of hand meshes. Motivated
by this, we apply 2000 fitting steps for a clip and split the fitting steps
into five stages with 400 steps in each stage to formulate our iterative
fitting pipeline. In the meantime, the Limited-memory BFGS (Nocedal & Wright,
1999) with a strong Wolfe line is applied to our optimization. In the first
three stages, all the loss and parameters are optimized together. The weights
$w_{body}=w_{hand}=1$ are applied for $L_{J}$ to obtain a good body pose
estimation. In the last two stages, we will first extract a mean pose from the
record of the previous optimization to gain a stable body shape and freeze it
as a fixed shape, as the signer will not change in a video by default.
Subsequently, to obtain accurate and detailed hand meshes, we will enlarge the
$w_{hand}$ to 2 to reach the final holistic mesh reconstruction with a natural
and accurate hand pose.

###  B.1 Qualitative Analysis of SignAvatars Dataset

#### Comparison with State-of-the-arts

To showcase the effectiveness of our annotation pipeline and its capability of
recovering accurate hand & holistic mesh annotation. We have conducted
experiments against previous state-of-the-art monocular methods including
SMPLify-x (Pavlakos et al., 2019), OSX (Lin et al., 2023), PyMAF-X (Zhang et
al., 2023a). As shown in Fig. 9, our method generates significantly more
natural body movement, as well as accurate hand pose and better pixel-mesh
aligned body shape ($\beta$). Please refer to our supplementary videos for
more details.

As we can see from the comparison with state-of-the-art methods, our method
can generate significantly natural and plausible hand poses. Subsequently, the
biomechanical constraints can serve as a prior for eliminating the implausible
poses, which happens frequently in complex interacting-hands scenarios for
other monocular capture methods.

###  B.2 Evaluation Protocols

In this subsection, we will elaborate on the computational details of our used
evaluation protocol. To start with, our evaluation relies on a text-motion
embedding model following prior arts (Zhang et al., 2023b; Tevet et al., 2022;
Lee et al., 2023). For simplicity, we use the same symbols and notations as in
our Secs. 3.1 and 4. Through the GRU embedding layer, we embed our motion
representation $M_{1:T}$ and linguistic feature $E^{l}_{1:s}$ into $f_{m}\in
R^{d}$ and $f_{l}\in R^{d}$ with the same dimensions to apply contrastive loss
and minimize the feature distances, where $d=512$ is used in our experiments.
After motion and prompt feature extraction, we compute each of the evaluation
metrics, which are summarized below:

  * •

Frechet Inception Distance (FID) ($\downarrow$), the distributional distance
between the generated motion and the corresponding real motion based on the
extracted motion feature.

  * •

Diversity, the average Euclidean distance in between the motion features of
$N_{D}=300$ randomly sampled motion pairs.

  * •

R-precision ($\uparrow$), the average accuracy at top-$k$ positions of sorted
Euclidean distances between the motion embedding and each GT prompt embedding.

  * •

Multimodality, average Euclidean distance between the motion feature of
$N_{m}=10$ pairs of motion generated with the same single input prompt.

  * •

Multimodal Distance (MM-Dist) ($\downarrow$), average Euclidean distance
between each generated motion feature and its input prompt feature.

  * •

MR-precision ($\downarrow$), the average accuracy at top-$k$ positions of
sorted Euclidean distance between a generated motion feature and 16 motion
samples from dataset (1 positive + 15 negative).

We now provide further details in each of those. For simplicity, we denote the
dataset length as $N$ below.

#### Frechet Inception Distance (FID)

is used to evaluate the distribution distance between the generated motion and
the corresponding real motion:

$FID=||\mu_{gt}-\mu_{pred}||_{2}-Tr(C_{gt}+C_{pred}-2(C_{gt}C_{pred})^{1/2})$
(7)

where $\mu_{gt},\mu_{pred}$ are the mean values for the features of real
motion and generated motion, separately. $C,Tr$ are the covariance matrix and
trace of a matrix.

Diversity is used for evaluating the variance of the generated SL motion.
Specifically, we randomly sample $N_{D}=300$ motion feature pairs
$\\{f_{m},f_{m}^{{}^{\prime}}\\}$ and compute the average Euclidean distance
between them by:

$Diversity=\frac{1}{N_{D}}\sum_{i}^{N_{D}}||f_{m}^{i}-f_{m}^{i^{\prime}}||$
(8)

Multimodality is leveraged to measure the diversity of the SL motion generated
from the same prompts. Specifically, we compute the average Euclidean distance
between the extracted motion feature of $N_{m}=10$ pairs
$\\{f_{m}^{j},f_{m}^{j^{\prime}}\\}$ of motion generated with the same single
input prompt. Through the full dataset, it can be written as:

$Multimodality=\frac{1}{N*N_{m}}\sum_{i}^{N}\sum_{j}^{N_{M}}||f_{m}^{ij}-f_{m}^{ij^{\prime}}||$
(9)

Multimodal Distance (MM-Dist) is applied to evaluate the text-motion
correspondency. Specifically, it computes the average Euclidean distance
between each generated motion feature and its input prompt feature:

$MM\textsc{-}Dist=\frac{1}{N}\sum_{i}^{N}||f_{m}^{i}-f_{l}^{i}||$ (10)

##  Appendix C Discussion

###  C.1 Related Work

In this section, we present more details about the related work as well as the
open problems.

#### Background

Existing SL datasets, and dictionaries are typically limited to 2D, which is
ambiguous and insufficient for learners as introduced in Lee et al. (2023),
different signs could appear to be the same in 2D domain due to depth
ambiguity. In that, 3D avatars and dictionaries are highly desired for
efficient learning (Naert et al., 2020), teaching, and many downstream tasks.
However, The creation of 3D avatar annotation for SL is a labor-intensive,
entirely manual process conducted by SL experts and the results are often
unnatural (Aliwy & Ahmed, 2021). As a result, there is not a unified large-
scale multi-prompt 3D sign language holistic motion dataset with precise hand
mesh annotations. The lack of such 3D avatar data is a huge barrier to
bringing these meaningful applications to hearing-impaired communities, such
as 3D sign language production (SLP), 3D sign language recognition (SLR), and
many downstream tasks such as digital simultaneous translators between spoken
language and sign language in VR/AR.

#### Open problems

Overall, the open problems chain is: 1) Current 3D avatar annotation methods
for sign language are mostly done manually by SL experts and are labor-
intensive. 2) Lack of generic automatic 3D expressive avatar annotation
methods with detailed hand pose. 3) Due to the lack of a generic annotation
method, there is also a lack of a unified large-scale multi-prompt 3D co-
articulated continuous sign language holistic motion dataset with precise hand
mesh annotations. 4) Due to the above constraints, it is difficult to extend
sign language applications to highly desired 3D properties such as 3D SLR, 3D
SLP, which can be used for many downstream applications like virtual
simultaneous SL translators, 3D dictionaries, etc.

According to the problem chain, we will introduce the SOTA from three aspects:
3D holistic mesh annotation pipeline, 3D sign language motion dataset, and 3D
SL applications.

3D holistic mesh annotation: There are a lot of prior works for reconstructing
holistic human body from RGB images with parametric models like SMPL-X
(Pavlakos et al., 2019), Adam (Joo et al., 2018). Among them, TalkSHOW (Yi et
al., 2023) proposes a fitting pipeline based on SMPLify-X (Pavlakos et al.,
2019) with a photometric loss for facial details. OSX (Lin et al., 2023)
proposes a time-consuming finetune-based weakly supervision pipeline to
generate pseudo-3D holistic annotations. However, such expressive parametric
models have rarely been applied to the SL domain. Kratimenos et al. (2021) use
off-the-shelf methods to estimate holistic 3D mesh on the GSLL sign-language
dataset (Theodorakis et al., 2014). In addition to that, only a concurrent
work (Forte et al., 2023) can reconstruct 3D holistic mesh annotation from
isolated-sign videos with HamNoSys annotations with pre-defined interpolation
and symmetric loss based on the rules of HamNoSys. However, it can not be
generalized to multiple-sign videos or videos without isolated-sign-specific
HamNoSys annotation. These methods cannot deal with the challenging self-
occlusion, hand–hand and hand–body interactions which makes them insufficient
for complex interacting hand scenarios such as sign language. There is not yet
a generic annotation pipeline that is sufficient to deal with complex
interacting hand cases in continuous and co-articulated SL videos.

Sign Language Datasets While there have been many well-organized continuous SL
motion datasets (Duarte et al., 2021; Albanie et al., 2021; Camgoz et al.,
2018; Hanke et al., 2020; Huang et al., 2018) with 2D videos or 2D keypoints
annotations, the only existing 3D SL motion dataset with 3D holistic mesh
annotation is in (Forte et al., 2023), which is purely isolated sign based and
HamNoSys-based and not sufficient for tackling real-world applications in
natural language scenarios. There is not yet a unified large-scale multi-
prompt 3D SL holistic motion dataset with continuous and co-articulated signs
and precise hand mesh annotations.

SL applications Regarding the SL applications, especially sign language
production (SLP), Arkushin et al. (2023) can generate 2D motion sequences from
HamNoSys. Saunders et al. (2020) and Saunders et al. (2021) are able to
generate 3D keypoint sequences with glosses. The avatar approaches are often
hand-crafted and produce robotic and unnatural movements. Apart from them,
there are also early avatar approaches (Ebling & Glauert, 2016; Efthimiou et
al., 2010; Bangham et al., 2000; Zwitserlood et al., 2004; Gibet et al., 2016)
with a pre-defined protocol and character.

###  C.2 Licensing

Our dataset will first be released under the CC BY-NC-SA (Attribution-
NonCommercial-Share-Alike) license for research purposes. We also elaborate on
the license of the data source we used in our dataset collection:

#### How2Sign (Duarte et al., 2021)

Creative Commons Attribution-NonCommercial 4.0 International License.

#### DGS Corpus (Prillwitz et al., 2008)

is under CC BY-NC license.

#### Dicta-Sign (Matthes et al., 2012)

is under CC-BY-NC-ND 4.0 license.

#### WLASL (Li et al., 2020)

Computational Use of Data Agreement (C-UDA-1.0)


Australia, and recognise their continued cultural and spiritual connection to
the land, waterways, cosmos, and community. We pay our deepest respects to all
Elders, ancestors and descendants of the Giabal, Jarowair, and Kambuwal
nations, upon whose lands the Minerva-Australis facility at Mt Kent is
situated.

## References

  * Adams (2010) Adams, E. R. 2010, PhD thesis, Massachusetts Institute of Technology
  * Adams et al. (2010) Adams, E. R., López-Morales, M., Elliot, J. L., Seager, S., & Osip, D. J. 2010, ApJ, 721, 1829, doi: 10.1088/0004-637X/721/2/1829
  * Adams et al. (2011) —. 2011, ApJ, 728, 125, doi: 10.1088/0004-637X/728/2/125
  * Adams et al. (2021) Adams, E. R., Jackson, B., Johnson, S., et al. 2021, , 2, 152, doi: 10.3847/PSJ/ac0ea0
  * Addison et al. (2019) Addison, B., Wright, D. J., Wittenmyer, R. A., et al. 2019, PASP, 131, 115003, doi: 10.1088/1538-3873/ab03aa
  * Addison et al. (2021) Addison, B. C., Wright, D. J., Nicholson, B. A., et al. 2021, MNRAS, 502, 3704, doi: 10.1093/mnras/staa3960
  * Alonso et al. (2004) Alonso, R., Brown, T. M., Torres, G., et al. 2004, ApJ, 613, L153, doi: 10.1086/425256
  * Alonso et al. (2008) Alonso, R., Auvergne, M., Baglin, A., et al. 2008, A&A, 482, L21, doi: 10.1051/0004-6361:200809431
  * Alsubai et al. (2019) Alsubai, K., Tsvetanov, Z. I., Pyrzas, S., et al. 2019, AJ, 157, 224, doi: 10.3847/1538-3881/ab19bc
  * Anderson et al. (2012) Anderson, D. R., Collier Cameron, A., Gillon, M., et al. 2012, MNRAS, 422, 1988, doi: 10.1111/j.1365-2966.2012.20635.x
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Astropy Collaboration et al. (2022) Astropy Collaboration, Price-Whelan, A. M., Lim, P. L., et al. 2022, 935, 167, doi: 10.3847/1538-4357/ac7c74
  * Bailey & Goodman (2019) Bailey, A., & Goodman, J. 2019, MNRAS, 482, 1872, doi: 10.1093/mnras/sty2805
  * Baluev et al. (2019) Baluev, R. V., Sokov, E. N., Jones, H. R. A., et al. 2019, MNRAS, 2246, doi: 10.1093/mnras/stz2620
  * Barker et al. (2024) Barker, A. J., Efroimsky, M., Makarov, V. V., & Veras, D. 2024, MNRAS, 527, 5131, doi: 10.1093/mnras/stad3530
  * Bayliss et al. (2018) Bayliss, D., Gillen, E., Eigmüller, P., et al. 2018, MNRAS, 475, 4467, doi: 10.1093/mnras/stx2778
  * Bouma et al. (2020) Bouma, L. G., Winn, J. N., Howard, A. W., et al. 2020, ApJ, 893, L29, doi: 10.3847/2041-8213/ab8563
  * Bradley et al. (2019) Bradley, L., Sipőcz, B., Robitaille, T., et al. 2019, astropy/photutils: v0.6, Zenodo, doi: 10.5281/zenodo.2533376
  * Chen et al. (2023) Chen, D.-C., Xie, J.-W., Zhou, J.-L., et al. 2023, arXiv e-prints, arXiv:2311.00305, doi: 10.48550/arXiv.2311.00305
  * Chen et al. (2021) Chen, G., Pallé, E., Parviainen, H., et al. 2021, MNRAS, 500, 5420, doi: 10.1093/mnras/staa3555
  * Ciceri et al. (2016) Ciceri, S., Mancini, L., Southworth, J., et al. 2016, MNRAS, 456, 990, doi: 10.1093/mnras/stv2698
  * Collier Cameron & Jardine (2018) Collier Cameron, A., & Jardine, M. 2018, MNRAS, 476, 2542, doi: 10.1093/mnras/sty292
  * Coppejans et al. (2013) Coppejans, R., Gulbis, A. A. S., Kotze, M. M., et al. 2013, PASP, 125, 976, doi: 10.1086/672156
  * Craig et al. (2017) Craig, M., Crawford, S., Seifert, M., et al. 2017, astropy/ccdproc: v1.3.0.post1, Zenodo, doi: 10.5281/zenodo.1069648
  * Davoudi et al. (2021) Davoudi, F., MirshafieKhozani, P., Paki, E., et al. 2021, Astronomy Letters, 47, 638, doi: 10.1134/S1063773721090024
  * Dawson & Johnson (2018) Dawson, R. I., & Johnson, J. A. 2018, ARA&A, 56, 175, doi: 10.1146/annurev-astro-081817-051853
  * De et al. (2023) De, K., MacLeod, M., Karambelkar, V., et al. 2023, Nature, 617, 55, doi: 10.1038/s41586-023-05842-x
  * Delrez et al. (2016) Delrez, L., Santerne, A., Almenara, J. M., et al. 2016, MNRAS, 458, 4025, doi: 10.1093/mnras/stw522
  * Eastman et al. (2010) Eastman, J., Siverd, R., & Gaudi, B. S. 2010, PASP, 122, 935, doi: 10.1086/655938
  * Efroimsky & Makarov (2022) Efroimsky, M., & Makarov, V. V. 2022, Universe, 8, 211, doi: 10.3390/universe8040211
  * Espinoza et al. (2019) Espinoza, N., Rackham, B. V., Jordán, A., et al. 2019, MNRAS, 482, 2065, doi: 10.1093/mnras/sty2691
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. 2013, PASP, 125, 306, doi: 10.1086/670067
  * Fox & Wiegert (2022) Fox, C., & Wiegert, P. 2022, MNRAS, 516, 4684, doi: 10.1093/mnras/stac2575
  * Gillon et al. (2010) Gillon, M., Lanotte, A. A., Barman, T., et al. 2010, A&A, 511, A3, doi: 10.1051/0004-6361/200913507
  * Gillon et al. (2012) Gillon, M., Triaud, A. H. M. J., Fortney, J. J., et al. 2012, A&A, 542, A4, doi: 10.1051/0004-6361/201218817
  * Goldreich & Soter (1966) Goldreich, P., & Soter, S. 1966, Icarus, 5, 375, doi: 10.1016/0019-1035(66)90051-0
  * Guillot & Havel (2011) Guillot, T., & Havel, M. 2011, A&A, 527, A20, doi: 10.1051/0004-6361/201015051
  * Guillot & Showman (2002) Guillot, T., & Showman, A. P. 2002, A&A, 385, 156, doi: 10.1051/0004-6361:20011624
  * Gulbis et al. (2011) Gulbis, A. A. S., Bus, S. J., Elliot, J. L., et al. 2011, PASP, 123, 461, doi: 10.1086/659636
  * Hagey et al. (2022) Hagey, S. R., Edwards, B., & Boley, A. C. 2022, AJ, 164, 220, doi: 10.3847/1538-3881/ac959a
  * Hamer & Schlaufman (2019) Hamer, J. H., & Schlaufman, K. C. 2019, AJ, 158, 190, doi: 10.3847/1538-3881/ab3c56
  * Harre et al. (2023) Harre, J. V., Smith, A. M. S., Barros, S. C. C., et al. 2023, A&A, 669, A124, doi: 10.1051/0004-6361/202244529
  * Hebb et al. (2010) Hebb, L., Collier-Cameron, A., Triaud, A. H. M. J., et al. 2010, ApJ, 708, 224, doi: 10.1088/0004-637X/708/1/224
  * Hoyer et al. (2016) Hoyer, S., López-Morales, M., Rojo, P., Minniti, D., & Adams, E. R. 2016, MNRAS, 455, 1334, doi: 10.1093/mnras/stv2362
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science and Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Husnoo et al. (2012) Husnoo, N., Pont, F., Mazeh, T., et al. 2012, MNRAS, 422, 3151, doi: 10.1111/j.1365-2966.2012.20839.x
  * Ivshina & Winn (2022) Ivshina, E. S., & Winn, J. N. 2022, ApJS, 259, 62, doi: 10.3847/1538-4365/ac545b
  * Jackson et al. (2023) Jackson, B., Adams, E. R., & Morgenthaler, J. P. 2023, AJ, 166, 142, doi: 10.3847/1538-3881/acef00
  * Jackson et al. (2008) Jackson, B., Greenberg, R., & Barnes, R. 2008, ApJ, 678, 1396, doi: 10.1086/529187
  * Johnson et al. (2007) Johnson, J. A., Fischer, D. A., Marcy, G. W., et al. 2007, ApJ, 665, 785, doi: 10.1086/519677
  * Kanodia & Wright (2018) Kanodia, S., & Wright, J. 2018, Research Notes of the American Astronomical Society, 2, 4, doi: 10.3847/2515-5172/aaa4b7
  * Kokori et al. (2022) Kokori, A., Tsiaras, A., Edwards, B., et al. 2022, Experimental Astronomy, 53, 547, doi: 10.1007/s10686-020-09696-3
  * Lang et al. (2010) Lang, D., Hogg, D. W., Mierle, K., Blanton, M., & Roweis, S. 2010, 139, 1782, doi: 10.1088/0004-6256/139/5/178210.48550/arXiv.0910.2233
  * Liddle (2007) Liddle, A. R. 2007, MNRAS, 377, L74, doi: 10.1111/j.1745-3933.2007.00306.x
  * Maciejewski et al. (2021) Maciejewski, G., Fernández, M., Aceituno, F., et al. 2021, A&A, 656, A88, doi: 10.1051/0004-6361/202142424
  * Mannaday et al. (2022) Mannaday, V. K., Thakur, P., Southworth, J., et al. 2022, AJ, 164, 198, doi: 10.3847/1538-3881/ac91c2
  * Matsakos & Königl (2015) Matsakos, T., & Königl, A. 2015, ApJ, 809, L20, doi: 10.1088/2041-8205/809/2/L20
  * Metzger et al. (2012) Metzger, B. D., Giannios, D., & Spiegel, D. S. 2012, MNRAS, 425, 2778, doi: 10.1111/j.1365-2966.2012.21444.x
  * Millholland & Laughlin (2018) Millholland, S., & Laughlin, G. 2018, ApJ, 869, L15, doi: 10.3847/2041-8213/aaedb1
  * Miyazaki & Masuda (2023) Miyazaki, S., & Masuda, K. 2023, AJ, 166, 209, doi: 10.3847/1538-3881/acff71
  * Morello et al. (2020) Morello, G., Claret, A., Martin-Lagarde, M., et al. 2020, The Journal of Open Source Software, 5, 1834, doi: 10.21105/joss.01834
  * Morello et al. (2020) Morello, G., Claret, A., Martin-Lagarde, M., et al. 2020, The Astronomical Journal, 159, 75, doi: 10.3847/1538-3881/ab63dc
  * Morgenthaler (2022) Morgenthaler, J. P. 2022, jpmorgen/BigMultiPipe, 0.2.0, Zenodo, doi: 10.5281/zenodo.7485043
  * Morgenthaler (2023a) —. 2023a, jpmorgen/ccdmultipipe, 0.0.2, Zenodo, doi: 10.5281/zenodo.7507738
  * Morgenthaler (2023b) —. 2023b, Io Input/Output facility (IoIO) control, reduction, and analysis software, v1.1.1, Zenodo, doi: 10.5281/zenodo.7703222
  * Morgenthaler et al. (2019) Morgenthaler, J. P., Rathbun, J. A., Schmidt, C. A., Baumgardner, J., & Schneider, N. M. 2019, ApJL, 871, L23, doi: 10.3847/2041-8213/aafdb7
  * Moyano et al. (2017) Moyano, M., Almeida, L. A., von Essen, C., Jablonski, F., & Pereira, M. G. 2017, MNRAS, 471, 650, doi: 10.1093/mnras/stx1612
  * Murgas et al. (2014) Murgas, F., Pallé, E., Zapatero Osorio, M. R., et al. 2014, A&A, 563, A41, doi: 10.1051/0004-6361/201322374
  * Nielsen et al. (2019) Nielsen, L. D., Bouchy, F., Turner, O., et al. 2019, A&A, 623, A100, doi: 10.1051/0004-6361/201834577
  * Osip et al. (2004) Osip, D. J., Phillips, M. M., Bernstein, R., et al. 2004, in Ground-based Instrumentation for Astronomy, ed. A. F. M. Moorwood & M. Iye, Vol. 5492, International Society for Optics and Photonics (SPIE), 49 – 59, doi: 10.1117/12.552414
  * Öztürk & Erdem (2019) Öztürk, O., & Erdem, A. 2019, MNRAS, 486, 2290, doi: 10.1093/mnras/stz747
  * Patra et al. (2017) Patra, K. C., Winn, J. N., Holman, M. J., et al. 2017, AJ, 154, 4, doi: 10.3847/1538-3881/aa6d75
  * Patra et al. (2020) —. 2020, AJ, 159, 150, doi: 10.3847/1538-3881/ab7374
  * Peluso et al. (2023) Peluso, D. O., Esposito, T. M., Marchis, F., et al. 2023, PASP, 135, 015001, doi: 10.1088/1538-3873/acaa58
  * Penev et al. (2018) Penev, K., Bouma, L. G., Winn, J. N., & Hartman, J. D. 2018, AJ, 155, 165, doi: 10.3847/1538-3881/aaaf71
  * Penev et al. (2012) Penev, K., Jackson, B., Spada, F., & Thom, N. 2012, ApJ, 751, 96, doi: 10.1088/0004-637X/751/2/96
  * Petrucci et al. (2018) Petrucci, R., Jofré, E., Ferrero, L. V., et al. 2018, MNRAS, 473, 5126, doi: 10.1093/mnras/stx2647
  * Petrucci et al. (2020) Petrucci, R., Jofré, E., Gómez Maqueo Chew, Y., et al. 2020, MNRAS, 491, 1243, doi: 10.1093/mnras/stz3034
  * Poddaný et al. (2010) Poddaný, S., Brát, L., & Pejcha, O. 2010, New A, 15, 297, doi: 10.1016/j.newast.2009.09.001
  * Pu & Lai (2019) Pu, B., & Lai, D. 2019, MNRAS, 488, 3568, doi: 10.1093/mnras/stz1817
  * Rauer et al. (2010) Rauer, H., Erikson, A., Kabath, P., et al. 2010, AJ, 139, 53, doi: 10.1088/0004-6256/139/1/53
  * Rosário et al. (2022) Rosário, N. M., Barros, S. C. C., Demangeon, O. D. S., & Santos, N. C. 2022, A&A, 668, A114, doi: 10.1051/0004-6361/202244513
  * Schwarz (1978) Schwarz, G. 1978, Annals of Statistics, 6, 461
  * Sedaghati et al. (2017) Sedaghati, E., Boffin, H. M. J., MacDonald, R. J., et al. 2017, Nature, 549, 238, doi: 10.1038/nature23651
  * Sun et al. (2023) Sun, L., Gu, S., Wang, X., et al. 2023, MNRAS, 520, 1642, doi: 10.1093/mnras/stad204
  * Sun et al. (2018) Sun, Z., Ji, J.-h., & Dong, Y. 2018, Chinese Astron. Astrophys., 42, 101, doi: 10.1016/j.chinastron.2018.01.007
  * Tregloan-Reed et al. (2013) Tregloan-Reed, J., Southworth, J., & Tappert, C. 2013, MNRAS, 428, 3671, doi: 10.1093/mnras/sts306
  * Tsiaras et al. (2016) Tsiaras, A., Waldmann, I. P., Rocchetto, M., et al. 2016, ApJ, 832, 202, doi: 10.3847/0004-637X/832/2/202
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261, doi: 10.1038/s41592-019-0686-2
  * Vissapragada et al. (2022) Vissapragada, S., Chontos, A., Greklek-McKeon, M., et al. 2022, The Astrophysical Journal Letters, 941, L31, doi: 10.3847/2041-8213/aca47e
  * Wang et al. (2023) Wang, W., Zhang, Z., Chen, Z., et al. 2023, arXiv e-prints, arXiv:2310.17225, doi: 10.48550/arXiv.2310.17225
  * Wang et al. (2021) Wang, X.-Y., Wang, Y.-H., Wang, S., et al. 2021, ApJS, 255, 15, doi: 10.3847/1538-4365/ac0835
  * Weinberg et al. (2024) Weinberg, N. N., Davachi, N., Essick, R., et al. 2024, ApJ, 960, 50, doi: 10.3847/1538-4357/ad05c9
  * Weinberg et al. (2017) Weinberg, N. N., Sun, M., Arras, P., & Essick, R. 2017, ApJ, 849, L11, doi: 10.3847/2041-8213/aa9113
  * Wittenmyer et al. (2022) Wittenmyer, R. A., Clark, J. T., Trifonov, T., et al. 2022, AJ, 163, 82, doi: 10.3847/1538-3881/ac3f39
  * Yang & Wei (2022) Yang, F., & Wei, X. 2022, Publications of the Astronomical Society of the Pacific, 134, 024401, doi: 10.1088/1538-3873/ac495a
  * Yee et al. (2020) Yee, S. W., Winn, J. N., Knutson, H. A., et al. 2020, ApJ, 888, L5, doi: 10.3847/2041-8213/ab5c16
  * Yeh et al. (2023) Yeh, L.-C., Jiang, I.-G., & A-thano, N. 2023, arXiv e-prints, arXiv:2310.08953, doi: 10.48550/arXiv.2310.08953
  * Zhou et al. (2019) Zhou, G., Bakos, G. Á., Bayliss, D., et al. 2019, AJ, 157, 31, doi: 10.3847/1538-3881/aaf1bb