Quo Vadis, Motion Generation? From Large Language Models to Large Motion Models
Under review as a conference paper at ICLR 2025

QUO VADIS, MOTION GENERATION? FROM LARGE
LANGUAGE MODELS TO LARGE MOTION MODELS

Anonymous authors
Paper under double-blind review

ABSTRACT

Inspired by recent success of LLMs, the field of human motion understanding has
increasingly shifted towards the development of large motion models. Despite
some progress, current works remain far from achieving truly generalist mod-
els, largely due to the lack of large-scale, high-quality motion data. To address
this, we present MotionBase, the first million-level motion generation benchmark,
offering 15 times the data volume of the previous largest dataset, and featuring
multimodal data with hierarchically detailed text descriptions. By leveraging this
vast dataset, our large motion model demonstrates strong performance across a
broad range of motions, including unseen ones. Through systematic investiga-
tion, we underscore the importance of scaling both data and model size, with
synthetic data and pseudo labels playing a crucial role in mitigating data acqui-
sition costs. Moreover, our research reveals the limitations of existing evalua-
tion metrics, particularly in handling out-of-domain text instructions — an is-
sue that has long been overlooked. In addition, we introduce a 2D lookup-free
approach for motion tokenization, which preserves motion information and ex-
pands codebook capacity, further enhancing the representative ability of large
motion models. The release of MotionBase and the insights gained from this
study are expected to pave the way for the development of more powerful and
versatile motion generation models. Our code and database will be released at
https://anonymous.4open.science/r/MotionBase.

1

INTRODUCTION

Motion generation is an emerging field with diverse applications in video games, filmmaking, and
robotics animation. At the forefront of this area is text-to-motion generation (T2M) (Ahn et al.,
2018; Ahuja & Morency, 2019), which plays a crucial role in translating natural language into human
motions. State-of-the-art T2M models typically rely on a combination of the motion quantization
methods (e.g., VQ (Van Den Oord et al., 2017)), along with a text encoder (e.g., CLIP (Radford
et al., 2021)) and decoder (e.g., GPT-2 (Radford et al., 2019)) to generate motion sequences from
detailed textual instructions. Despite the availability of a few high-quality datasets (Guo et al.,
2022a; Lin et al., 2024) curated in recent years, their limited size restricts current methods to a
narrow range of scenarios, creating performance bottlenecks when addressing diverse or unseen
motions, as illustrated in Figure 1 (RIGHT).

The rapid advancement of large language models (LLMs) (Touvron et al., 2023a) in multimodal
learning has been significantly bolstered by the availability of vast data resources (Zheng et al.,
2024; Xu et al., 2024). In contrast, the volume of motion data remains considerably smaller than that
of visual-text data, as illustrated in Figure 1 (LEFT). This disparity primarily arises from the high
costs associated with motion data collection, which often requires specialized wearable devices and
substantial human labor for annotation. Consequently, developing a state-of-the-art (SoTA) large
motion model based on LLMs presents a significant challenge and remains an unresolved issue.
While some recent efforts (Jiang et al., 2023) have explored this direction, the effectiveness of large
motion models has yet to be fully demonstrated.

In this paper, we aim to address the question: “Can a large motion model be a promising direction
for motion generation?” To tackle this, we have developed a systematic data collection scheme that
led to the creation of MotionBase, the first large-scale dataset containing over one million motion

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: LEFT: Curves showing the effects of scaling up large motion models. MotionBase is the
first large text-to-motion dataset comparable in scale to visual benchmarks like ImageNet. RIGHT:
While existing models perform well on constrained datasets like Motion-X and HumanML3D, they
struggle with out-of-domain concepts on MotionBase, exhibiting limited generalization.

sequences — 15 times larger than the previous largest dataset. This initiative provides a solid foun-
dation for building robust, universally applicable large motion models and offers a comprehensive
testbed for future research.

Building on the solid foundation of MotionBase, we can now conduct a comprehensive investiga-
tion into the effectiveness of large motion models. This research aims to firstly identify key factors
driving their advancement and offer valuable insights for future model design, including: ❶ scal-
ing both data and model size significantly reduces joint prediction errors on critical metrics while
improving generalization to novel motions. ❷ Despite observable domain gaps, synthetic and static
data, as well as pseudo motion labels are becoming increasingly essential and effective, especially
given the high cost of acquiring ground truth motion data. ❸ Existing metrics show limitations when
faced with out-of-domain text instructions. Notably, the widely used metric, FID, fails to accurately
capture the alignment between ground truth and generated motions. Our findings highlight the need
for a more robust and equitable evaluation framework that enhances open-set generalization.

In addition to these factors, we argue that large motion models are further constrained by inad-
equate motion representation. Most approaches rely on transforming motion into discrete tokens
via vector quantization (VQ), which are then processed by autoregressive models to generate mo-
tion sequences. While these methods have produced impressive results, they suffer from two major
drawbacks. ❶ Information loss: The current VQ process inevitably leads to the loss of critical
information. Given a motion clip with D-dimensional features M = {m1, m2, ..., mT }, where
mi ∈ RD, VQ compresses it into a list of 1D embeddings of size ⌊T /α⌋ × d, where α is the tempo-
ral downsampling ratio and d is the codebook dimension. Unlike images, which consist of uniform
RGB pixel values, each motion state mi contains a set of distinct features (e.g., joint position, ve-
locity, foot-ground contact). Using a single 1D embedding to represent such complex motion states
is insufficient. This not only results in the loss of vital information but also limits the model’s ability
to flexibly generate motion at a part-level. ❷ Limited Codebook Size: Existing VQ are limited
by a small codebook, meaning that all possible human motions must be selected from these limited
options. Consequently, these 1D embeddings fail to capture the vast diversity of human motion.

To address this issue, we propose treating a motion clip as a 2D image with a single channel, rep-
resented as M ∈ RT ×D×1. By expanding the dimensionality of the motion clip from 1D to 2D,
we enhance the encoder’s capacity, improving its ability to represent complex motions while retain-
ing more critical information after tokenization. Although increasing the size of the codebook is a
straightforward way to enhance its expressiveness, this approach often leads to “codebook collapse,"
particularly when training samples are scarce. To mitigate this, we introduce a finite scalar quan-
tizing method inspired by Mentzer et al. (2023), which enables learning a large motion vocabulary
without requiring a lookup for corresponding tokens in the codebook for each entry. As a result,
we expand the motion codebook by at least two orders of magnitude, boosting its representational
capacity while maintaining efficiency.

2

607080FID (test on all)100110120The size of the model parametersGPT2-mediumLlama2Llama3.1Llama29013013B0.8B7B8B13B0.8B7B8BMotion-XHumanML3DMotion-base0.5MImageNetMotion-base140150Someone is standing and playing the piano.A man kicks something or someone with his left leg.(a) Motion-X(b) HumanML3DThe person is standing still, looking forward. Upper body: The person's right arm hangs relaxed by their side, while the left arm is bent at the elbow, with the hand placed on their stomach or lower chest area. The shoulders are squared and the torso is upright. Lower body: Both feet are planted firmly on the ground, with legs slightly apart. The person's weight appears to be evenly distributed between both legs.The right arm is not hanging down.The left arm is not bent.(c) MotionbaseThe person is standing still, looking forward. Upper body: The person's right arm hangs relaxed by their side, while the left arm is bent at the elbow, with the hand placed on their stomach or lower chest area. The shoulders are squared and the torso is upright. Lower body: Both feet are planted firmly on the ground, with legs slightly apart. The person's weight appears to be evenly distributed between both legs.The right arm is not hanging down.The left arm is not bent.(c) MotionbaseThe person is gesturing with their right hand. Upper body: The right arm is extended forward with the hand open and fingers pointing outward. The left arm hangs by their side. The torso is slightly twisted to the left. Lower body: The left leg is slightly forward with the foot flat on the ground. The right leg is back, with the heel slightly raised, suggesting a shift in weight to the left leg.The fingers are not pointing outward.(d) MotionbaseThe person is gesturing with their right hand. Upper body: The right arm is extended forward with the hand open and fingers pointing outward. The left arm hangs by their side. The torso is slightly twisted to the left. Lower body: The left leg is slightly forward with the foot flat on the ground. The right leg is back, with the heel slightly raised, suggesting a shift in weight to the left leg.The fingers are not pointing outward.(d) MotionbaseUnder review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

We summarize our main contributions as follows. (1) MotionBase: We introduce MotionBase, the
first large-scale motion generation benchmark containing over one million motions with detailed
textual descriptions, significantly advancing the capability to effectively train motion generation
models. (2) Key Insights: Our research identifies critical factors affecting the effectiveness of large
motion models, emphasizing the importance of scaling both data and model size. Additionally, we
uncover limitations in the current evaluation metrics, particularly when handling diverse and unseen
motions. (3) Novel Motion Quantization: We propose a novel motion quantization approach that
represents motion clips as 2D images and constructs a finite-scale codebook without requiring token
lookups. This method retains essential information and expands the capacity of the motion encoder,
enhancing the ability of large motion models to leverage large-scale motion data.

2 RELATED WORK

2.1 LARGE LANGUAGE MODELS AND MULTI-MODALITY

Substantial advancements have been made in enhancing LLMs (Brown et al., 2020; Raffel et al.,
2020; Chowdhery et al., 2022) with the ability to understand and respond to human instructions,
through a technique known as instruction tuning (Ouyang et al., 2022). Recent research has extended
these capabilities to the multimodal domain (Ye et al., 2023; Zheng et al., 2023), with notable work
by Liu et al. (2023), who pioneered visual instruction tuning to create a highly adaptable visual
assistant. Additionally, Li et al. (2023a) integrated multimodal context directly into instruction
data to further enhance model performance. Subsequent studies (Zhang et al., 2023b; Zhao et al.,
2023) expanded this research by scaling up instructional datasets and incorporating image-rich text.
Notably, Dai et al. (2023) developed InstructBLIP, based on BLIP-2 (Li et al., 2023b), which features
an advanced visual feature extraction mechanism to improve performance across vision-language
tasks. Despite these breakthroughs, the application of multimodal models to human motion remains
less competitive compared to current state-of-the-art (SoTA) methods, although recent initiatives are
beginning to explore this domain (Jiang et al., 2023; Zhang et al., 2024b).

2.2 VECTOR QUANTIZATION

Vector quantization (VQ) has been highly successful in generating high-quality images (Van
Den Oord et al., 2017) and videos (Gupta et al., 2022; Yan et al., 2021). VQ-VAE first converts
images into discrete representations and autoregressively models their distribution. Building on
this, Lee et al. (2022) introduced residual quantization (RQ), which encodes images into a stacked
map of discrete codes, efficiently reducing the spatial resolution of features. You et al. (2022) further
developed hierarchical vector quantization (HQ), employing a pyramid scheme with two-level codes
for image encoding. Most existing motion generation approaches have adopted VQ or its variants to
quantize human motions. However, the small codebook size in traditional VQ methods limits their
ability to generalize and accurately represent the diversity of human motions. Although increas-
ing the codebook size can improve representational capacity, it often leads to codebook collapse.
Recently, Mentzer et al. (2023) demonstrated that discrete codes can be obtained via scalar quanti-
zation, where each scalar entry is independently quantized to the nearest integer through rounding.
Similarly, Yu et al. (2023) introduced a lookup-free codebook that maps videos into compact discrete
tokens, utilizing all codes without auxiliary losses and expanding the codebook size.

2.3 HUMAN MOTION GENERATION

The task of motion generation involves creating human motion based on various inputs, such as text
descriptions (Guo et al., 2022b; Petrovich et al., 2022), action labels (Cervantes et al., 2022; Guo
et al., 2020) or motion prefixes (Liu et al., 2022; Mao et al., 2019). Among these, text-to-motion
(T2M) generation has received the most attention due to the ease and flexibility of using natural
language as input. Early approaches (Fragkiadaki et al., 2015; Ghosh et al., 2017; Gopalakrishnan
et al., 2019) rely on deterministic motion modeling, which often produce averaged, blurry results.
To overcome this, researchers introduce stochastic methods using models like GANs (Cai et al.,
2018; Wang et al., 2020) or VAEs (Aliakbarian et al., 2020). For instance, T2M-GPT (Zhang et al.,
2023a) extends the temporal VAE to capture the probabilistic relationship between text and mo-
tion. Recently, Guo et al. (2024) proposed integrating residual quantization and masked modeling

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Table 1: Comparison with existing human motion datasets. More details can be found in our ap-
pendix. In the table, B, H, and F refer to body, hand, and face, respectively. “part” indicates that the
text captions include fine-grained descriptions of body parts, while “body” means the descriptions
are not as detailed. “multi” and “single” specify whether the dataset contains multi-person scenarios
or only single-person data. Our MotionBase is the largest motion generation dataset and benchmark
to date, featuring at least 15× more data than previous datasets, along with additional modalities.

SEQ NUMBER MOTION TEXT RGB DEPTH BBOX PERSON

KIT (Plappert et al., 2016)
HumanML3D (Guo et al., 2022a)
MotionX (Lin et al., 2024)

MotionBase-V1

5.7K
29.2K
81.1K

>1M

B
B
B,H,F

B,H

body
body
body

part

(cid:37)
(cid:37)
(cid:33)

(cid:33)

(cid:37)
(cid:37)
(cid:37)

(cid:33)

(cid:37)
(cid:37)
(cid:37)

(cid:33)

single
single
single

multi

to improve traditional vector quantization (VQ). Lu et al. (2023) designed a hierarchical VQVAE
to separately encode body and hand motions. To better align with a motion auto-encoder, Motion-
CLIP (Tevet et al., 2022) incorporates CLIP (Radford et al., 2021) as the text encoder, bringing
in more robust text priors. Additionally, Zhang et al. (2024b) and Jiang et al. (2023) explored the
development of unified models based on LLMs which accept multimodal conditions (e.g., vision,
text, and pose), enabling the generation of subsequent, preceding, or “in-between” motions. De-
spite leveraging the power of LLMs, these large motion models remain limited to in-domain text
instructions and do not yet perform as competitively as existing SoTA methods.

In this work, we aim to bridge the gap between large language models and generalized, reliable
large motion models. To achieve this, We begin by introducing MotionBase — a novel, large-scale
dataset designed to support extensive pretraining and comprehensive fair evaluation.

3 MOTIONBASE DATASET

Data is the foundation of large motion models. With advancements in fields like human pose detec-
tion, we are now able to extract high-quality motion sequences from vast amounts of online videos,
including datasets like InternViD (Wang et al., 2023) and WebVid (Bain et al., 2021). In its initial
public release, our MotionBase contains over one million motion clips, each annotated with fine-
grained automatic pseudo labels. A comparison with existing benchmarks is presented in Table 1.
Our data collection pipeline involves the following key steps in order.
❶ Source Video Collection and Cleaning: We begin by collecting over 20 million videos from
publicly available datasets and online platforms such as YouTube. To ensure quality and relevance,
we filter out videos that do not contain human figures.
❷ 2D-3D Keypoint Estimation: Keypoints are essential for capturing the skeletal structure of
human motion.
Initially, we estimate whole-body 2D keypoints with confidence scores using a
pretrained model (Xu et al., 2022). To further enhance motion accuracy, we estimate precise 3D
keypoints with another pretrained model (Sárándi et al., 2023) trained on large 3D datasets, Fol-
lowing the method of Lin et al. (2024), we apply temporal smoothing and enforce 3D bone length
constraints during triangulation, improving the stability and consistency of the keypoint estimations.
❸ Incorporating Additional Modalities: A comprehensive understanding of human motion ben-
efits from the inclusion of diverse modalities such as RGB and depth data. To enrich MotionBase,
we provide annotations for these additional modalities. Furthermore, MotionBase includes videos
featuring multi-person scenarios, with each motion sequence grounded in its corresponding video
through object-level bounding boxes. Although this paper primarily focuses on the text-to-motion
task, these additional modalities open avenues for future research in other areas.
❹ Local-Global Pose Estimation: We begin by registering the body model SMPL-X (Pavlakos
et al., 2019) for each frame in MotionBase, which leverages keypoints based on progressive learning-
based mesh fitting method (Lin et al., 2024). Specifically, we predict SMPL-X parameters using a
pretrained body mesh recovery method, OSX (Lin et al., 2023), followed by iterative optimization
to fit the parameters to the target 2D and 3D joint positions. After fitting, we apply global motion
optimization based on Yuan et al. (2022) to refine both global motions and camera poses simulta-

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 2: Examples from MotionBase, which encompasses a diverse range of human motions,
including both long-term clips and static snapshots. It features various scenes, ranging from outdoor
environments to indoor settings, and includes both clean, single-person scenarios as well as crowded,
multi-person scenes. Additionally, MotionBase comprises a mix of real-world data and synthetic
data generated by game engines. For more details about MotionBase, please refer to Appendix A.

neously, ensuring alignment with the video evidence. Finally, for motions with noisy or occluded
input data, we reconstruct complete and plausible motions using RoHM (Zhang et al., 2024a).
❺ Hierarchical Motion Descriptions: Existing benchmarks face inherent limitations in their text
descriptions. Previous studies (Guo et al., 2022a) typically use a single sentence to describe whole-
body motions, neglecting finer details of individual body parts, such as the arms or legs. This
approach restricts the model’s ability to perform more nuanced body comprehension and flexible
part-level motion control (e.g., raising only the left arm). Moreover, the richness of text labels often
varies across different motions; for example, a large portion of the Motion-X dataset provides only
action labels. In contrast, MotionBase offers hierarchical textual annotations for each video inspired
by Pi et al. (2023). We carefully design a prompt format and use Gemini-1.5-pro (Reid et al., 2024)
to generate detailed descriptions for individual body parts (e.g., left arm, right leg), assigning a
dedicated sentence to each. Additionally, we summarize the overall body movement in a paragraph
containing 1–3 sentences, providing a more comprehensive description of the motion.

4 SCALING UP LARGE MOTION MODEL

4.1 OVERALL ARCHITECTURE

Similar to previous LLM-based multimodal models, we treat motion as a foreign language. The
overall framework is presented in Figure 11 in Appendix B. Our large motion model, built on a
pre-trained LLM, functions as a generative model that connects a motion tokenizer with the LLM
backbone Θ. The motion tokenizer encodes raw motion clip features M into token embeddings
V = {v1, v2, ..., vn} ∈ Rn×d, where n denotes the number of motion tokens and d represents the
dimensionality of each token. To integrate motion tokens into the LLM framework, we incorporate
K discrete codes in the motion codebook as additional vocabulary for the LLM. Additionally, we
introduce two special tokens, <mot> and </mot>, to signify the start and end of motion sequences
within the input/output streams. The LLM backbone Θ is built on a decoder-only architecture using

5

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

causal transformers. The model generates outputs Y = {y1, y2, ..., ym} in an auto-regressive man-
ner, where Y corresponds to the generated motion sequence based on the provided motion-text input
tokens. In this work, each motion-text pair in the MotionBase dataset is framed as an instruction-
following instance {XQ, XM }, representing a question-answer interaction between the user and the
motion model. The entire instructional dataset adheres to this unified format. To train our model,
we optimize the negative log-likelihood over the predicted tokens which is defined as follows:

L(Θ) = −

L
(cid:88)

j=1

log PΘ(yj|desc, ˆy1:j−1),

(1)

where ˆy and y denote the input and target token sequences, respectively. Θ represents the model
parameters, and L is the length of the target sequence. The input description, desc, can be empty
depending on the instruction provided.

4.2

2D LOOKUP-FREE MOTION QUANTIZATION

Similar to visual tokenization, motion tokenization is a process that compresses motion signals into
a series of discrete tokens, typically involving an encoder E, a decoder D and a codebook C. We
propose a 2D lookup-free quantization method as a key component for building large motion models.

2D Motion Quantization. Traditional motion quantizers use 1D embeddings to represent motion
at each timestamp, which inevitably results in the loss of crucial information. Furthermore, this
approach limits the quantizer’s ability to generate and interpret part-level motions. To address these
limitations, we treat the motion sequence M = {m1, m2, ..., mT } as a single-channel image, rep-
resenting each motin sequence as M ∈ RT ×D×1. Each motion embedding mi is divided into P
components, capturing distinct features of motion, such as root orientation, joint rotation and foot
contact. Our motion encoder then converts M into a feature map E(M) ∈ R⌊T /α⌋×P ×d, where α
denotes the temporal downsampling ratio. This approach ensures that each body part is tokenized
separately, allowing for more granular, part-level motion encoding and decoding.

Lookup-Free Quantization. Traditional motion quantizers are often constrained by small code-
book sizes, restricting their ability to capture the full diversity of human motion. A common ap-
proach is to expand the motion vocabulary. However, excessively enlarging the codebook can result
in “codebook collapse”, where only a small subset of tokens in the codebook is used, offering min-
imal performance improvements. In some cases, an overly large vocabulary can even degrade the
model’s overall performance. To address this, a more effective way is to reduce the dimensionality
of code embeddings (Mentzer et al., 2023), which limits the representational capacity of individual
tokens and encourages more efficient learning across a larger vocabulary. Similar to Yu et al. (2023),
we reduce the embedding dimension of the codebook to zero by replacing the codebook C ∈ RK×d
with an integer set C with |C| = K. Specifically, C is the Cartesian product of single-dimensional
variables C =×d
i=1Ci, where Ci = {−1, 1} and d is equal to log2 K. Given a feature vector
z ∈ Rd, our quantizer Q(·) converts each dimension of the quantized representation into:

Q(zi) = arg mincik

||zi − cik|| = −1{zi ≤ 0} + 1{zi > 0},

(2)

The token index is computed as Index(z) =
where cij denotes the j-th value of Ci.
(cid:80)d
i=1 2i−11{zi > 0}. To train the tokenizer, we employ a standard combination of reconstruc-
tion, perceptual, and commitment losses, along with an entropy penalty to promote better codebook
utilization (Yu et al., 2023). Importantly, we exclude the use of GAN loss, as it was found to nega-
tively impact training stability.

5 EXPERIMENTS

5.1 EXPERIMENTAL SETUP

Datasets. Our investigation first is conducted on the following text-to-motion datasets: Hu-
manML3D (Guo et al., 2022a) and Motion-X (Lin et al., 2024). HumanML3D comprises 14,616
motion clips sourced from the AMASS dataset (Mahmood et al., 2019), paired with 44,970 textual
descriptions. Motion-X, a more recent dataset, includes approximately 81,000 motion clips. To

6

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

validate our conclusions on larger-scale data, we also carry out experiments on the proposed Mo-
tionBase dataset with two variants: MotionBase-0.5 and MotionBase-1.0. MotionBase-0.5 contains
500,000 clips, while MotionBase-1.0 encompasses the full scope of our collected data, with over 1
million clips. Following standard practice, each dataset is split into training, validation, and test sets
in proportions of 85%, 5%, and 15%, respectively.

Evaluation Metrics.
For the motion generation task, we employ the following metrics in our
experiments following Guo et al. (2022a). (1) Frechet Inception Distance (FID): This metric assesses
overall motion quality by measuring the distributional difference between the high-level features of
generated motions and real motions. (2) Motion-retrieval Precision (R-Precision) and Multimodal
Distance (MMDist): These metrics evaluate the semantic alignment between the textual input and
generated motions. R-Precision measures the top-1/2/3 retrieval accuracy, while MMDist computes
the distance between matched text and motion pairs. Additionally, we validate our motion tokenizer
by conducting experiments on the motion reconstruction task. This is measured using both Mean
Per Joint Position Error (MPJPE) and FID. MPJPE quantifies the average distance (in millimeters)
between the predicted joint positions and the ground truth positions across all joints in the skeleton.
Implementation Details. For the motion tokenizer, we implement a VQ codebook C ∈ R1024×512
with an embedding dimensionality of d = 512, and the resulting discrete codes are incorporated as
additional vocabulary for the LLM. In comparison, our lookup-free codebook has a size of 216 =
16384, where the least frequently used tokens from the LLM’s codebook are mapped to represent
motion codes. The motion encoder E operates with a temporal downsampling rate of α = 4. We
experiment with four LLM architectures to build our large motion model: GPT2-medium (Radford
et al., 2019), Llama-2-7b, Llama-2-13b (Touvron et al., 2023b), and Llama3.1-8b (Dubey et al.,
2024). The motion tokenizer is trained with a learning rate of 1e-4 and a batch size of 256 over
300K iterations. For training the large motion model, full parameter tuning is performed on 8×A800
GPUs, with a batch size of 1024, over 300 epochs. The learning rate is set to 2e-4 for GPT2-medium
and 2e-5 for the Llama models. Further details are provided in the appendix due to space limitation.

Table 2: Comparisons under different model and data sizes. All experiments are conducted using
the same pretrained VQ model for consistency. Additionally, we re-train the motion autoencoder
and text encoder (Guo et al., 2022a) separately on the Motion-X and MotionBase datasets, using
their respective data to train the motion autoencoder for each dataset’s evaluation.

Motion-X

MotionBase

Decoder

#Inst.

#Param.

R@1 ↑

R@3 ↑

Real

GPT-2
GPT-2
GPT-2
GPT-2

LLaMA-2
LLaMA-2
LLaMA-2
LLaMA-2

LLaMA-3
LLaMA-3
LLaMA-3
LLaMA-3

LLaMA-2
LLaMA-2
LLaMA-2
LLaMA-2

-

0.02M
0.08M
0.5M
1M

0.02M
0.08M
0.5M
1.0M

0.02M
0.08M
0.5M
1M

0.02M
0.08M
0.5M
1.0M

-

355M
355M
355M
355M

7B
7B
7B
7B

8B
8B
8B
8B

13B
13B
13B
13B

0.496

0.206
0.468
0.358
0.357

0.207
0.471
0.372
0.351

0.217
0.483
0.363
0.354

0.225
0.486
0.375
0.359

0.821

0.402
0.791
0.618
0.614

0.405
0.794
0.627
0.602

0.418
0.802
0.625
0.611

0.436
0.805
0.636
0.612

FID ↓

0.038

54.017
0.096
4.852
5.083

53.354
0.159
4.908
5.582

54.004
0.103
4.798
5.100

53.447
0.132
4.792
5.370

R@1 ↑

R@3 ↑

0.290

0.037
0.055
0.252
0.264

0.041
0.074
0.256
0.263

0.039
0.071
0.256
0.266

0.040
0.074
0.259
0.298

0.563

0.109
0.155
0.533
0.542

0.109
0.185
0.522
0.536

0.102
0.183
0.533
0.557

0.107
0.186
0.520
0.599

FID ↓

0.011

125.824
124.230
0.636
0.516

113.189
127.664
1.084
0.545

117.561
125.310
0.512
0.394

117.594
126.999
0.511
0.595

5.2 DISCUSSION OF SCALING UP MOTION GENERATION

In this section, we investigate the impact of model size and data scale on motion generation perfor-
mance. We utilize the motion autoencoder (Guo et al., 2022a) retrained on Motion-X and Motion-
Base datasets to evaluate performance on their respective test sets. We categorize our training data

7

Under review as a conference paper at ICLR 2025

into four scales: 0.02M (HumanML3D only), 0.08M (Motion-X only), 0.5M (MotionBase-0.5), and
1M (MotionBase-1.0). To ensure fair comparison, we employ the same VQ as the motion tokenizer,
maintaining consistency across experiments to validate our conclusions.

Does increasing model size benefit motion generation? Yes. As shown in Table 2, our results
demonstrate that increasing model size leads to significant performance improvements when pro-
vided with the same amount of training data. Specifically, Llama2-13b outperforms Llama2-7b,
which in turn surpasses GPT2-medium, illustrating a clear trend of performance gains as model ca-
pacity increases. This suggests that models with larger size are better equipped to capture diverse,
complex patterns and relationships within human motions.

Does increasing data scale benefit motion generation? Yes. In Table 2, when using the same foun-
dation model, increasing the scale of training data leads to substantial improvement on MotionBase
test set, aligning with our expected scaling laws. This improvement is particularly pronounced in
the R-precision metric, emphasizing the critical role of data scale in enhancing semantic alignment
between generated motions and text prompts. However, contrary to our expectations, we observe
a noticeable performance decline on Motion-X test set if not trained on Motion-X (0.08M). We
attribute this to the limitations of the retrieval-based evaluation model, as discussed in Section 5.4.

-

Real

0.511

Decoder

0.797 0.002

MLD
MotionDiffuse

R@1 ↑ R@3 ↑ FID ↓ MMDist ↓

Table 3: Comparison with existing SoTA methods on the
HumanML3D benchmark. Results marked with ∗ repre-
sent values reproduced using the officially released code,
while unmarked results are taken from the original papers.

Does the large motion model perform
SoTA competitively? We evaluate
our large motion model on the widely
adopted HumanML3D benchmark. We
compare its performance against a va-
riety of SoTA approaches. This in-
cludes diffusion-based methods such as
MLD (Chen et al., 2023) and Motion-
Diffuse (Zhang et al., 2022), as well
as the GPT-based T2M-GPT (Zhang
et al., 2023a). We also compare against
LLM fine-tuning methods like Mo-
tionGPT (Jiang et al., 2023; Zhang
et al., 2024b), MotionLLM (Wu et al.,
2024), and AvatarGPT (Zhou et al.,
2024). As shown in Table 3, our model,
which utilizes Llama-2-13B as the de-
coder and calculates the loss over the
entire concatenated sequence of input
text, achieves SOTA performance. Our large motion model significantly outperforms other LLM-
based methods such as MotionGPT and AvatarGPT, as well as the earlier T2M-GPT. In particular,
we observe substantial improvements in key metrics such as R@1, R@3, and MMDist, highlighting
our model’s ability to generate motion sequences that are better aligned with text descriptions and
of higher quality.

0.492
T2M-GPT
MotionGPT1,∗
0.409
MotionGPT1
0.492
MotionGPT2,∗ Llama-2-13B 0.367
MotionGPT2,∗ Llama-1-13B 0.363
MotionGPT2
Llama-1-13B 0.411
Gemma-2b
0.482
MotionLLM
Llama-1-13B 0.389
AvatarGPT

0.775 0.141
0.667 0.162
0.778 0.232
0.654 0.571
0.633 0.592
0.696 0.542
0.770 0.491
0.623 0.567

3.121
3.992
3.096
3.981
4.029
3.584
3.138
-

0.772 0.473
0.782 0.630

GPT-2
T5
T5

Llama-2-13B 0.519

0.803 0.166

0.481
0.491

3.196
3.113

2.974

2.964

Ours

-
-

Slow convergence of large motion models. To
evaluate the convergence speed of large motion mod-
els, we train GPT-2, Llama2-7b, and Llama3.1-8b
for 300 epochs on Motion-X. The training curve of
with R@1 performance is illustrated in Figure 3.
We obverse that all large motion models nearly con-
verge by 200 epochs, with larger models converg-
ing faster. Initializing these models with pre-trained
weights proves beneficial for speeding up conver-
gence. Compared to large multimodal models like
LLaVA (Liu et al., 2023), large motion models re-
quire more epochs to capture the complex represen-
tations of motion sequences. We attribute the slow
convergence of these models to the limited represen-
tation capacity of the motion tokenizer, which con-
tains only 512 motion tokens. This suggests the need
to optimize the motion tokenizer and expand its rep-

8

Figure 3: Training curves with Y-axis denot-
ing R@1 retrieval accuracy. All these mod-
els are trained for 300 epochs at most and are
evaluated every 1000 steps.

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Under review as a conference paper at ICLR 2025

resentation space. To address this, we explore 2D-LFQ quantization method as a promising alterna-
tive.

Does Static and Synthetic Data help? Yes, the ad-
dition of static image data and synthesized data both
contribute to improvements, as illustrated in Table 4,
more analysis can be found in Appendix C.1.

Table 4: Ablation of the effectiveness of syn-
thetic and static data, which takes about 28%
and 44% of all data, respectively.

Real

0.290

0.011

0.563

FID ↓

TRAIN SET

R@1 ↑ R@3 ↑

0.111
0.120
0.264

w/o static & syn
w/o static
MotionBase

Do large motion models outperform in out-of-
distribution setup? Yes. We present the results
in Table 5. This ablation is essential for further val-
idating the generalization capabilities of large mo-
tion models, as the improvements observed in Ta-
ble 2 may stem from the inclusion of additional in-
domain data from Motion-X. In this setup, we select
four subsets from MotionBase, comprising 90K samples (UNSEEN-90K), for evaluation, while the
remaining 38 subsets are used for training. This ensures that the test set consists entirely of out-
of-domain (OOD) samples. We compare the performance of models trained on HumanML3D, Mo-
tionX, and Motion-#38, all utilizing the GPT2-medium architecture, where #N denotes the number
of training subsets. All models are trained using the GPT2-medium. The results on the OOD test set
clearly demonstrate that the model trained on MotionBase significantly outperforms those trained
on HumanML3D and MotionX, particularly in terms of R@1 and R@3 metrics. These findings
strongly highlight the superior generalization ability of large motion models when handling unseen
OOD data, especially when trained on diverse, large-scale datasets. However, we once again observe
unexpected results with the FID metric, which will be discussed further in Section 5.4.

57.719
55.983
0.516

0.248
0.252
0.542

Figure 4: Comparison with different motion quantization on Motion-X (left) and MotionBase
(right). Note that we only show MPJPE (↓) results here. FID results is shown in Appendix C.9.

5.3 DISCUSSION OF MOTION QUANTIZATION

Table 5: Ablation of out-of-domain evaluation on
UNSEEN-90K dataset, where #N denotes we use
N subsets of MotionBase for training.

In this section, we investigate the impact of
different motion quantization methods. We
compare our proposed 2D lookup-free quan-
tization (2D-LFQ) against
two commonly
used approaches: residual vector quantization
(RVQ) and vector quantization (VQ), across
various codebook sizes ranging from 28 to
216. The number of parameters for RVQ/VQ
and 2D-LFQ are 19.43M and 108.35M, re-
spectively. As shown in Figure 4, 2D-LFQ
demonstrates significant improvements over
both RVQ and VQ. Notably, as the codebook
size increases, 2D-LFQ continues to enhance performance, while RVQ and VQ experience dimin-
ishing returns or performance degradation with larger codebooks. Our deeper analysis attributes
these gains to better codebook utilization by 2D-LFQ. Figure 5 illustrates that the utilization rates

HumanML3D
MotionX
MotionBase-#38

204.833
178.368
10.613

0.032
0.042
0.136

0.101
0.119
0.321

TRAIN SET

R@1 ↑

R@3 ↑

FID ↓

0.349

0.147

0.005

Real

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

MPJPEMPJPEUnder review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

for VQ and RVQ begin to decline once the codebook size exceeds 210, which corresponds to the
peak performance for these methods, whereas the utilization of 2D-LFQ continues to increase with
larger codebooks. Additionally, we conduct further experiments to validate the benefits of 2D mo-
tion encoding in Appendix C.9.

5.4 LIMITATION OF AUTOMATED METRIC

As mentioned earlier, the FID scores in Table 2
and Table 5 yield unexpected results. Specifically,
when evaluating on Motion-X and UNSEEN-90K,
FID achieves its best performance when trained
on Motion-X, significantly outperforming both the
smaller HumanML3D and the larger-scale Motion-
Base.
In this section, we aim to investigate this
anomaly. FID, a standard metric widely used for
generation tasks, is typically measured by a pre-
In traditional image generation,
trained evaluator.
FID is calculated using a well-trained, robust visual
encoder like InceptionNet (Szegedy et al., 2015),
which is trained on millions of images. However, the
evaluator currently used to compute FID for motion
generation is a simple motion autoencoder with a
very small parameter scale (Guo et al., 2022a). Since
this motion autoencoder is trained on limited data consisting of only 20K motions, we argue that it
may lack the generalization needed for robust performance, leading to difficulties in reliably cap-
turing the complex semantic alignment between text and motion.Similar unexpected results occur
in motion reconstruction as well. As show in Table 6, the FID score on HumanML3D is two or-
ders of magnitude higher when comparing 2D-LFQ and VQ-VAE, despite the former achieving a
much lower MPJPE. When tested on MotionBase, 2D-LFQ obtains the highest FID score even while
achieving the best MPJPE. We observe the same issue with other metrics like MMDist, as discussed
in Appendix C.1. Notably, Voas et al. (2023) have mentioned that existing metrics are sensitive to
the quality of the embedding space and do not always align with human perception. These findings
highlight the need for a more robust and fair metric for large motion models moving forward.

Figure 5: Comparison of codebook utiliza-
tion for different motion quantization.

Table 6: Robustness investigation of the evaluation metrics on the motion reconstruction task.

HumanML3D

Motion-X

MotionBase

Tokenizer

#Num.

#Param.

FID ↓ MPJPE ↓

FID MPJPE

FID MPJPE

VQ-VAE
RQ-VAE
2D-LFQ

512
512
16384

19.43M 0.078
19.43M 0.05
108.35M 1.769

69.2
37.5
45.6

0.852
0.568
0.295

106.4
56.9
54.1

4.366
4.026
7.853

123.6
78.2
64.1

6 CONCLUSION

In this paper, we explore how to advance the field of large-scale motion generation. To this end, we
introduce a large-scale motion dataset named MotionBase, which includes detailed text descriptions
and rich modality annotations, providing a strong foundation for effectively training large motion
models. Our research highlights key findings, such as the impact of scaling both data and model size.
Additionally, we identify potential limitations in the current evaluation metrics, particularly when
assessing diverse and unseen motions. To enhances the benefits large motion models can derive from
extensive motion data, we propose a novel motion quantization approach that treats motion clips as
2D images and constructs a finite-scale codebook, eliminating the need for token lookups. We hope
that this research offers valuable direction for future work in large-scale motion generation.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Hyemin Ahn, Timothy Ha, Yunho Choi, Hwiyeon Yoo, and Songhwai Oh. Text2action: Generative
adversarial synthesis from language to action. In 2018 IEEE International Conference on Robotics
and Automation (ICRA), pp. 5915–5920. IEEE, 2018.

Chaitanya Ahuja and Louis-Philippe Morency. Language2pose: Natural language grounded pose
forecasting. In 2019 International Conference on 3D Vision (3DV), pp. 719–728. IEEE, 2019.

Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould.
A stochastic conditioning scheme for diverse human motion prediction. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5223–5232, 2020.

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and
image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference
on computer vision, pp. 1728–1738, 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.

Haoye Cai, Chunyan Bai, Yu-Wing Tai, and Chi-Keung Tang. Deep video generation, prediction and
completion of human action sequences. In Proceedings of the European conference on computer
vision (ECCV), pp. 366–382, 2018.

Pablo Cervantes, Yusuke Sekikawa, Ikuro Sato, and Koichi Shinoda. Implicit neural representations
for variable length human motion generation. In European Conference on Computer Vision, pp.
356–372. Springer, 2022.

Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your
commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 18000–18010, 2023.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam
Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm:
Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.

Jihoon Chung, Cheng-hsin Wuu, Hsuan-ru Yang, Yu-Wing Tai, and Chi-Keung Tang. Haa500:
Human-centric atomic action dataset with curated videos. In Proceedings of the IEEE/CVF inter-
national conference on computer vision, pp. 13465–13474, 2021.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang,
Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language
models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha
Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.

Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models
for human dynamics. In Proceedings of the IEEE international conference on computer vision,
pp. 4346–4354, 2015.

Partha Ghosh, Jie Song, Emre Aksan, and Otmar Hilliges. Learning human motion models for long-
In 2017 International Conference on 3D Vision (3DV), pp. 458–466. IEEE,

term predictions.
2017.

Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander G Ororbia. A neural
temporal model for human motion prediction. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 12116–12125, 2019.

Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and
Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the
28th ACM International Conference on Multimedia, pp. 2021–2029, 2020.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating
diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 5152–5161, 2022a.

Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for
the reciprocal generation of 3d human motions and texts. In European Conference on Computer
Vision, pp. 580–597. Springer, 2022b.

Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Gener-
ative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 1900–1910, 2024.

Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martín, and Li Fei-Fei.
Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206.11894,
2022.

Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a
foreign language. Advances in Neural Information Processing Systems, 36:20067–20079, 2023.

Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image
generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 11523–11532, 2022.

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan
arXiv preprint

Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning.
arXiv:2306.05425, 2023a.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-
arXiv preprint

image pre-training with frozen image encoders and large language models.
arXiv:2301.12597, 2023b.

Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li. One-stage 3d whole-body mesh
In Proceedings of the IEEE/CVF Conference on

recovery with component aware transformer.
Computer Vision and Pattern Recognition, pp. 21159–21168, 2023.

Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang.
Motion-x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural
Information Processing Systems, 36, 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv

preprint arXiv:2304.08485, 2023.

Zhenguang Liu, Shuang Wu, Shuyuan Jin, Shouling Ji, Qi Liu, Shijian Lu, and Li Cheng.

In-
vestigating pose representations and motion contexts modeling for 3d motion prediction. IEEE
transactions on pattern analysis and machine intelligence, 45(1):681–697, 2022.

Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung
Shum. Humantomato: Text-aligned whole-body motion generation. In Forty-first International
Conference on Machine Learning, 2023.

Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-
time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pp. 10895–10904, 2023.

Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black.
Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF interna-
tional conference on computer vision, pp. 5442–5451, 2019.

Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. Learning trajectory dependen-
cies for human motion prediction. In Proceedings of the IEEE/CVF international conference on
computer vision, pp. 9489–9497, 2019.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu,
and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn
supervision. In 2017 international conference on 3D vision (3DV), pp. 506–516. IEEE, 2017.

Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza-

tion: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023.

OpenAI. GPT-4o mini: advancing cost-efficient intelligence. https://openai.com/index/

gpt-4o-mini-advancing-cost-efficient-intelligence/, 2024.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol-
low instructions with human feedback. Advances in neural information processing systems, 35:
27730–27744, 2022.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios
Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single
image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pp. 10975–10985, 2019.

Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from
textual descriptions. In European Conference on Computer Vision, pp. 480–497. Springer, 2022.

Huaijin Pi, Sida Peng, Minghui Yang, Xiaowei Zhou, and Hujun Bao. Hierarchical generation of
human-object interactions with diffusion probabilistic models. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pp. 15061–15073, 2023.

Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big

data, 4(4):236–252, 2016.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language

models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual
models from natural language supervision. In International conference on machine learning, pp.
8748–8763. PMLR, 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text
transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-
baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem-
ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint
arXiv:2403.05530, 2024.

István Sárándi, Alexander Hermans, and Bastian Leibe. Learning 3d human pose estimation from
dozens of datasets using a geometry-aware autoencoder to bridge between skeleton formats. In
Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2956–
2966, 2023.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du-
mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In
Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9, 2015.

Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-
body human grasping of objects. In Computer Vision–ECCV 2020: 16th European Conference,
Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pp. 581–600. Springer, 2020.

Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Ex-
posing human motion generation to clip space. In European Conference on Computer Vision, pp.
358–374. Springer, 2022.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and
efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko-
lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda-
tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in

neural information processing systems, 30, 2017.

Jordan Voas, Yili Wang, Qixing Huang, and Raymond Mooney. What is the best automated metric
for text to motion generation? In SIGGRAPH Asia 2023 Conference Papers, pp. 1–11, 2023.

Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan
Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understand-
ing and generation. arXiv preprint arXiv:2307.06942, 2023.

Zhenyi Wang, Ping Yu, Yang Zhao, Ruiyi Zhang, Yufan Zhou, Junsong Yuan, and Changyou Chen.
In
Learning diverse stochastic human-action generators by learning smooth latent transitions.
Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 12281–12288, 2020.

Qi Wu, Yubo Zhao, Yifan Wang, Yu-Wing Tai, and Chi-Keung Tang. Motionllm: Multimodal
motion-language learning with large language models. arXiv preprint arXiv:2405.17013, 2024.

Boshen Xu, Ziheng Wang, Yang Du, Sipeng Zheng, Zhinan Song, and Qin Jin. Egonce++: Do
arXiv preprint

egocentric video-language models really understand hand-object interactions?
arXiv:2405.17719, 2024.

Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer base-
lines for human pose estimation. Advances in Neural Information Processing Systems, 35:38571–
38584, 2022.

Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using

vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen
Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models
with multimodality. arXiv preprint arXiv:2304.14178, 2023.

Tackgeun You, Saehoon Kim, Chiheon Kim, Doyup Lee, and Bohyung Han. Locally hierarchi-
cal auto-regressive modeling for image generation. Advances in Neural Information Processing
Systems, 35:16360–16372, 2022.

Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong
Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–
tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.

Ye Yuan, Umar Iqbal, Pavlo Molchanov, Kris Kitani, and Jan Kautz. Glamr: Global occlusion-
aware human mesh recovery with dynamic cameras. In Proceedings of the IEEE/CVF conference
on computer vision and pattern recognition, pp. 11038–11049, 2022.

Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu,
Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete
In Proceedings of the IEEE/CVF conference on computer vision and pattern
representations.
recognition, pp. 14730–14740, 2023a.

Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei
Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint
arXiv:2208.15001, 2022.

Siwei Zhang, Bharat Lal Bhatnagar, Yuanlu Xu, Alexander Winkler, Petr Kadlecek, Siyu Tang, and
Federica Bogo. Rohm: Robust human motion reconstruction via diffusion. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14606–14617, 2024a.

14

Under review as a conference paper at ICLR 2025

Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun.
Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint
arXiv:2306.17107, 2023b.

Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai
Yu, and Wanli Ouyang. Motiongpt: Finetuned llms are general-purpose motion generators. In
Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7368–7376, 2024b.

Bo Zhao, Boya Wu, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint

arXiv:2307.04087, 2023.

Sipeng Zheng, Yicheng Feng, Zongqing Lu, et al. Steve-eye: Equipping llm-based embodied agents
In The Twelfth International Conference on Learning

with visual perception in open worlds.
Representations, 2023.

Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, and Zongqing Lu. Unicode: Learning a
unified codebook for multimodal large language models. arXiv preprint arXiv:2403.09072, 2024.

Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All-in-one framework for motion under-
standing planning generation and beyond. In Proceedings of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition, pp. 1357–1366, 2024.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

Appendices

A ADDITIONAL DETAILS OF MOSEBASE

In this section, we provide more details about Motionbase that are not included in the main paper
due to spatial limitations.

A.1 STATISTIC ANALYSES

MotionBase contains over 1 million motion sequences from 42 different public datasets and web
videos on the Internet. Subsets of MotionX, including Animation, Perform, Dance, Aist, Kungfu,
GRAB (Taheri et al., 2020), Music, Idea400 (Lin et al., 2024), HAA500 (Chung et al., 2021), Game
Motion, and Fitness, are included in MotionBase. Recognizing the high cost of collecting and anno-
tating videos, we also see the untapped potential of images for motion understanding. Consequently,
MotionBase incorporates image data by repeating each image across 64 frames and treating it as a
motion sequence. For the datasets with long-range videos, such as MPI-INF-3DHP (Mehta et al.,
2017), we segment the footage into sub-clips with random durations ranging from 10 seconds to one
minute. Figure 6 and Figure 7 illustrate the scale and length distributions of MotionBase.

Figure 6: The scale distribution of motion sequences across subsets of MotionBase.

A.2 PROMPT OF MOTION DESCRIPTION

In this paper, we use Gemini-1.5-pro (Reid et al., 2024) and GPT-4o-mini (OpenAI, 2024) as large
multimodal models (LMM) to generate textual annotations for video and image data, respectively.
For each person-centric sample, we first crop and track the person’s body using the corresponding
bounding box(es). The LMM is then tasked with focusing on the person’s physical movements and
positions in the global space to generate detailed descriptions. Unlike previous datasets, we provide

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Figure 7: The length distribution across different subsets of MotionBase

more granular motion descriptions by dividing the body into upper and lower sections, prompting
the LMM to generate part-specific descriptions (“part-level”). Additionally, an overall summary of
the entire body’s movement (“whole-body”) is also produced. Figure 8 illustrates the prompt used
to caption human motion sequences in MotionBase.

A.3 WORD DISTRIBUTION ANALYSIS

To further explore the annotated motion text, we generate word clouds from the entire text corpus
in MotionBase. Since the annotations in MotionBase consist of both whole-body and part-level
descriptions, we create separate word clouds for general labels and more detailed annotations, as
shown in Figure 9 and Figure 10, respectively. In Figure 9, we observe that the whole-body anno-
tations primarily highlight high-level motion activities, such as standing, sitting, and walking. In
contrast, Figure 10 shows that part-level annotations focus more on specific body movements, in-
cluding the torso, shoulders, legs, and arms. We believe that this hierarchical structure of annotations
will enhance the understanding of motion.

B ADDITIONAL OVERVIEW OF MODEL ARCHITECTURE

Due to space limitations in the main paper, we provide the overview of our model architecture in
Figure 11 in this appendix. Following most LMMs, our large motion model consists of two stages:
pre-training and fine-tuning. During the pre-training stage, we train a motion encoder, a motion
decoder, and a motion codebook to represent motions using discrete tokens. With this motion to-
kenizer, we fine-tune an autoregressive language model to predict motion tokens. In the inference
stage, the input text is processed by the language model to generate motion tokens in an autoregres-
sive manner, which are then decoded into natural motion by the pre-trained motion decoder.

17

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Figure 8: Prompt examples to label human motions in the video. We use Gemini-1.5-pro and GPT-
4o-mini to generate motion descriptions for the video and image data, respectively. We provide
“whole-body” (UP) and “part-level” (DOWN) labels for each sample in the dataset.

18

Begin by providing a general overview of the person's current action (e.g., walking, sitting, interacting) within the BBOX area. Then, proceed with a detailed breakdown, focusing exclusively on the physical movements and positions of the person within the BBOX. For the upper body, describe the position and movement of the arms, hands, shoulders, and torso. For the lower body, detail the position and movement of the legs, feet, and overall balance. Ensure the description strictly covers physical actions without mentioning facial expressions, clothing, or environmental elements outside the BBOX.Example:The person is standing still, observing something in front of them.lUpper body： Their arms hang relaxed by their sides, with the shoulders slightly back and the chest open. The torso is upright, with minimal movement, indicating a calm, neutral stance.lLower body： Both feet are planted firmly on the ground, shoulder-width apart. The knees are slightly bent, and their weight is evenly distributed between both legs.The person is standing within the designated area, engaging in a conversation seemingly directed toward someone positioned off-camera to the left. **Upper Body:*** **Arms:**  Initially held loosely at the sides, the arms transition to various positions throughout the interaction. At times, they rise to chest level with palms open, suggesting an explanatory gesture.  Occasionally, one or both arms extend outwards, indicating direction or emphasis. * **Hands:** Hand movements correspond with arm gestures. Palms face upwards and outwards during open-handed motions, then relax to a neutral position when the arms are at rest. * **Shoulders:** Shoulders remain relatively relaxed throughout, with subtle shifts in position reflecting the arm movements. They don't appear tense or raised, implying a generally comfortable stance.* **Torso:** The torso largely remains stationary, facing forward, with slight turns coinciding with the shifting weight distribution of the lower body.**Lower Body:*** **Legs:**  Legs maintain a comfortable stance, slightly apart, with the weight appearing balanced. There's a subtle shift in weight distribution as they adjust their stance. * **Feet:** Feet remain planted on the ground, primarily shoulder-width apart. The positioning suggests a grounded and stable stance.  * **Overall Balance:** The individual appears balanced and at ease throughout the interaction, with movements suggesting engagement in the conversation rather than discomfort or restlessness.Under review as a conference paper at ICLR 2025

Figure 9: Word cloud of whole-body textual annotation in MotionBase.

Figure 10: Word cloud of part-level textual annotation in MotionBase.

C ADDITIONAL EXPERIMENTAL RESULTS

In this section, we provide more experimental analysis which can not be presented in our main paper
due to space limitation.

Table 7: Ablation of the effectiveness of synthetic data and static data.

TRAIN SET

R@1 ↑

R@3 ↑

FID ↓ MMDist ↓

Real

w/o static & syn
w/o static
MotionBase

0.290

0.111
0.120
0.264

0.563

0.248
0.252
0.542

0.011

57.719
55.983
0.516

3.480

8.412
8.175
4.007

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Figure 11: Overview of the large motion model, which can be divided into two stages. In the first
stage(left), we pre-train a motion VQ-VAE to quantify motion sequences into tokens. In the second
stage(right), we fine-tune an autoregressive language model to predict motion tokens.

Table 8: Results on the test set with synthetic and static data filtered out.

TRAIN SET

R@1 ↑

R@3 ↑

FID ↓ MMDist ↓

Real

w/o static & syn
w/o static
MotionBase

0.196

0.167
0.166
0.168

0.474

0.396
0.393
0.399

0.006

1.740
1.780
1.614

1.647

2.323
2.356
2.300

C.1 ABLATION OF SYNTHESIS AND STATIC DATA

For handling static data, our core strategy is to introduce specific language prompts during training.
Specifically, by adding language markers such as "keep the action still," we explicitly guide the
model to understand the distinction between static and dynamic actions. Prompt-based methods
can effectively differentiate between different motion distributions. To validate this approach, we
conduct a series of ablation experiments. We train GPT2-medium on three variations of MotionBase:
without synthetic data, without image data, and without both synthetic data and image data. The
model is trained for 300 epochs with a learning rate of 2e-4. Using the VQ-VAE and retrieval model
trained on MotionBase, we test on the MotionBase test set and a subset of the test set where static
and synthetic data are filtered out. The results are shown in Table 7 and Table 8. Our findings
indicate that incorporating both static data (i.e., image data) and synthetic data leads to performance
improvements in terms of R-Precision.

Table 9: Comparison of evaluations using different encoder models.

EM_Humanml3d

EM_Motion-X

Decoder

#Inst.

#Param. R@1 ↑ R@3 ↑

FID ↓ R@1 ↑ R@3 ↑

FID ↓

Real

GPT-2
GPT-2

-

-

0.02M 355M
0.08M 355M

LLaMA-2
LLaMA-2

LLaMA-3
LLaMA-3

LLaMA-2
LLaMA-2

0.02M
0.08M

0.02M
0.08M

0.02M
0.08M

7B
7B

8B
8B

13B
13B

0.511

0.466
0.462

0.497
0.474

0.500
0.499

0.519
0.504

0.797

0.752
0.744

0.778
0.758

0.783
0.786

0.803
0.790

0.002

0.101
0.208

0.214
0.452

0.173
0.264

0.166
0.393

0.496

0.358
0.362

0.378
0.376

0.380
0.393

0.395
0.400

0.821

0.651
0.656

0.671
0.673

0.675
0.696

0.695
0.700

0.038

0.050
0.754

0.122
0.518

0.094
0.591

0.105
0.637

C.2 ABLATION OF DIFFERENT ENCODER MODELS

Table 9 presents the evaluation results on the HumanML3D test set using different encoder mod-
els (EM). We employ the same dual-encoder architecture (Guo et al., 2022a) but trained it on two

20

Motion TokensAutoregressive Language ModelOutput Motion TokensA womanisblowinga balloonwhilewalkingMotion EncoderMotion DecoderMotion Codebook<motion_id_0><motion_id_1><motion_id_2><motion_id_3><motion_id_4><motion_id_5>Motion TokensUnder review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Table 10: Comparison between fine-tuning and learning from scratch on the Motion-X test set.

#Inst

Real

0.02M
0.02M

0.08M
0.08M

From Sctrach R@1 ↑ R@3 ↑

FID ↓ MMDist ↓

-

Yes
No

Yes
No

0.496

0.035
0.206

0.460
0.468

0.821

0.103
0.402

0.782
0.791

0.038

16.904
54.017

0.113
0.096

2.438

9.280
8.218

2.862
2.798

Table 11: Results of different loss calculation methods on the HumanML3D test set.

Loss Calculation R@1 ↑ R@3 ↑

FID ↓ MMDist ↓

Real

Motion Seq Loss
Whole Seq Loss

0.511

0.388
0.466

0.797

0.650
0.752

0.002

0.680
0.101

2.974

3.919
3.234

distinct datasets: HumanML3D and Motion-X, where HumanML3D is a subset of Motion-X. The
results highlight the limited generalization ability of the encoder model. When using the model
trained on the larger Motion-X dataset, performance metrics on HumanML3D decrease. This sug-
gests that training on the broader Motion-X dataset negatively impacts R-Precision performance
on the HumanML3D subset. Furthermore, when the encoder model is trained on Motion-X, in-
creasing the training data size for the text-to-motion model leads to significant performance gains.
Conversely, when using the encoder model trained on HumanML3D, the performance of the text-
to-motion model degrades as the training data size increases. This might be attributed to inherent
limitations in the encoder model itself.

C.3 ABLATION OF LEARNING FROM SCRATCH VS. FINE-TUNING

We compare the performance of fine-tuning GPT-2 against training it from scratch (random ini-
tialization). As shown in Table 10, fine-tuned models consistently outperform those trained from
scratch, particularly when trained on HumanML3D and evaluated on MotionX. The improvement
of pretrained LLM highlights the importance of text pre-training in enhancing the model’s under-
standing of text descriptions and improving its generalization capabilities.

C.4 ABLATION OF DIFFERENT LOSS CALCULATION STRATEGIES

We also investigate the impact of different loss calculation strategies on model performance: We
compare two strategies: 1) calculating the loss solely on the output motion tokens, and 2) calcu-
lating the loss on both the input text and the output motion tokens. As shown in Table 11, our
results indicate that the second strategy yields better performance. This improvement compared to
the first alternative is likely due to the strategy’s ability to prevent catastrophic forgetting of text
understanding. Additionally, it helps mitigate overfitting to motion patterns in the training data,
thereby enhancing the model’s generalization ability.

C.5 ABLATION STUDY ON HIERARCHICAL TEXT AND BASIC TEXT

To investigate the effectiveness of hierarchical text representation, we conduct a series of ablation
experiments. As shown in Table 12, we compare the training results using hierarchical text with both
basic and detailed descriptions, against the results using only basic descriptions. The experimental
results demonstrate that hierarchical text can effectively enhance the model’s semantic understand-
ing, thereby improving the semantic matching of generated motions.

It is worth noting that the evaluation results for hierarchical text are sometimes overestimated, even
surpassing the ground truth. We hypothesize that this is because the evaluator itself is a network

21

Under review as a conference paper at ICLR 2025

Table 12: Results of Hierarchical Text and Basic Text on MotionBase.

Training text

R@1 ↑ R@3 ↑

FID ↓ MMDist ↓

Real

Basic text
Hierarchical text

0.290

0.264
0.302

0.563

0.542
0.603

0.011

0.516
0.521

3.480

4.007
3.523

Table 13: Results of LoRA and full parameter fine-tuning on MotionBase.

Training method R@1 ↑ R@3 ↑

FID ↓ MMDist ↓

Real

LoRA
Full Param

0.290

0.249
0.264

0.563

0.520
0.542

0.011

1.896
0.516

3.480

3.869
4.007

model trained on the training set to fit its distribution, and may exhibit bias on the test set. If the
generated text-motion data aligns better with the training set distribution, the evaluation metrics
might even outperform the ground truth on the test set. Therefore, how to quantitatively evaluate
motion generation performance remains an interesting research topic worthy of further exploration.

C.6 ABLATION STUDY ON LORA AND FULL PARAMETER FINE-TUNING

We conduct an ablation study comparing LoRA and full parameter fine-tuning. As shown in Ta-
ble 13, LoRA fine-tuning struggles to achieve competitive results. We attribute this limitation to
the introduction of new motion tokens, which necessitate substantial parameter adjustments for the
language model to comprehend these additional tokens. The constrained nature of LoRA fine-tuning
appears insufficient to effectively address these demands.

C.7 EXPERIMENTAL COMPARISON WITH T2M-GPT ON MOTIONBASE

We train the T2M-GPT model on the MotionBase dataset and compare it with a model based on
GPT-2 medium. As shown in Table 14, despite comparable parameter counts, the T2M-GPT method
struggles to produce competitive results. Because of the inherent limitations of CLIP’s text encod-
ing capabilities, models trained this way struggle to understand a wider range of motion-related
language. We believe that large motion models based on decoder-only LLMs, which jointly train
text tokens and motion tokens, achieve better text-motion semantic alignment and stronger motion
generation capabilities.

C.8 ABLATION OF MOTION GENERATION BASED ON LFQ

To validate the applicability of the LFQ quantization method for motion generation, we conducted
experiments summarized in Table 15. These experiments include data scaling with GPT-2 and pa-
rameter scaling using 0.02M training samples. The results are consistent with our initial conclusions,
confirming robust performance across scaling scenarios. Furthermore, LFQ demonstrates a slight

Table 14: Results of T2M-GPT and GPT-2 on MotionBase.

Model

Real

-

T2M-GPT
GPT-2 Medium

380M
355M

#Param. R@1 ↑ R@3 ↑

FID ↓ MMDist ↓

0.290

0.243
0.264

0.563

0.504
0.542

0.011

1.909
0.516

3.480

4.593
4.007

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Figure 12: Comparison with different motion quantization on the Motion-X (left) and MotionBase
dataset (right). The Y-axis denotes FID (↓).

performance advantage over VQ when evaluated with GPT-2. Given that LFQ utilizes a significantly
larger codebook, which increases training difficulty, we anticipate that further improvements could
be achieved by scaling both model parameters and training data.

Table 15: Ablation of motion generation using LFQ and VQ under different setups.

Motion-X

MotionBase

Decoder

GPT-2-VQ

GPT-2-LFQ
GPT-2-LFQ
GPT-2-LFQ

GPT-2-LFQ
LLaMA-2-LFQ
LLaMA-2-LFQ

#Inst.

1M

0.02M
0.08M
1M

0.02M
0.02M
0.02M

#Param.

R@1 ↑

R@3 ↑

355M

355M
355M
355M

355M
7B
13B

0.357

0.166
0.332
0.394

0.166
0.225
0.206

0.614

0.341
0.558
0.628

0.341
0.383
0.351

FID ↓

5.083

76.214
6.245
4.275

76.214
68.542
71.238

R@1 ↑

R@3 ↑

0.264

0.042
0.062
0.326

0.042
0.062
0.085

0.542

0.085
0.144
0.607

0.085
0.140
0.184

FID ↓

0.516

136.254
128.071
0.452

136.254
125.082
119.036

C.9 ABLATION OF MOTION QUANTIZATION

First, we provide additional FID results on Motion-X in Figure 12. It is worth noting that while our
motion quantizer performs worse than RQ-VAE on the smaller HumanML3D dataset, it surpasses
both VQ and RQ when evaluated on the larger Motion-X and MotionBase benchmarks, as can be
seen in Table 6. This suggests that our approach offers a greater advantage when applied to larger
datasets, highlighting its improved generalization compared to previous methods.

To further validate the effectiveness of our 2D quantization strategy, we compare the 2D-LFQ
method with its 1D counterpart (which is identical to VQ except for the quantization strategy). The
results, shown in Table 16, demonstrate that 2D quantization in LFQ significantly outperforms the
1D version. This highlights the superior ability of 2D quantization to enhance the representational
capacity of the motion tokenizer.

Table 16: Ablation of 2D motion quantization vs. its 1D version.

HumanML3D

Motion-X

MotionBase

Tokenizer

#Num.

#Param.

FID ↓ MPJPE ↓

FID MPJPE

FID MPJPE

1D-LFQ
2D-LFQ

16384
16384

19.43M 3.85
108.35M 1.769

52.5
45.6

2.783
0.295

78.9
54.1

10.358
7.853

80.1
64.1

23

Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

D DATASET CONSTRUCTION PIPELINE

Our data collection pipeline is a multi-stage process designed to curate a large-scale, high-quality,
and richly annotated multimodal motion dataset. The detailed steps are outlined below:

Video Data Collection and Cleaning: We amass over 20 million videos from publicly available
datasets like InternVid and WebVid, as well as online platforms such as YouTube. To maintain data
relevance and quality, we employ a pretrained human detection model to filter out videos lacking
human presence.

2D and 3D Keypoint Estimation: We estimate 2D human keypoints and their corresponding con-
fidence scores using the pretrained VitPose model (Xu et al., 2022). To further refine motion in-
formation, we leverage a pretrained 3D keypoint estimation model (Sárándi et al., 2023) trained on
extensive 3D datasets. Following the methodology of Lin et al. (2024), we apply temporal smooth-
ing and 3D bone length constraints during triangulation to enhance the stability and consistency of
the keypoint estimations.

Multimodal Information Integration: For a more comprehensive understanding of human motion,
MotionBase incorporates RGB, depth data, and annotations for multi-person scenarios. In multi-
person sequences, each motion is grounded to its respective video via object-level bounding boxes.
While this work primarily focuses on text-to-motion tasks, these additional modalities pave the way
for future research in related areas.

Local-Global Pose Estimation: We fit the SMPL-X body model (Pavlakos et al., 2019) to each
frame in MotionBase using a progressive learning-based mesh fitting approach (Lin et al., 2024).
Specifically, we predict SMPL-X parameters using the pretrained OSX method (Lin et al., 2023),
followed by iterative optimization to align the parameters with the target 2D and 3D joint positions.
Subsequently, we apply a global motion optimization technique based on Yuan et al. (2022) to refine
both global motions and camera poses, ensuring consistency with the video evidence. Finally, for
motion sequences with noisy or occluded input data, we employ RoHM (Zhang et al., 2024a) to
reconstruct complete and plausible motions.

Single-Frame Pose Expansion: To enhance dataset diversity and scale, we expand single-frame
pose data into multi-frame sequences. We achieve this using the PHC (Luo et al., 2023) strategy
and the pre-trained motion completion model MotionGPT (Jiang et al., 2023). The PHC strategy
ensures the physical plausibility of the generated motion sequences, while MotionGPT provides
motion priors to enhance naturalness and fluidity.

Hierarchical Motion Descriptions: MotionBase features hierarchical text annotations to address
limitations in existing dataset descriptions. Leveraging the Gemini-1.5-pro large language model
(Reid et al., 2024) and a carefully crafted prompt format, we generate detailed descriptions for
individual body parts (e.g., left arm, right leg), dedicating a sentence to each. Furthermore, we sum-
marize the overall body movement with 1-3 sentences, providing a more holistic motion description.

E DATASET QUALITY EVALUATION

E.1 MOTION DATA QUALITY

To ensure dataset quality, we conduct multifaceted evaluations of the motion data.

Refinement using a Reinforcement Learning-based Strategy: We use PHC to train a reinforce-
ment learning-based policy model that refines the raw motion data, ensuring conformity to physical
laws and enhancing realism. This policy takes raw motion sequences as input, treats them as target
poses, and generates new motion sequences satisfying physical laws in a simulated environment,
thereby eliminating issues such as jitter and foot sliding. While this strategy may encounter chal-
lenges with drastic movements, it effectively improves data quality for most motion sequences.

Data Diversity: A key advantage of the MotionBase dataset is its scale and diversity. We collect
over one million motion sequences from multiple sources (including InternVid and internet videos),
encompassing a wide range of motion types. This diversity supports the training of more generaliz-
able motion models.

24

Under review as a conference paper at ICLR 2025

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Figure 13: Quantitative examples of motions generated by our large motion model.

E.2 TEXT DESCRIPTION QUALITY

To ensure text description quality, we employ a multi-level evaluation approach.

Automatic Evaluation based on Large Language Models: We automatically evaluate text de-
scriptions in MotionBase using large language models such as Gemini-1.5-pro (Reid et al., 2024)
and GPT-4. We use a 1-to-5 scoring system based on these criteria:

• 1 point (Very Poor): The description is vague, irrelevant to the motion content, or contains

severe grammatical errors.

• 2 points (Poor): The description lacks specifics, detail, or contains obvious errors.
• 3 points (Fair): The description basically reflects the motion content but lacks detail and

may contain minor errors.

• 4 points (Good): The description is accurate and detailed, clearly expressing the motion

process.

• 5 points (Excellent): The description is precise, comprehensive, and fluent, providing an

in-depth analysis of motion details.

We use GPT-4o to score text descriptions from MotionBase, MotionX, and HumanML3D. Mo-
tionBase achieves an average score of 3.837, while MotionX and HumanML3D score 1.386 and
1.703, respectively, indicating higher quality in MotionBase’s text descriptions. To further evaluate
consistency between text descriptions and motion content, we also input text descriptions and cor-
responding rendered motion videos into the Gemini-Pro model for scoring. MotionBase achieves
an average score of 3.82, while MotionX and HumanML3D score 2.25 and 3.08, respectively, again
confirming the quality advantage of MotionBase’s text descriptions.

Consistency Check of Hierarchical Descriptions: MotionBase provides hierarchical text descrip-
tions, including overall, local detail, and rule-based descriptions. We use GPT-4 and manual checks

25

Person falls to the ground in a sitting motion and then pops back up in a standing position.A person squats down then jumps.A man full-body sideways jumps to his left.A woman is blowing a balloon while walking.A person is building blocks and walking at the same time.The person performs Lunges Of Crossover Reverse Lunge.Text PromptGenerated Motion SequencesUnder review as a conference paper at ICLR 2025

to ensure consistency across different levels, guaranteeing logical coherence and informational com-
pleteness.

F ADDITIONAL QUALITATIVE RESULTS

We provide some examples to visualize the human motions predicted by our large motion model
trained on MotionBase, as illustrated in Figure 13. As can be seen, our large motion model is capable
of generating motion sequences that align well with the input texts, demonstrating the effectiveness
of the MotionBase dataset.

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

26

