# STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion
Retargeting

Zenghao Chai, Chen Tang, Yongkang Wong, , and Mohan Kankanhalli Z. Chai, Y.
Wong, and M. Kankanhalli are with the School of Computing, National University
of Singapore, Singapore, 117417. Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>C. Tang is with the
Department of Computer Science and Technology, Tsinghua University, China,
100084. Email<EMAIL_ADDRESS>Kankanhalli is the corresponding
Author. 0000-0003-3709-4947 0000-0002-0108-6729 0000-0002-1239-4428
0000-0002-4846-2015

###### Abstract

The creation of 4D avatars (_i.e_., animated 3D avatars) from text description
typically uses text-to-image (T2I) diffusion models to synthesize 3D avatars
in the canonical space and subsequently applies animation with target motions.
However, such an optimization-by-animation paradigm has several drawbacks. (1)
For pose-agnostic optimization, the rendered images in canonical pose for
naïve Score Distillation Sampling (SDS) exhibit domain gap and cannot preserve
view-consistency using only T2I priors, and (2) For post hoc animation, simply
applying the source motions to target 3D avatars yields translation artifacts
and misalignment. To address these issues, we propose Skeleton-aware Text-
based 4D Avatar generation with in-network motion Retargeting (STAR). STAR
considers the geometry and skeleton differences between the template mesh and
target avatar, and corrects the mismatched source motion by resorting to the
pretrained motion retargeting techniques. With the informatively retargeted
and occlusion-aware skeleton, we embrace the skeleton-conditioned T2I and
text-to-video (T2V) priors, and propose a hybrid SDS module to coherently
provide multi-view and frame-consistent supervision signals. Hence, STAR can
progressively optimize the geometry, texture, and motion in an end-to-end
manner. The quantitative and qualitative experiments demonstrate our proposed
STAR can synthesize high-quality 4D avatars with vivid animations that align
well with the text description. Additional ablation studies shows the
contributions of each component in STAR. The source code and demos are
available at: https://star-avatar.github.io.

###### Index Terms:

Computer Graphics, Text-to-Avatar Generation, Digital Humans, 4D Avatar

##  I Introduction

Recent years have witnessed an impressive progress of text-to-image (T2I)
generative models [1, 2, 3, 4]. By leveraging the versatile T2I diffusion
priors [1, 2], the creation of 3D content [5, 6, 7, 8, 9] (especially human-
like characters [10, 11, 12, 13, 14]) from arbitrary text description have
received great attention and interest in the intersection of computer vision
and graphics communities.

In an extended application, the creation of 4D avatar – synthesizing
animatable characters with authentic human motions – has been found valuable
in film and gaming industry. Existing text-based 4D avatar generation follows
a typical optimization-by-animation paradigm (see Fig. 1). The state-of-the-
art methods [10, 15, 16, 17] first resort to the use of image priors [1] for
generating 3D avatars from a text description, and subsequently apply the
desired human motion from audio [18, 19, 20], video [21, 22], or text [23, 24,
25] to generate 4D avatars. However, we observe that such a typical paradigm
encounters several challenges.

\begin{overpic}[trim=113.81102pt 0.0pt 85.35826pt
0.0pt,clip,width=433.62pt,grid=false]{figure/cmp_pipe.jpg} \end{overpic}
Figure 1: Typical Optimization-by-Animation Pipeline (Top) _v.s_. Ours
(Bottom) for Text-based 4D Avatar Generation. Top. Previous methods (_e.g_.,
TADA [10]) first optimize the 3D representation in fixed canonical pose and
subsequently apply motions for animation, which lack shape diversity and
exhibit animation artifacts. Bottom. The proposed STAR for 4D avatar
generation. It leverages in-network motion retargeting and hybrid SDS to
jointly update geometry, texture, and motions to achieve vivid 4D avatar from
only text description.

\begin{overpic}[trim=426.79134pt 0.0pt 284.52756pt
0.0pt,clip,width=433.62pt,grid=false]{figure/teaser1-v2.jpg} \end{overpic}

\begin{overpic}[trim=426.79134pt 0.0pt 284.52756pt
0.0pt,clip,width=433.62pt,grid=false]{figure/teaser2-v2.jpg} \end{overpic}

Figure 2:  Examples of Generated 4D Avatars. We propose STAR to produce high-
fidelity 4D avatars from only text description. For each sample, we showcase:
Left. Face and full-body of the textured 3D avatar and its normal map, and
Right. Randomly selected frames of the 4D animation in two different camera
views. Best viewed in color and zoom-in.

First, existing methods [10, 13, 17, 26] fail to address the pose distribution
difference between the rendered 2D avatar in canonical space and that of T2I
models. This is because T2I priors are trained in a majority of natural human
poses. Therefore, such canonical pose-driven optimization suffers from a
potential domain gap and is thus detrimental to obtaining expressive 3D
avatars. In contrast, while several methods [11, 15, 27] learn deformable 3D
representations [28] with sampled pose [29], they implicitly suggest the
source template and target avatar posses similar geometry and skeleton
structures. Thus, it requires additional regularization to constrain the 3D
representations to achieve reasonable deformations, which sacrifices the
geometry diversity. Second, the optimization step based on T2I Score
Distillation Sampling (SDS) lacks stability and may potentially deteriorate
into local optimum. The reason is that generated identity and/or appearance
with different viewpoints exhibit frame inconsistency and content drift from
naïve T2I priors, _e.g_., inconsistent background, appearance, and
illumination. Finally, due to the skeleton and geometry difference between the
source template and target avatar, animation artifacts (_e.g_., body
intersection) happen when simply copying the source motion from motion
generative models [20, 23, 24] to create a target 4D avatar.

To address the above challenges, we propose a Skeleton-aware Text-based 4D
Avatar Generation with in-network motion Retargeting (STAR) model. STAR
seamlessly synthesizes high-fidelity 4D avatar with authentic animations from
only text description (see Fig. 2). Specifically, the optimization process of
STAR is skeleton-aware, geometry-aware, and motion-aware. STAR progressively
optimizes the geometry, appearance, and motion such that it compounds
realistic 4D avatar to align the text description, with three crucial insights
beyond prior art. First, inspired by [24, 30], we perform in-network motion
retargeting of the initialized motions. As a result, the updated motions and
skeleton possess precise structural information of the target 3D avatar while
maintaining the semantics of the given text description. Second, to eliminate
the disturbance of temporal-inconsistency and view-agnostic SDS with naïve T2I
models, we resort to the training-free text-to-video (T2V) diffusion models
[31] to provide frame-consistency priors. By coherently conditioning on the
retargeted, occlusion-aware skeleton with ControlNet [32], STAR provides
informative skeleton conditions that contributes to high-quality avatars of
the optimization stage. Finally, to improve the geometry-appearance alignment
among different views and poses, we propose a hybrid SDS, which is a compound
of multi-view SDS and masked sequential SDS, as well as accompanied with the
elaborately designed hierarchical regularization, to train the 3D
representation that stabilizes the optimization process and achieve
satisfactory 4D avatar.

The aforementioned insights and techniques successfully address the challenges
of text-based 4D avatar creation. The proposed STAR seamlessly generates vivid
4D avatars from only text description, and is flexible to deploy with
arbitrary motions generated from either text, audio, or videos, while
preserving the motion transfer quality of the 4D avatar. Experimental results
demonstrate STAR synthesizes high-fidelity 4D avatar which aligns well with
the given text description. Both quantitative and qualitative experiments
demonstrate our STAR achieves state-of-the-art generation results compared to
previous methods with the optimization-by-animation pipeline.

In summary, our main contributions are as follows:

  * •

To the best of our knowledge, we are the earliest work to analyze the
influence of missing and/or mismatched motion in text-to-avatar generation. We
propose a novel end-to-end model STAR to address these drawbacks in 4D avatar
generation.

  * •

We systemically design the STAR pipeline with integration of text-to-motion,
motion retargeting, and text-to-3D techniques with crucial modifications. This
enables STAR to generate high-quality avatars with vivid animations from text
descriptions.

  * •

We combine the capabilities of skeleton-conditioned T2I and T2V priors,
equipped with occlusion-aware, coarse-to-fine hybrid SDS, and hierarchical
regularization to improve the optimization process and address the view-
inconsistency and facial-blur issues.

  * •

Quantitative and qualitative experiments validate STAR can generate vivid 4D
avatars and outperform the optimization-by-animation baselines. Code and
trained models will be released for research purposes.

##  II Related Work

The tasks of 3D content generation with text description have received
increasing attention in recent years. A feasible solution is to leverage the
2D models prior [1, 2, 4, 33] pretrained on
$\langle\textit{image},\textit{text}\rangle$ pairs to optimize the 3D
generation. This section reviews methods that are related to this work.
Comprehensive surveys can be found in [34, 35, 36].

Text-driven 3D Content Generation. The seminal works (_e.g_., DreamField [37],
CLIP-mesh [38], AvatarCLIP [39], CLIP-Face [40], Text2Mesh [41]) optimize the
underlying 3D representation (_e.g_., NeRF [42], 3DGS [43], or mesh) via CLIP
[44] space similarity between rendered images and text embeddings. However,
these methods exhibit less realism since CLIP only provides high-level
semantics. Recent advances resort to the versatile T2I diffusion [1] priors
for text-guided 3D generation. The pioneering work, DreamFusion [5] and SJC
[45], propose Score Distillation Sampling (SDS) for text-to-3D generation.
Follow-up works such as ProlificDreamer [46], PGC [47], DreamTime [48] revisit
and improve the naïve SDS for better generation quality. One line of work
improves the training pipeline with elaborately designed branches. Magic3D [6]
proposes a coarse-to-fine pipeline to generate fine-grained 3D meshes.
Fantasia3D [49] disentangles the generation of geometry and texture, Guide3D
[14], MVDream [50], MVD-Fusion [51] and TEXTure [52] leverage the multi-view
images or features for improving view-consistency. More recently, by
leveraging the hybrid text-driven image [1, 2, 53], video [54, 55, 56, 57],
and 3D-aware [8, 50, 58] diffusion models, MAV3D [59], 4D-fy [60], and AYG
[61] focus on generate 4D dynamic scenes using deformable 4D representations
[62, 63, 64].

Text-driven 3D Avatar Generation. By integrating explicit [29, 65] and/or
implicit [28, 62, 66] 3D human priors for full-body 3D human creation from
text description, early works such as AvatarCLIP [39] and CLIP-Actor [67]
optimize the 3D representations [42, 66] through CLIP similarly [44]. Due to
the limited guidance from CLIP, these models exhibit noisy geometry and blurry
texture. Recent advances [10, 11, 13, 17, 68, 69] integrate the power of
diffusion models with SDS and achieve promising progress with reasonable
details of text guided 3D avatar generation, which follow the similar paradigm
of text-to-3D content generation [5]. AvatarCraft [17] and DreamHuman [11]
leverage the pretrained T2I models and SMPL [65] body prior to train implicit
3D representations [28, 42, 66], with coarse-to-fine and semantic zoom
techniques. TADA [10] uses SMPL-X [29] model and learns explicit mesh for
avatar generation. To improve the optimization process with more complex
avatars, state-of-the-art methods resort to ControlNet [70] with elaborately
designed branches to provide view-consistent guidance such that improves the
plausible details. Existing methods uses skeleton (_e.g_., DreamWaltz [15] and
HumanGaussian [16]), keypoint (_e.g_., DreamAvatar [71] and AvatarBooth [72]),
DensePose [73] (_e.g_., AvatarVerse [26] and AvatarStudio [27]) or
normal/depth (_e.g_., HumanNorm [12]) as conditions with modified SDS. A
common trend of the above work is either to use a fixed canonical pose or a
sampled template pose for optimization. As a result, it is challenging to
generate personalized structures and high-fidelity avatars.

Character Animation & Motion Retargeting. One line work of text-driven avatar
generation focuses on generating animatable 3D avatars [10, 15, 17, 67] for 4D
content. These methods typically apply the prepared motions [20, 23, 74, 75,
76, 77] to animate the canonical avatar for 4D content creation. For example,
by leveraging the text-driven human generation models [78, 79, 80], which
synthesize realistic human-like motion from textual descriptions, TADA [10]
can directly animate the avatars for its explicit mesh representation, while
AvatarCraft [17], DreamWaltz [15] and HumanGaussian [16] use additional
deformable modules to translate or query the motion deformation into
corresponding implicit representations. However, these methods neglect the
shape and skeleton difference between the template shape and target avatar,
which thereby introduce artifacts if simply copy the source motion for avatar
animation. In the field of computer graphics, motion retargeting [81, 82, 83,
84] is a long-standing space-time optimization problem that identifies
features of the source motions as kinematic constraints. To improve the
retargeting quality, recent works use deep learning techniques [30, 85, 86,
87, 88] to learn motion representations from skeleton and/or geometry
constraints.

To the best of our knowledge, while there have been impressive progress in the
area of text-based generation, motion retargeting has not been investigated to
address the animation artifacts for text-to-avatar generation. In the next
section, we embrace skeleton-aware and geometry-aware motion retargeting to
produce 4D avatars.

##  III Method

###  III-A Preliminary

Avatar Representation. In Eq. 1, we use the re-topologized SMPL-X [29] with
per-vertex displacement $\delta$ and UV texture $\alpha$ to represent a
textured 3D avatar $\boldsymbol{m}(\beta,\psi,\delta;\boldsymbol{q},\alpha)$.

$\displaystyle\boldsymbol{m}(\beta,\psi,\delta;\boldsymbol{q},\alpha)$
$\displaystyle=\mathcal{W}\left(\bf{T}(\beta,\psi,\delta;\boldsymbol{q}),\mathcal{J}(\beta);\boldsymbol{q},\omega,\alpha\right)$
(1) $\displaystyle\bf{T}(\beta,\psi,\delta;\boldsymbol{q})$
$\displaystyle=\bar{\boldsymbol{m}}+\boldsymbol{b}_{s}(\beta)+\boldsymbol{b}_{e}(\psi)+\boldsymbol{b}_{p}(\boldsymbol{q})+\delta,$
(2)

where $\beta$ and $\psi$ represent the shape and expression parameters.
$\boldsymbol{q}$ indicates the sampled pose from the motions $\mathcal{Q}$.
$\mathcal{W}$ represents linear blending skinning function [89], with
predefined blend weights $\omega$. $\mathcal{J}$ represents the 3D joint
location regressor. $\bar{\boldsymbol{m}}$ is the shape template,
$\boldsymbol{b}_{s},\boldsymbol{b}_{e}$ and $\boldsymbol{b}_{p}$ are shape,
expression, and pose bases, respectively. In this work, we denote the
learnable parameters as $\Theta\coloneqq\\{\beta,\psi,\delta,\alpha\\}$ for
simplicity.

Score Distillation Sampling (SDS) [5] leverages the pretrained 2D diffusion
model to minimize the difference between the predicted noise
$\epsilon_{\phi}(\boldsymbol{x}_{t};y,\tau)$ and Gaussian noise
$\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, such that it computes
gradient to optimize the learnable parameter $\Theta$:

$\nabla_{\Theta}\mathcal{L}_{\text{SDS}}(\phi,\boldsymbol{x})=\mathbb{E}_{\tau,\epsilon}\left[w(\tau)(\epsilon_{\phi}(\boldsymbol{x}_{\tau};y,\tau)-\epsilon)\frac{\partial\boldsymbol{x}}{\partial\Theta}\right],$
(3)

where $\boldsymbol{x}\coloneqq g(\Theta;\boldsymbol{q},\boldsymbol{\pi})$
represents the differentiable rendering of the 3D model, given a body pose
$\boldsymbol{q}$ and camera view $\boldsymbol{\pi}$. $y$ indicates the text
condition, $w(\tau)$ is the weight function correlated to noise level $\tau$.

\begin{overpic}[trim=142.26378pt 184.9429pt 28.45274pt
0.0pt,clip,width=433.62pt,grid=false]{figure/pipeline2.jpg} \end{overpic}
Figure 3: Overview of the Proposed STAR. Left. Given a text description, we
initialize the human motion with pretrained text-to-motion model [24]. Note
that the typical optimization-by-animation paradigm easily yields deteriorated
body structures and animation artifacts for 4D avatar generation. Right. We
eliminate the potential pose distribution bias in the SDS-based optimization
by integrating the retargeted motion for animation. With the personalized and
occlusion-aware skeleton, we leverage the hybrid T2I and T2V diffusion models
to provide 3D consistent priors that progressively optimize the geometry,
texture, and motion to produce 4D avatar in an end-to-end manner.

###  III-B Motivation and Overview

Previous methods typically exhibit an optimization-by-animation paradigm to
create 4D avatars, which firstly resort to SDS [5] in Eq. 3 or its variants
[15, 26, 27] to train the 3D representation by performing the denoising
process of rendered canonical avatars. However, the diffusion models are
pretrained to estimate images with diverse poses, which exhibit pose-
distribution bias in the optimization stage and hinder the generation
fidelity. In the animation stage, we notice that simply assigning the motions
to the canonical avatar yields artifacts, as it ignores the skeleton and
geometry difference between the source template and the target avatar. These
two key issues make the text-to-avatar models unable to generate vivid 4D
avatars with both shape diversity and motion quality.

To mitigate these issues, we propose the Skeleton-aware Text-based 4D Avatar
generation with in-network motion Retargeting model (STAR). Fig. 3 shows the
overview pipeline of the proposed method, which contains three crucial
components. First, the optimization process is skeleton-aware and motion-
aware. By leveraging the sampled human motions generated from text
descriptions [24], we make the rendered 3D avatar align better with natural
pose distribution in diffusion priors. Second, to address the artifacts when
simply copying the source motions into target geometry, we employ in-network
motion retargeting [30] to progressively update the initial motions such that
the motions faithfully reflect the personalized structure of the target avatar
during the training stage. Third, to improve the optimization consistency,
instead of naïve T2I-based SDS, we propose a hybrid SDS approach that employs
both T2I [1] and T2V [31] priors with occlusion-aware, skeleton-conditioned
ControlNet [32] to provide strong view-consistency supervisions and improve
the generation quality. The following sections describe each component in
detail.

###  III-C Skeleton-Aware Text-based 4D Avatar Generation

Motion Initialization. To synthesize an initial motion that aligns with the
given text description, we leverage a pretrained text-to-motion model
$\mathcal{M}$ [24] to translate the text description into motion sequences
$\mathcal{Q}^{s}=\\{\boldsymbol{q}^{s}_{0},\cdots,\boldsymbol{q}^{s}_{\ell}\\}$
with length $\ell=196$. However, the initial motion is based on a source
template mesh $\boldsymbol{m}^{s}$ with joints $\boldsymbol{j}^{s}$. It
exhibits geometry and skeleton mismatch when we simply copy $\mathcal{Q}^{s}$
to animate an avatar $\boldsymbol{m}^{t}$.

In-Network Motion Optimization. Intuitively, sampling and rendering avatars
with random poses for training can potentially eliminate the distribution bias
such that it improves the synthesized quality and optimization stability.
However, we observe that simply applying a pose $\boldsymbol{q}^{s}_{i}$ to
the target avatar yields the intersection artifacts of body regions,
especially for those with shape-variant bodies. As a result, it leads to
counterproductive drawbacks for motion-aware optimization (as demonstrated in
Sec. IV-D).

To simultaneously eliminate motion artifacts and pose inconsistency, and
improve the generation quality, we leverage a pretrained motion retargeting
model
$\mathcal{F}=\\{\Delta{\mathcal{F}}_{\boldsymbol{m}},\Delta{\mathcal{F}}_{\boldsymbol{j}}\\}$
[30] to correct the initial motion conditioned by both skeleton and geometry
of a target avatar $\boldsymbol{m}^{t}$. At optimization step $T$, we retarget
the source motion $\mathcal{Q}^{s}$ to the target motion $\mathcal{Q}^{t}_{T}$
conditioned by the canonical mesh $\boldsymbol{m}^{*t}_{T}$ and its joints
$\boldsymbol{j}^{*t}_{T}$, with geometry-aware and skeleton-aware residuals
$\delta\mathcal{Q}_{T}$,

$\mathcal{Q}^{t}_{T}=\mathcal{Q}^{s}+\underbrace{\Delta{\mathcal{F}}_{\boldsymbol{m}}(\boldsymbol{m}^{s},\boldsymbol{m}^{*t}_{T})+\Delta{\mathcal{F}}_{\boldsymbol{j}}(\boldsymbol{j}^{s},\boldsymbol{j}^{*t}_{T})}_{\delta\mathcal{Q}_{T}}.$
(4)

In this way, we obtain the animated geometry $\boldsymbol{m}^{t}_{T}$ and
joints $\boldsymbol{j}^{t}_{T}$ using the retargeted motion
$\mathcal{Q}^{t}_{T}$. The 3D geometry and motions w.r.t. the text description
are updated progressively during optimization, which in return, provide
precisely rendered images with diverse motions for the diffusion process.

###  III-D Hybrid SDS with Frame Consistency

Skeleton-conditioned ControlNet with Occlusion Detection. Vanilla SDS [5]
leverages view-dependent prompts for text-to-image diffusion models to provide
sparse supervision signals. Thus, the generation results tend to be imprecise
and will potentially introduce Janus (multi-face) problems (see Fig. 4).
Inspired by the recent success of ControlNet [32] in text-to-avatar generation
[15, 16, 26, 27], we use the skeleton from retargeted human motions as precise
conditions.

To eliminate the visibility ambiguity of the rendered skeleton during the
diffusion process, we leverage the informative rasterization process in
differentiable rendering [90] to select visible skeleton $\boldsymbol{j}^{o}$.
Specifically, given camera view $\boldsymbol{\pi}$, we denote the vertices of
$\boldsymbol{m}^{t}$ as $\boldsymbol{v}^{t}$ and the $i$-th joints
$\boldsymbol{j}^{t}_{i}$ are visible _iff_ $\boldsymbol{j}^{t}_{i}$’s
$k$-nearest neighbor is located in the visible vertices
$\boldsymbol{v}^{\prime}\subseteq\boldsymbol{v}$, indicated by rasterization
step in rendering. Then, we leverage the occlusion-aware skeleton
$\boldsymbol{j}^{o}$ as condition for ControlNet [32] to compute the revised
T2I SDS as

$\displaystyle\nabla_{\Theta}$
$\displaystyle\mathcal{L}_{\text{SDS}}(\phi,\boldsymbol{x}^{*})=$ (5)
$\displaystyle\mathbb{E}_{\tau,\epsilon}\left[\omega(\tau)\left(\epsilon_{\phi}(\boldsymbol{x}^{*}_{\tau};\boldsymbol{j}^{o},y,\tau)-\epsilon\right)\frac{\partial\boldsymbol{x}^{*}}{\partial\Theta}\right],$

where $\boldsymbol{x}^{*}\coloneqq
g(\Theta;\boldsymbol{q}^{*},\boldsymbol{\pi})$ indicates the rendered images
with retargeted motions $\boldsymbol{q}^{*}\in\mathcal{Q}^{t}_{T}$ at training
step $T$.

Masked Sequential SDS. While the skeleton-conditioned ControlNet can provide
precise view-dependent knowledge, we observe that diffusion models potentially
exhibit instability across different views. To improve the frame-consistency,
apart from leveraging the pretrained T2I models and the given retargeted
motion sequence $\mathcal{Q}^{t}$, we additionally resort to the training-free
T2V models [31] to present frame-consistent priors of constitute views during
the 4D avatar creation steps. However, empirical observation shows that while
the T2V model [31] can achieve plausible consistency in the body parts, it
yields blurry regions in the fine-grained structures (_e.g_., the face
regions). Therefore, we select facial mask and propose a masked sequential SDS
by eliminating the potential error on the face regions. Specifically, we
obtain the non-facial mask $\boldsymbol{x}^{*}_{\text{mask}}$ in rendering of
given camera view $\boldsymbol{\pi}$ and eliminate the facial regions for
sequential SDS, as Eq. 6 shows.

$\displaystyle\nabla_{\Theta}$
$\displaystyle\mathcal{L}^{\text{seq}}_{\text{SDS}}(\phi,\boldsymbol{x}^{*},\boldsymbol{x}^{*}_{\text{mask}};\mathcal{Q}^{\prime}\subseteq\mathcal{Q}^{t})=$
(6)
$\displaystyle\mathbb{E}_{\mathcal{Q}^{\prime},\tau,\epsilon}\left[\boldsymbol{x}^{*}_{\text{mask}}\odot\omega(\tau)\left(\epsilon^{*}_{\phi}(\boldsymbol{x}^{*}_{\tau};\boldsymbol{j}^{o},y,\tau,l)-\epsilon\right)\frac{\partial\boldsymbol{x}^{*}}{\partial\Theta}\right],$

where $\mathcal{Q}^{\prime}$ is the sampled $l$-length motion from
$\mathcal{Q}^{t}$ for T2V model, with noise predictor
$\epsilon^{*}_{\phi}(\cdot)$ modified from T2I model $\epsilon_{\phi}(\cdot)$,
$\odot$ indicates Hadamard product.

###  III-E Hierarchical Regularization

To prevent the model from learning noisy geometry, we propose simple yet
effective regularization terms, namely hierarchical geometry regularization,
to improve both the global structure and the fine-grained local details of 3D
geometry. Specifically, $\mathcal{L}_{\text{shp}}$ in Eq. 7 regulates the
shape parameters $\beta$ to ensure a reasonable global structure of the
generated avatar.

$\mathcal{L}_{\text{shp}}=\left\|\beta\right\|_{2}^{2}.$ (7)

In addition, to improve the smoothness of geometry details from per-vertex
displacement $\delta$, we use Laplacian smoothness
$\mathscr{L}(\boldsymbol{m}^{t})$ to regulate $\delta$ as Eq. 8 illustrates.

$\mathcal{L}_{\text{lap}}=\left\|\boldsymbol{m}^{t}-\mathscr{L}(\boldsymbol{m}^{t})\right\|_{2}^{2}.$
(8)

In addition to the aforementioned global constraints, we observe that SDS can
be sensitive to facial structures, especially the eye and mouth regions.
Therefore, in Eq. 9, we use the predefined facial keypoints to preserve the
relative positions and penalize the intersections of eye and lip contours,
such that regulate $\boldsymbol{m}^{t}$ with reasonable facial structure.

$\displaystyle\mathcal{L}_{\text{face}}$
$\displaystyle=\mathds{1}\left(\delta(v^{*}_{u},v^{*}_{l})<0\right)||v^{*}_{u}-v^{*}_{l}||^{2}_{2}$
(9)
$\displaystyle+\mathds{1}(||v^{*}_{e}-v^{*}_{f}||^{2}_{2}<r^{2})||v^{*}_{e}-v^{*}_{f}||^{2}_{2},$

where $v^{*}_{u},v^{*}_{l}$ indicate the paired upper-and-lower lip vertices,
$v^{*}_{e},v^{*}_{f}$ indicate vertices of eyeball and forehead regions,
respectively, $r$ indicates the eyeball radius. $\mathds{1}(\cdot)$ is the
indicate function that penalizes _iff_ $v^{*}_{u}$ lies below the lower lip
$v^{*}_{l}$ or $v^{*}_{e}$ intersected with $v^{*}_{f}$.

Finally, the overall hierarchical geometry regularization
$\mathcal{L}_{\text{reg}}$ is weighted summarize of the above terms,

$\mathcal{L}_{\text{reg}}=\mathcal{L}_{\text{shp}}+\mathcal{L}_{\text{lap}}+\mathcal{L}_{\text{face}},$
(10)

where we eliminate the weights of each term for clarity. The weights are
empirically set to balance the smoothness and diversity, and keep consistent
for different text prompts in our experiments.

###  III-F Overall Optimization Algorithm

Algorithm 1 Training pipeline of STAR for text-to-4D avatar generation.

Input :

Text description $y$

Require :

Learning rate $\eta$; Pretrained T2I prior $\epsilon_{\phi}$; Pretrained
motion generation model $\mathcal{M}$; Pretrained motion retargeting model
$\mathcal{F}$; Training step $T^{*}$

Output :

4D Avatar
$\mathrm{M}^{*}\coloneqq\\{\boldsymbol{m}|\boldsymbol{m}(\Theta^{*}_{T};\boldsymbol{q}),\forall\boldsymbol{q}\in\mathcal{Q}^{t}_{T}\\}$

$\mathcal{Q}^{t}_{0}=\mathcal{Q}_{s}\leftarrow\mathcal{M}(y)$ ;

# Motion Initialization while  _$T\leq T^{*}$_ do

Sample motion $\mathcal{Q}^{\prime}_{T}\subseteq\mathcal{Q}^{t}_{T}$ and
random pose $\boldsymbol{q}_{T}\in\mathcal{Q}^{t}_{T}$; Generate animated 4D
avatar:
$\mathrm{M}_{T}\coloneqq\\{\boldsymbol{m}|\boldsymbol{m}(\Theta_{T};\boldsymbol{q}),\forall\boldsymbol{q}\in\mathcal{Q}^{\prime}_{T}\\}$
;

# Eq. 1 Sample random camera view $\boldsymbol{\pi}$; Render video
$\mathrm{X}_{T}=\\{\boldsymbol{x}|\boldsymbol{x}\coloneqq
g(\Theta_{T};\boldsymbol{q},c),\forall\boldsymbol{q}\in\mathcal{Q}^{\prime}_{T})\\}$;
Calculate gradient for T2V step, s.t. $\mathrm{M}_{T},\mathrm{X}_{T}$:
$\nabla_{\Theta}\mathcal{L}_{\text{T2V}}=\nabla_{\Theta}\mathcal{L}_{\text{SDS}}^{\text{seq}}+\nabla_{\Theta}\mathcal{L}_{\text{reg}}$
;

# Eq. 6 & Eq. 10 Update parameters
$\Theta_{T}\leftarrow\Theta_{T}-\eta\nabla_{\Theta}\mathcal{L}_{\text{T2V}}$;
Sample $n$ camera views
${\Pi}=\\{\boldsymbol{\pi}_{1},\cdots,\boldsymbol{\pi}_{n}\\}$; while
_$\boldsymbol{\pi}\in\Pi$_ do

Sample random pose $\boldsymbol{q}_{T}\in\mathcal{Q}^{t}_{T}$; Generate
animated 3D avatar:
$\boldsymbol{m}_{T}=\boldsymbol{m}(\Theta_{T};\boldsymbol{q}_{T})$ ;

# Eq. 1 Render image $\boldsymbol{x}_{T}\coloneqq
g(\Theta_{T};\boldsymbol{q}_{T},c)$; Calculate gradient for T2I step, s.t.
$\boldsymbol{m}_{T},\boldsymbol{x}_{T}$:
$\nabla_{\Theta}\mathcal{L}_{\text{T2I}}=\nabla_{\Theta}\mathcal{L}_{\text{SDS}}+\nabla_{\Theta}\mathcal{L}_{\text{reg}}$
;

# Eq. 5 & Eq. 10 Update parameters
$\Theta_{T}\leftarrow\Theta_{T}-\eta\nabla_{\Theta}\mathcal{L}_{\text{T2I}}$;

end while

Update motion
$\mathcal{Q}^{t}_{T}\leftarrow\mathcal{Q}^{s}+\delta\mathcal{Q}_{T}$, s.t.
$\Theta_{T}$ ;

# Eq. 4 $T\leftarrow T+1$;

end while

Return :
$\mathrm{M}^{*}\coloneqq\\{\boldsymbol{m}|\boldsymbol{m}(\Theta^{*}_{T};\boldsymbol{q}),\forall\boldsymbol{q}\in\mathcal{Q}^{t}_{T}\\}$.

The overall training pipeline is illustrated in Algo. 1. The proposed STAR
embraces the capability of T2I and T2V priors, with crucial motion-driven,
occlusion-aware, and structure-aware information to provide multi-view and
frame-consistency supervision signals for the SDS process. In this way, STAR
generates high-quality 4D avatar with vivid animations that faithfully aligns
with text descriptions.

##  IV Experiments

###  IV-A Implementation Details

We implement STAR in PyTorch [91] and train the model on a single NVIDIA A100
GPU with 40GB memory. For the SDS guidance, we use Stable Diffusion v1.5 [1]
as T2I and T2V priors, with classifier free guidance [92] set to $100$. We
follow [10] to sample camera poses for both head and body. In each training
step, we leverage the full-body with random camera views for the T2V process
to guarantee the global structure. We also randomly sample heads or bodies for
the T2I process to improve the fidelity of facial appearances. We use Adam
optimizer [93] to train the model for $15,000$ steps. We set $k$-nearest
neighbor as $k=20$ and $50$ for face/body occlusion detection.

###  IV-B Quantitative Comparison

Experiment Setting. We compare STAR with the state-of-the-art (SOTA) methods
[10, 15, 16, 17]. Specifically, we use 26 representative prompts varying from
cartoon characters and realistic humans for quantitative comparisons. For each
prompt, we obtain the “A-pose” avatars of all evaluated methods. For the 4D
avatars, STAR seamlessly generates the 4D avatar end-to-end from the text
description. For previous optimization-by-animation methods, we use the motion
generated from pretrained text-to-motion models [24] to produce animated
results.

Evaluation Metrics. We quantitatively evaluate all methods using the CLIPScore
[94] (based on CLIP ViT-B/16) and VQAScore [95] (based on CLIP-FlanT5-XXL).
CLIPScore evaluates the cosine similarity between the rendered images and the
text descriptions, while VQAScore defines the alignment score with VQA
probability (_i.e_., “Yes” answer to “Does this figure show {text}?”
question). Specifically, we evaluate the avatar fidelity by comparing scores
between the object text and rendered the canonical avatar from $8$ evenly
distributed horizontal views, and the animation fidelity by comparing per-
frame scores between videos and the text description (including motion
description).

TABLE I: Quantitative Comparisons of the Canonical 3D Avatar and 4D Avatar. We
use CLIPScore and VQAScore metrics, higher score indicates better text-3D
alignment.

Evaluation Metric | Canonical 3D Avatar | 4D Avatar
---|---|---
CLIPScore$\uparrow$ | VQAScore$\uparrow$ | CLIPScore$\uparrow$ | VQAScore$\uparrow$
AvatarCraft [17] | 32.09$\pm$2.83 | 0.56$\pm$0.15 | 32.66$\pm$2.16 | 0.49$\pm$0.14
DreamWaltz [15] | 35.74$\pm$2.80 | 0.76$\pm$0.12 | 35.05$\pm$2.89 | 0.57$\pm$0.18
TADA [10] | 36.88$\pm$2.07 | 0.79$\pm$0.16 | 35.57$\pm$1.77 | 0.61$\pm$0.15
HumanGaussian [16] | 34.97$\pm$1.93 | 0.79$\pm$0.06 | 34.87$\pm$2.02 | 0.59$\pm$0.15
STAR (Ours) | 37.28$\pm$2.09 | 0.83$\pm$0.10 | 36.42$\pm$2.52 | 0.62$\pm$0.13

TABLE II: User Study Results. We show the average percentage ($\%$) of each
baseline and our STAR being selected as “best” according to three aspects: Q1.
Geometry Quality, Q2. Appearance Quality, and Q3. Motion Quality.

Methods | Geo. Qual.$\uparrow$ | App. Qual.$\uparrow$ | Mot. Qual.$\uparrow$
---|---|---|---
AvatarCraft [17] | 3.95 | 4.50 | 7.56
DreamWaltz [15] | 23.89 | 22.48 | 23.75
TADA [10] | 17.36 | 17.20 | 16.77
HumanGaussian [16] | 11.32 | 12.65 | 12.43
STAR (Ours) | 43.48 | 43.17 | 39.49

User Study. Since CLIPScore and VQAScore only consider per-frame text-image
similarity, these metrics are not able to estimate the visual quality in the
temporal dimension. To further demonstrate the quality of STAR for 4D avatar
generation, we use Amazon Mechanical Turk (AMT) to enroll $142$ workers with
diverse backgrounds to perform a user study. Each worker is required to rate a
total of $26$ set of generated results. For each sample, we present the 4D
avatar of different methods and corresponding text prompts. The users are
required to pick the best results based on three criteria: Q1. geometry
quality, Q2. appearance quality, and Q3. motion quality.

Results. The results of evaluation metrics and user study are presented in
Tab. I and Tab. II, respectively. Tab. I demonstrates that STAR achieves the
best performance in both metrics. Compared to previous methods which
separately train the canonical avatar and apply animations, our end-to-end
training framework (see Algo. 1) contributes better text-3D alignment. With
the elaborate-designed in-network motion retargeting mechanism, STAR generates
better 3D avatars while ensuring more vivid animations compared to the
previous methods that posse typical optimization-by-animation paradigm. As
shown in Tab. II, the recruited evaluators further confirm that STAR’s avatars
align better with human perception. STAR consistently achieves the highest
agreement on geometry, texture, and motion quality compared to the evaluated
methods.

\begin{overpic}[trim=56.9055pt 284.52756pt 483.69684pt
56.9055pt,clip,width=433.62pt,grid=false]{figure/cmp_cano.jpg} \end{overpic}
Figure 4: Qualitative Comparisons of the Canonical 3D Avatar. For each method,
we showcase examples of different prompts varying from real-world human to
cartoon characters. The proposed STAR exhibits better text-geometry/texture
alignment while ensuring shape variance and view consistency. As result, it
presents visually better results compared to prior art.

\begin{overpic}[trim=0.0pt 0.0pt 284.52756pt
0.0pt,clip,width=433.62pt,grid=false]{figure/cmp_anim1_v2.jpg}\end{overpic}

\begin{overpic}[trim=0.0pt 0.0pt 284.52756pt
0.0pt,clip,width=433.62pt,grid=false]{figure/cmp_anim2_v2.jpg}\end{overpic}

Figure 5: Qualitative Comparisons of 4D Avatars. For each sample, we showcase
random selected frames of each avatar. The proposed STAR generates vivid 4D
avatars which faithfully align with the given text description. The avatars
generated by the comparison methods either show deteriorated body structures
or animation artifacts.

###  IV-C Qualitative Comparison

In addition to the quantitative results, we present qualitative comparisons of
the “A-pose” textured avatars in Fig. 4 with both frontal and back views. Fig.
5 shows 4D avatars with randomly selected frames. The examples in Fig. 4
demonstrate STAR achieves better generation quality in terms of geometry
quality, diversity and text alignment compared to the evaluated methods. Note
that AvatarCraft [17] uses a fixed template human body to train the NeuS [66]
and Instant-NGP [62] representation, therefore lacks shape variance and
fidelity. Although HumanGaussian [16] and DreamWaltz [15] train the 3D static
representation by incorporating the skeleton from canonical pose as
conditions, they still suffer from Janus problem (_e.g_., Groot in different
views) or view-inconsistency (_e.g_., Hulk in different views). TADA [10]
possesses the same 3D representation as STAR, however, it exhibits unnatural
structures of the generated avatars, due to the pose-agnostic optimization
step and missing structural regularization. As a comparison, STAR is conceived
with the crucial thinking that personalized motion plays crucial role in 4D
avatar generation, with the in-optimization motions, hybrid SDS, and
hierarchical regularization, STAR generates high-fidelity 3D avatars and shows
better visualized quality compared to prior arts.

Fig. 5 shows more samples of generated 4D avatar 8 randomly selected frames.
Existing methods with the optimization-by-animation paradigm ignore the
personalized structure of 3D avatar, which in return yields artifacts in the
animated bodies (_e.g_., body piercing with arms). For implicit 3D
representation [43, 62, 66], previous methods [15, 16, 17] require additional
efforts to train deformable fields or build correspondence for animation but
still exhibit blurry, especially with challenging poses (_e.g_., Captain
Marvel’s examples). For explicit 3D representation [29], while such mesh-based
representation is flexible to animate with arbitrary motions without the
requirements of fine-tuning, the animated avatars of TADA [10] are still
unsatisfactory since the canonical avatar do not preserve reasonable body
structures. Simply copying motions introduces additional artifacts. As a
comparison, STAR produces 4D avatars end-to-end, distinguishing the shape-
level and skeleton-level differences through motion retargeting. This allow
STAR to generate reasonable 3D avatars while possessing accurate and vivid 4D
animations.

###  IV-D Ablation Studies

We conduct detailed ablation studies to demonstrate the effectiveness of each
component in STAR. We subsequently compare the results without (w/o) the
following components: A1. w/o hybrid SDS, instead with only T2I-based SDS
(_i.e_., Eq. 5); A2. w/o skeleton occlusion detection and facial mask for our
revised SDS, instead, simply stack T2V and T2I models with skeleton-
conditioned SDS; A3. w/o pose sampling and motions, instead with only
canonical pose and T2I-based SDS; A4. w/o in-network motion retargeting,
instead with fixed initialized motions; A5. w/o hierarchical regularization.
In each ablation, we preserve other parts unchanged and remove only necessary
components for comparison. The qualitative results are shown in Fig. 6.

\begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,width=124.45004pt,grid=false]{figure/ablation_wo_t2v_crop.jpg}\end{overpic}
(a) A1: w/o hybrid SDS

\begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,width=168.58882pt,grid=false]{figure/ablation_wo_mask_v2_crop.jpg}\end{overpic}
(b) A2: w/o revised SDS

\begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,width=118.84586pt,grid=false]{figure/ablation_wo_pose_crop.jpg}\end{overpic}
(c) A3: w/o pose sampling

\begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,clip,width=215.04338pt,grid=false]{figure/ablation_wo_retarget_v2_crop.jpg}\end{overpic}
(d) A4: w/o in-network motion retargeting

\begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,clip,width=196.89427pt,grid=false]{figure/ablation_wo_reg_crop.jpg}\end{overpic}
(e) A5: w/o hierarchical regularization

Figure 6: Ablation Study of STAR. We qualitatively demonstrate the necessity
of each component in STAR. For each ablation, we showcase: Left. Generated 3D
avatar from STAR, Right. Results after removing the stated component.

Fig<EMAIL_ADDRESS>shows the necessity of our hybrid SDS. T2V prior
provides view-consistent supervision and contributes to textural alignment and
fidelity. Without T2V supervision, the generated avatar (_e.g_., “Thin
handsome boy”) exhibits unnatural textures (_e.g_., eyes) and lack fine
details (_e.g_., collars). However, Fig<EMAIL_ADDRESS>demonstrates that simply stacking T2V and T2I is harmful to avatar fidelity,
without the elaborate-designed skeleton-conditioned SDS (Eq. 5) and masked
sequential SDS (Eq. 6), the generated avatars exhibit Janus problems and less
expressiveness of facial appearance (_e.g_., confusion of head for “A cute boy
with facial painting”).

Intuitively, while directly adding motion retargeting can address the
artifacts for pose-hoc animation, our ablation study shows the importance of
in-network motion retargeting in optimization stage. First, Fig.
<EMAIL_ADDRESS>shows that optimization with a fixed canonical pose
is harmful to the body structure. As we discussed before, the lack of pose
diversity yields potential distribution bias of the pretrained diffusion
models. This makes the SDS-based optimization learn flawed geometry and fail
to capture reasonable body scale (_e.g_., disproportionate head and limbs of
“Slim businesswoman”). In addition, Fig<EMAIL_ADDRESS>shows
comparisons of simply sampling the initial motion to improve the pose
diversity for optimization, as attempted by previous methods [11, 69, 27].
However, such intuitive sampling ignores personalized skeleton and geometry,
making the 3D representation learns a template-like geometry that compromises
the mismatched motions (_e.g_., “Tyrion Lannister” in fact is with Dwarfism).
In contrast, STAR progressively updates mesh and motion to achieve diverse 3D
avatar with vivid animations, which better aligns with the given description.

Finally, Fig<EMAIL_ADDRESS>demonstrates the effectiveness of our
simple-yet-effective hierarchical regularization, where the shape constraint
imposes the avatar to follow a reasonable body structure. For example, the
Laplacian smoothness loss contributes to fine details (_e.g_., belts of
“Superman”), and the facial regularization improves the misaligned lips and
intersected eyes.

##  V Limitations & Future Work

Section IV has shown that STAR can produce high-quality 4D avatar. However,
there are still several limitations. For example, as we use SMPL-X [29] as 3D
human prior, STAR cannot represent out-of-distribution text descriptions such
as animals, and the per-vertex displacement is hard to represent hair strands,
eyelashes, and complex accessories. We leave it for future work to integrate
implicit representation for learning more complex avatars.

##  VI Conclusion

In this paper, we propose STAR to generate high-quality avatars with vivid
animations from only text descriptions. STAR is conceived by the crucial
thinking that motion diversity and fidelity play crucial roles in text-based
4D avatar generation. We consider the skeleton and geometry difference between
the source template and the target avatar, by proposing in-network motion
retargeting to progressively optimize geometry, texture, and motion. We also
propose a novel hybrid occlusion-aware SDS by leveraging the power of
skeleton-conditioned T2I and T2V models, with simple-yet-effective
hierarchical regularization to achieve high-fidelity 4D avatar generation
quality. Both quantitative and qualitative experiments demonstrate STAR can
generate vivid 4D avatar and outperforms previous methods. In-depth ablations
show the effectiveness of each component in STAR.

## References

  * [1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _CVPR_ , 2022, pp. 10 684–10 695.
  * [2] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, S. K. S. Ghasemipour, R. G. Lopes, B. K. Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, “Photorealistic text-to-image diffusion models with deep language understanding,” in _NeurIPS_ , 2022, pp. 36 479–36 494.
  * [3] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _CVPR_ , 2019, pp. 4401–4410.
  * [4] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Mcgrew, I. Sutskever, and M. Chen, “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models,” in _ICML_ , ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 16 784–16 804.
  * [5] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “DreamFusion: Text-to-3D using 2D Diffusion,” in _ICLR_ , 2023.
  * [6] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin, “Magic3D: High-resolution text-to-3D content creation,” in _CVPR_ , 2023, pp. 300–309.
  * [7] J. Zhang, X. Li, Z. Wan, C. Wang, and J. Liao, “Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields,” _IEEE Trans. Vis. Comput. Graph._ , 2024.
  * [8] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick, “Zero-1-to-3: Zero-shot one image to 3d object,” in _ICCV_ , 2023, pp. 9298–9309.
  * [9] S. Szymanowicz, C. Rupprecht, and A. Vedaldi, “Splatter image: Ultra-fast single-view 3d reconstruction,” _CVPR_ , 2024.
  * [10] T. Liao, H. Yi, Y. Xiu, J. Tang, Y. Huang, J. Thies, and M. J. Black, “TADA! Text to Animatable Digital Avatars,” in _3DV_ , 2024.
  * [11] N. Kolotouros, T. Alldieck, A. Zanfir, E. G. Bazavan, M. Fieraru, and C. Sminchisescu, “DreamHuman: Animatable 3D Avatars from Text,” in _NeurIPS_ , 2023.
  * [12] X. Huang, R. Shao, Q. Zhang, H. Zhang, Y. Feng, Y. Liu, and Q. Wang, “HumanNorm: Learning normal diffusion model for high-quality and realistic 3d human generation,” in _CVPR_ , 2024.
  * [13] Y. Xu, Z. Yang, and Y. Yang, “SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance,” _arXiv preprint arXiv:2312.08889_ , 2023.
  * [14] Y. Cao, Y.-P. Cao, K. Han, Y. Shan, and K.-Y. K. Wong, “Guide3D: Create 3D Avatars from Text and Image Guidance,” _arXiv preprint arXiv:2308.09705_ , 2023.
  * [15] Y. Huang, J. Wang, A. Zeng, H. CAO, X. Qi, Y. Shi, Z.-J. Zha, and L. Zhang, “DreamWaltz: Make a Scene with Complex 3D Animatable Avatars,” in _NeurIPS_ , 2023.
  * [16] X. Liu, X. Zhan, J. Tang, Y. Shan, G. Zeng, D. Lin, X. Liu, and Z. Liu, “HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting,” in _CVPR_ , 2024.
  * [17] R. Jiang, C. Wang, J. Zhang, M. Chai, M. He, D. Chen, and J. Liao, “AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control,” in _ICCV_ , 2023, pp. 14 371–14 382.
  * [18] J. Tseng, R. Castellon, and K. Liu, “EDGE: Editable dance generation from music,” in _CVPR_ , 2023, pp. 448–458.
  * [19] J. Li, D. Kang, W. Pei, X. Zhe, Y. Zhang, L. Bao, and Z. He, “Audio2Gestures: Generating diverse gestures from audio,” _IEEE Trans. Vis. Comput. Graph._ , 2023.
  * [20] H. Yi, H. Liang, Y. Liu, Q. Cao, Y. Wen, T. Bolkart, D. Tao, and M. J. Black, “Generating holistic 3d human motion from speech,” in _CVPR_ , 2023, pp. 469–480.
  * [21] Z. Cai, W. Yin, A. Zeng, C. Wei, Q. Sun, Y. Wang, H. E. Pang, H. Mei, M. Zhang, L. Zhang, C. C. Loy, L. Yang, and Z. Liu, “SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation,” _NeurIPS_ , 2023.
  * [22] J. Lin, A. Zeng, H. Wang, L. Zhang, and Y. Li, “One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer,” in _CVPR_ , 2023, pp. 21 159–21 168.
  * [23] B. Jiang, X. Chen, W. Liu, J. Yu, G. YU, and T. Chen, “MotionGPT: Human motion as a foreign language,” in _NeurIPS_ , 2023.
  * [24] M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Z. Liu, “MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model,” _TPAMI_ , 2024.
  * [25] X. Gao, Y. Yang, Z. Xie, S. Du, Z. Sun, and Y. Wu, “GUESS: GradUally Enriching SyntheSis for Text-Driven Human Motion Generation,” _IEEE Trans. Vis. Comput. Graph._ , 2024.
  * [26] H. Zhang, B. Chen, H. Yang, L. Qu, X. Wang, L. Chen, C. Long, F. Zhu, D. Du, and M. Zheng, “AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Pose,” in _AAAI_ , vol. 38, no. 7, 2024, pp. 7124–7132.
  * [27] J. Zhang, X. Zhang, H. Zhang, J. H. Liew, C. Zhang, Y. Yang, and J. Feng, “AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text,” _arXiv preprint arXiv:2311.17917_ , 2023.
  * [28] T. Alldieck, H. Xu, and C. Sminchisescu, “imGHUM: Implicit generative models of 3D human shape and articulated pose,” in _ICCV_ , 2021, pp. 5461–5470.
  * [29] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black, “Expressive body capture: 3d hands, face, and body from a single image,” in _CVPR_ , 2019, pp. 10 975–10 985.
  * [30] J. Zhang, J. Weng, D. Kang, F. Zhao, S. Huang, X. Zhe, L. Bao, Y. Shan, J. Wang, and Z. Tu, “Skinned motion retargeting with residual perception of motion semantics & geometry,” in _CVPR_ , 2023, pp. 13 864–13 872.
  * [31] B. Peng, X. Chen, Y. Wang, C. Lu, and Y. Qiao, “Conditionvideo: Training-free condition-guided video generation,” in _AAAI_ , vol. 38, no. 5, 2024, pp. 4459–4467.
  * [32] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _ICCV_ , 2023, pp. 3836–3847.
  * [33] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , 2022.
  * [34] C. Li, C. Zhang, A. Waghwase, L.-H. Lee, F. Rameau, Y. Yang, S.-H. Bae, and C. S. Hong, “Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era,” _arXiv preprint arXiv:2305.06131_ , 2023.
  * [35] H. Lee, M. Savva, and A. X. Chang, “Text-to-3D Shape Generation,” _Computer Graphics Forum_ , vol. 43, no. 2, 2024.
  * [36] J. Liu, X. Huang, T. Huang, L. Chen, Y. Hou, S. Tang, Z. Liu, W. Ouyang, W. Zuo, J. Jiang _et al._ , “A Comprehensive Survey on 3D Content Generation,” _arXiv preprint arXiv:2402.01166_ , 2024.
  * [37] A. Jain, B. Mildenhall, J. T. Barron, P. Abbeel, and B. Poole, “Zero-shot text-guided object generation with dream fields,” in _CVPR_ , 2022, pp. 867–876.
  * [38] N. Mohammad Khalid, T. Xie, E. Belilovsky, and T. Popa, “CLIP-Mesh: Generating textured meshes from text using pretrained image-text models,” in _SIGGRAPH Asia_ , 2022, pp. 25:1–25:8.
  * [39] F. Hong, M. Zhang, L. Pan, Z. Cai, L. Yang, and Z. Liu, “AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars,” _TOG_ , vol. 41, no. 4, pp. 1–19, 2022.
  * [40] S. Aneja, J. Thies, A. Dai, and M. Nießner, “ClipFace: Text-guided Editing of Textured 3D Morphable Models,” in _SIGGRAPH_ , 2023, pp. 70:1–70:11.
  * [41] O. Michel, R. Bar-On, R. Liu, S. Benaim, and R. Hanocka, “Text2Mesh: Text-driven neural stylization for meshes,” in _CVPR_ , 2022, pp. 13 492–13 502.
  * [42] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” in _ECCV_ , ser. Lecture Notes in Computer Science, vol. 12346. Springer, 2020, pp. 405–421.
  * [43] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3D Gaussian splatting for real-time radiance field rendering,” _TOG_ , vol. 42, no. 4, pp. 1–14, 2023.
  * [44] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” in _ICML_ , ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, pp. 8748–8763.
  * [45] H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich, “Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation,” in _CVPR_ , 2023, pp. 12 619–12 629.
  * [46] Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu, “ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation,” _NeurIPS_ , 2023.
  * [47] Z. Pan, J. Lu, X. Zhu, and L. Zhang, “Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping,” in _ICLR_ , 2024.
  * [48] Y. Huang, J. Wang, Y. Shi, B. Tang, X. Qi, and L. Zhang, “DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation,” in _ICLR_ , 2024.
  * [49] R. Chen, Y. Chen, N. Jiao, and K. Jia, “Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3d content creation,” in _ICCV_ , 2023, pp. 22 246–22 256.
  * [50] Y. Shi, P. Wang, J. Ye, L. Mai, K. Li, and X. Yang, “MVDream: Multi-view Diffusion for 3D Generation,” in _ICLR_ , 2024.
  * [51] H. Hu, Z. Zhou, V. Jampani, and S. Tulsiani, “MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation,” in _CVPR_ , 2024.
  * [52] E. Richardson, G. Metzer, Y. Alaluf, R. Giryes, and D. Cohen-Or, “TEXTure: Text-Guided Texturing of 3D Shapes,” in _SIGGRAPH_ , 2023, pp. 54:1–54:11.
  * [53] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “SDXL: Improving latent diffusion models for high-resolution image synthesis,” in _ICLR_ , 2024.
  * [54] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, “Align your latents: High-resolution video synthesis with latent diffusion models,” in _CVPR_ , 2023, pp. 22 563–22 575.
  * [55] Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai, “AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning,” in _ICLR_ , 2024.
  * [56] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans, “Imagen Video: High Definition Video Generation with Diffusion Models,” _arXiv preprint arXiv:2210.02303_ , 2022.
  * [57] D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng, “MagicVideo: Efficient Video Generation With Latent Diffusion Models,” _arXiv preprint arXiv:2211.11018_ , 2022.
  * [58] Y. Liu, C. Lin, Z. Zeng, X. Long, L. Liu, T. Komura, and W. Wang, “SyncDreamer: Generating Multiview-consistent Images from a Single-view Image,” in _ICLR_ , 2024.
  * [59] U. Singer, S. Sheynin, A. Polyak, O. Ashual, I. Makarov, F. Kokkinos, N. Goyal, A. Vedaldi, D. Parikh, J. Johnson, and Y. Taigman, “Text-to-4D dynamic scene generation,” in _ICML_ , ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 31 915–31 929.
  * [60] S. Bahmani, I. Skorokhodov, V. Rong, G. Wetzstein, L. Guibas, P. Wonka, S. Tulyakov, J. J. Park, A. Tagliasacchi, and D. B. Lindell, “4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling,” in _CVPR_ , 2024.
  * [61] H. Ling, S. W. Kim, A. Torralba, S. Fidler, and K. Kreis, “Align your Gaussians: Text-to-4d with dynamic 3D Gaussians and composed diffusion models,” in _CVPR_ , 2024.
  * [62] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” _TOG_ , vol. 41, no. 4, pp. 1–15, 2022.
  * [63] A. Cao and J. Johnson, “Hexplane: A fast representation for dynamic scenes,” in _CVPR_ , 2023, pp. 130–141.
  * [64] J. Luiten, G. Kopanas, B. Leibe, and D. Ramanan, “Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis,” in _3DV_ , 2024.
  * [65] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “SMPL: a skinned multi-person linear model,” _TOG_ , vol. 34, no. 6, pp. 1–16, 2015.
  * [66] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, “NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction,” _NeurIPS_ , pp. 27 171–27 183, 2021.
  * [67] K. Youwang, K. Ji-Yeon, and T.-H. Oh, “CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes,” in _ECCV_ , ser. Lecture Notes in Computer Science, vol. 13663, 2022, pp. 173–191.
  * [68] S. Huang, Z. Yang, L. Li, Y. Yang, and J. Jia, “AvatarFusion: Zero-shot Generation of Clothing-Decoupled 3D Avatars Using 2D Diffusion,” in _ACM MM_ , 2023, pp. 5734–5745.
  * [69] Y. Yuan, X. Li, Y. Huang, S. De Mello, K. Nagano, J. Kautz, and U. Iqbal, “GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning,” in _CVPR_ , 2024.
  * [70] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _ICCV_ , 2023, pp. 3836–3847.
  * [71] Y. Cao, Y.-P. Cao, K. Han, Y. Shan, and K.-Y. K. Wong, “DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models,” in _CVPR_ , 2024.
  * [72] Y. Zeng, Y. Lu, X. Ji, Y. Yao, H. Zhu, and X. Cao, “AvatarBooth: High-Quality and Customizable 3D Human Avatar Generation,” _arXiv preprint arXiv:2306.09864_ , 2023.
  * [73] R. A. Güler, N. Neverova, and I. Kokkinos, “DensePose: Dense Human Pose Estimation in the Wild,” in _CVPR_ , 2018, pp. 7297–7306.
  * [74] M. Petrovich, M. J. Black, and G. Varol, “Action-conditioned 3d human motion synthesis with transformer vae,” in _ICCV_ , 2021, pp. 10 985–10 995.
  * [75] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-or, and A. H. Bermano, “Human motion diffusion model,” in _ICLR_ , 2023.
  * [76] Y. Yuan and K. Kitani, “DLow: Diversifying Latent Flows for Diverse Human Motion Prediction,” in _ECCV_ , ser. Lecture Notes in Computer Science, vol. 12354. Springer, 2020, pp. 346–364.
  * [77] Y. Shafir, G. Tevet, R. Kapon, and A. H. Bermano, “Human motion diffusion as a generative prior,” in _ICLR_ , 2024.
  * [78] M. Petrovich, M. J. Black, and G. Varol, “TEMOS: Generating diverse human motions from textual descriptions,” in _ECCV_ , ser. Lecture Notes in Computer Science, vol. 13682. Springer, 2022, pp. 480–497.
  * [79] C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng, “Generating diverse and natural 3d human motions from text,” in _CVPR_ , 2022, pp. 5152–5161.
  * [80] C. Ahuja and L.-P. Morency, “Language2pose: Natural language grounded pose forecasting,” in _3DV_ , 2019, pp. 719–728.
  * [81] M. Gleicher, “Retargetting motion to new characters,” in _SIGGRAPH_ , 1998, pp. 33–42.
  * [82] Y. Kim, H. Park, S. Bang, and S.-H. Lee, “Retargeting human-object interaction to virtual avatars,” _IEEE Trans. Vis. Comput. Graph._ , vol. 22, no. 11, pp. 2405–2412, 2016.
  * [83] Y. Pan, S. Tan, S. Cheng, Q. Lin, Z. Zeng, and K. Mitchell, “Expressive talking avatars,” _IEEE Trans. Vis. Comput. Graph._ , 2024.
  * [84] B. Ji, Y. Pan, Y. Yan, R. Chen, and X. Yang, “StyleVR: Stylizing Character Animations with Normalizing Flows,” _IEEE Trans. Vis. Comput. Graph._ , 2023.
  * [85] J. Lim, H. J. Chang, and J. Y. Choi, “PMnet: Learning of Disentangled Pose and Movement for Unsupervised Motion Retargeting,” in _BMVC_ , 2019, p. 136.
  * [86] R. Villegas, J. Yang, D. Ceylan, and H. Lee, “Neural kinematic networks for unsupervised motion retargetting,” in _CVPR_ , 2018, pp. 8639–8648.
  * [87] K. Aberman, P. Li, D. Lischinski, O. Sorkine-Hornung, D. Cohen-Or, and B. Chen, “Skeleton-aware networks for deep motion retargeting,” _TOG_ , vol. 39, no. 4, pp. 62–1, 2020.
  * [88] L. Hu, Z. Zhang, C. Zhong, B. Jiang, and S. Xia, “Pose-aware attention network for flexible motion retargeting by body part,” _IEEE Trans. Vis. Comput. Graph._ , no. 1, pp. 1–17, 2023.
  * [89] J. Lewis, M. Cordner, and N. Fong, “Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation,” in _SIGGRAPH_ , 2000, pp. 165–172.
  * [90] J. Johnson, N. Ravi, J. Reizenstein, D. Novotny, S. Tulsiani, C. Lassner, and S. Branson, “Accelerating 3D Deep Learning with PyTorch3D,” in _SIGGRAPH Asia 2020 courses_ , 2020, pp. 1–1.
  * [91] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _NeurIPS_ , 2019.
  * [92] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in _NeurIPS Workshop on Deep Generative Models and Downstream Applications_ , 2021.
  * [93] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2015.
  * [94] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi, “CLIPScore: a reference-free evaluation metric for image captioning,” in _EMNLP_ , 2021.
  * [95] Z. Lin, D. Pathak, B. Li, J. Li, X. Xia, G. Neubig, P. Zhang, and D. Ramanan, “Evaluating text-to-visual generation with image-to-text generation,” _arXiv preprint arXiv:2404.01291_ , 2024.


#### 37 LinExpr Class

LinExpr object contains operations related to variables for building linear
constraints, and provides the following methods:

##### LinExpr()

> Synopsis
>

>> LinExpr(arg1=0.0,arg2=None)

>
> Description
>

>> Create a _LinExpr Class_ object.

>>

>> If argument arg1 is constant, argument arg2 is None, then create a _LinExpr
Class_ object and initialize it using argument arg1. If argument arg1 is _Var
Class_ or _LinExpr Class_ object, and argument arg2 is constant or considered
to be constant 1.0 when argument arg2 is None, then initialize the newly
created _LinExpr Class_ object using arguments arg1 and arg2. If argument arg1
is list object and argument arg2 is None, then argument arg1 contains a list
of variable-coefficient pairs and initialize the newly created _LinExpr Class_
object using arguments arg1 and arg2. For other forms of arguments, call
method addTerms to initialize the newly created _LinExpr Class_ object.

>
> Arguments
>

>> arg1

>>

>>> Optional, 0.0 by default.

>>

>> arg2

>>

>>> Optional, None by default.

>
> Example

  * #CreateanewLinExprobjectandinitializeitto0.0 expr0=LinExpr() #CreateaLinExprobjectandinitializeitto2*x+3*y expr1=LinExpr([(x,2),(y,3)]) #CreateaLinExprobjectandinitializeittox+2*y expr2=LinExpr([x,y],[1,2])

##### LinExpr.setCoeff()

> Synopsis
>

>> setCoeff(idx,newval)

>
> Description
>

>> Set new coefficient value of a variable based on its index in an
expression.

>
> Arguments
>

>> idx

>>

>>> Index of the variable in the expression, starting with 0.

>>

>> newval

>>

>>> New coefficient value of the variable.

>
> Example

  * #Setthecoefficientforthetermatindex0inexpressionexprto1.0 expr.setCoeff(0,1.0)

##### LinExpr.getCoeff()

> Synopsis
>

>> getCoeff(idx)

>
> Description
>

>> Retrieve the coefficient of a variable by its index from an expression.

>
> Arguments
>

>> idx

>>

>>> Index of the variable in the expression, starting with 0.

>
> Example

  * #Retrievethecoefficientforthetermatindex1fromexpressionexpr coeff=expr.getCoeff(1)

##### LinExpr.getVar()

> Synopsis
>

>> getVar(idx)

>
> Description
>

>> Retrieve the variable by its index from an expression. Return a _Var Class_
object.

>
> Arguments
>

>> idx

>>

>>> Index of the variable in the expression, starting with 0.

>
> Example

  * #Retrievethevariableforthetermatindex1fromexpressionexpr x=expr.getVar(1)

##### LinExpr.getConstant()

> Synopsis
>

>> getConstant()

>
> Description
>

>> Retrieve the constant term from an expression.

>
> Example

  * #Retrievetheconstanttermfromlinearexpressionexpr constant=expr.getConstant()

##### LinExpr.getValue()

> Synopsis
>

>> getValue()

>
> Description
>

>> Retrieve the value of an expression computed using the current solution.

>
> Example

  * #Retrievethevalueofexpressionexprforthecurrentsolution. val=expr.getValue()

##### LinExpr.getSize()

> Synopsis
>

>> getSize()

>
> Description
>

>> Retrieve the number of terms in an expression.

>
> Example

  * #Retrievethenumberoftermsinexpressionexpr exprsize=expr.getSize()

##### LinExpr.setConstant()

> Synopsis
>

>> setConstant(newval)

>
> Description
>

>> Set the constant term of linear expression.

>
> Arguments
>

>> newval

>>

>>> Constant term to be set.

>
> Example

  * #Setconstanttermoflinearexpression'expr'to2.0 expr.setConstant(2.0)

##### LinExpr.addConstant()

> Synopsis
>

>> addConstant(newval)

>
> Description
>

>> Add a constant to an expression.

>
> Arguments
>

>> newval

>>

>>> Constant to add.

>
> Example

  * #Addconstant2.0tolinearexpression'expr' expr.addConstant(2.0)

##### LinExpr.addTerm()

> Synopsis
>

>> addTerm(var,coeff=1.0)

>
> Description
>

>> Add a new term to current expression.

>
> Arguments
>

>> var

>>

>>> Variable to add.

>>

>> coeff

>>

>>> Magnification coefficient for added term. Optional, 1.0 by default.

>
> Example

  * #Addtermxtolinearexpression'expr' expr.addTerm(x)

##### LinExpr.addTerms()

> Synopsis
>

>> addTerms(vars,coeffs)

>
> Description
>

>> Add a single term or multiple terms into an expression.

>>

>> If argument vars is _Var Class_ object, then argument coeffs is constant;
If argument vars is _VarArray Class_ object or list, then argument coeffs is
constant or list; If argument vars is dictionary or _tupledict Class_ object,
then argument coeffs is constant, dict, or _tupledict Class_ object.

>
> Arguments
>

>> vars

>>

>>> Variables to add.

>>

>> coeffs

>>

>>> Coefficients for variables.

>
> Example

  * #Addterm2*x+2*ytolinearexpression'expr' expr.addTerms([x,y],[2.0,3.0])

##### LinExpr.addExpr()

> Synopsis
>

>> addExpr(expr,coeff=1.0)

>
> Description
>

>> Add new expression to the current one.

>
> Arguments
>

>> expr

>>

>>> Expression or expression builder to add.

>>

>> coeff

>>

>>> Magnification coefficients for the added expression. Optional, 1.0 by
default.

>
> Example

  * #Addlinearexpression2*x+2*yto'expr' expr.addExpr(x+y,2.0)

##### LinExpr.clone()

> Synopsis
>

>> clone()

>
> Description
>

>> Create a deep copy of the expression.

>
> Example

  * #Createadeepcopyofexpressionexpr exprcopy=expr.clone()

##### LinExpr.reserve()

> Synopsis
>

>> reserve(n)

>
> Description
>

>> Pre-allocate space for linear expression object.

>
> Arguments
>

>> n

>>

>>> Number of terms to be allocated.

>
> Example

  * #Allocate100termsforlinearexpression'expr' expr.reserve(100)

##### LinExpr.remove()

> Synopsis
>

>> remove(item)

>
> Description
>

>> Remove a term from a linear expression.

>>

>> If argument item is constant, then remove the term stored at index i of the
expression; otherwise argument item is _Var Class_ object.

>
> Arguments
>

>> item

>>

>>> Constant index or variable of the term to be removed.

>
> Example

  * #Removethetermwhoseindexis2fromlinearexpressionexpr expr.remove(2) #Removethetermwhosevariableisxfromlinearexpressionexpr expr.remove(x)

#### 38 QuadExpr Class

QuadExpr object contains operations related to variables for building linear
constraints, and provides the following methods:

##### QuadExpr()

> Synopsis
>

>> QuadExpr(expr=0.0)

>
> Description
>

>> Create a _QuadExpr Class_ object.

>>

>> Argument expr is constant, _Var Class_, _LinExpr Class_ object or _QuadExpr
Class_ object.

>
> Arguments
>

>> expr

>>

>>> Optional, 0.0 by default.

>
> Example

  * #CreateanewQuadExprobjectandinitializeitto0.0 quadexpr0=QuadExpr() #CreateaQuadExprobjectandinitializeitto2*x+3*y quadexpr1=QuadExpr([(x,2),(y,3)]) #CreateaQuadExprobjectandinitializeittox*x+2*x*y quadexpr2=QuadExpr(x*x+2*x*y)

##### QuadExpr.setCoeff()

> Synopsis
>

>> setCoeff(idx,newval)

>
> Description
>

>> Set new coefficient value of a term based on its index in a quadratic
expression.

>
> Arguments
>

>> idx

>>

>>> Index of the term in the quadratic expression, starting with 0.

>>

>> newval

>>

>>> New coefficient value of the term.

>
> Example

  * #Setthecoefficientforthetermatindex0inquadraticexpressionquadexprto1.0 quadexpr.setCoeff(0,1.0)

##### QuadExpr.getCoeff()

> Synopsis
>

>> getCoeff(idx)

>
> Description
>

>> Retrieve the coefficient of a term by its index from a quadratic
expression.

>
> Arguments
>

>> idx

>>

>>> Index of the term in the quadratic expression, starting with 0.

>
> Example

  * #Retrievethecoefficientforthetermatindex1fromquadraticexpressionquadexpr coeff=quadexpr.getCoeff(1)

##### QuadExpr.getVar1()

> Synopsis
>

>> getVar1(idx)

>
> Description
>

>> Retrieve the first variable of a quadratic term by its index from an
expression. Return a _Var Class_ object.

>
> Arguments
>

>> idx

>>

>>> Index of the quadratic term in the expression, starting with 0.

>
> Example

  * #Retrievethefirstvariableofaquadratictermatindex1fromquadraticexpressionquadexpr x=quadexpr.getVar1(1)

##### QuadExpr.getVar2()

> Synopsis
>

>> getVar2(idx)

>
> Description
>

>> Retrieve the second variable of a quadratic term by its index from an
expression. Return a _Var Class_ object.

>
> Arguments
>

>> idx

>>

>>> Index of the quadratic term in the expression, starting with 0.

>
> Example

  * #Retrievethefirstvariableofaquadratictermatindex1fromquadraticexpressionquadexpr y=quadexpr.getVar2(1)

##### QuadExpr.getLinExpr()

> Synopsis
>

>> getLinExpr()

>
> Description
>

>> Retrieve the linear terms (if exist) fronm quadratic expression. Return a
_LinExpr Class_ object.

>
> Example

  * #Retrievethelineartermsfromaquadraticexpressionquadexpr linexpr=quadexpr.getLinExpr()

##### QuadExpr.getConstant()

> Synopsis
>

>> getConstant()

>
> Description
>

>> Retrieve the constant term from a quadratic expression.

>
> Example

  * #Retrievetheconstanttermfromquadraticexpressionquadexpr constant=quadexpr.getConstant()

##### QuadExpr.getValue()

> Synopsis
>

>> getValue()

>
> Description
>

>> Retrieve the value of a quadratic expression computed using the current
solution.

>
> Example

  * #Retrievethevalueofquadraticexpressionquadexprforthecurrentsolution. val=quadexpr.getValue()

##### QuadExpr.getSize()

> Synopsis
>

>> getSize()

>
> Description
>

>> Retrieve the number of terms in a quadratic expression.

>
> Example

  * #Retrievethenumberoftermsinquadraticexpressionquadexpr exprsize=quadexpr.getSize()

##### QuadExpr.setConstant()

> Synopsis
>

>> setConstant(newval)

>
> Description
>

>> Set the constant term of quadratic expression.

>
> Arguments
>

>> newval

>>

>>> Constant to set.

>
> Example

  * #Setconstanttermofquadraticexpression'quadexpr'to2.0 quadexpr.setConstant(2.0)

##### QuadExpr.addConstant()

> Synopsis
>

>> addConstant(newval)

>
> Description
>

>> Add a constant to a quadratic expression.

>
> Arguments
>

>> newval

>>

>>> Constant to add.

>
> Example

  * #Addconstant2.0toquadraticexpression'quadexpr' quadexpr.addConstant(2.0)

##### QuadExpr.addTerm()

> Synopsis
>

>> addTerm(coeff,var1,var2=None)

>
> Description
>

>> Add a new term to current quadratic expression.

>
> Arguments
>

>> coeff

>>

>>> Magnification coefficient for added term. Optional, 1.0 by default.

>>

>> var1

>>

>>> The first variable for added term.

>>

>> var2

>>

>>> The second variable for added term, defaults to None, i.e. add a linear
term.

>
> Example

  * #Addtermxtoquadraticexpression'quadexpr' quadexpr.addTerm(1.0,x)

##### QuadExpr.addTerms()

> Synopsis
>

>> addTerms(coeffs,vars1,vars2=None)

>
> Description
>

>> Add a single term or multiple terms into a quadratic expression.

>>

>> If argument vars is _Var Class_ object, then argument vars2 is _Var Class_
object or None, argument coeffs is constant; If argument vars is _VarArray
Class_ object or list, then argument vars2 is _VarArray Class_ object, list or
None, argument coeffs is constant or list; If argument vars is dictionary or
_tupledict Class_ object, then argument vars2 is dictionary, _tupledict Class_
object or None, argument coeffs is constant, dictionary, or _tupledict Class_
object.

>
> Arguments
>

>> coeffs

>>

>>> Coefficients for terms.

>>

>> vars1

>>

>>> The first variable of each term.

>>

>> vars2

>>

>>> The second variable of each term, defaults to None, i.e. add a linear
term.

>
> Example

  * #Addterm2*x+3y+2*x*x+3*x*ytoquadraticexpression'quadexpr' #Note:MixedformatissupportedbyaddTermsyet. quadexpr.addTerms([2.0,3.0],[x,y]) quadexpr.addTerms([2.0,3.0],[x,x],[x,y])

##### QuadExpr.addLinExpr()

> Synopsis
>

>> addLinExpr(expr,mult=1.0)

>
> Description
>

>> Add new linear expression to the current quadratic expression.

>
> Arguments
>

>> expr

>>

>>> Linear expression or linear expression builder to add.

>>

>> mult

>>

>>> Magnification coefficient for the added expression. Optional, 1.0 by
default.

>
> Example

  * #Addlinearexpression2*x+2*yto'quadexpr' quadexpr.addLinExpr(x+y,2.0)

##### QuadExpr.addQuadExpr()

> Synopsis
>

>> addQuadExpr(expr,mult=1.0)

>
> Description
>

>> Add new quadratic expression to the current one.

>
> Arguments
>

>> expr

>>

>>> Expression or expression builder to add.

>>

>> mult

>>

>>> Magnification coefficients for the added expression. Optional, 1.0 by
default.

>
> Example

  * #Addquadraticexpressionx*x+2*yto'quadexpr' quadexpr.addQuadExpr(x*x+2*y,2.0)

##### QuadExpr.clone()

> Synopsis
>

>> clone()

>
> Description
>

>> Create a deep copy of the expression.

>
> Example

  * #Createadeepcopyofquadraticexpressionquadexpr exprcopy=quadexpr.clone()

##### QuadExpr.reserve()

> Synopsis
>

>> reserve(n)

>
> Description
>

>> Pre-allocate space for quadratic expression object.

>
> Arguments
>

>> n

>>

>>> Number of terms to be allocated.

>
> Example

  * #Allocate100termsforquadraticexpression'expr' expr.reserve(100)

##### QuadExpr.remove()

> Synopsis
>

>> remove(item)

>
> Description
>

>> Remove a term from a quadratic expression.

>>

>> If argument item is constant, then remove the term stored at index i of the
expression; otherwise argument item is _Var Class_ object.

>
> Arguments
>

>> item

>>

>>> Constant index or variable of the term to be removed.

>
> Example

  * #Removethetermwhoseindexis2fromquadraticexpressionquadexpr quadexpr.remove(2) #Removethetermsoneofwhichvariableisxfromquadraticexpressionquadexpr quadexpr.remove(x)

#### 39 PsdExpr Class

PsdExpr object contains operations related to variables for building positive
semi-definite constraints, and provides the following methods:

##### PsdExpr()

> Synopsis
>

>> PsdExpr(expr=0.0)

>
> Description
>

>> Create a _PsdExpr Class_ object.

>
> Arguments
>

>> expr

>>

>>> Optional, 0.0 by default. Can be a constant, _Var Class_, _LinExpr Class_
object or _PsdExpr Class_ object.

>
> Example

  * #CreateanewPsdExprobjectandinitializeitto0.0 expr0=PsdExpr() #CreateaPsdExprobjectandinitializeitto2*x+3*y expr1=PsdExpr(2*x+3*y)

##### PsdExpr.setCoeff()

> Synopsis
>

>> setCoeff(idx,mat)

>
> Description
>

>> Set new symmetric matrix coefficient for a positive semi-definite variable
based on its index in the expression.

>
> Arguments
>

>> idx

>>

>>> Index of the positive semi-definite variable in the expression, starting
with 0.

>>

>> mat

>>

>>> New symmetric matrix coefficient of the positive semi-definite variable.

>
> Example

  * #Setsymmetricmatrixforthepositivesemi-definitevariableatindex0inexpression"expr"tomat expr.setCoeff(0,mat)

##### PsdExpr.getCoeff()

> Synopsis
>

>> getCoeff(idx)

>
> Description
>

>> Retrieve the symmetric matrix coefficient of a positive semi-definite
variable by its index from the expression.

>
> Arguments
>

>> idx

>>

>>> Index of the positive semi-definite variable in the expression, starting
with 0.

>
> Example

  * #Retrievethesymmetricmatrixcoefficientforthepositivesemi-definitevariableatindex1fromexpressionexpr mat=expr.getCoeff(1)

##### PsdExpr.getPsdVar()

> Synopsis
>

>> getPsdVar(idx)

>
> Description
>

>> Retrieve a positive semi-definite variable by its index from the
expression. Return a _PsdVar Class_ object.

>
> Arguments
>

>> idx

>>

>>> Index of the positive semi-definite variable in the expression, starting
with 0.

>
> Example

  * #Retrievethepositivesemi-definitevariableatindex1fromexpressionexpr x=expr.getPsdVar(1)

##### PsdExpr.getLinExpr()

> Synopsis
>

>> getLinExpr()

>
> Description
>

>> Retrieve the linear terms (if exist) fronm positive semi-definite
expression. Return a _LinExpr Class_ object.

>
> Example

  * #Retrievethelineartermsfromapositivesemi-definiteexpressionexpr linexpr=expr.getLinExpr()

##### PsdExpr.getConstant()

> Synopsis
>

>> getConstant()

>
> Description
>

>> Retrieve the constant term from a positive semi-definite expression.

>
> Example

  * #Retrievetheconstanttermfromexpressionexpr constant=expr.getConstant()

##### PsdExpr.getValue()

> Synopsis
>

>> getValue()

>
> Description
>

>> Retrieve the value of a positive semi-definite expression computed using
the current solution.

>
> Example

  * #Retrievethevalueofpositivesemi-definiteexpressionexprforthecurrentsolution. val=expr.getValue()

##### PsdExpr.getSize()

> Synopsis
>

>> getSize()

>
> Description
>

>> Retrieve the number of terms in a positive semi-definite expression.

>
> Example

  * #Retrievethenumberoftermsinexpressionexpr exprsize=expr.getSize()

##### PsdExpr.setConstant()

> Synopsis
>

>> setConstant(newval)

>
> Description
>

>> Set the constant term of positive semi-definite expression.

>
> Arguments
>

>> newval

>>

>>> Constant to set.

>
> Example

  * #Setconstanttermofexpression'expr'to2.0 expr.setConstant(2.0)

##### PsdExpr.addConstant()

> Synopsis
>

>> addConstant(newval)

>
> Description
>

>> Add a constant to a positive semi-definite expression.

>
> Arguments
>

>> newval

>>

>>> Constant to add.

>
> Example

  * #Addconstant2.0toexpression'expr' expr.addConstant(2.0)

##### PsdExpr.addTerm()

> Synopsis
>

>> addTerm(var,mat)

>
> Description
>

>> Add a new term to current positive semi-definite expression.

>
> Arguments
>

>> var

>>

>>> The positive semi-definite variable to add.

>>

>> mat

>>

>>> The symmetric matrix coefficient for the positive semi-definite variable.

>
> Example

  * #Addpositivesemi-definitetermC1*Xtoexpression'expr' expr.addTerm(X,C1)

##### PsdExpr.addTerms()

> Synopsis
>

>> addTerms(vars,mats)

>
> Description
>

>> Add a single term or multiple positive semi-definite terms into a positive
semi-definite expression.

>>

>> If argument vars is _PsdVar Class_ object, then argument mats is _SymMatrix
Class_ object; If argument vars is _PsdVarArray Class_ object or list, then
argument mats is _SymMatrixArray Class_ or list;

>
> Arguments
>

>> vars

>>

>>> The positive semi-definite variables to add.

>>

>> mats

>>

>>> The symmetric matrices of the positive semi-definite terms.

>
> Example

  * #AddtermsC1*X1+C2*X2toexpression'expr' expr.addTerms([X1,X2],[C1,C2])

##### PsdExpr.addLinExpr()

> Synopsis
>

>> addLinExpr(expr,mult=1.0)

>
> Description
>

>> Add new linear expression to the current positive semi-definite expression.

>
> Arguments
>

>> expr

>>

>>> Linear expression or linear expression builder to add.

>>

>> mult

>>

>>> Magnification coefficient for the added expression. Optional, 1.0 by
default.

>
> Example

  * #Addlinearexpression2*x+2*yto'expr' expr.addLinExpr(x+y,2.0)

##### PsdExpr.addPsdExpr()

> Synopsis
>

>> addPsdExpr(expr,mult=1.0)

>
> Description
>

>> Add new positive semi-definite expression to the current one.

>
> Arguments
>

>> expr

>>

>>> Positive semi-definite expression or positive semi-definite expression
builder to add.

>>

>> mult

>>

>>> Magnification coefficient for the added positive semi-definite expression.
Optional, 1.0 by default.

>
> Example

  * #Addpositivesemi-definiteexpressionC*Xto'expr' expr.addPsdExpr(C*X)

##### PsdExpr.clone()

> Synopsis
>

>> clone()

>
> Description
>

>> Create a deep copy of the expression.

>
> Example

  * #Createadeepcopyofexpressionexpr exprcopy=expr.clone()

##### PsdExpr.reserve()

> Synopsis
>

>> reserve(n)

>
> Description
>

>> Pre-allocate space for positive semi-definite expression object.

>
> Arguments
>

>> n

>>

>>> Number of terms to be allocated.

>
> Example

  * #Allocate100termsforpositivesemi-definiteexpression'expr' expr.reserve(100)

##### PsdExpr.remove()

> Synopsis
>

>> remove(item)

>
> Description
>

>> Remove a term from a positive semi-definite expression.

>>

>> If argument item is constant, then remove the term stored at index i of the
expression; otherwise argument item is _PsdVar Class_ object.

>
> Arguments
>

>> item

>>

>>> Constant index or _PsdVar Class_ variable of the term to be removed.

>
> Example

  * #Removethetermwhoseindexis2frompositivesemi-definiteexpressionexpr expr.remove(2) #Removethetermsoneofwhichvariableisxfrompositivesemi-definiteexpressionexpr expr.remove(x)

#### 40 CoptError Class

CoptError Class provides operations on error. An exception of the CoptError is
thrown when error occurs in a method call corresponding to the underlying
interface of solver. The following attributes are provided to retrieve error
information:

  * •

CoptError.retcode

> Error code.

  * •

CoptError.message

> Error message.

### 3 Helper Functions and Utilities

Helper functions and utilities are encapsulated based on Python’s basic data
types, providing easy-to-use data types to facilitate the rapid construction
of complex optimization models. This section will explain its functions and
usages.

#### 1 Helper Functions

##### multidict()

> Synopsis
>

>> multidict(data)

>
> Description
>

>> Split a single dictionary into keys and multiple dictionaries. Return keys
and dictionaries.

>
> Arguments
>

>> data

>>

>>> A Python dictionary to be applied. Each key should map to a list of $n$
values.

>
> Example

  * keys,dict1,dict2=multidict({ "hello":[0,1], "world":[2,3]})

##### quicksum()

> Synopsis
>

>> quicksum(data)

>
> Description
>

>> Build expressions efficiently. Return a _LinExpr Class_ object.

>
> Arguments
>

>> data

>>

>>> Terms to add.

>
> Example

  * expr=quicksum(m.getVars())

#### 2 tuplelist Class

The tuplelist object is an encapsulation based on Python lists, and provides
the following methods:

##### tuplelist()

> Synopsis
>

>> tuplelist(list)

>
> Description
>

>> Create and return a _tuplelist Class_ object.

>
> Arguments
>

>> list

>>

>>> A Python list.

>
> Example

  * tl=tuplelist([(0,1),(1,2)]) tl=tuplelist([('a','b'),('b','c')])

##### tuplelist.add()

> Synopsis
>

>> add(item)

>
> Description
>

>> Add an item to a _tuplelist Class_ object

>
> Arguments
>

>> item

>>

>>> Item to add. Can be a Python tuple.

>
> Example

  * tl=tuplelist([(0,1),(1,2)]) tl.add((2,3))

##### tuplelist.select()

> Synopsis
>

>> select(pattern)

>
> Description
>

>> Get all terms that match the specified pattern. Return a _tuplelist Class_
object.

>
> Arguments
>

>> pattern

>>

>>> Specified pattern.

>
> Example

  * tl=tuplelist([(0,1),(0,2),(1,2)]) tl.select(0,'*')

#### 3 tupledict Class

The tupledict class is an encapsulation based on Python dictionaries, and
provides the following methods:

##### tupledict()

> Synopsis
>

>> tupledict(args,kwargs)

>
> Description
>

>> Create and return a _tupledict Class_ object.

>
> Arguments
>

>> args

>>

>>> Positional arguments.

>>

>> kwargs

>>

>>> Named arguments.

>
> Example

  * d=tupledict([(0,"hello"),(1,"world")])

##### tupledict.select()

> Synopsis
>

>> select(pattern)

>
> Description
>

>> Get all terms that match the specified pattern. Return a _tupledict Class_
object.

>
> Arguments
>

>> pattern

>>

>>> Specified pattern.

>
> Example

  * d=tupledict([(0,"hello"),(1,"world")]) d.select()

##### tupledict.sum()

> Synopsis
>

>> sum(pattern)

>
> Description
>

>> Sum all terms that match the specified pattern. Return a _LinExpr Class_
object.

>
> Arguments
>

>> pattern

>>

>>> Specified pattern.

>
> Example

  * expr=x.sum()

##### tupledict.prod()

> Synopsis
>

>> prod(coeff,pattern)

>
> Description
>

>> Filter terms that match the specified pattern and multiply by coefficients.
Return a _LinExpr Class_ object.

>
> Arguments
>

>> coeff

>>

>>> Coefficients. Can be a dict or a _tupledict Class_ object.

>>

>> pattern

>>

>>> Specified pattern.

>
> Example

  * coeff=dict([(1,0.1),(2,0.2)]) expr=x.prod(coeff)

#### 4 ProbBuffer Class

The ProbBuffer is an encapsulation of buffer of string stream, and provides
the following methods:

##### ProbBuffer()

> Synopsis
>

>> ProbBuffer(buff)

>
> Description
>

>> Create and return a _ProbBuffer Class_ object.

>
> Arguments
>

>> buff

>>

>>> Size of buffer, defaults to None, i.e. the buffer size is 0.

>
> Example

  * #Createabufferofsize100 buff=ProbBuffer(100)

##### ProbBuffer.getData()

> Synopsis
>

>> getData()

>
> Description
>

>> Get the contents of buffer.

>
> Example

  * #Printthecontentsinbuffer print(buff.getData())

##### ProbBuffer.getSize()

> Synopsis
>

>> getSize()

>
> Description
>

>> Get the size of the buffer.

>
> Example

  * #Getthesizeofthebuffer print(buff.getSize())

##### ProbBuffer.resize()

> Synopsis
>

>> resize(sz)

>
> Description
>

>> Resize the size of the buffer.

>
> Arguments
>

>> sz

>>

>>> New size of buffer.

>
> Example

  * #Resizethesizeofbufferto100 buff.resize(100)

## Index