# Text-Driven Diverse Facial Texture Generation
via Progressive Latent-Space Refinement

Chi Wang12, Junming Huang12, Rong Zhang3, Qi Wang1, Haotian Yang2,
Haibin Huang2, Chongyang Ma2, Weiwei Xu1
1 State Key Lab of CAD&CG, Zhejiang University
2 Kuaishou Technology 3 Zhejiang Gongshang University

###### Abstract

Automatic 3D facial texture generation has gained significant interest
recently. Existing approaches may not support the traditional physically based
rendering pipeline or rely on 3D data captured by Light Stage. Our key
contribution is a progressive latent space refinement approach that can
bootstrap from 3D Morphable Models (3DMMs)-based texture maps generated from
facial images to generate high-quality and diverse PBR textures, including
albedo, normal, and roughness. It starts with enhancing Generative Adversarial
Networks (GANs) for text-guided and diverse texture generation. To this end,
we design a self-supervised paradigm to overcome the reliance on ground truth
3D textures and train the generative model with only entangled texture maps.
Besides, we foster mutual enhancement between GANs and Score Distillation
Sampling (SDS). SDS boosts GANs with more generative modes, while GANs promote
more efficient optimization of SDS. Furthermore, we introduce an edge-aware
SDS for multi-view consistent facial structure. Experiments demonstrate that
our method outperforms existing 3D texture generation methods regarding photo-
realistic quality, diversity, and efficiency.

Figure 1: Our method can faithfully generate a variety of facial textures from
text prompts for photo-realistic rendering. From left to right: (a)
unconditional generation results, (b) multi-view rendering results using our
generated PBR texture from an uncommon prompt “scar”, (c) relighting results
using our generated PBR texture of “Barack Obama”.

##  1 Introduction

3D facial texture generation has gained widespread attention recently for its
potential to boost industries, including Augmented Reality (AR), Virtual
Reality (VR), gaming, and films. The traditional pipeline of creating textures
typically involves UV mapping, material production, or texture synthesis.
Recently, with the advancement of large-scale pre-trained models, such as
Stable Diffusion [49] and the Contrastive Language-Image Pre-Training (CLIP)
model, textures can be generated through a more flexible and efficient
pathway,  _i.e_. , text-driven texture generation. Pioneers have made
effective attempts to obtain facial textures under the guidance of text
prompts [2, 23]. However, these methods encounter challenges in generating
disentangled UV texture maps, such as albedo, roughness, and normal, which are
essential to physically based realistic rendering.

This paper focuses on text-driven physically based rendering (PBR) texture
generation to achieve flexible and high-fidelity 3D face creation. Previous
researches employ CLIP loss [40, 32] or Score Distillation Sampling (SDS) [46,
11] to generate PBR textures with the guidance of given text prompts.
Nevertheless, they require extra efforts to retrain the generation model for
different 3D objects or text prompts. DreamFace [71] proposes a diffusion-
based generation model in both the latent space and image space, yielding
high-quality diffuse texture maps. This method is supervised with explicit
ground truth PBR data captured by multi-view photometric systems similar to
Light Stage [37]. The data collection process requires significant resource
allocation. Recently, several approaches [3] utilize 3D Morphable Models
(3DMMs) [5, 14] to conveniently synthesize UV textures from large-scale facial
image datasets [29], which simplifies the data collection. However, it is
unclear how to seamlessly integrate these scalable datasets into PBR textures
generation.

Figure 2: The key idea of latent space refinement approach. The latent space
is expanded progressively to handle more text prompts.

In this work, we propose a progressive latent space refinement approach that
can bootstrap from 3DMM-based texture maps to generate high-quality and
diverse PBR textures. Our key idea is demonstrated in Fig. 2. Our approach
starts with initializing a base latent space using the FFHQ-UV dataset [3].
Since FFHQ-UV is entangled with weak specular highlights, directly applying
them in the generative model training may introduce obvious artifacts in PBR
textures, such as light spots. To address this problem, we intend to design a
generative network combined with differentiable rendering [41] and well-
designed regularization terms to disentangle the textures. Specifically, we
employ a GAN model to establish the initial latent space. Once the model is
capable of generating reasonable results, we proceed with latent space
refinement for text-guided generation and introduce an improved edge-aware SDS
to enhance the diversity. In the refinement step, the GAN-generated results
serve as initializations for SDS, promoting mutual benefits between GAN and
SDS. This choice is also due to that it is unclear how to effectively
integrate differential rendering with the noisy texture output during the
training of denoising network in the diffusion model.

We construct a framework called PBRGAN with three stages to achieve text-
driven, diverse facial PBR texture generation. (1) The first stage introduces
a self-supervised training scheme to initialize a base latent space for
disentangled PBR texture generation. We propose regularization terms regarding
material properties, _e.g_. albedo variance or normal deviation, to suppress
the specular artifacts. (2) The second stage boosts the latent space with
cross-modal generation capability via CLIP-based alignment. It enables our
model to generate reasonable results for text attributes resembling the FFHQ
distribution. (3) In the last stage, we design a bootstrapping framework to
distill relevant knowledge of the pre-trained diffusion model to refine the
latent space. It leverages the strengths of both GAN and SDS. SDS guides GANs
with more generation patterns, while GANs provide better initial positions for
SDS. We further strengthen the SDS loss with explicit facial feature-line
prior and introduce an edge-aware SDS (EASDS) loss based on ontrolNet [72].
EASDS helps to align key facial features and effectively improve structural
accuracy. After the three stages, PBRGAN can handle uncommon text conditions,
such as a knife scar or purple lips, and permits texture generation in a
single feedforward process. In summary, our contributions are as follows:

  * •

We propose a bootstrapping approach to refine the latent space for text-driven
facial PBR texture generation. It introduces a progressive knowledge-
distilling process to handle text conditions by leveraging the mutual benefits
of GANs and SDS.

  * •

We present a three-stage framework to expand the generative capability. In
addition, we design a novel ControlNet-based edge-aware SDS loss to align the
facial structures.

  * •

Our framework can generate high-quality facial PBR texture maps. Given a
specific text prompt, the response time of our method is lower than state-of-
the-art (SOTA) methods based on diffusion models.

##  2 Related Work

#### 3D-aware image synthesis.

Generative models have been developed rapidly in recent years [20, 15, 22,
49]. GAN-based methods, especially StyleGAN [28, 29, 27], have demonstrated
powerful capabilities to generate high-fidelity 2D images. Building upon this,
some approaches explored integrating 3D information into image generation to
achieve 3D-aware image synthesis. Visual object network [74] proposes a fully
differentiable 3D-aware generative model for image and shape synthesis with a
disentangled 3D representation. Liao et al. [34] consider the generative
process into a 3D content creation stage and a 2D rendering stage, yielding a
3D controllable image synthesis model. However, 3D data collections are
indispensable for these methods. Follow-up works introduce neural
representations and leverage 2D data to address the dataset limitation [42,
55, 69, 13, 52]. Pi-GAN [7] and ShadeGAN [44] represent scenes as view-
consistent neural radiance fields and map 3D coordinates to pixel values as a
3D prior. EG3D [6] and Next3D [59] adopt feature triplanes for more efficient
3D representation. Another series of approaches is dedicated to improving the
controllability of 3D-aware generative models [36, 12, 60]. Nonetheless, these
methods only generate 2D images and are difficult to use in the classic 3D
content creation pipeline.

Figure 3: The pipeline of PBRGAN. (a) We generate disentangled PBR textures by
leveraging entangled FFHQ-UV textures and differentiable rendering. (b) We
align the latent space with the text space under the guidance of CLIP to
achieve text-guided generation. (c) We amalgamate GAN and SDS and further
expand the latent space to handle more text prompts.

#### Text-to-3D generation.

Generating 3D contents from 2D images has made rapid progress [19, 43, 51, 17,
53, 16, 56] in recent years. But its controllability needs to be further
improved. With the advent of large-scale cross-modal pre-training models, the
text-guided 3D generative model [50, 23, 25, 2, 19] has become a frontier
research topic. Many researchers utilize the CLIP model [47] to ensure the
generated 3D contents match the text prompts. Text2Mesh [39] optimizes the
color attributes and geometric displacements of individual vertices
simultaneously. CLIP-NeRF [63] presents a Nerf-based framework to provide
users with flexible control over 3D content using either a text prompt or an
exemplar image. However, these methods may fail to generate high-quality
results.

Recently, the powerful generative capabilities of diffusion models have been
integrated into text-to-3D generation[64, 38, 35, 61]. DreamFusion [46] adopts
pre-trained text-to-image diffusion models as priors and generates realistic
3D models via SDS optimization. ProlificDreamer [67] further extends SDS to
Variational Score Distillation, which optimizes the distribution of 3D scenes.
TEXTure [48] iteratively renders the object from different viewpoints and
applies a depth-to-image 2D diffusion model to generate a complete unwrapped
UV texture map. However, the above methods generate textures in entangled
representations, which are incompatible with the PBR pipeline.

#### PBR texture generation.

To fit the existing computer graphics production pipeline, PBR texture
generation is essential for photorealistic results. Some methods [70, 33, 8,
31] employ 3DMM [4, 18, 62, 19] to generate physical-based texture maps with
the guidance of reference images or random noises. In recent contributions,
CLIP loss and SDS loss have been applied to text-driven PBR texture
generation, such as CLIP-mesh [40], TANGO [32], and Fantasia3D [11]. However,
these models must be retrained for each 3D object or each text prompt, which
is time-consuming.

DreamFace [71] presents a diffusion model-based scheme to generate
personalized 3D faces with text guidance. It can achieve physically-based
rendering by decomposing compact latent space into diffuse albedo, specular
intensity, and normal maps. The decomposing model relies on a captured high-
quality PBR texture dataset, which is usually collected for commercial
purposes and has not been released to the public. Besides, the inference
efficiency of DreamFace is limited by the diffusion process. In this paper, we
expand the $\mathcal{W_{+}}$ feature space of StyleGAN to support text-to-3D
PBR texture generation with no captured datasets. It will benefit from both
the efficiency of GANs and the controllability of text-based generation.

##  3 Our Method

In this section, we present PBRGAN, a progressive latent space refinement
approach for text-driven facial texture generation with PBR compatibility and
efficient inference. Given the geometry of a 3D face, PBRGAN aims to generate
the appearance under the guidance of text prompts. The appearance is
decomposed into three components, _i.e_. , albedo, normal, and roughness maps
in the texture space without any supervision on explicitly disentangled data.
As illustrated in Fig. 3, the whole pipeline comprises three stages. We start
by training a PBR StyleGAN (Sec. 3.1) to initialize the PBRGAN latent space.
Then we align it with the text space using a CLIP-based loss (Sec. 3.2).
Finally, we further refine the PBRGAN latent space for more uncommon text
conditions (Sec. 3.3).

###  3.1 PBR StyleGAN

Given an arbitrary 3D facial geometry, we propose a self-supervised learning
scheme to generate disentangled UV textures,  _e.g_. , the
diffuse/specular/normal maps, via leveraging a non-PBR facial texture dataset
FFHQ-UV [3]. FFHQ-UV is constructed with even illumination conditions. Its
textures are entangled with weak specular highlights. Directly applying them
to train the generation model may introduce light spot artifacts. To avoid
this issue, we do not utilize FFHQ-UV data as supervision. Instead, we train a
disentangling model with differentiable rendering and additional
regularization terms.

As diffusion models typically involve iterative noise injection and denoising
processes, the changes in the image are subtle at each iteration. The non-
deterministic and complex nature of its iterative gradient computation process
presents challenges for integrating diffusion models with differentiable
rendering during the denoising process. It hinders us from directly training a
diffusion model with FFHQ-UV output. Therefore, we construct a PBR texture
generator based on StyleGAN2 [29] to establish an initial latent space. The
following sections will provide detailed introductions to the network
architecture and training strategy.

#### Network structure.

Fig. 3(a) demonstrates the first stage of our PBRGAN framework. Given a
randomly initialized latent code $z\in\mathbb{R}^{512}$, a latent mapper
$\mathcal{M}_{z}$ transforms $z$ into an intermediate space $\mathcal{W_{+}}$
to produce a new latent embedding $w\in\mathbb{R}^{16*512}$. Then $w$
modulates the parameters of $\mathcal{G}$ through adaptive instance
normalization (AdaIN) [24]. To enable StyleGAN with physically-based
rendering, a generator $\mathcal{G}$ outputs a seven-channel appearance map in
the unwrapped UV space. Following the setting of Munkberg et al. [41], the
output can be split into $T=(T_{a},T_{r},N)$, where $T_{a}$ is the albedo map,
$T_{r}$ are roughness parameters, and $N$ describes tangent space normal
perturbations. For the geometry, we generate random facial meshes by sampling
in the parameter space of the 3DMM-based model HIFI3D++ [5]. It can provide
abundant geometric information with enhanced diversity.

#### Differentiable-rendering-based training.

We introduce a differentiable rendering module $\mathcal{R}$ to train the
PBRGAN. $\mathcal{R}$ takes the predicted PBR textures, a 3D geometry $s$, and
a random pose $p$ as input and produces a 2D facial image
$I_{z}=\mathcal{R}(T_{a},T_{r},N,s,p)$. For the ground truth, $\mathcal{R}$
utilizes an entangled texture map $\hat{T}$ in FFHQ-UV to render another image
$\hat{I}=\mathcal{R}(\hat{T},s,p)$. Then PBRGAN can be supervised by the
distance between $I_{z}$ and $\hat{I}$. Besides, $I_{z}$ and $\hat{I}$ are
input to a global discriminator and a patch-based local discriminator for
detail enhancement. We employ the discriminator training technique of
Diffusion-GAN [66] to improve the diversity of the results.

#### Regularization terms.

Moreover, we design two regularization terms to stabilize the training. They
introduce priors of material properties to guide the texture disentanglement.
Specifically, we propose $\mathcal{J}_{a}$ for the albedo and
$\mathcal{J}_{n}$ for the normal map. $\mathcal{J}_{a}$ is used to reduce the
specular highlights by constraining the albedo of the skin region to be
smooth. $\mathcal{J}_{n}$ constrains the predicted normals should stick to a
standard surface normal of the UV space. The regularization term $\mathcal{J}$
can be represented as:

$\displaystyle\mathcal{J}$ $\displaystyle=\mathcal{J}_{a}+\mathcal{J}_{n}$ (1)
$\displaystyle=|\nabla(T_{a}\odot M_{skin})|+|T_{n}-N|$

where $M_{skin}$ is the mask of the skin region, $\odot$ is elment-wise
multiplication, $\nabla$ represents the total variation, and $N$ is a standard
normal map with $[0,0,1]$ for all pixels. $\mathcal{J}_{a}$ is calculated in
both the RGB space and the hue channel of the HLS color space.

###  3.2 Text and Latent Alignment

With the emergence of pre-trained cross-modal models, text prompts can provide
a flexible and user-friendly interface for 3D content generation. Existing
approaches typically rely on fine-tuning the generative models to establish
consistency between the results and the texts[46]. However, it usually
requires a lot of time for optimization. To avoid this issue, we extend
StyleGAN to support text-based PBR texture generation. The key idea is
correlating the text embeddings with the $\mathcal{W_{+}}$ latent space of
PBRGAN.

Recent advances based on GAN inversion techniques and cross-modal models have
led to a more feasible solution for aligning text with latent space. GAN-
inversion-based image editing algorithms [73, 1, 54] invert given images back
into the latent space of GANs and manipulate them by varying the latent code
in different interpretable directions, such as age and expression. In the
meantime, the CLIP model has made remarkable progress in establishing cross-
modal correlations between image and text space [45]. Therefore, we employ the
CLIP model as an anchor point for alignment between text space and
$\mathcal{W_{+}}$ space. We will introduce the text prompt generation and the
pipeline in the following.

#### Text prompt generation.

We define a text prompt dictionary with geometry-independent facial attributes
to construct the text space. The dictionary consists of common facial
attributes, such as “eyebrow shape” or “red lips”, and uncommon attributes
like “scar” or “purple lips”. We formulate different text prompts through
randomly sampling attributes from the dictionary. For more details, please
refer to our supplementary materials.

#### The alignment framework.

As shown in Fig. 3(b), we integrate an additional text embedding module into
the PBRGAN to achieve the alignment. It consists of a CLIP embedding module
$\mathcal{E}$ and a text mapper $\mathcal{M}_{t}$ with a twining structure of
the latent mapper $\mathcal{M}_{z}$ in Sec. 3.1. In this stage, the parameters
of $\mathcal{G}$ and $\mathcal{E}$ are frozen. $\mathcal{M}_{t}$ is optimized
to map the text embedding space $\mathcal{W^{\prime}_{+}}$ to
$\mathcal{W_{+}}$. To train $\mathcal{M}_{t}$, we random sample a latent $z$
and produce an image $I_{z}=\mathcal{R}(\mathcal{G}(z),s,p)$. Then $I_{z}$ is
fed into $\mathcal{E}$ as input to get the embedding $t_{z}$ in the CLIP
space. The text mapper $\mathcal{M}_{t}$ takes $t_{z}$ as input and transforms
it into $w^{\prime}\in\mathcal{W^{\prime}_{+}}$. $w^{\prime}$ is utilized to
generate a facial image, which can be represented as:

$I_{t}=\mathcal{R}(\mathcal{G}(\mathcal{M}_{t}(\mathcal{E}(I_{z}))),s,p).$ (2)

#### The alignment loss functions.

The space alignment is supervised by two loss functions, _i.e_. , a loss
$\mathcal{L}_{\mathcal{W_{+}}}$ in the $\mathcal{W_{+}}$ space and a loss
$\mathcal{L}_{CLIP}$ in the CLIP space. $\mathcal{L}_{\mathcal{W_{+}}}$ is the
Manhattan distance between $\mathcal{W^{\prime}_{+}}$ and $\mathcal{W_{+}}$.
$\mathcal{L}_{CLIP}$ is a cycle consistency loss to constrain the rendered
image $I_{t}$ to have a consistent CLIP embedding with $t_{z}$. The overall
loss function is as follows:

$\mathcal{L}=|\mathcal{M}_{t}(\mathcal{E}(I_{t}))-\mathcal{M}_{z}(z)|+\alpha|\mathcal{E}(I_{z})-\mathcal{E}(I_{t})|,$
(3)

where $\alpha$ is the weight for $\mathcal{L}_{CLIP}$. In our experiments,
$\alpha$ is set to $0.05$. Once the training of the alignment module is done,
we can generate PBR textures for a given text prompt $t$ through a feedforward
pass $\mathcal{G}(t)$.

###  3.3 Latent Space Refinement

After the alignment, PBRGAN is proficient at generating images from common
text prompts resembling the FFHQ distribution. It still faces limitations in
producing reasonable textures using uncommon prompts, _e.g_. , “a prominent
knife scar on the face”. To generate textures that match these prompts,
existing methods [64, 38] employ a pre-trained diffusion model as an image
prior and perform parameters optimization using the SDS loss [46]. Such
approaches require inefficient model optimization for each input prompt.

In the third stage of PBRGAN, we propose a latent space refinement approach to
amalgamate GANs and SDS so that they can benefit from each other mutually. SDS
can provide more generation pattern guidance for GANs to break through space
limitations, while GANs can produce better initial positions for the
optimization of SDS to reduce the computational cost. Meanwhile, we expand the
latent space to efficiently generate more diverse PBR textures by leveraging
the generative capability of SDS.

#### GANs and SDS amalgamation.

To amalgamate GANs and SDS, we use results generated by GANs as the
optimization target of the SDS loss (amalgamation part of Fig. 3(c)).
Furthermore, we modify the basic SDS loss into an enhanced edge-aware version
(EASDS) by introducing a key facial structure alignment module and a pre-
trained ControlNet $\phi$ [72]. The EASDS has four steps (Fig. 4):

Figure 4: Explanation of our edge-aware SDS (EASDS). We sample from different
viewpoints and employ soft edges detected from a template texture as feature-
line conditions for ControlNet.

  * (1)

Optimization target preparation. We run an inference procedure of PBRGAN to
generate the texture maps $T$ for a given text prompt $\tau$. $T$ will be
taken as the target parameter for the optimization of EASDS.

  * (2)

Camera pose setting. We place a series of predefined viewpoints around the
face model to fit our differentiable- rendering-based pipeline.

  * (3)

Feature-line extraction. We render reference facial images $I_{ref}$ from the
viewpoints with a random facial geometryand a predefined template texture.
Then a pre-trained edge detection network HEDNet [68] is exploited to detect
soft edges $E$ from the renerderred image.

  * (4)

EASDS optimization. For a given viewpoint, the rasterizer renders a facial
image $I_{\tau}=\mathcal{R}(T,s,p)$ with the same geometry of (3). The rough
HED edges associated with this viewpoint from the former step are taken as
feature-line priors to ControlNet. We optimize $T$ with the ControlNet
gradient through the rendered image $I_{\tau}$. The ControlNet $\phi$ use
$\tau$ and $E$ to provide a score function $\hat{\epsilon}_{\phi}$ as the
optimization guidance for $T$:

$\nabla_{T}\mathcal{L}_{\mathrm{EASDS}}(\phi,I_{\tau})\triangleq\mathbb{E}_{t,\epsilon}\left[w(t)\left(\hat{\epsilon}_{\phi}\left(\mathbf{z}^{I_{\tau}}_{t};\tau,t,E\right)-\epsilon\right)\frac{\partial
I_{\tau}}{\partial T}\right]$ (4)

where $w(t)$ is a timestep-related weighting function, $\epsilon$ is the
sample noise, $\mathbf{z}^{I_{\tau}}_{t}$ is the noisy image based on
${I_{\tau}}$. The optimization starts from the front view and iteratively
proceeds to other viewpoints. With feature lines extracted from the same 3D
reference model, EASDS ensures that the images rendered from different
perspectives maintain multi-view consistency in facial features.

Figure 5: Qualitative comparison results. From left to right: the input text
prompts, the results generated with ClipFace, DreamFace, Fantasia3D, ours, and
ours-EASDS.

For special text prompts of specific individuals, such as Barack Obama,
Avatar, Iron Man, _etc_. , PBRGAN will produce the corresponding texture maps
via direct optimization based on EASDS.

#### Latent space expansion.

For some uncommon attributes that FFHQ-UV does not possess, such as “scar” or
“purple lips”, we expect to distill relevant knowledge of the diffusion model
and inject it into the latent space of PBRGAN. It will expand the latent space
and boost PBRGAN for better generative diversity. Specifically, we first
construct two kinds of text prompts. Common prompts ${\tau_{c}}$ only consist
of common attributes, while uncommon prompts ${\tau_{uc}}$ involve uncommon
attributes. Then, we generate a series of facial PBR textures based on these
prompts through EASDS. Afterward, we employ the generated textures as the
manufactured ground truth data to finetune the PBRGAN.

As illustrated in the expansion part of Fig. 3(c), the finetuning procedure
has two training paths: one is an unconditional generation path from random
sampled $z$, which can stabilize the training; the other is a conditional
generation path based on mixed prompts of ${\tau_{c}}$ and ${\tau_{uc}}$. For
the latter one, we introduce a conditional discriminator [26] to match the
generated results with the input conditions. The conditional discriminator is
initialized with parameters of the global discriminator (see Sec. 3.1). It
takes the rendered images and text embeddings as input and produces true value
only when the image is matched with the prompt. After network training, the
generative space of PBRGAN is expanded to conditions beyond the FFHQ
distribution.

##  4 Experiments

| Method | Rep | Data | Inf Time / min | Metrics | User Study
---|---|---|---|---|---|---
| hyperIQA [58] $\uparrow$ | CLIPIQA$+$ [65] $\uparrow$ | TOPIQface [10] $\uparrow$ | CLIPSCORE [21] $\uparrow$
w/o prompt | ClipFace [2] | Texture | FFHQ |  $<$ 0.01 (1x 3090) | 0.63 | 0.55 | 0.58 | - | -
Ours. | PBR | FFHQUV |  $<$ 0.01 (1x3090) | 0.74 | 0.57 | 0.64 | - | -
w/ prompt | TANGO [32] | PBR | - | 10 (1x 3090) | 0.50 | 0.45 | 0.35 | 0.67 | 1.61
Fantasia3D [11] | PBR | - | 45 (8x 3090) | 0.66 | 0.52 | 0.57 | 0.74 | 2.20
ClipFace [2] | Texture | FFHQ | 90 (1x 3090) | 0.63 | 0.56 | 0.52 | 0.77 | 2.36
DreamFace [71] | PBR | LightStage | 5 (1x A6000) | 0.76 | 0.63 | 0.77 | 0.72 | 2.96
Ours | PBR | FFHQUV |  $<$ 0.01 (1x3090) | 0.72 | 0.60 | 0.73 | 0.76 | 2.80
Ours-EASDS | PBR | FFHQUV | 1 (1x 3090) | 0.68 | 0.58 | 0.74 | 0.74 | 3.34

Table 1: Qualitative comparison results. We conduct experiments with or
without prompts. The best results are highlighted as 1st and 2nd. Our method
wins first place in the human perceptual study and inference time. We also
achieve the second-best in other metrics for text-conditional generation. Note
that DreamFace needs captured PBR data to train and spends five times as much
inference time as ours.

In this paper, we focus on text-driven facial PBR texture generation. The
experiments are conducted on a Linux server with 8 Tesla V100 GPUs and an
Intel Xeon Platinum 8268 CPU. We demonstrate details of experiments,
comparisons with state-of-the-art methods, and ablation studies.

###  4.1 Experimental Setup

#### Dataset preparation.

We use the FFHQ-UV dataset [3] for experiments. FFHQ-UV is a large-scale
3DMMs-based facial UV-texture dataset derived from a natural image dataset
FFHQ. It contains $50,000+$ texture UV-maps with even illuminations and
cleaned facial regions. For the facial geometry, We randomly sample
$\numprint{10000}$ facial mesh data from HIFI3D++ [5]. Since HIFI3D++ contains
no eyeballs, we employ the albedo and geometry of the eyeballs produced from
the pipeline of FFHQ-UV [3]. To get rid of the influence of hair, we fill the
hair area with the average skin color and smooth the border region.

#### Implementation details.

We train PBRGAN using rendered images of a resolution of $512\times 512$ and
PBR texture maps of the same size. The detailed settings are as follows:

  * •

The first stage (Sec. 3.1): learning rate = $0.0025$, batch size = $8$. We use
NVDIFFREC [41] as our differentiable rendering module.

  * •

The second stage (Sec. 3.2): learning rate = $1$, batch size = $32$. For CLIP
supervision, we adopt the ‘ViT-B/32’ pre-trained variant. To get better CLIP
embedding of the rendered results, we use 3D geometries with eyeballs here.

  * •

The third stage (Sec. 3.3): learning rate = $0.0025$, batch size = $8$. For
EASDS supervision, we utilize the HED Boundary-conditional ControlNet [72]
with the base model Stable Diffusion v1.5 [49]. We predefine $20$ viewpoints
for the EASDS optimization. To feed viewpoint information into the EASDS
optimization, a viewpoint-relevant positional prompt, structured as “in the
front/back/left/right/bottom/vertical view”, is appended to the original
prompt. In addition, We remove the face-related prompts, such as lips or
eyebrows, when optimized in the back view since SDS tends to generate human
faces on the back of the head in order to fit these prompts.

###  4.2 Comparison and Analysis

We compare our method with four state-of-the-art texture generation methods,
including Tango [32], Fantasia3D [11], ClipFace [2], and DreamFace [71].
ClipFace can be further divided into two parts: unconditional sampling and
conditional optimization, while all the other methods only support conditional
generation. Therefore, we first compare our method with ClipFace on the
unconditional generation task and then make a comparison with all methods on
the text-conditional generation task.

#### Metrics.

We use the following metrics111We use an implementation of IQA provided by
[9]. from three perspectives to evaluate our method.

  * •

Non-reference image quality assessment (IQA) metrics: Non-reference IQA aims
to evaluate the quality of images without relying on a reference image. We
adopt MUSIQ [30], HyperIQA [58], and CLIPIQA+ [65] for the generated image
quality evaluation.

  * •

Task-specific metrics: We utilize a tailored metric for face quality
measurement, _i.e_. TOPIQface [10], which is trained with face IQA dataset
GFIQA [57], to evaluate the quality of the rendered facial images.

  * •

Prompt-aware metrics: We use the CLIPSCORE [21] to measure the alignment
between the given prompt and the corresponding generated face.

For these metrics, higher scores indicate better image quality or higher text-
image consistency.

#### Unconditional texture generation.

For the text-unrelated generation task, we use non-reference-IQA and task-
specific metrics to evaluate our method with $\numprint{1000}$ randomly
sampled latent code $z$. As shown in the “without prompt” part of Table 1, our
method achieves a higher score by $10.3\%$ on TOPIQface, which shows the
effectiveness of our PBRGAN. Our method also outperforms ClipFace in terms of
the non-reference-IQA metrics, yielding better image quality with enhanced
sharpness and intricate details. It demonstrates the effectiveness of our
latent space representation.

#### Text-driven texture generation.

For the conditional generation task, our methods were evaluated by randomly
sampling $30$ prompts from our dictionary to cover different attributes.
Specifically, we first synthesize the images with these prompts using all
methods. The images are combined with corresponding text prompts to generate
text-image pairs. Then, we adopt all the above metrics to evaluate the image
quality and text-image consistency of these pairs.

Since PBRGAN is capable of generating PBR texture maps with straightforward
inferences, our method strikes a balance between efficiency and performance.
As illustrated in the “with prompt” part of Tab. 1, PBRGAN based on EASDS
optimization (Ours-EASDS) is able to generate textures at $1$\- minute level.
It shows that the amalgamation of GAN and SDS can reduce the optimization time
by a large margin. For the quality evaluation, Ours-EASDS achieves comparable
results to DreamFace with tenuous distinction and outperforms all other
methods. Note that DreamFace is trained with captured PBR texture maps, while
our method involves no ground truth PBR data. Further expanding the latent
space with EASDS-based textures (Ours) may slightly diminish partial
quantitative metrics, but it decreases the inference time to less than $0.5$
seconds.

Fig. 5 demonstrates the qualitative comparison results. Although ClipFace can
generate facial textures with most designated attributes, its results have
obvious artifacts (2nd Column). Fantasia3D is designed for generic-object
texture generation. Applying it to facial texture generation may cause
disharmonious patterns (4th Column). DreamFace is able to generate relatively
realistic results. However, it fails to handle uncommon attributes like “scar”
or “purple lips” (3rd Column). Our methods are capable of producing reasonable
facial texture maps with diverse prompts.

#### User study.

We conducted a user study to evaluate the perceptual quality of the results.
We generate $30$ text-image pairs for each method of TANGO, Fantasia3D,
ClipFace, DreamFace, OurS and Ours-EASDS. $150$ participants are recruited to
evaluate the results. We ask every participant to grade each text-image pair
according to the image quality and text-image consistency. The grading is
quantified as a preference score to assess the overall quality from 1 (least
low quality) to 5 (most high quality). As the last column of Tab. 1 shows,
ours-EASDS is superior to other methods.

###  4.3 Ablation Study

To explore the effects of our modules, we conducted ablation experiments for
EASDS and GAN-SDS amalgamation.

#### EASDS.

Figure 6: Ablation study results of EASDS. Note that there are artifacts and
misalignments in the results of SDS. The second row shows the generated
textures using challenging geometry under the guidance of “Barack Obama”.
Benefiting from EASDS’s alignment of feature lines, our method is capable of
generating better textures tailored for extreme geometries.

We compare our EASDS with the classical SDS [46], which has no edge priors.
Specifically, given a text prompt, we generate an initial PBR texture with our
PBRGAN of the second stage. Afterward, we perform SDS-based and EASDS-based
optimization separately to generate different texture maps. As the Fig. 6
shows, there are some misalignments between the geometry edges and texture
edges in the SDS-based results. With additional feature-line priors, our EASDS
can enhance facial structural accuracy and generate better PBR textures than
SDS.

#### GAN and SDS amalgamation.

The classical SDS-based texture generation methods _e.g_. Fantasia3D [11] are
optimized from randomly initialized parameters. The initial optimization point
is distant from the target. In this paper, we utilize the results generated by
PBRGAN of the second stage as the initial point for EASDS, which is beneficial
for shortening the optimization path and generating better results (Fig. 7).
The amalgamation can reduce the required optimization time for each prompt
from $70$ to $20$ seconds.

Figure 7: The influence of the EASDS initialization.

##  5 Conclusion

In this work, we present a progressive latent space refinement approach to
generate high-quality and diverse facial PBR textures under the guidance of
text prompts. Our approach constructs a multi-stage framework to bootstrap
from 3DMM-based textures to reduce the reliance on ground truth PBR data. We
design a self-supervised scheme with differentiable rendering and carefully
crafted regularization terms to disentangle UV textures. To refine the latent
space, we introduce a knowledge distillation process utilizing GANs and SDS.
We also propose a ControlNet-based and edge-aware SDS to increase the accuracy
of facial structures. The experiments demonstrate that our method can
efficiently create high-quality and diverse facial PBR textures. In the
future, we aim to explore the generation of accurate facial geometry in
conjunction with diverse PBR textures.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In IEEE Conf. Comput. Vis. Pattern Recog., pages 8296–8305, 2020.
  * [2] Shivangi Aneja, Justus Thies, Angela Dai, and Matthias Nießner.  Clipface: Text-guided editing of textured 3d morphable models.  In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.
  * [3] Haoran Bai, Di Kang, Haoxian Zhang, Jinshan Pan, and Linchao Bao.  Ffhq-uv: Normalized facial uv-texture dataset for 3d face reconstruction.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 362–371, 2023.
  * [4] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 157–164. ACM, 2023.
  * [5] Zenghao Chai, Haoxian Zhang, Jing Ren, Di Kang, Zhengzhuo Xu, Xuefei Zhe, Chun Yuan, and Linchao Bao.  Realy: Rethinking the evaluation of 3d face reconstruction.  In Eur. Conf. Comput. Vis., pages 74–92. Springer, 2022.
  * [6] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 16123–16133, 2022.
  * [7] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 5799–5809, 2021.
  * [8] Anpei Chen, Zhang Chen, Guli Zhang, Kenny Mitchell, and Jingyi Yu.  Photo-realistic facial details synthesis from single image.  In Int. Conf. Comput. Vis., pages 9429–9439, 2019.
  * [9] Chaofeng Chen and Jiadi Mo.  IQA-PyTorch: Pytorch toolbox for image quality assessment.  [Online]. Available: https://github.com/chaofengc/IQA-PyTorch, 2022.
  * [10] Chaofeng Chen, Jiadi Mo, Jingwen Hou, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin.  Topiq: A top-down approach from semantics to distortions for image quality assessment.  arXiv preprint arXiv:2308.03060, 2023.
  * [11] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  arXiv preprint arXiv:2303.13873, 2023.
  * [12] Kangle Deng, Gengshan Yang, Deva Ramanan, and Jun-Yan Zhu.  3d-aware conditional image synthesis.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 4434–4445, 2023.
  * [13] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong.  Gram: Generative radiance manifolds for 3d-aware image generation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 10673–10683, 2022.
  * [14] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, 2019.
  * [15] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 12873–12883, 2021.
  * [16] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  Get3d: A generative model of high quality 3d textured shapes learned from images.  Adv. Neural Inform. Process. Syst., 35:31841–31854, 2022.
  * [17] Lin Gao, Tong Wu, Yu-Jie Yuan, Ming-Xian Lin, Yu-Kun Lai, and Hao Zhang.  Tm-net: Deep generative networks for textured meshes.  ACM Trans. Graph., 40(6):1–15, 2021.
  * [18] Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Pérez, and Christian Theobalt.  Reconstruction of personalized 3d face rigs from monocular video.  ACM Trans. Graph., 35(3):1–15, 2016.
  * [19] Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou.  Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 1155–1164, 2019.
  * [20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [21] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  CLIPScore: a reference-free evaluation metric for image captioning.  In EMNLP, 2021.
  * [22] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Adv. Neural Inform. Process. Syst., 33:6840–6851, 2020.
  * [23] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu.  AvatarCLIP: zero-shot text-driven generation and animation of 3D avatars.  ACM Trans. Graph., 41(4):1–19, July 2022.
  * [24] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In Int. Conf. Comput. Vis., pages 1501–1510, 2017.
  * [25] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 867–876, 2022.
  * [26] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  NIPS, 33:12104–12114, 2020.
  * [27] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  Adv. Neural Inform. Process. Syst., 34:852–863, 2021.
  * [28] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 4401–4410, 2019.
  * [29] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 8110–8119, 2020.
  * [30] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang.  Musiq: Multi-scale image quality transformer.  In Int. Conf. Comput. Vis., pages 5148–5157, 2021.
  * [31] Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer, Abhijeet Ghosh, and Stefanos Zafeiriou.  Avatarme++: Facial shape and brdf inference with photorealistic rendering-aware gans.  IEEE Trans. Pattern Anal. Mach. Intell., 44(12):9269–9284, 2021.
  * [32] Jiabao Lei, Yabin Zhang, Kui Jia, et al.  Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition.  Adv. Neural Inform. Process. Syst., 35:30923–30936, 2022.
  * [33] Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al.  Learning formation of physically-based face attributes.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 3410–3419, 2020.
  * [34] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger.  Towards unsupervised learning of generative models for 3d controllable image synthesis.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 5871–5880, 2020.
  * [35] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 300–309, 2023.
  * [36] Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and SY Kung.  3d-fm gan: Towards 3d-controllable face manipulation.  In Eur. Conf. Comput. Vis., pages 107–125. Springer, 2022.
  * [37] Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, Paul E Debevec, et al.  Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination.  Rendering Techniques, 2007(9):10, 2007.
  * [38] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 12663–12673, 2023.
  * [39] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 13492–13502, 2022.
  * [40] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa.  Clip-mesh: Generating textured meshes from text using pretrained image-text models.  In SIGGRAPH Asia 2022 conference papers, pages 1–8, 2022.
  * [41] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler.  Extracting triangular 3d models, materials, and lighting from images.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 8280–8290, 2022.
  * [42] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In Int. Conf. Comput. Vis., pages 7588–7597, 2019.
  * [43] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger.  Texture fields: Learning texture representations in function space.  In Int. Conf. Comput. Vis., pages 4531–4540, 2019.
  * [44] Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai.  A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis.  Adv. Neural Inform. Process. Syst., 34:20002–20013, 2021.
  * [45] Justin NM Pinkney and Chuan Li.  clip2latent: Text driven sampling of a pre-trained stylegan using denoising diffusion and clip.  In Brit. Mach. Vis. Conf., page 594. BMVA Press, 2022.
  * [46] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [48] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or.  Texture: Text-guided texturing of 3d shapes.  arXiv preprint arXiv:2302.01721, 2023.
  * [49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 10684–10695, 2022.
  * [50] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan.  Clip-forge: Towards zero-shot text-to-shape generation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 18603–18613, 2022.
  * [51] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  Adv. Neural Inform. Process. Syst., 33:20154–20166, 2020.
  * [52] Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger.  Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids.  Adv. Neural Inform. Process. Syst., 35:33999–34011, 2022.
  * [53] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  Adv. Neural Inform. Process. Syst., 34:6087–6101, 2021.
  * [54] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in gans.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 1532–1540, 2021.
  * [55] Yichun Shi, Divyansh Aggarwal, and Anil K Jain.  Lifting 2d stylegan for 3d-aware face generation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 6258–6266, 2021.
  * [56] Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai.  Texturify: Generating textures on 3d shape surfaces.  In Eur. Conf. Comput. Vis., pages 72–88. Springer, 2022.
  * [57] Shaolin Su, Hanhe Lin, Vlad Hosu, Oliver Wiedemann, Jinqiu Sun, Yu Zhu, Hantao Liu, Yanning Zhang, and Dietmar Saupe.  Going the extra mile in face image quality assessment: A novel database and model.  IEEE Trans. Multimedia, 2023.
  * [58] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang.  Blindly assess image quality in the wild guided by a self-adaptive hyper network.  In IEEE Conf. Comput. Vis. Pattern Recog., June 2020.
  * [59] Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu.  Next3d: Generative neural texture rasterization for 3d-aware head avatars.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 20991–21002, 2023.
  * [60] Feitong Tan, Sean Fanello, Abhimitra Meka, Sergio Orts-Escolano, Danhang Tang, Rohit Pandey, Jonathan Taylor, Ping Tan, and Yinda Zhang.  Volux-gan: A generative model for 3d face synthesis with hdri relighting.  In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–9, 2022.
  * [61] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng.  Dreamgaussian: Generative gaussian splatting for efficient 3d content creation.  arXiv preprint arXiv:2309.16653, 2023.
  * [62] Luan Tran and Xiaoming Liu.  Nonlinear 3d face morphable model.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 7346–7355, 2018.
  * [63] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.  Clip-nerf: Text-and-image driven manipulation of neural radiance fields.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 3835–3844, 2022.
  * [64] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 12619–12629, 2023.
  * [65] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy.  Exploring clip for assessing the look and feel of images.  In AAAI, 2023.
  * [66] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou.  Diffusion-gan: Training gans with diffusion.  arXiv preprint arXiv:2206.02262, 2022.
  * [67] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  arXiv preprint arXiv:2305.16213, 2023.
  * [68] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In ICCV, pages 1395–1403, 2015.
  * [69] Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou.  3d-aware image synthesis via learning structural and textural representations.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 18430–18439, 2022.
  * [70] Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and Hao Li.  High-fidelity facial reflectance and geometry inference from an unconstrained image.  ACM Trans. Graph., 37(4):1–14, 2018.
  * [71] Longwen Zhang, Qiwei Qiu, Hongyang Lin, Qixuan Zhang, Cheng Shi, Wei Yang, Ye Shi, Sibei Yang, Lan Xu, and Jingyi Yu.  Dreamface: Progressive generation of animatable 3d faces under text guidance.  arXiv preprint arXiv:2304.03117, 2023.
  * [72] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In ICCV, pages 3836–3847, 2023.
  * [73] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  In Eur. Conf. Comput. Vis., pages 592–608. Springer, 2020.
  * [74] Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman.  Visual object networks: Image generation with disentangled 3d representations.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Adv. Neural Inform. Process. Syst., volume 31. Curran Associates, Inc., 2018.