TCDiff: Triple Condition Diffusion Model with 3D Constraints for Stylizing Synthetic Faces
Bernardo Biesseck Pedro Vidal Luiz Coelho Roger Granada David Menotti|









    


TCDiff: Triple Condition Diffusion Model with
 3D Constraints for Stylizing Synthetic Faces
    SIBGRAPI Paper ID: 
 
    
===========================================================================================



[
979-8-3503-7603-6/24/$31.00  2024 IEEE]

§ ABSTRACT 
A robust face recognition model must be trained using datasets that include a large number of subjects and numerous samples per subject under varying conditions (such as pose, expression, age, noise, and occlusion).
Due to ethical and privacy concerns, large-scale real face datasets have been discontinued, such as MS1MV3, and synthetic face generators have been proposed, utilizing GANs and Diffusion Models, such as SYNFace, SFace, DigiFace-1M, IDiff-Face, DCFace, and GANDiffFace, aiming to supply this demand. 
Some of these methods can produce high-fidelity realistic faces, but with low intra-class variance, while others generate high-variance faces with low identity consistency. 
In this paper, we propose a Triple Condition Diffusion Model (TCDiff) to improve face style transfer from real to synthetic faces through 2D and 3D facial constraints, enhancing face identity consistency while keeping the necessary high intra-class variance.
Face recognition experiments using 1k, 2k, and 5k classes of our new dataset for training outperform state-of-the-art synthetic datasets in real face benchmarks such as LFW, CFP-FP, AgeDB, and BUPT. Our source code is available at: .












§ INTRODUCTION


In recent years, the availability of large face recognition datasets containing thousands of real faces, such as CASIA-WebFace <cit.>, VGGFace2 <cit.>, MS1MV3 <cit.>, WebFace260M <cit.>, and Glint360K <cit.>, has contributed to remarkable advancements in face recognition across various challenging domains, including pose, age, occlusions, and noise. 
With such data, deep neural networks trained with sophisticated angular margin loss functions, such as SphereFace <cit.>, CosFace <cit.>, ArcFace <cit.>, CurricularFace <cit.>, MagFace <cit.> and AdaFace <cit.>, have achieved impressive performances on different benchmarks.

However, datasets of this nature present critical ethical, annotation, and bias problems <cit.>. 
Furthermore, the long-tailed distribution of samples in many datasets poses additional challenges, necessitating careful network architecture and loss function design to ensure the robustness of model generalization. 
These challenges also make it difficult to explore facial attribute influences like expression, pose, and illumination. In contrast, learning-based face recognition models encode facial images into fixed-dimensional embedding vectors, enabling various tasks like identification and verification.
While publicly available datasets have driven recent progress, they come with associated problems.
Synthetic datasets offer a potential solution, providing privacy benefits, virtually unlimited data generation, and control over demographic characteristics. 
This contrasts with real-world datasets, which are constrained by privacy regulations and representational biases.

Due to such advantages, synthetic faces in face recognition have attracted attention for their potential to mitigate privacy concerns and long-standing dataset biases, such as long-tail distributions and demographic imbalances. Recently, face recognition competitions using synthetic faces have been held, such as FRCSyn <cit.> <cit.> and SDFR <cit.>, showing the increasing interest of the research community on this topic.



Generating synthetic faces and manipulating attributes such as pose, expression, age, noise, and occlusion with high visual fidelity and identity consistency is a challenging task due to the ill-posed nature of representing 3D objects in 2D planes. Therefore, 3D facial constraints might improve model learning and reduce facial inconsistencies.

In this paper, we propose a Triple Condition Diffusion Model (TCDiff) to stylize a synthetic identity face with real style attributes from real faces, such as pose, expression, age, noise, occlusion, shadow, hair, etc., with 2D and 3D consistency constraints, aiming to enhance intra-class identity consistency. 
Fig. <ref> presents an overview of our method and the constraints computed with identity image (X_id), style image (X_sty), and stylized image (X̂_0). 
Our experimental results show that enhancing intra-class identity consistency improves synthetic dataset quality when training face recognition models with few classes.





§ RELATED WORK


Face synthesis has emerged as a prominent area of research, driven by advancements in deep generative models like GANs <cit.> and Diffusion Models <cit.>. Such methods excel at generating high-quality facial images with unique identities. However, they often lack the intra-class variance necessary to train powerful face recognition models. In this regard, recent approaches explore such variance in real-face datasets, mixing synthetic and real images to generate multiple samples from the same synthetic subject.

SynFace <cit.> proposes a Mixup Face Generator designed to create synthetic face images with different identities. To mitigate the domain gap between the synthetic and real face data, the method incorporates a Domain Mixup module regularized by an angular margin loss. In contrast, SFace <cit.> trains a StyleGAN2-ADA <cit.> using identity labels as conditional constraints. 
These constraints are similarly regularized by an angular margin loss.

DigiFace-1M <cit.> employs the 3DMM-based model FaceSynthetics <cit.> to generate multiple synthetic faces, varying their expression and pose parameters. The rendering pipeline from FaceSynthetics further enhances flexibility by allowing modifications in the background and illumination settings in the images. After generating original images, data augmentation techniques including flipping, cropping, adding noise, blurring, and warping are applied to improve the face recognition performance. Despite the 3D consistency in images, this dataset has limitations in appearance due to the intrinsic synthetic texture of faces.

GANDiffFace <cit.> combines the strengths of GANs and Diffusion Models to produce realistic faces while incorporating intra-class variance. Initially, synthetic faces are generated using StyleGAN3 <cit.> trained on the FFHQ dataset <cit.>, and grouped based on by extracted facial attributes such as pose, expression, illumination, gender, and race. Support Vector Machine (SVM) classifiers are trained to distinguish each group. The normal vectors concerning the resulting separation hyperplanes are used as directions to edit facial attributes of faces in latent space. Despite achieving realistic appearances for the generated synthetic faces, there remains a gap in the evaluation performance using real face testing benchmarks of face recognition models, trained on their synthetic dataset.

IDiffFace <cit.> uses a Diffusion Model trained on the FFHQ <cit.> real face dataset to create new synthetic faces. These faces are generated using a U-Net-based architecture enriched with residual and attention blocks to encourage the model to improve the intra-class variance generation ability. To prevent overfitting and ensure diverse outputs, they also propose a Contextual Partial Dropout (CPD) technique.

DCFace <cit.> introduces an approach to minimize the distribution gap between synthetic and real face datasets by employing a diffusion model. This model integrates visual constraints to transfer the stylistic characteristics of real faces onto synthetic faces, thereby enhancing the intra-class variance. Initially trained on the CASIA-WebFace <cit.> real face dataset, the model utilizes statistics derived from intermediate features of images, assuming these contain style information to be transferred to any other identity. Despite these efforts, some artifacts and identity inconsistencies persist in the synthetic output.

Fig. <ref> illustrates some samples generated by the aforementioned methods. Each row corresponds to samples of distinct synthetic subjects, while columns contain different samples of the same subject. While the limited number of images may not be sufficient to provide an accurate visual analysis, they offer some initial intuitions about their characteristics, such as intra-class variance and identity consistency.






§ FACE STYLE TRANSFER


Image style transfer is a technique that generates novel images by merging the content of one image with the stylistic elements of another  <cit.>. This process leverages Convolutional Neural Networks (CNNs) pre-trained on image classification tasks to extract hierarchical intermediate feature representations. The content of an image is captured by the higher layers of the network, which encode the image's structure and objects, while the style is represented by the correlations between feature maps in the lower layers, known as Gram matrices. To achieve style transfer, the technique minimizes a loss function that combines the content loss, which measures the difference between content representations of the original and generated images, and style loss, which quantifies the difference between style representations of the original and generated images. By iteratively adjusting a white noise image based in this combined loss function, the network gradually synthesizes the final output that maintains the content of one image while adopting the style of another.

In the face recognition field, a person's identity can be expressed mainly by facial parts such as eyes, nose, lips, eyebrows, etc., and their spatial position in the face. Meanwhile, style is related to facial pose, expression, age, noise, occlusion, color, etc <cit.>. Achieving a perfect disentanglement of identity and style representation remains a significant challenge in deep learning. Existing style transfer methods aim to manage this tradeoff depending on the final goal <cit.>.




§ PROPOSED APPROACH


For face style extraction, we adopt the proposed model E_sty of DCFace <cit.>, which uses intermediate feature maps I_sty∈ℝ^C × H × W extracted with a pre-trained and fixed weights face recognition model F_s from a given input face image X_sty, where C, H and W are the number of channels, height, and width of the feature maps, respectively. Each feature map is divided into a grid k × k and each element I_sty^k_i∈ℝ^C ×H/k×W/k is mapped on the mean and variance of I_sty^k_i as


    Î^k_i = BN(Conv(ReLU(Dropout(I_sty^k_i)))),


    μ_sty^k_i = SpatialMean(Î^k_i),    σ_sty^k_i = SpatialStd(Î^k_i),


    s^k_i = LN((W_1 ⊙μ_sty^k_i + W_2 ⊙σ_sty^k_i ) + P_emb),


    E_sty(X_sty) := s = [s^1, s^2, s^k_i, ..., s^k × k, s'],


where s' corresponds to Î_sty^k_i being a global feature, where k = 1.  P_emb∈ℝ^50 × C is a learned position embedding <cit.> that makes the model learn to extract patches styles according to their locations in X_sty . BN and LN are BatchNorm <cit.> and LayerNorm <cit.> operations.

The face style embedding E_sty extracted from X_sty is then applied to an identity image X_id using a U-Net denoising diffusion probabilistic model (DDPM) <cit.> face mixer ϵ_θ(X_t, t, E_id(X_id), E_sty(X_sty)), whose architecture is illustrated in Fig. <ref>. X_t is a noisy version of X_sty at time-step t and E_id is a face recognition model ResNet50 <cit.> responsible for extracting a discriminant identity embedding.




Given an identity image X_id and a style image X_sty, a new stylized image X̂_0 is obtained as


    X̂_0 = (X_t - √(1 - α̅)ϵ_θ(X_t, t, X_id, X_sty)) / √(α̅_t).


where α̅_t is a pre-set variance scheduling scalar <cit.>.

To train the face mixer ϵ_θ, we first employ the mean squared error (MSE) loss L_MSE between style image X_sty and stylized image X̂_0


    L_MSE = 1/MN∑_i=1^M∑_j=1^N( X_sty(i,j) - X̂_̂0̂(i,j) )^2


to enforce the model to preserve relevant style features of X_sty in X̂_̂0̂. Additionally, to balance identity and style features of X_id and X_sty in X̂_0, we also employ an identity loss L_ID through cosine similarity (CS)


    L_ID =    - γ_t CS(F(X_id), F(X̂_0)) 
       - (1-γ_t)CS(F(X_sty), F(X̂_0))


where γ_t ∈ℝ  |   0 ≤γ_t ≤ 1.

To enhance intra-class identity consistency when stylizing synthetic faces, we also propose to add a 3D facial shape loss L_3D


    L_3D = √(∑ (x_id^3D - x̂_0^3D)^2)


which computes the Euclidean Distance between the 3DMM <cit.> shape feature vectors x_id^3D and x̂_0^3D, obtained from X_id and X̂_0. Due to the lack of large datasets containing both 2D and 3D scanned representations of real facial, we obtained 3D Morphable Model (3DMM) coefficients during training using the 3D face reconstruction model MICA <cit.>.





The shape of a face in a 3DMM representation is described by the positions of a set of 3D vertices S = (x_1, y_1, z_1, x_2, y_2, z_2, …, x_n, y_n, z_n)^T ∈ℝ^3n. These vertices form a mesh that captures the geometric structure of the face. Mathematically, the shape vector S can be expressed as a linear combination of a mean shape S̅ and a set of shape basis vectors S_i:


    S = S̅ + ∑_i=1^nα_i S_i,


where α_i are the shape coefficients that determine how much each basis vector S_i contributes to the final shape. The shape vector S consists of the 3D coordinates of all vertices in the mesh.

Similarly, the face texture is described by a vector T = (R_1, G_1, B_1, R_2, G_2, B_2, …, R_n, G_n, B_n)^T ∈ℝ^3n, which is a linear combination of a mean texture T̅ and a set of texture basis vectors T_i:


    T = T̅ + ∑_i=1^mβ_i T_i,


where β_i are the texture coefficients that control the contribution of each texture basis vector T_i. The texture vector T consists of the RGB color values for each vertex in the mesh.

The basis vectors S_i and T_i, for shape and texture, are derived from a Principal Component Analysis (PCA) over a set of real 3D face scans, which allows representing new faces within the training set variance.

To obtain the 3DMM facial shape coefficients x_id^3D and x̂_0^3D from identity face X_id and stylized face X̂_̂0̂, the MICA <cit.> method uses the face embedding produced by a state-of-the-art 2D face recognition network <cit.> as input to a small mapping network z = M(ArcFace(I)) ∈ℝ^300. Therefore, x_id^3D = M(X_id) and x̂_0^3D = M(X̂_̂0̂).

Finally, our total loss function L_T is defined as


    L_T = L_MSE + λ_id L_ID + λ_3D L_3D,


where λ_id and λ_3D are scaling parameters to balance the importance of 2D and 3D facial identity constraints.




§ EXPERIMENTS


This section presents our experimental setup, datasets, obtained results, and qualitative analysis. To fairly evaluate the robustness of our proposed TCDiff face mixer, we adopted the same protocol of DCFace <cit.> by using the real faces dataset CASIA-WebFace <cit.> as the training set and a grid 5 × 5 for style feature extraction. Our model was trained for 10 epochs with batch=16 using AdamW Optimizer <cit.> with the learning rate of 1e-4 on one GPU NVIDIA GeForce RTX 3090. We set λ_ID=0.05 and varied λ_3D={0.001, 0.005, 0.01, 0.05} to analyse the impact of 3D consistency constraints.

After training TCDiff, we selected the same 10k distinct synthetic identities of DCFace <cit.>, which were generated using the publicly released unconditional DDPM <cit.> trained on FFHQ <cit.>. Each synthetic identity was stylized with 50 randomly chosen real faces of CASIA-WebFace <cit.>, resulting in a new synthetic dataset of 500k images. Fig. <ref> shows 1 synthetic face image X_id, 16 real faces X_sty from CASIA-WebFace, and their corresponding 16 new samples of X_id.

We choose ResNet50 <cit.> backbone and ArcFace <cit.> loss as Face Recognition (FR) model to evaluate the quality of the proposed synthetic dataset in cross-dataset scenarios using seven different datasets in face verification (1:1) task: LFW <cit.> CFP-FF <cit.>, CPLFW <cit.>, CFP-FP <cit.>, AgeDB <cit.>, CALFW <cit.>, and BUPT-CBFace <cit.>.
These datasets are commonly applied in FR to validate or test models. Each dataset contains a verification protocol consisting of face pairs labeled as genuine (same person) or impostor (different person).

LFW (6k pairs) and CFP-FF (7k pairs) protocols are mainly focused on frontal face verification, representing controlled scenarios. Otherwise, CPLFW (6k pairs) and CFP-FP (7k pairs) contain faces with more varied poses to simulate in-the-wild scenarios. AgeDB (7k pairs) and CALFW (6k pairs) focus on comparing faces with large age differences, while BUPT-CBFace (8k pairs) contains the same number of pairs of 4 distinct ethnic groups: Asian, Caucasian, African, and Indian. All protocols were split into 10-fold containing 50% of genuine and 50% of impostor pairs. Following the cross-validation method, we use 9 folds to select the best threshold and 1 for the final test.




§ DISCUSSION


In this section, we present a qualitative and quantitative analysis of the results obtained with the synthetic datasets generated with DCFace <cit.> and our proposed model TCDiff for the face recognition task.

Even with few samples, we can visually observe in Fig. <ref> that synthetic faces stylized by DCFace <cit.> have low intra-class consistency, compared to its corresponding original synthetic identity X_id. For instance, the stylized male faces 0, 8, and 10 seem to belong to distinct identities. 
The same happens with the stylized female faces 2 and 9. 
Otherwise, our face mixer ϵ_θ tends to preserve identity regions, such as eyes, nose, and mouth of original X_id in stylized faces to enhance the intra-class consistency imposed by L_3D when λ_3D varies from 0.001 to 0.01.
Stylized faces are completely degraded when λ_3D = 0.05 and this setting was ignored in our experiments.



Such a qualitative analysis is quantitatively confirmed in Fig. <ref>, where intra-class cosine similarities are presented.
Blue bars show the distributions of all ∑_i=0^M C^N_i_2 similarities, where M is the number of classes and N_i is the number of samples of the i-th class, while orange bars show the distributions of the mean intra-class similarities.
One can observe higher similarities in faces stylized with our model TCDiff, indicating a higher intra-class consistency due to the shape of eyes, nose, lips, skin color, pose, expression, and facial accessories. 

The quality of a face dataset for FR task is assessed not just by the images themselves, but by the performances of FR models trained on it. Results in Table <ref> show that enhancing intra-class consistency improves synthetic datasets quality when training with 1k, 2k, and 5k classes. We hypothesize that such an identity consistency improves inter-class separability when the number of distinct identities are low.

However, this improvement is surpassed when training with 10k classes, indicating that the inter-class variability is also an important property of synthetic datasets for face recognition.
ResNet50 performed slightly better on dataset CFP-FF <cit.> when training with stylized images by our model TCDiff (λ_3D=0.001), due to the greater existence of frontal faces in such a dataset.




§ CONCLUSION AND FUTURE WORK


In this work we propose TCDiff, a face style transfer trained with 2D and 3D facial constraints, aiming to improve the quality of synthetic datasets for face recognition. By increasing the importance of 3D constraints, our model can preserve identity features of the input synthetic face to be stylized, which enhances intra-class identity consistency.

This behavior contributes to increasing the quality of small synthetic datasets and might be explored in the future for more classes, as the interest in this field is growing due to the advantages of synthetic data. As a future step, facial expression and pose constraints might be added to the face styler model, aiming to balance better the importance of identity and style features in newly generated samples.



§ ACKNOWLEDGMENT


This work was supported by
a tripartite-contract, i.e., unico - idTech, UFPR (Federal University of Paraná), and FUNPAR (Fundação da Universidade Federal do Paraná).  
We thank the Federal Institute of Mato Grosso (IFMT), Pontes e Lacerda, for supporting Bernardo Biesseck, and also thank the National Council for Scientific and Technological Development (CNPq) (# 315409/2023-1) for supporting Prof. David Menotti.
ANONYMOUS.








IEEEtran






