Experimental Evaluation sec:experiments We first investigate the nature of
dataset bias by exploring each facial attribute and focus on the facial
attributes which induce much bias. Furthermore, we empirically investigate the
effectiveness of our method on two fashion tasks involving bias — (1) sex
classification and (2) facial attribute classification. In sex classification,
we concurrently manipulate multiple femininity/masculinity attributes to
generate paired images with female and male appearance and construct sex-level
balanced dataset. Meanwhile, in facial attribute classification, in the basis
of sex-level balanced dataset, we further consider the other non-sex-related
facial attributes to construct the semi-synthetic dataset which is balanced in
terms of both sex and facial attribute. In both experiments, we first show
that the classification performance of the model trained with synthetic
datasets achieves comparable performance as the model trained on the original
dataset. Then, we conduct a comprehensive fairness evaluation with a wide
range of bias assessment metrics
directional_bias_amplification,SensitiveNets,divergence_between_score_distributions,correlation_distance,RLB
to show that fairness has been improved after training with synthetic datasets
compared to the model solely trained on the original dataset. Further, we
demonstrate that our method outperforms both strategic resampling
debias_resampling1 and balanced dataset construction method with synthetic
images hat_glasses_correlation to address dataset bias, and several debiasing
models debias_adversarial_uniform_confusion_ACC_unbalanced_dataset2_LAOFIW,
Image_caption_Bias_amplification_yz,debias_domain_independent_training_BA_usage
to address task bias. Finally, we present an ablation study to evaluate
different factors influencing the performance of our method. Although we
discuss sex as the protected attribute in this section, our method is general
purpose and can be used for all protected attributes if the labels are
available.

Attributes Study subsec:attributes_study Before presenting the improvement on
fairness, we first conduct in-depth evaluation of dataset bias at the facial
attribute level for CelebA dataset CelebA, which is a face dataset containing
202,599 images of celebrity faces and 40 binary attributes per image. Since
our method produces a facial attribute-level balanced synthetic dataset, it is
valuable to study dataset bias at facial attribute level instead of the
general overall accuracy for all facial attributes. Following
hat_glasses_correlation, we train a multitask facial attribute classifier with
ResNet-50 ResNet50 to recognize facial attributes on the sex-level balanced
CelebA dataset. The main results are shown in tab:multi_attributes_recognition
as baseline and full results are presented in the appendix.

As shown in tab:multi_attributes_recognition by results of baseline, solely
balancing across sex does not guarantee fairness in facial attribute
classification since even with balanced training across females and males, the
imbalance of facial attribute across sex still exists. Although we know
classification models tend to learn distinguishable representations from
positive samples instead of negative samples negative_positive, there are
insufficient positive samples of some specific facial attributes in the
minority group. Inspired by the categorization in hat_glasses_correlation
where they only categorize a subset of facial attributes, we summarize all 40
facial attributes in CelebA dataset into three groups — (1) unbiased
attributes which do not yield much bias (the difference of AP between female
and male is less than 5%), (2) masculinity/femininity attributes, which are
considered as AOI in subsec:sex_classfication to construct sex-level balanced
dataset, and (3) non-sex-related facial attributes but inducing much bias even
with sex-level balanced training, which are considered as AOI appending
masculinity/femininity attributes in subsec:attribute_recognition to construct
both sex-level and facial attribute-level balanced dataset.

##  Synthetic Attribute-level Balanced Datasets

To construct the synthetic dataset $\mathcal{D}_{syn}$, we train the
generative model solely on the training set of CelebA dataset containing
162,770 images to ensure the isolation from the testing set since the images
generated by trained GAN are used for the following tasks. For the generative
model choice to generate high-quality images (which is elaborated in
appendix), we use StyleGAN2 [styleGan2] since _style mixing regularization_ in
StyleGAN2 facilitates our goal to assign different specific facial attributes
and mitigate the interference from other unassigned facial attributes. We
persist in training the generative model to generate more real images. As a
standard metric, FID is used to evaluate sample qualities of generated images.
FID of the presented results in the paper is $3.56$, which is comparable to
existing GANs [FID_comparable1, FID_comparable2] trained on CelebA dataset and
posted recently. Although the benchmark for FID of image generation on CelebA
dataset is $2.71$ [FID_benchmark_CelebA_64], our objective is to utilize
generated images to improve fairness of existing datasets rather than improve
image quality of generated images. To assign attributes, we use
$\texttt{intra\\_threshold}=2\sqrt{2}$ and inter_threshold $=\sqrt{2}$. In
general, we empirically found that any settings for these two hyperparameters
among the recommended range $[\sqrt{2},2\sqrt{2}]$ will not significantly
affect the performance of downstream classification tasks, which confirms with
our theoretical discussion in subsec:ablation_study. Furthermore, for
resolution spectrum separation, StyleGAN2 [styleGan2] have already provided
reference to assign different attributes in different resolutions, high-level
attributes (hair style, face shape) in coarse spatial resolution
$(4^{2}-8^{2})$, and racial appearance (colors of eyes, hair, skin) in fine
spatial resolution $(16^{2}-1024^{2})$. The dimensionality of the latent space
is 512 and the resolution of generated images is set to be $256^{2}$ so that
by [styleGan2] we generally have a 14-layer 512-dimensional latent space to
assign attributes. Specifically, we leave $4^{2}$ original for the basic
identity construction. For resolution choices to assign attributes, we assign
face shape attributes (Chubby, Big Nose, Pointy Nose, High Cheekbones and
Double Chin) in $8^{2}$, fine face shape attributes (Bags Under Eyes, Wavy
Hair and Straight Hair) in $16^{2}$, hair color attributes (Black Hair, Blond
Hair, Brown Hair and Gray Hair) in $32^{2}-64^{2}$, and skin color attributes
(Pale Skin) in $128^{2}-256^{2}$. Besides, we assign _masculinity/femininity
attributes_ in $8^{2}-16^{2}$. As shown in fig:sex, the generated images are
listed in pairs with highlighted _femininity/masculinity attributes_ , and
similar non-sex facial attributes (skin color and hair color) and other photo
settings (lighting, pose and background). Besides, in fig:CAT, we controllably
assign _femininity/masculinity attributes_ and another facial attribute.
Furthermore, in fig:grid, we concurrently assign sex and multiple facial
attributes and generally keep other unassigned facial attributes unchanged,
which demonstrates the advantage that attribute assignment does not interfere
with each other.

##  Sex Classification

As in [FairFace, PPB_metric_Joy], the imbalance of training dataset may hamper
the accuracy of sex classification, we therefore validate the fairness
improvement of our synthetic datasets on sex classification in this section.
To help downstream classification models to well learn the difference between
female and male (the useful information for sex classification), given an
identity seed, we only modify _femininity/masculinity attributes_ and leave
other facial attributes unchanged, as shown in fig:sex, which also ensures
that there are sufficient samples with large diversity of facial attributes
from identity seeds in both female group and male group. Experiment setup.
ResNet-50 [ResNet50] is used as the backbone for sex classifier. The baseline
model is trained with original CelebA dataset [CelebA] under two settings —
(1) Original to keep the training set of CelebA dataset fully original with
the whole set of 162,770 images, and (2) Balanced to balance the training set
across sex. For training with synthetic dataset, we also conduct two types of
experiments — (1) Same size to combine one half of balanced original training
set with a same size balanced synthetic dataset to be mixed dataset so that
the number of mixed dataset is the same as the number of training images in
the balanced original dataset, and (2) Supplement to supplement the lack of
male images compared with female images in the original dataset so that the
mixed dataset is balanced in the whole dataset scale. For comparison with the
other attribute-level balance method [hat_glasses_correlation], we conduct
experiments under the original setting in their paper (the original CelebA
dataset combined with the balanced synthetic dataset with 160,000 sex pairs of
images) and two same settings of ours. The testing set is the original testing
CelebA dataset for all experiments. Evaluation protocol. To demonstrate the
effectiveness of our method on sex classification and facial attribute
classification, we use accuracy and fairness as the main evaluation metrics.
First, we compare the classification performance between the model trained on
synthetic datasets and the original dataset to verify that the testing
accuracy on the real testing dataset is preserved at the same level, which
demonstrates that the generated images yield proper facial attributes in the
metrics of the real dataset. Meanwhile, we use several fairness metrics for a
comprehensive fairness comparison since different types of metrics assess bias
in different directions. According to the taxonomy of bias assessment metrics
in [RLB], we select several representative metrics in these categories. We
select Bias Amplification (BA) [directional_bias_amplification,
debias_domain_independent_training_BA_usage] among _information leakage_
-based metrics and two types of metrics based on _statistical dependence_ ,
Distance Correlation (${dcor}^{2}$) [correlation_distance,
BR_net_correlation_distance_bias_metric] and Representation-Level Bias (RLB)
[RLB] to increase the diversity of metrics and comprehensively evaluate
fairness in different ways. Results. As shown in tab:sex_classification, we
can see the testing accuracy of synthetic dataset is kept at the same level
under both two settings, which demonstrates that the generated images yield
proper _femininity/masculinity attributes_ in the original dataset.
Furthermore, although the classification accuracy is already saturated, with
additional images under the Supplement setting, we improve the accuracy in
minority groups. On the other hand, for fairness evaluation compared with
other datasets, our method further improves fairness. Although
[hat_glasses_correlation] uses more images than the original dataset, our
method yields even better results in both accuracy and fairness metrics with
fewer training images. We elaborate the discussion on the relation between the
size of the synthetic dataset and performance of our method in the appendix.

[width=0.5]sections/figure/grid.png

Figure : Examples of generated images with multiple assigned facial
attributes.

##  Facial Attribute Classification

Having found that the remaining _dataset bias_ of facial attribute
classification on CelebA dataset stems from imbalance at both _protected
attribute_ level and facial attribute level as discussed in
subsec:attributes_study, we construct the synthetic dataset which is balanced
in both these two levels to study the effectiveness of our method for all non-
sex-related facial attributes, and compare our method with methods based on
sex-level balance [debias_resampling1] and facial attribute-level balance
[hat_glasses_correlation]. Furthermore, we highlight the comparisons with
debiasing models
[debias_adversarial_uniform_confusion_ACC_unbalanced_dataset2_LAOFIW,
Image_caption_Bias_amplification_yz,
debias_domain_independent_training_BA_usage]. Experiment Setup. We train a
multitask facial attribute classifier with ResNet-50 [ResNet50]. The baseline
is trained with sex-level balanced CelebA dataset constructed from original
CelebA dataset. Our method combines a portion of original dataset without
surplus samples of some facial attributes and the synthetic dataset which
supplements insufficient samples of AOI so that the mixed dataset is balanced
at both sex level and facial attribute level, as illustrated in fig:idea.
Evaluation protocol. We use Average Precision (AP) as the performance metric.
For bias assessment, beside BA, ${dcor}^{2}$ and RLB, Difference in Equal
Opportunity (DEO) [SensitiveNets, HSIC] as a metric version of _equal
opportunity_ and KL-divergence between score distribution (KL)
[divergence_between_score_distributions, hat_glasses_correlation] as a
stronger notion stemming from _equalized odds_ are used to verify the
achievements of EO,EOS. Results. In tab:multi_attributes_recognition, we
present one representative facial attribute for the mutually exclusive facial
attributes, Wavy Hair and Straight Hair, and full results are presented in
appendix. As AP is preserved at the same level for all AOIs, our method
outperforms other methods for Blond Hair and High Cheekbones. In fairness
comparison with baseline, all three methods introduce fairness under all
metrics in a same trend. As pointed by [RLB], ${dcor}^{2}$ and RLB, the
metrics based on _statistical dependence_ are more consistent and stable. In
this sense, although strategically resampling
[debias_resampling_representation_bias_colored_MNIST] outperforms our method
under BA, it yields second worse performance on the metrics based on
_statistical dependence_. Furthermore, with better DEO and KL, we verify the
achievement of EO,EOS in sec:method, which is unreachable for sex-level
balanced dataset and strategically resampling. According to results of
imbalanced training, due to the existence of the facial attribute-level skew
in the dataset, the classifier trained with these skew datasets may be biased
already. In this sense, compared with [hat_glasses_correlation], our method is
more stable in all facial attributes since we assign label directly rather
than rely on such unreliable facial attribute classifier trained on original
dataset as in [hat_glasses_correlation], which may be harmful for fairness.
Compared with tab:sex_classification, the range of two metrics based on
_statistical dependence_ in sex classification are clearly higher than the
range in the facial attribute classification since they directly reveal the
statistical dependence between learned representations to predict sex and sex
labels themselves, which better assess the bias for sex classification. We
also conduct an overall comparison of average results over all non-sex-related
AOIs (7 facial attributes in the paper and 7 mutually exclusive facial
attributes in appendix) with several debiasing models based on adversarial
forgetting
[debias_adversarial_uniform_confusion_ACC_unbalanced_dataset2_LAOFIW], domain
adaptation [Image_caption_Bias_amplification_yz] and domain independent
[debias_domain_independent_training_BA_usage]. We train a ResNet-50 [ResNet50]
to classify all non-sex-related AOIs on the original CelebA dataset as
baseline model. For the multi-label classification, mean Average Precision
(mAP) is used to evaluate the overall performance. In tab:comparison of
classification performance comparison, domain independent method outperforms
other methods. In parallel, without the in-process debiasing techniques usage,
our method yields comparable classification performance and outperforms other
methods in all bias assessment metrics, which demonstrates a comprehensive
fairness improvement. In general, there is a tradeoff between classification
performance and fairness performance, although domain independent method
outperforms other methods in mAP, our method improves fairness mostly at an
acceptable expense of classification performance. Furthermore, the proposed
synthetic dataset is model-agnostic as compared under different backbones in
appendix.

##  Ablation Study

_Intra-class similarity_ and _inter-class difference_. In this section, we
will discuss _intra-class similarity_ and _inter-class difference_ involving
two hyperparameter intra_threshold and inter_threshold. Considered the task to
generate female and male images for which, given _masculinity attributes
seeds_ $S_{male}$ and _femininity attributes seeds_ $S_{female}$, _intra-class
similarity_ is deprecated when intra_threshold is small since there may be no
dimension indice where $\forall e_{i},e_{j}\in S_{male},|e_{i}-e_{j}|<$
intra_threshold, $A^{intra}_{male}$ is empty. By contrast, _inter-class
difference_ will be deprecated when $inter\\_threshold$ is large. For example,
given an identity latent vector $e_{ID}$ whose generated image is female, when
_intra-class similarity_ and _inter-class difference_ are both deprecated, the
generated image is same as the prime image generated from $e_{ID}$. As shown
in fig:deprecated, without _intra-class similarity_ , the generated image is
apparently female. Further, when we apply _intra-class similarity_ and
deprecate _inter-class difference_ , the generated image is transferred to be
male with few _femininity attributes_ (hair) since the generated model has
learned _masculinity attributes_ but has not fully suppressed _femininity
attributes_. On the other hand, if intra_threshold is large or inter_threshold
is small, $A^{intra}_{male}$ or $B^{inter}_{male}$ contains all dimension
indices so that _intra-class similarity_ or _inter-class difference_ is
overemphasized. As shown in fig:overemphasized, compared with generated images
under proper threshold settings, the generated images under strong threshold
settings in the top row are cookie-cutter without the randomness from identity
seeds. In the whole process, _intra-class similarity_ contributes mostly and
_inter-class difference_ plays an auxiliary role. Since the random latent
vector is simulated under standard normal distribution $N(0,1)$ and the
difference between two latent vectors follows normal distribution
$N(0,\sqrt{2})$ where standard deviation is $\sqrt{2}$, the proper range for
two thresholds is $[\sqrt{2},2\sqrt{2}]$ so that $A_{Y}^{inter}$ may contain
the dimension indices in the main lope representing attribute similarity among
$S_{Y}$, and $B_{Y}^{intra}$ may exclude the dimension indices in the main
lope representing attribute similarity of $S_{Y}$ and $S_{\bar{Y}}$, where
$S_{\bar{Y}}$ is the set of latent vectors without _Attribute of Interest_
$\mathcal{Y}$.

[][Deprecated.][width=.78]sections/figure/intra_inter.png

[][Overemphasized.][width=.6]sections/figure/too_strong.png

Figure : The role of intra-class similarity and inter-class difference.