k-Same-Siamese-GAN: k-Same Algorithm with Generative Adversarial Network for Facial Image De-identification with Hyperparameter Tuning and Mixed Precision Training
Yi-Lun Pan Min-Jhih Huang Kuo-Teng Ding Ja-Ling Wu Jyh-Shing Jang









k-Same-Siamese-GAN: k-Same Algorithm with Generative Adversarial Network for Facial Image De-identification with Hyperparameter Tuning and Mixed Precision Training
    Yi-Lun Pan, Min-Jhih Huang, Kuo-Teng Ding, Ja-Ling Wu, Jyh-Shing Jang

Department of Computer Science & Information Engineering, National Taiwan University, Taiwan

No. 1, Sec. 4, Roosevelt Rd., Taipei 10617, Taiwan

{d06922016,r07922049,r07922009,wjl,jang}@csie.ntu.edu.tw






    
===========================================================================================================================================================================================================================================================================================






For a data holder, such as a hospital or a government entity, who has a privately held collection of personal data, in which the revealing and/or processing of the personal identifiable data is restricted and prohibited by law. Then, “how can we ensure the data holder does conceal the identity of each individual in the imagery of personal data while still preserving certain useful aspects of the data after de-identification?" becomes a challenge issue. In this work, we propose an approach towards high-resolution facial image de-identification, called k-Same-Siamese-GAN, which leverages the k-Same-Anonymity mechanism, the Generative Adversarial Network, and the hyperparameter tuning methods. Moreover, to speed up model training and reduce memory consumption, the mixed precision training technique is also applied to make kSS-GAN provide guarantees regarding privacy protection on close-form identities and be trained much more efficiently as well. Finally, to validate its applicability, the proposed work has been applied to actual datasets - RafD and CelebA for performance testing. Besides protecting privacy of high-resolution facial images, the proposed system is also justified for its ability in automating parameter tuning and breaking through the limitation of the number of adjustable parameters.





§ INTRODUCTION

The protection of facial images is getting higher and higher awareness recently, especially when personal images and videos are easily captured by pervasively equipped high resolution visual devices, such as smart phones and surveillance cameras. These devices did simplify the image- and video- footage capturing tasks a lot; nevertheless, attention should be paid to the misuse of the captured imagery data especially when they are stored on a datacenter. De-identification <cit.> is one of the basic methods aims at protecting the privacy of imagery data and granting their legal usage, at the same time.

In facial image de-identification problems, there are two kinds of dilemma. One is to make the surrogate images as different from the original image as possible so as to ensure the removal of any personal identifiable attribute. The work <cit.> presented a facial image de-identification method with expression preservation capabilities based on variational adaptive filters, where the filtering process preserves most of the important facial image characteristics (i.e., eyes, gazed regions, lips and their corners).

The other dilemma is to make surrogate images retain as much structural information in the original image as possible so that the image utility remains. Therefore, the existing facial image de-identification procedures based mostly on the k-same algorithms <cit.> and leveraged some facial feature preserving techniques, such as the Active Appearance Models (AAMs) and the Principal Component Analysis (PCA) method <cit.>, to explicitly construct morphing-faces for preserving the facial utility attributes like age, race, and gender, as much as possible. Unfortunately, the results are usually too fake to be applied in real world, due to the corresponding poor visual quality and unnatural appearance. Since only traditional de-identification algorithms are involved in the above words, it is our belief that there is still a large room left for performance enhancing with the aid of advanced Deep Neural Network (DNN) related architectures. Moreover, even if DNN-related algorithms and/or architectures are applied, the associated large time spent and huge amount of memory consuming in GPGPUs become the critical obstacles needed to be conquered.

To tackle the above-mentioned obstacles, in this work, a novel GAN-based facial image de-identification system is proposed, which relies on the formal privacy protection ability provided by the above-mentioned k-Same approaches. To release the above-mentioned obstacle-issues in applying DNNs, we not only designed an effective label-scheme function for generating de-identified image set automatically, but also took appropriate hyperparameter tuning into account. In addition, for enhancing the re-identification ability, a recently proposed image recognition NN, called the Siamese Network <cit.>, has been modified and integrated into our kSS-GAN. 










Comparing with related works, the proposed facial image de-identification system provides better privacy protection ability, surrogate image quality, and training performance by leveraging the superiority of the mixed precision training (MPT) technique. The main contributions of this work can then be summarized as follows. (1) A novel GAN-based de-identification algorithm for protecting the privacy and reserving the utility of high-resolution facial images at the same time, is proposed. (2) A novel labelling scheme to automatically generate the de-identified image set, is proposed. (3) By combining with the k-Same algorithm, the guarantees of the proposed system against privacy invading of imagery data, is provided. (4) With the aid of MPT, the training process of the proposed work has been speeded up a lot. (5) By Integrating with appropriate automatic Machine Learning (autoML) toolkits, such as Advisor <cit.>, better hyperparameters turning is achieved.










§ RELATED WORKS



 §.§ k-Anonymity and Face De-identification


The concept of k-Anonymity was addressed by Sweeney <cit.> and had been applied to de-identify the entries of a relational database. Sweeney said a table is conformed to the k-Anonymity (after de-identification) if and only if each sequence of values, for each attribute, in the table appears at least k times. Since there are k copies of the same sequence in the data set, the probability of connecting a single sequence to the original sequence is bounded by 1/k. The k-Anonymity model inspired a series of the so-called k-Same de-identification algorithms. The following present brief reviews of some highly relevant articles to this study and analyze their pros and cons.


Gross  <cit.> proposed a model-based face de-identification method. The weaknesses of this work include - (1) the group of surrogates cannot be extended, in other words, k-value is limited, (2) the problem of ghosting effects cannot be well-solved, and (3) the most fatal weakness is that the background information of the raw images must be removed so as to align each image with one another. To overcome these challenges, we leverage the k-Same-M positive property (will be detailed later) and integrating AAM into the k-Anonymity parameter space for keeping the visual information and facial expression related attributes as much as possible. <cit.> addressed a very useful table-based anonymity scheme. It helps k-Anonymity algorithms provide guarantees to protect privacy with limited number of Quasi-identifiers. However, its main drawback lies in dealing only with specific personal data. That is, it focuses only on the text database and cannot be applied to media dataset directly. In this work, we imitate the idea of the former table-based anonymity algorithm and design a novel labeling scheme for generating de-identified image set, automatically. 


Meden  <cit.> proposed the k-Same-Net scheme, whose key idea is to combine the k-Anonymity algorithm with GNN architectures. Although the obtained result is the state-of-the-art in this research area, there are some weaknesses still. First, while selecting the cluster centroids (i.e., similar images), traditional PCA algorithm was used; therefore, certain amount of computation is inevitable. Second, it takes quite a long training time when GNN architecture is applied. Another fatal weakness of this approach is that the original images must be down sampled during training and synthesizing, which impairs the quality of surrogate images a lot.





 §.§ StarGAN and Siamese Net


StarGAN is a multi-domain image-to-image translation framework, within a single dataset or between multiple datasets, and it has shown remarkable results as reported in <cit.>. 


Siamese Net is a special type of NN architectures designed for recognizing one-shot images. Instead of being a model assimilating to classify its inputs, it learns to distinguish between two input images. A Siamese Net <cit.> consists of two identical NNs, each taking care of one of the input images. The last layer outputs of the two networks are fed to a contrastive loss function, which is used to calculate the similarity of the two images. During the training, the two sister networks share the weights and optimize the contrastive loss function to make similar images as close as possible, and vice versa.

Hence, we use this network as the designed similarity arbitrator to prevent the Generator from producing images that alike to the original one too much.







 §.§ Mixed Precision Training (MPT)



The cost to train or use the model will be grown along with the depth of the network. Therefore, to increase the value of the designed system in real applications, we have to find some ways to reduce the cost, especially in training. The most intuitive way to build a cost-effective model from its original complex form is to simplify the associated model architecture by pruning and/or quantization; nevertheless, in this way, relinquishing in performance is an inevitable side effect <cit.>. Literally, MPT means there are more than one floating-point datatypes involved during model training. With the aid of MPT, we can nearly halve the memory requirement and, on recently available GPUs, speed up 2 to 3 times in the training process. Moreover, integrating MPT into a training process introduces no change to any model structure and produces nearly the same performance as that of its original counterpart.





§ SYSTEM IMPLEMENTATION

Before discussing the proposed system in detail, as illustrated in Figure <ref>, an explanation of the architecture of kSS-GAN for face de-identification, is given. First of all, Radboud Faces Database (RaFD) <cit.> and CelebA <cit.> datasets are chosen as the input image sets for generating new groups of image sets. Due to the concatenation with the modified loss function of Siamese-GAN, the auto label-scheme helps generate a label for each image, according to some specific features, such as facial expression, gender, group id and so on. Finally, we can get the k-same/k-anonymity de-identified faces as the system outputs.


 §.§ The Implementation of kSS-GAN






kSS-GAN consists of three functional modules: Face Recognition module, Cluster Generating module, and Candidate Clustering modules. The task of Face Recognition module is to align faces and call a distance measuring function to calculate the similarity of the target image to the whole image set. The Cluster Generating module is used to control the image grouping function for selecting k-identity faces and call the auto-labelling function to replace or add specific features for generating candidate clusters. Thus, this module generates the required group ID and labels appropriate scheme (such as facial expressions, color of hairs, and so on) for one-hot encoding. This is the reason why, with the aid of auto-labelling scheme, we can generate de-identified image set automatically and flexibly. The Candidate Clusters module is responsible for communicating with Siamese-GAN to conduct training. At the same time, the hyperparameter tuning tool is activated to find the most appropriate parameter combinations, and MPT is activated as well to speed up the training process and reduce the memory consumption. The schematic diagram of the whole proposed system is sketched in Figure <ref>.



The operations of kSS-GAN and Siamese-GAN are detailed in Algorithm <ref> and Algorithm <ref>, respectively. Each image is sampled from the input data and denoted as x ∼P_r, where P_r is the associated probability. The latent variables z ∼ p(z) of each sampled image can be obtained, similarly. The latent variables combined with uniformly distributed random numbers are treated as the input and fed to kSS-GAN. In fact, the target image at the input layer of kSS-GAN is generated based on the latent variables with random numbers ϵ∼ U[0,1] and the one-hot encoding; therefore, direct controlling the strength of image features at different scales. Combining with noises injected directly into the kSS-GAN network, this kind of architectural arrangement leads to automatic, unsupervised separation of high-level attributes (e.g., pose, identity, facial expressions) from stochastic variations (e.g., freckles, hair) in the generated images, and enables intuitive scale-specific mixing and interpolation operations.



 §.§ The Implementation of Siamese-GAN
 

In Star-GAN, a generated image must go through the reconstruction process, as shown in Figure <ref> (R1), to ensure its relevance to the original one. The authors of Star-GAN then applied Cycle Consistency Loss, defined in <cit.> to the generator, which can be written as


    ℒ_rec = 𝔼_x,c,c'[  x - G(G(x,c),c') _1 ]

where G takes the translated image G(x,c) and the original domain label c' as inputs and tries to reconstruct the original image x. The L_1 norm was chosen as the reconstruction loss in Equation (<ref>).

In our work, however, the generated images must be deviated from the original one so as to achieve the so-called K-Anonymity. The newly proposed Siamese-GAN is depicted in Figure <ref>, where (R1) stands for the information flow of the reconstruction process of Star-GAN and (R2) denotes that of the newly proposed Siamese-GAN. In (R2), we feed input and reconstructed images into Siamese-GAN to generate the corresponding feature vectors. Then, the calculated similarity is used to guide GAN producing a fake image which is different from the original one. 










In the rest of this section, the involved feature-based similarity measure, reconstruction loss, adversarial loss, domain classification loss, and the overall loss function will be detailed in sequence.



  §.§.§ Feature-Based Similarity


To overcome the shortages associated with Star-GAN, we modify its reconstruction process from (R1) to (R2), as depicted in Figure <ref>, for ensuring that the translated image do distinguishable from the original one. Further, we also apply the Cycle Consistency Loss to the generator and use the feature-based similarity kernel <cit.> to measure the reconstruction loss, that is


    S(f_real, f_fake) = 1/2(1+f_real^𝐓 f_fake)

where f_real and f_fake are the feature maps of the real and the fake images, respectively. Since f_real and f_fake have been L_2-norm normalized, the inner product f_real^𝐓  f_fake is actually the Cosine similarity between the two originally unnormalized feature vectors. The Cosine similarity is chosen because it is bounded and its gradient with respect to f_real and f_fake, as compared with other bounded similarity functions like the Bhattacharyya coefficient, can be calculated much easily.



  §.§.§ Reconstruction Loss


To get the feature vector of an image, as shown in Figure <ref>, we include the Siamese model <cit.> into the reconstruction process. Siamese model will learn facial features of the real and the fake images during the training process and then project the input and the reconstructed images back to the feature space. The so-obtained feature vectors are fed into the feature-based similarity measuring function, as mentioned in the previous sub-section. On the bases of the calculated similarity and the following new reconstruction loss function, cf. Equation (<ref>), a similarity score of the two images will be obtained.


    ℒ_rec = 𝔼_x,c,c'[ S(x, G(G(x,c), c')]




  §.§.§ Adversarial Loss


On the other hand, to make the generated (fake) images look close to the original (real) ones, we introduce an adversarial loss function, as given in Equation (<ref>), where G generates an image G(x, c) according to the input image x and the target domain label c, while D works to distinguish the real image from the fake one. The whole process behaves like a minmax game. That is, the generator G attempts to minimize the distinguishable objective function, while the discriminator D tries its best to maximize it.


    ℒ_adv = 𝔼_x[ log( D(x) ) ] + 𝔼_x,c[ log(1 - D(G(x,c))) ]





  §.§.§ Domain Classification Loss


For a given input image x and a target domain label c, we have to translate x into an output image y which will be correctly classified into the target domain with label c. To achieve this goal, we add an ancillary classifier into D, just like Star-GAN, and impose two domain-specific classification loss functions, as defined in Equations (<ref>) and (<ref>), to optimize D and G, respectively. The domain classification loss function for the discriminator D is defined as


    ℒ_real_cls = 𝔼_x,c'[- log( D_cls(c'|x) ) ]

while the domain classification loss function for the generator G is defined as


    ℒ_fake_cls = 𝔼_x,c'[- log( D_cls(c'|G(x,c)) ) ]

where D_cls(c'|x) denotes the probability distribution over the domain labels computed by D. In other words, G tries to minimize this objective function for generating images that can be classified to the target domain with label c and D learns to classify a real image x into its corresponding original domain with label c'.



  §.§.§ Overall Loss Function


Finally, the loss functions for optimizing G and D can now be written, respectively, as

    L_D  = -L_adv + λ_cls*L_real_cls


and

    L_G  = L_adv + λ_cls*L_fake_cls + λ_rec*L_rec

where λ_cls and λ_rec respectively are hyperparameters controlling the relative importance of domain classification loss and the reconstruction loss, as compared with the adversarial loss. Without loss of generality, we use λ_cls = 1 and λ_rec = 10 in all experiments of this work.



 §.§ The Implementation of MPT


As pre-described, MPT is used to reduce the training time and memory consumption. The kernels of MPT are the floating-point operation with 16-bit precision (FP16) and the Tensor-Core which can accelerate matrix operations and halve the GPU memory consumption. We first cast the input images to FP16 and ingest forward propagation through the model. After that, we convert the model output to FP32 for evaluating the loss and then scale the loss back to FP16 to cover wider representable range. The casting of forward model output to FP32 is a must for preserving small values that are, otherwise, quantized to zero in FP16.



Last but not the least, we have to update the FP32 master-weight and start a new iteration. The FP32 master-weight prevents the updated weight gradients from becoming zero if its magnitude is smaller than 2^-24. The above-mentioned workflow is illustrated in Figure <ref>.


§ THE EVALUATION OF HYPERPARAMETER PERFORMANCE




 §.§ Preliminaries of the Experiments and the testing Datasets


















The Radboud Faces Database (RaFD) is used to benchmark the proposed work. RaFD contains high-quality images of 67 subjects, each of which has eight different facial expressions (i.e., anger, disgust, fear, happiness, sadness, surprise, contempt, and neutrality). 

The other tested dataset is CelebFaces Attributes Dataset (CelebA). 

It has large diversities, large quantities, and rich annotations, including 10,177 identities, 202,599 face images, 5 landmark locations, and 40 binary attribute annotations per image. 



 §.§ The Evolution of Hyperparameter Performances

Three specific solvers of the open-source tool, Advisor, are used to test the proposed system, including Particle Swarm, Random Search, and Hyperopt <cit.>. Since the Cluster Generator module in kSS-GAN is to choose specific features for generating proper candidate clusters, we use Advisor to search for better hyperparameter combinations. As illustrated in Figure <ref>, the x-axe stands for the number of tuning iteration and the y-axe denotes the value of the loss function; clearly, Particle Swarm and Hyperopt provide much lower loss than that of the Random Search. After getting the hyperparameter combination associated with the minimum loss, we use that particular parameter combination to train the model. For comparison, as shown in Figure <ref>, we take the loss of the default combination (i.e., without using any solver) into account. Based on the resultant loss of the training process, the combinations of hyperparameters recommended by Advisor do provide lower system loss than that of the native approach.







 §.§ Experimental Results - the effectiveness of MPT


To illustrate the effects of MPT, we trained the model with the two different GPU platforms listed in Table <ref>, which are Volta V100 and Pascal P100, respectively. The memory consumption before including MPT is quite high; actually, the corresponding memory utility rate is approaching to 100%. In contrast, by using MPT, the memory utility rates of the system reduce to 36% for V100 and 37% for P100, respectively. In MPT, FP16 is one of the computing data types which enabled us to leverage the extremely high speed FP16 matrix operations in Tensor-Core. Roughly speaking, with FP16 computing in Tensor-Core, the training process can be speeded up almost 2 to 3 times, as compared with the case without using MPT in V100. Although not every GPU has Tensor-Core, such as P100, MPT can still reduce lots of memory consumption in training.





 §.§ The Evolution of kSS-GAN for Face De-identification

For checking the effectiveness of kSS-GAN in facial image de-identification, we examine the quality of the fake images generated by K-Same-M, K-Same-Net, and kSS-GAN, respectively. For the same criteria k = 3, we listed the original and the generated images of K-Same-M, K-Same-Net, and kSS-GAN (from top to bottom) in Figure <ref>. One should aware of that the schemes differ not only in the generated images’ visual quality but also in the amount of information content retained. In other words, the quality (represented by the naturalness and the smoothness) of the generated image plays the dominating role, if privacy protection (i.e., facial image de-identification) is the only application target. However, under certain application scenarios the identity re-identification is a must, then besides image quality, the amount of retained information associated with the original image becomes very crucial. 


In terms of image quality, as shown in Figure <ref>, k-same-Net does provide the best result. However, its re-identification capability has never been addressed. The proposed approach, as shown in the bottom of Figure <ref>, achieves the goals of providing anonymity guarantee, producing nearly natural and realistic de-identification results, and supporting the ability for re-identifying the original ID if necessary. That is, even though the details of the generated facial image still have some defects, especially in the regions with lights and shadows, which impair the quality of the output image a few. Nevertheless, from Figure <ref>, the proposed kSS-GAN provides the highest possibility of ID re-identification from the output images. Moreover, we also find that there are no ghosting effects in the generated images. Thus, we apply Blurry Compression Test to compare the three k-same approaches with the blur detection from OpenCV <cit.>. Figure <ref> shows the comparison results, where the x-axe gives the value of k, and y-axe represents the degree of sharpness (i.e., the variance of Laplacian values). The higher the degree of sharpness is the more the clearness will be. From Figure <ref>, the image generated by kSS-GAN is the clearest one in each level of the same k.







According to the previous experiments, when the k value is small, the generated surrogate images appear unnatural in the proposed kSS-GAN algorithm. On the other hand, when the value of k is increased to 30, as shown in Figure <ref>, it is found that the naturalness of surrogate images synthesized by the proposed kSS-GAN algorithm has been greatly improved.



§ CONCLUSION AND FUTURE WORK


In this work, we proposed a novel approach towards facial image de-identification, called kSS-GAN, which leverages the techniques of k-Same-Anonymity mechanism, GAN and hyperparameter tuning. In order to speed the whole training process up and reduce the memory consumption, the well-performed MPT and the open-sourced hyperparameter tuning tools have also been included in the proposed system. Experimental results showed that kSS-GAN achieved the goals of providing anonymity guarantee, producing nearly natural and realistic de-identification results, and supporting the ability for re-identifying the original ID if necessary. In the future, we will apply the designed approach to video streams for generating the surrogate images in real time; in other words, we will tackle the following critical issues, such as real time object detection from videos, enhancing the quality of images generated by kSS-GAN, and finding effective methods to provide higher model compression. So far, we have realized a proof of concept (POC) prototype system, as shown in the snapshots of Figure <ref>, when it is applied to a video captured from the youtube (<https://youtu.be/MoOpwZme5mc>, <https://youtu.be/BgsjZGbJLZ4>).






ieee




