Attribute-Guided Deep Polarimetric Thermal-to-visible Face Recognition
Seyed Mehdi Iranmanesh Nasser M. Nasrabadi
cs.CV











Attribute-Guided Deep Polarimetric Thermal-to-visible Face Recognition
    Seyed Mehdi Iranmanesh, Nasser M. Nasrabadi

West Virginia University

{seiranmanesh}@mix.wvu.edu, {nasser.nasrabadi}@mail.wvu.edu
    December 30, 2023
======================================================================================================================================

empty



	In this paper, we present an attribute-guided deep coupled learning framework to address the problem of matching polarimetric thermal face photos against a gallery of visible faces. The coupled framework contains two sub-networks, one dedicated to the visible spectrum and the second sub-network dedicated to the polarimetric thermal spectrum. Each sub-network is made of a generative adversarial network (GAN) architecture.  We propose a novel Attribute-Guided Coupled Generative Adversarial Network (AGC-GAN) architecture which utilizes facial attributes to improve the thermal-to-visible face recognition performance. The proposed AGC-GAN exploits the facial attributes and leverages multiple loss functions in order to learn rich discriminative features in a common embedding subspace. To achieve a realistic photo reconstruction while preserving the discriminative information, we also add a perceptual loss term to the coupling loss function. An ablation study is performed to show the effectiveness of different loss functions for optimizing the proposed method. Moreover, the superiority of the model compared to the state-of-the-art models is demonstrated using polarimetric dataset. 

 



§ INTRODUCTION





In recent years, there has been significant amount of research in Heterogeneous Face Recognition (HFR) <cit.>. The main issue in HFR is to match the visible face image to a face image that has been captured in another domain such as in the infrared spectrum <cit.>, or the polarimetric thermal <cit.> domain. Infrared images are categorized into two major groups of reflection and emission. The reflection category, which contains near infrared (NIR) and shortwave infrared (SWIR) bands, is more informative about the facial details and it is very similar to the visible imagery. Due to this reflective phenomenology of the NIR and SWIR, there has been a significant performance on NIR-to-visible face recognition accuracy <cit.> and to some extent for SWIR-to-visible face recognition accuracy <cit.>. 



The emission category contains the midwave infrared (MWIR) and longwave infrared (LWIR) bands and it is less informative <cit.> compared with the reflection category. Due to the significant phenomenological differences between the distribution of thermal and visible imagery, matching a thermal face against a gallery of visible faces becomes a challenging task. However, thermal-to-visible recognition is highly demanding because in the thermal data no active illumination is needed at night-time or low-light environments since the thermal imagery is based on the emission and originates from the underlying skin and depends on the individual's physiology. Many recent approaches on cross-modal problem have mainly focused on closing the gap between the two different modalities, but they have not explored the soft biometrics i.e., facial attributes. 


Recently, via an emerging technology <cit.>, the polarization state information of thermal emission has been exploited to provide additional geometrical and textural details, especially around the nose and the mouth, which complements the textural details of the conventional intensity-based thermal images. This additional information which is not available in the conventional intensity-based thermal imaging <cit.>, is utilized in recent algorithms to enhance the cross thermal-to-visible face recognition <cit.>. Fig. <ref> shows a visible image and its corresponding conventional thermal (S_0) and polarimetric state information (S_1 and S_2) images. 

There are some informative traits, such as age, gender, ethnicity, race, and hair color, which are not distinctive enough for the sake of recognition, but still can act as complementary information to other primary information, such as face and fingerprint. These traits, which are known as soft biometrics, can improve recognition algorithms while they are much cheaper and faster to acquire. They can be directly used in a unimodal system for some applications. Soft biometric traits have been utilized jointly with hard biometrics (face photo) for different tasks such as person identification or face recognition <cit.>. However, the soft biometric traits are considered to be available both during the training and testing phases. Our approach looks at this problem in a different way. We consider the case when soft biometric information does not exist during the testing phase, and our method can predict them directly in a multi-tasking paradigm. 




Multi-task learning (MTL) and deep learning techniques has been vastly applied in computer vision and biometrics problems <cit.>. MTL basically attempts to solve correlated tasks concurrently with the help of knowledge sharing between tasks. In <cit.> MTL is employed to predict attributes such as age, gender, race, etc.  Face photo can be viewed as having some positive or negative hidden relationship with some of its soft facial biometric traits.  


In this paper, we propose an Attribute-Guided Coupled Generative Adversarial Network (AGC-GAN), which considers the Convolutional Neural Network (CNN) weight sharing followed by the dedicated weights which are responsible for learning the representative features for each specific face attribute. The network also tries to find the common embedding space between the polarimetric thermal and visible faces utilizing a coupling structure and an adversarial training. Optimizing the coupled network with the help of the facial attributes leads to a  more discriminative embedding subspace and can be utilized to enhance the performance of the main task which is the heterogeneous face recognition.

To summarize, the following are our main contributions: 


	
  ∙ A novel polarimetric thermal-to-visible face recognition system is proposed based on an attribute-guided coupled GAN to synthesize visible faces from the polarimetric thermal images. 
	
	
  ∙ A multi-tasking framework is proposed to predict facial attributes from the polarimetric thermal faces. To the best of our knowledge, no such demonstration has been proposed in the literature. 
	
	
  * ∙ Extensive experiments are conducted on ARL polarimetric facial database <cit.> and the proposed method is compared to recent state-of-the-art methods. 








§ PRELIMINARIES
















 §.§ Conditional Generative Adversarial Networks


The generative adversarial network consists of two networks, namely a generator and discriminator which compete with each other in a minimax game <cit.>. Conditional adversarial networks is an extension of generative adversarial networks in which both the generator and discriminator are conditioned on some auxiliary information y. The extra information y can be any kind of information such as class label or data from other modalities. The objective of the conditional GAN is the same as the classical GAN as follows:



    Gmin Dmax E_x∼ P_data(x)[log D(x|y)]+
       E_z∼ P_z [log(1-D(G(z|y)))]  .




§ PROPOSED METHOD


The proposed AGC-GAN is illustrated in Fig. <ref>. The proposed approach consists of two coupled generators and two discriminators. Each generator is also responsible to predict facial attributes in a multi-tasking paradigm. In the following subsections we explain these modules in detail.



 §.§ Deep Coupled Framework


The final objective of the proposed model is the identification of the polarimetric test faces against a gallery of visible faces which we do not have access to them during the training phase. For this reason, we couple two U-net networks <cit.> one is dedicated to the visible spectrum (Vis-GAN) and the other network is dedicated to the polarimetric spectrum (Pol-GAN). Each network performs a non-linear transformation of the input space. The final objective of our proposed AGC-GAN is to find the global deep latent features representing the relationship between polarimetric face images and their corresponding visible ones. In order to find a common latent embedding subspace between these two different domains we couple two networks (Vis-GAN and Pol-GAN) via a contrastive loss function <cit.>. This loss function (ℓ_cont) pulls the genuine pairs (i.e., a visible face image with its own corresponding polarimetric face images) towards each other in a common latent feature subspace and push the impostor pairs (i.e., a visible face image of a subject with another subject's polarimetric face images) apart from each other (see Fig. <ref>). 
Similar to <cit.>, our contrastive loss is of the form:

    ℓ_cont   (z_1(y^i_vis),z_2(y^j_pol),y_cont)=
       (1-y_cont)L_gen(D(z_1(y^i_vis),z_2(y^j_pol))+
        y_contL_imp(D(z_1(y^i_vis),z_2(y^j_pol))   ,

where y^i_vis is the input for the Vis-GAN (i.e., visible face image), and y^j_pol is the input for the pol-GAN (i.e., polarimetric face images). y_cont is a binary label, L_gen and L_imp represent the partial loss functions for the genuine and impostor pairs, respectively, and D(z_1(y^i_ vis),z_2(y^j_pol)) indicates the Euclidean distance between the embedded data in the common latent feature subspace. z_1(.) and z_2(.) are the deep CNN-based embedding functions, which transform y^i_vis and y^j_pol into a common latent embedding subspace. The binary label, y_cont, is assigned a value of 0 when both modalities, i.e., visible and polarimetric, form a genuine pair, or, equivalently, the inputs are from the same class (cl^i =cl^j). On the contrary, when the inputs are from different classes, which means they form an impostor pair, y_cont is equal to 1. In addition, L_gen and L_imp are defined as follows: 

    L_gen(D(z_1(y^i_vis),z_2(y^j_pol)))= 
       12  ||z_1(y^i_vis)-z_2(y^j_pol)||^2_2
    		    for     cl^i=cl^j   ,

and

    L_imp(D(z_1(y^i_vis), z_2(y^j_pol)))=
       12   max(0,m-||z_1(y^i_vis)-z_2(y^j_pol)||^2_2)  for   cl^i ≠ cl^j,


where m is the contrastive margin. Therefore, the coupling loss function can be written as:

    L_cpl   = 1/N^2 ∑_i=1^N∑_j=1^Nℓ_cont(z_1(y^i_vis), z_2(y^j_pol), y_cont),


where N is the number of training samples. It should be noted that the contrastive loss function (<ref>) considers the subjects' labels implicitly. Therefore, it has the ability to find a discriminative embedding space by employing the data labels in contrast to some other metrics such as the Euclidean distance. This discriminative embedding subspace would be useful in identifying a polarimetric probe photo against a gallery of visible face photos. 




 §.§ Multi-Attribute Prediction and Identification Task:


The objective of this model is to predict a set of attributes using a visible or polarimetric face images. Therefore, in this architecture a visible face image (polarimetric face images) is presented to the network as an input and a set of attributes are predicted. Suppose the input is a visible image y^i_vis∈ Y, and its class label is cl^i ∈ CL for i=1,…,N where N is the number of the training samples. Let the soft biometric traits, contain T different facial attributes or binary class labels. Therefore, in this framework we denote them as cl^t for t=1,…,T . Learning multiple CNNs separately is not optimal since different tasks may have some hidden relationships with each other and may share some common features. This is supported by <cit.> where they train a CNN features for the face recognition task and they use it directly for the facial attribute prediction. Therefore, our network shares a big portion of its parameters among different attribute prediction tasks in order to enhance the performance of the recognition task. Thus, the loss function is as follows: 


    L_avis   = 1/N ∑_i=1^N∑_t=1^Tℓ(f^t_vis(z_1(y_vis^i)× w_vis^t), cl^i,t)  ,


where ℓ is a proper loss function (e.g., cross entropy) and f^t_vis(.) is a binary classifier for the attribute t operated on the bottleneck of Vis-GAN (see Fig. <ref>). w_vis^t represents the remaining parameters which are learned separately for each facial attribute task. 


The same procedure is performed in the other network (Pol-GAN) with polarimetric thermal images as input. The Pol-GAN network is also responsible to estimate a set of facial attributes. Therefore, the loss function is: 


    L_apol =     1/N ∑_j=1^N∑_t=1^Tℓ (f^t_pol(z_2(y^j_pol)× w_pol^t), cl^j,t)  ,



where ℓ is the cross entropy loss function and f^t_pol(.) is a binary classifier for the attribute t operated on the bottleneck of Pol-GAN (see Fig. <ref>). w_pol^t represents the remaining features which are assigned separately for each facial attribute prediction task. The total attribute prediction loss function is: 



    L_a = L_avis + L_apol.








 §.§ Generative Adversarial Loss


Let G_vis and G_pol denote the generators that synthesize the corresponding visible image from the visible and its polarimetric images, respectively. To synthesize the output and to make sure that the synthesized images generated by the two generators are indistinguishable from the corresponding ground truth visible image, we utilized the GAN loss function in <cit.>. As it is shown in Fig. <ref>, the first generator G_vis is responsible to generate a visible image when the network is conditioned on a visible image. On the other hand, the second generator G_pol tries to generate the visible image from the polarimetric images which is a more challenging task compared to the first generator. Therefore, the total loss for the coupled GAN is as follows: 




    L_GAN = L_vis + L_pol  ,


where the GAN loss function for the Vis-GAN sub-network is given as:



    L_vis = G_vismin D_vismax E_x^i∼ P_vis(x)[log D(x^i|y^i_vis)]+
       E_z∼ P_z [log(1-D(G(z|y^i_vis)))]  ,


where y^i_vis is the visible image used as condition for the Vis-GAN and x^i is the real visible data. It should be noted that for the Vis-GAN the real data x^i and the condition y^i_vis are the same. Similarly the loss for the Pol-GAN is given as: 



    L_pol = G_polmin D_polmax E_x^j∼ P_vis(x)[log D(x^j|y^j_pol)]+
       E_z∼ P_z [log(1-D(G(z|y^j_pol)))]  ,




where y^j_pol is the polarimetric images used as condition for the Pol-GAN and x^j is the real visible data. It should be noted that x^i is the same as x^j if they refer to the same person (cl^i=cl^j) and otherwise they are not the same.



 §.§ Overall Loss Function
 

The proposed approach contains the following loss functions: the Euclidean L_E_vis and L_E_pol losses which are enforced on the recovered visible images from the Vis-GAN and Pol-GAN sub-networks, respectively, are defined as follows:



    L_E_vis= ||G_vis(z|y^i_vis)-x^i||^2_2 , 
       
    	L_E_pol= ||G_pol(z|y^j_pol)-x^j||^2_2 , 
       
    	L_E= L_E_vis + L_E_pol.


The L_GAN (<ref>) loss is also added to generate sharper images. In addition, based on the success of the perceptual loss in low-level vision tasks <cit.>, a perceptual loss is added to the Pol-GAN sub-network to generate a more realistic face photos as follows: 



    L_P_pol = 1C_pW_pH_p∑_c=1^C_p∑_w=1^W_p∑_h=1^H_p
       ||V(G_pol(z|y^j_pol))^c,w,h-V(x^j)^c,w,h|| ,




where x^j is the ground truth visible image, G_pol(z|y^j_pol) is the output of Pol-GAN generator. V(.) represents a non-linear CNN transformation and C_p, W_p, H_p are the dimension of a particular layer in V. It should be noted that the perceptual loss is just used in the Pol-GAN sub-network. 
Similarly, we utilized a perceptual attribute loss which measures the difference between the facial attributes of the synthesized images and the real image. To extract attributes from a given visible face, we fine-tune the pre-trained VGG-Face <cit.> on ten annotated facial attributes as tabulated in Table <ref>. This network (attribute predictor) is trained separately from AGC-GAN. Afterward, this attribute predictor is utilized for perceptual attribute loss on Vis-GAN and Pol-GAN as follows:



    L_pavis = ||A(G_vis(z|y_vis^i))-A(x^i)||_2^2, 
       
    	L_papol = ||A(G_pol(z|y_pol^j))-A(x^j)||_2^2,  
       
    	L_pa = L_pavis + L_papol ,



where A is the fine-tuned VGG-Face attribute predictor network. L_pa is the total perceptual attribute loss function which composed of the perceptual attribute losses for the Vis-GAN (L_pavis) and Pol-GAN (L_papol) sub-networks. 

Finally, the coupling loss function (<ref>) is added to train both networks Vis-GAN and Pol-GAN jointly to make the embedding subspace of the mentioned networks as close as possible and to preserve a more discriminative and distinguishable shared space. Therefore, the total loss function is as follows: 



    L_T = L_cpl+ λ_1 L_E+ λ_2 L_GAN
       λ_3 L_a + λ_4 L_Ppol + λ_5 L_pa,


where λ_1, λ_2, λ_3, λ_4, and λ_5 are the hyper-parameters which weight different loss terms in the total loss function. 







 §.§ Testing Phase


During the testing phase, only the Pol-GAN is used. For a given test probe y_pol^t, Pol-GAN sub-network of the proposed AGC-GAN is employed to synthesize the visible image G_pol(z|y^t_pol)= x̂^t_vis. Eventually, the identification of face recognition is done, by calculating the minimum Euclidean distance between the synthesized image from the polarimetric prob against the visible gallery images as follows:     


    x^t^*_vis = x^t_ visargmin   ||x^t_vis,x̂^t_vis||   ,


where x̂^t_vis is the synthesized probe face image and x^t^*_vis is the selected matching visible face image within the gallery of face images. In addition, the Pol-GAN sub-network can be employed to predict the facial attributes from the polarimetric face probe. The predicted facial attributes can also be used to narrow the search or identify a person of interest in a visible gallery of faces.     



 §.§ Implementation Details


A U-net structure <cit.> is employed as the network for the generator since it is able to address the vanishing gradient problem as well as capturing large receptive field. Also, a patch-based discriminator <cit.> is used in the proposed method and it is trained iteratively with the generator. The entire network is trained in Pytorch. For the sake of training AGC-GAN, the hyper-parameters for all the loss functions are set to one except for the perceptual loss L_p_pol and perceptual attribute loss L_pa  which is set to 0.5.  For training we used Adam optimizer <cit.> with the first-order momentum of 0.5, the learning rate of 0.0002, and batch size of 4. For the generator the ReLU activation, and for the discriminator the Leaky ReLU activation with the slope of 0.2 is considered. The perceptual loss is assessed on the relu3-1 layer of the pre-trained VGG model <cit.>. In order to fine-tune the attribute predictor network utilized for perceptual attribute loss, we manually annotated images with the attributes tabulated in Table <ref>.



§ EXPERIMENTS

We evaluate the proposed face recognition method by comparing with several recent works <cit.> on the ARL Multi-modal Face database <cit.>.

Polarimetric Thermal Face dataset <cit.> comprises polarimetric thermal face images and their corresponding visible spectrum faces related to 60 subjects. Data was collected at three different distances: Range 1 (2.5 m), Range 2 (5 m), and Range 3 (7.5 m). At each range two different conditions, including baseline and expression are considered. 


To increase the correlation between the two modalities of visible and thermal, each modality was preprocessed. We applied a band-pass filter so called difference of Gaussians (DoG), to emphasize the edges in addition to removing high and low frequency noise. 

We pass S_0, S_1, and S_2 to the Pol-GAN's three channels as the input as shown in Fig. <ref>. To train the network, the genuine and impostor pairs are constructed. The genuine pair is constructed for the same subject photos from the two different modalities. For the impostor pair, a different subject is selected for each modality. In general, the number of the generated impostor pairs are significantly larger than the genuine pairs. For the sake of balancing the training set, we consider the same number of genuine and impostor pair. After training the model, during the testing phase, only the polarimetric sub-network is used for the evaluation. For a given probe, the Pol-GAN sub-network is used to synthesize the visible image. Afterwards, the Euclidean distance is used to match the synthesize image to its closest image from the gallery. The ratio of the number of correctly classified subjects and the entire number of subjects is computed as the identification rate. 

In each experiment the dataset is partitioned to the training and testing sets randomly. The same set of training and testing are used to evaluate PLS <cit.>, DPM <cit.>, CpNN <cit.>, PLS∘DPM <cit.>, PLS∘CpNN <cit.>, GAN-VFS <cit.>, and the proposed AGC-GAN network. Fig. <ref> shows the overall cumulative matching characteristics (CMC) curves for our proposed method and the other state-of-the-art methods over all the three different ranges as well as the expressions data at Range 1. For the sake of comparison, in addition to the polarimetric thermal-to-visible face recognition performance, Fig. <ref> also shows the results for the conventional thermal-to-visible face recognition for some of the methods, namely PLS∘DPM, PLS∘CpNN, CpNN, and AGC-GAN. In the conventional thermal-to-visible face recognition, all the mentioned methods exactly follow the same procedure as before, with only using S_0 modality. Fig. <ref> illustrates that exploiting the polarization information of the thermal spectrum enhances cross-spectrum face recognition performance compared to the conventional one. Fig. <ref> also shows the superior performance of our approach compared to the state-of-the-art methods. In addition, our method could achieve prefect accuracy (100%) at Rank-4 and above. 


Table <ref> tabulates the Rank-1 identification rates for five different scenarios: overall (which corresponds to Fig. <ref>), Range 1 expressions, Range 1 baseline, Range 2 baseline, and Range 3 baseline. In our proposed approach, exploiting polarization information enhance the Rank-1 identification rate by 2.24%, 5.16%, 4.65%, and 9.44% for Range 1 baseline, Range 1 expression, Range 2 baseline, and Range 3 baseline compared to the conventional thermal-to-visible face recognition. This table reveals that using deep coupled generative adversarial network technique with the contrastive loss function as well as utilizing facial attributes to transform different modalities into a distinctive common embedding subspace is superior to the other embedding techniques such as PLS∘CpNN. It also shows the effectiveness of our method in exploiting polarization information to improve the cross-spectrum face recognition problem.    



§ ABLATION STUDY


In order to illustrate the effect of adding different loss functions and their improvement in our proposed framework, we perform a study with the following evaluations using the polarimetic dataset: 1) Polar-to-visible using the coupled framework with using only L_cpl+ L_E losses, 2) Polar-to-visible using the proposed framework with L_cpl+ L_E+ L_GAN+L_p_pol+L_pa loss functions, and 3) Polar-to-visible with all the loss functions in the proposed framework (<ref>).


We plot the receiver operation characteristic (ROC) curves corresponding to the mentioned three different settings of the framework in the task of face verification. As it is shown in Fig. <ref> the L_GAN has an important rule in the enhancement of our proposed approach by transforming the polarimetric modality to the visible one. Moreover, adding facial attribute prediction loss enhances the face recognition performance. The reason behind this is because using facial attributes loss in addition to contrastive loss function leads to a more discriminative embedding space and this leads to a better face recognition performance. Consider a polarimetric subject with Id #2 (see Fig. <ref>). The contrastive loss function causes the corresponding visible images from Id#2 to move closer to Id#2's polarimetric and other Ids' visible images to move farther away. Now, using the contrastive loss function in conjunction with the attribute classification loss function makes Id #1 to move closer to Id #2 since they share the same set of attributes (see Fig. <ref>). In other words, it differentiates between different impostors of Id #2. The same procedure is performed for the other identities during the training process. Fig. <ref> visualizes the overall concept of adding facial attributes prediction loss function. As it is depicted, addition of attribute prediction loss leads to a more discriminative embedding subspace. This leads to a better face recognition performance as it is shown by the ROC curves in Fig. <ref>.    





§ ATTRIBUTE PREDICTION FROM POLARIMETRIC THERMAL


One of the benefits of the proposed AGC-GAN is predicting facial attributes directly from polarimetric thermal modality. These attributes can be utilized directly or can be fused with other modalities to enhance recognition performance. In order to illustrate the effectiveness of the proposed method we performed attribute prediction in four different scenarios: 1) Attribute prediction of visible images with the VGG-Face based attribute predictor (Network A in Sec. <ref>). 2) Attribute prediction of polarimteric images with the attribute predictor A. 3) Attribute prediction of polarimetric images with the fine-tuned attribute predictor A. In this case, we fine-tuned the attribute predictor A with the annotated polarimetric images and used it for the task of attribute prediction in the testing phase. 4) Attribute prediction of the polarimetric images using the Pol-GAN attribute predictor from the proposed AGC-GAN (see Fig. <ref>). Table <ref> shows the result of the prediction for the four mentioned frameworks. Although, fine-tuning the attribute predictor A increased the prediction performance (framework #3), but still its performance is less than our proposed framework. The proposed framework could outperform the other methods in predicting polarimetric face attributes and it has a comparable performance with the accuracy of predicted attributes from the visible face images (framework #1). 



§ CONCLUSION

We have introduced a novel approach to exploit facial attributes information for the purpose of polarimetric thermal-to-visible face recognition. The AGC-GAN contains two GAN-based sub-networks dedicated to visible and polarimetric input images. The proposed network is capable of transforming the visible and polarimetric thermal modalities into a common discriminative embedding subspace and synthesizing the visible photos from the embedding subspace. We simultaneously minimize the cost functions due to the facial attribute identification in addition to other cost functions in order to increase the inter-personal variations between different subjects with different sets of facial attributes in the latent feature subspace. This leads to a more discriminative embedding subspace. An ablation study was performed to demonstrate the enhancement obtained by different losses in the proposed method. We compared our method with state-of-the-art polarimetric thermal-to-visible face recognition methods and showed the superiority of our proposed method over them. 



ieee



