3D Face Mask Presentation Attack Detection Based on Intrinsic Image Analysis
Lei Li Zhaoqiang Xia Xiaoyue Jiang Yupeng Ma Fabio Roli Xiaoyi Feng






1]Lei Li
1]Zhaoqiang Xia
1]Xiaoyue Jiang
1]Yupeng Ma
2,1]Fabio Roli
1]Xiaoyi Feng

[1]School of Electronics and Information, Northwestern Polytechnical University, Xi'an, Shaanxi, China
[2]Department of Electrical and Electronic Engineering, University of Cagliari, Cagliari, Sardinia, Italy


  Face presentation attacks have become a major threat to face recognition systems and many countermeasures have been proposed in the past decade. However, most of them are devoted to 2D face presentation attacks, rather than 3D face masks. Unlike the real face, the 3D face mask is usually made of resin materials and has a smooth surface, resulting in reflectance differences. So, we propose a novel detection method for 3D face mask presentation attack by modeling reflectance differences based on intrinsic image analysis. In the proposed method, the face image is first processed with intrinsic image decomposition to compute its reflectance image. Then, the intensity distribution histograms are extracted from three orthogonal planes to represent the intensity differences of reflectance images between the real face and 3D face mask. After that, the 1D convolutional network is further used to capture the information for describing different materials or surfaces react differently to changes in illumination. Extensive experiments on the 3DMAD database demonstrate the effectiveness of our proposed method in distinguishing a face mask from the real one and show that the detection performance outperforms other state-of-the-art methods.



3D face PAD, Intrinsic Image Analysis, Deep Learning








§ INTRODUCTION

Face presentation attack detection (PAD) has become an important issue in face recognition, since the biometric technologies have been applied in various verification systems and mobile devices. For instance, Apple's face ID for iPhone X has been used to unlock cell phones and complete mobile payments. However, D. Oberhaus <cit.> reported that some researchers at the Vietnamese cybersecurity firm Bkav had made a mask that could fool the Face ID system. Considering such urgent security situation, an effective and reliable 3D face mask PAD method must be developed.

Based on different presented fake faces, four kinds of presentation attacks can be considered: printed photo, displayed image, replayed video and 3D mask. The printed photo, displayed image and replayed video attacks usually print or display a face image in 2D medium (e.g., paper and LCD screen). But for 3D face mask scenario, the attacker wears a 3D mask to deceive face recognition system. Compared to 2D face presentation attacks, 3D face masks contain the structural and depth information similar to real faces. Fig. <ref> shows an example of different face presentation attacks.


In the last decade, many face PAD approaches have been proposed <cit.>. However, the main focus of them has been on tackling the problem of 2D face presentation attacks, while 3D attacks have received much less attention. It is because detecting 3D face masks and distinguish them from real faces more challenging for common cameras. For instance, Liu et al. <cit.> estimated face depth by a CNN-RNN model and used the depth information to detect presented fake faces. This method can solve the 2D face PAD successfully. Whereas, this detection method cannot work well in a 3D mask attack as the mask also has depth information. In another work, Li et al. <cit.> tackled the 3D mask attacks by detecting the pulse from videos, which may be sensitive to camera settings and light conditions. Besides, it becomes easier to obtain 3D masks by attackers with the development of 3D printing. Thus, it is necessary to develop new methods for detecting 3D face mask.





In this work, we propose to analyze the intrinsic image for 3D face mask PAD. Intrinsic image decomposition can divide a single image into a reflectance image and a shading image <cit.>. For the reflectance image, it gives the albedo at each point, which describes the reflectance characteristics and is only decided by the material itself. In contrast, the shading image gives the total light flux incident at each point. Comparing the real face with face mask, we can find that their constituted materials are fairly different. Furthermore, due to the influence of hair follicles, the surface of 3D face mask is smoother than the surface of real face. More specifically, the face skin consists of three layers: epidermis, dermis and subcutaneous fat [https://www.webmd.com/skin-problems-and-treatments/ss/slideshow-skin-infections] as illustrated in Fig. <ref> (left). But for the 3D face mask, it is often made of resin materials as shown in Fig. <ref> (right). These material or surface differences can be revealed in reflectance images and further used for analyzing visual difference.





Our proposed method can be divided into three procedures: (i) the captured face image is first decomposed into a reflectance image and a shading image based on correlation-based intrinsic image extraction (CIIE) <cit.>; (ii) the intensity distribution histograms of reflectance images are calculated from three orthogonal planes and the 1D convolutional neural network (CNN) is introduced to extract the differences in reflectance images; (iii) the extracted features are fed into a Support Vector Machine (SVM) classifier to detect 3D face mask attack. The main contributions of this work are summarized:

♢ Unlike most previous works on 3D face mask PAD are based on analyzing visible cues (e.g. texture) directly, we propose a novel and appealing countermeasure from the view of intrinsic image decomposition.

♢ We design an intensity distribution feature for describing the difference of reflectance images between the real face and 3D face mask and introduce 1D CNN to represent different materials or surfaces react differently to the changes in variational illumination.

♢ We evaluate the proposed method on 3DMAD dataset and achieve the state-of-the-art performance.





§ RELATED WORK





 §.§ Methods for 2D Face PAD


Texture-analysis based methods. Due to the limitations of printers and display devices, the printed or replayed fake faces generally present blurred textures compared to the real ones. Thus, using texture information has been a natural approach to tackling 2D face presentation attack. For instance, many works employed hand-crafted features, such as local binary pattern (LBP) <cit.>, Haralick features <cit.> and scale-invariant feature transform (SIFT) <cit.>, and then used traditional classifiers, such as SVM and LDA. To capture the color differences in chrominance, the texture features extracted from different color spaces (e.g. RGB, HSV and YCbCr) are also been analyzed <cit.>. Apart from that, some feature descriptors or indexes are computed to describe the clarity of facial texture <cit.>. More recently, many attempts of using CNN-based features in face PAD <cit.>. While these methods are effective to typical 2D paper or replayed attacks, they become vulnerable when attackers wear a lifelike face mask.

Motion-analysis based methods.
Apart from analyzing the texture differences, motion is another important clue for printed photo and displayed image attacks. The most commonly used solutions are based on detecting aliveness signals, such as eye blinking <cit.>, head motion <cit.>, facial muscle twitching <cit.> and pulse <cit.>. Besides, there are methods relying on the planar object movements <cit.> and the optical flow fields <cit.>. Moreover, in order to capture the temporal texture variations from a video sequence as <cit.>, the deep networks with timing processing such as long short memory network (LSTM) and 3D CNN are used to detect face presentation attack <cit.>. These algorithms make it easy to detect the printed or displayed image attack. But when facing with 3D face mask attack with eye/mouth portion being cut, their performance will be greatly discounted.

Hardware based methods.
In recent years, various hardwares have been applied for the task of face PAD. For instance, the Kinect camera was used to capture depth information and detect fake faces <cit.>. The reason is that there is no any depth information in 2D presentation attacks. Also, the near-infrared (NIR) sensors and photodiodes are explored to enhance the optical difference between the real and fake faces <cit.>. More recently, light-field cameras allow exploiting disparity and depth information from a single capture. Therefore, some works introduced these kinds of cameras into face PAD <cit.>. Even though those hardware-based methods can achieve good performances for 2D face PAD, some of them might present operation restrictions in certain conditions. For instance, the sunlight can cause severe perturbations for NIR and depth sensors; wearable 3D masks are obviously challenging for those methods relying on depth data.



 §.§ Methods for 3D Face Mask PAD

3D face mask presentation attack is a new threat to face biometric systems and few countermeasures have been proposed compared to conventional 2D attacks. To address this issue, Erdogmus et al. <cit.> released the first 3D mask attack dataset (3DMAD). Different from printed or replayed fake faces, the attackers in 3DMAD wear 3D face masks of targeted persons. Inspired by <cit.>, Erdogmus et al. utilized multi-scale LBP based texture representation for 3D face mask PAD. In addition, Liu et al. <cit.> presented an rPPG-based method to cope with 3D mask attacks. In another work, Li et al. <cit.> tackled the 3D mask attacks based on pulse detection. These two methods mainly analyze the different heartbeat signals between real faces and 3D masks. However, they may be sensitive to camera settings and light conditions. More recently, Shao et al. <cit.> extracted optical flow features from a pre-trained VGG-net <cit.> to estimate subtle motion.



§ PROPOSED METHOD



 §.§ Rationale and Motivations

As aforementioned, the constituted materials of real face and 3D face mask are different and these differences can be revealed in reflectance images, which are shown in Fig. <ref> and <ref>. From the example, it can be found that the reflectance images of real face have low intensity and uniform distribution while the reflectance images of 3D face mask have high intensity and non-uniform distribution. Based on this observation, we propose a new method to discriminate the reflectance difference of real faces and 3D face masks.





In reflectance images of real and spoofing faces, reflectance images often present obvious intensity difference due to different materials and surface structures. To describe the intensity differences, we extract intensity histograms from the view of albedo intensity. Furthermore, the skin surface of the real face is rougher than the 3D face mask, which is caused by hair follicles, etc. When lighting changes or the face shake, the reflectance images of real face and 3D face mask also show different responses. For instance, the reflectance image of a smooth surface is more sensitive to lighting changes. So in addition to extracting the intensity histogram, we further use a 1D CNN to extract the difference in illumination sensitivity of the reflectance image.





Therefore, as shown in Fig. <ref>, our proposed approach is divided into four modules, i.e., Reflectance Extraction, Intensity Histograms, 1D CNNs and SVM classification. First, the frames of captured face video are decomposed into reflectance images and shading images. Then, the intensity histograms of reflectance images are calculated from three orthogonal planes (TOP) and combined into three histogram feature maps, which can describe the ratio and distribution of different intensities. After that, these TOP histogram feature maps are fed into three 1D CNNs to extract the difference in illumination sensitivity, respectively. Finally, an SVM classifier is employed to distinguish whether the concatenated deep features are oriented from a valid access or a 3D face mask.



 §.§ Intrinsic Image Decomposition

Based on the Lambertian theory <cit.>, the intensity value I(x,y) of every pixel in an image is the product of the shading S(x,y) and the reflectance R(x,y) at that point: I(x,y)=S(x,y)× R(x,y). The shading S(x,y) represents illumination changes and is decided by external light intensity. R(x,y) gives the albedo at each point. In addition, albedo completely describes the reflectance characteristics for Lambertian (perfectly diffusing) surfaces. Considering different materials of real face and 3D face mask, we will analyze the reflectance images of them. In the context, we utilize the CIIE algorithm <cit.> to decompose the original face image.

More specifically, the steerable filters are first used to decompose the face image into its constituent orientation/frequency bands. Then, the filter bank is applied to estimate the luminance components (LM_ij) and the variations in local amplitude (AM), texture (TM), and hue (HM), respectively. Based on this assumption that luminance variations are either due to reflectance or illumination but not both, the estimated shading and reflectance can be reconstructed as follows:

    S=F_S⊗{[ C_ij^rec_ shd× LM_ij if C_ij^rec_ shd > 0;                    0  if C_ij^rec_ shd≤ 0 ].


    R=F_S⊗{[ C_ij^rec_ ref× LM_ij if C_ij^rec_ ref > 0;                    0  if C_ij^rec_ ref≤ 0 ].

where ij is the index of LM and ⊗ is the reconstruction of steerable filters F_S with weighted LM_ij. C_ij^rec_ shd and C_ij^rec_ ref are correlation coefficients for reconstructed shading and reflectance, respectively.

During image reconstruction, some LM components will be set to zero and resulting images may lose their DC values. Meanwhile, some AM components will represent small variations in reflectance not shading. Thus two cost functions are used to optimize S and R, as illustrated in Eq. <ref> and Eq. <ref>.

    E_S(V_DC)= min_T_i∑ _iF_txt(I/S+V_DC, T_i)


    E_R= min_k∑ _iF_txt(L̂M̂_ij, T_k) + λ E_cst(F_adj)

where V_DC is the DC value to be optimized. The function F_txt(Î, T_i) represents texture consistency evaluated for image Î against the texture segmentation result. L̂M̂_ij is modelled as a normal distribution and E_cst(F_adj) constrains the maximum rescaling produced by the adjustment function. Fig. <ref> shows the reflectance image sequences of a real face and a 3D face mask.
















 §.§ Intensity Histograms

In order to represent different intensities and distributions of reflectance images, we extract the intensity histograms from three orthogonal planes (i.e., XY, XT and YT) as shown in Fig. <ref>, where T represents the time dimension.





Considering the three orthogonal planes, the reflectance image sequence can be denoted as ℝ={R_1,R_2,...,R_i,...R_n}, where R_i denotes the ith reflectance image in ℝ and n is the length of the sequence. In our algorithm, we set n=75, which can capture intensity variations <cit.>. Before extracting intensity histograms, the size of color reflectance images is first empirically normalized to 150× 150. For the XY plane, intensity histogram from the color channels of each reflectance image of ℝ is extracted and combined into a matrix, which is illustrated in Eq.<ref>.

    H_XY=
       [[ h(R_1^c1), h(R_1^c2), h(R_1^c3); h(R_2^c1), h(R_2^c2), h(R_2^c3);                   ..., ..., ...; h(R_n^c1), h(R_n^c2), h(R_n^c3) ] ]

where c1, c2 and c3 represent different color channels. h(·) means the operation of intensity histogram extraction and the dimension of H_XY is 75× 768.

However, on XT and YT planes, we extract a slice of ℝ every other row or column and count its intensity histograms also from different color channels. The reflectance image sequences from XT and YT can be written as 𝕏={X_1,X_2,...,X_i,...X_ṅ} and 𝕐={Y_1,Y_2,...,Y_i,...Y_n̈}, respectively. ṅ and n̈ are the sequence length and equal to n. Finally, the XT and YT intensity histogram matrices with the size of 75× 768 are obtained based on Eq. <ref> and Eq. <ref>.


    H_XT=
       [[ h(X_1^c1), h(X_1^c2), h(X_1^c3); h(X_2^c1), h(X_2^c2), h(X_2^c3);                   ..., ..., ...; h(X_ṅ^c1), h(X_ṅ^c2), h(X_ṅ^c3) ] ]



    H_YT=
       [[    h(Y_1^c1), h(Y_1^c2), h(Y_1^c3);    h(Y_2^c1), h(Y_2^c2), h(Y_2^c3);                      ..., ..., ...; h(Y_n̈^c1), h(Y_n̈^c2), h(Y_n̈^c3) ] ]







 §.§ 1D CNNs

Different materials and surfaces react differently to illumination changes. Therefore, apart from intensity location features, we also use 1D CNN to extract intensity variation information from H_XY, H_XT and H_YT. As shown in Fig. <ref>, the 1D CNNs module presents three parallel 1D CNNs. In each network, 1D operation is employed to extract intensity variation information. Compared to other deep networks <cit.>, the solution differs mainly in three aspects: (i) 1D convolutional filters are adopted to extract the temporal information of neighboring histograms; (ii) 1D pooling layers can effectively reduce the dimension after convolutional layers; (iii) The output features do not fuse the information between different intensity values.

The parameters of 1D CNN network are summarized in Table <ref>. For all convolutional layers except the 16_th layer, the size of convolutional filters is set to 3×1. This means the convolutional filter can extract the intensity variation in three adjacent histograms and reduce network parameters as <cit.>. After each convolutional layer, we use Rectified Linear Units (ReLU) to activate the convolutional outputs <cit.>. Then the pooling layers with size of 2×1 are utilized to gradually down-sample the results from ReLU layer. In the last pooling layer, the dimension of outputs is 1×768. Before feeding the outputs of last pooling layer into a fully connected (FC) layer, we introduce a transposition layer to reshape the size of outputs into 768× 1.

For face PAD, its essence is to classify whether the input is a real face or a presented 3D face mask. Thus, after FC layer, the most commonly used SoftMax loss function is employed to train the network parameters <cit.>. In testing stage, we extract the outputs of the last Conv layer as the finally features of H_XY, H_XT and H_YT denoted as C_XY, C_XT and C_YT. Fig. <ref> shows the intensity histogram matrixes and the learned intensity variation features of the reflectance image sequences of the real face and the 3D face mask in XY plane of RGB color space. After feature extraction, we concatenate C_XY, C_XT and C_YT into a feature vector with stronger representation, which are fed into an SVM classifier.

In training stage, the stochastic gradient descent (SGD) algorithm <cit.> is used to learn the network parameters. The learning rate is set to 10^-3. The momentum is set to 0.9 and weight decay 0.0005. In addition, we realize the 1D CNNs and SVM based on MatConvNet with the version 1.0-beta20 [http://www.vlfeat.org/matconvnet/] and liblinear with the version 1.96 [https://www.csie.ntu.edu.tw/∼cjlin/liblinear/], respectively.



§ EXPERIMENTAL SETUP













 §.§ Experimental Data

3D Mask Attack Database (3DMAD). 3DMAD [https://www.idiap.ch/dataset/3dmad] <cit.> is the first public-available database for 3D face mask PAD with 3D face masks from Thatsmyface[www.thatsmyface.com]. The database consists of 17 subjects, 3 sessions and total 255 video clips (76500 frames). Each subject has 15 videos with 10 real faces and 5 face masks. Videos are captured through Kinect sensors and are equipped with color and depth maps in 640 × 480 resolution. Fig. <ref> shows some color examples of real face and 3D face mask.





HKBU-MARs V1. HKBU-MARs V1 [http://rds.comp.hkbu.edu.hk/mars/] <cit.> is a supplementary database for 3D face mask PAD. It contains 120 videos from 8 subjects and masks. Two types of 3D masks are included: 6 from Thatsmyface and 2 from REAL-F [http://real-f.jp/en_the-realface.html]. Each subject corresponds to 10 genuine samples and 5 masked samples. All videos are recorded through Logeitech C920 web-camera in 1280 × 720 resolution under room lighting condition. Fig. <ref> shows some color examples of real face and 3D face mask.











 §.§ Evaluation Protocol

For training and testing, we use the leave-one-out cross-validation (LOOCV) mechanism as used in paper <cit.>. For 3DMAD database, one subject's data is left for testing while the data of the remaining 16 subjects is divided into two subject-disjoint halves as training and development sets in each fold of cross validations.

However, for KBU-MARs V1 database, we only use this database for inter-test as one person's video data is not published.

With performance evaluation, the results are reported in term of recently standardized ISO/IEC 30107-3 metrics <cit.>: Attack Presentation Classification Error Rate (APCER) and Bona Fide Presentation Classification Error Rate (BPCER). It is noted that the APCER and BPCER depend on the decision threshold estimated by Equal Error Rate (EER) on the development set. To compare the overall system performance in a single value, the Average Classification Error Rate (ACER) is computed, which is the average of the APCER and the BPCER <cit.>. To compare with existing works, we also report the results of EER and Area Under Curve (AUC) on the test set.




§ EXPERIMENTAL RESULTS AND DISCUSSION

In this part, the intensity histograms of different planes are firstly evaluated on the detection performance. Then, various color spaces are analyzed for 3D face mask PAD. Furthermore, we extract the intensity histograms from original face images and compare the detection results with intrinsic images. After that, the inter-test experiments are performed on HKBU-MARs V1 database. Finally, the performance of our method is compared against the state-of-the-art approaches.







 §.§ Three Orthogonal Planes

Table <ref> illustrates the results of the intensity histograms extracted from different planes and combined using different mechanisms. From the table, we can clearly find that all averaged EERs and all averaged AUCs of test sets are 0.0% and 100%. This means that our proposed method can accurately distinguish between real faces and 3D masks by the decision threshold determined by the data set itself. However, when the test set uses the threshold of the development set, the detection performance of the algorithm in the test set will decrease. In addition, for combining different intensity histograms, it is shown that simultaneously concatenating TOP histograms will not improve the detection accuracy. We conjecture that the reason may lie in that the feature dimension is too high for the classifier to overfit. Rather, when cascading the intensity histograms extracted from XT and YT planes, our algorithm achieve the best results with APCER=0.1%, BPCER=0.8% and ACER=0.4%. This reflects the fact that 3D face masks are more sensitive to the changes in lighting than real face skin. Fig. <ref> shows the detailed classification of the best case in LOOCV mechanism. For instance, there are some classification failures in the samples of the 2nd and 7th subjects.






















































 §.§ Different Color Spaces

Furthermore, we extract the histograms from different color spaces (i.e. RGB, YCbCr and HSV), and compare the detection results in Table <ref>. The table shows that the color is a very important clue for 3D face mask PAD. Compared to the RGB color space, the YCbCr and HSV display color and brightness in different channels. For instance, the HSV shows the brightness information in H channel and the color information in S and V channels. When the intensity histograms are extracted from the color channel and brightness channel respectively, the detection performance of our algorithm can be significantly improved. It can be observed in Table <ref> that the ACER of HSV is 22 times lower than the ACER of the RGB color space.












 §.§ Comparison with Original Face Image

In order to verify the validity of the proposed intrinsic image analysis, we extract the intensity histograms from original face images in HSV color space and calculate the intensity variational information also using 1D CNN. The comparison results are shown in Table . When extracting the intensity histograms from original face images, the EER of dev set and ACER of test set are 0.7% and 1.1%, respectively. Compared the results of our proposed intrinsic image analysis based method, the ACER is reduced by nearly three times. This shows that it is reasonable and effective to detect 3D face mask in intrinsic image space.





 §.§ Cross Database Evaluation


To gain insight into the generalization capabilities of our proposed detection method, we conduct a cross-database evaluation. To be more specific, the countermeasure is trained and tuned on 3DMAD and then tested on HKBU-MARs V1 database.

The obtained detection results are summarized in Table <ref>. During the experiment, the first 9 subjects in 3DMAD are used to train our algorithm, and the remaining 8 subjects are used as the development set. When classifying according to the threshold of 3DMAD database, all 3D face masks are detected incorrectly with APCER=100% and all real faces are correctly classified with BPCER=0%. The reason may lie in that the materials of face mask of these two databases are different. From these results, we conclude that our proposed method is only for detecting the 3D face masks with specific material.


























 §.§ Comparison with the State of the Art

Table <ref> gives a comparison with the state-of-the-art in 3D face mask PAD techniques proposed in the literature. It can be seen that our proposed intrinsic image analysis based method outperforms the-state-of-the-art algorithms on the EERs of development set test set. Especially compared with the pulse detection based algorithms <cit.>, the ACER of our proposed method is 0.4%, which is nearly 20 times better than the ACER obtained by <cit.>. As aforementioned, the reason may lie in the pulse detection based methods are sensitive to camera settings and light conditions. For LBP-TOP <cit.> and DCCTL <cit.>, their EERs are 1.4% and 0.6%, respectively. Even though the performances are better than pulse detection based methods, they can be interfered by the attacker's head shake. Unfortunately, for 3D face mask PAD, there is only one fully public database (i.e. 3DMAD) so far, which makes it difficult to fully verify the effectiveness of our algorithm.




§ CONCLUSION

In this article, we addressed the problem of 3D face mask PAD from the viewpoint of intrinsic image analysis. We designed a new feature for reflectance characteristic description. Apart from that, we also introduced 1D CNN to extract intensity variation features. Extensive experiments on 3DMAD database showed excellent results. However, deep features extracted from three orthogonal planes should have different weights, but in our approach we simply concatenate them together. Therefore, we will design a attention model to tackle this problem. For the generalization capability, we think developing a 3D face mask PAD method with interoperability is big open issue, which is far from the current state of the art. So we leave it to future work.




§ REFERENCES



