Recovering 3D geometry shape, albedo and lighting from a single image has wide applications in many areas, which is also a typical ill-posed problem. In order to eliminate the ambiguity, face prior knowledge like linear 3D morphable models (3DMM) learned from limited scan data are often adopted to the reconstruction process. However, methods based on linear parametric models cannot generalize well for facial images in the wild with various ages, ethnicity, expressions, poses, and lightings. Recent methods aim to learn a nonlinear parametric model using convolutional neural networks (CNN) to regress the face shape and texture directly. However, the models were only trained on a dataset that is generated from a linear 3DMM. Moreover, the identity and expression representations are entangled in these models, which hurdles many facial editing applications. In this paper, we train our model with adversarial loss in a semi-supervised manner on hybrid batches of unlabeled and labeled face images to exploit the value of large amounts of unlabeled face images from unconstrained photo collections. A novel center loss is introduced to make sure that different facial images from the same person have the same identity shape and albedo. Besides, our proposed model disentangles identity, expression, pose, and lighting representations, which improves the overall reconstruction performance and facilitates facial editing applications, e.g., expression transfer. Comprehensive experiments demonstrate that our model produces high-quality reconstruction compared to state-of-the-art methods and is robust to various expression, pose, and lighting conditions.
 3D face reconstruction from 2D images enables many exciting applications, such as face recognition (Blanz & Vetter, 2003; Paysan et al., 2009; , face puppetry , face reenactment (Thies et al., 2016; Garrido et al., 2015) , virtual make-up , etc. However, 3D face shape and texture inference from 2D images, especially from a single image, is an ill-posed problem since some 3D information is lost after the imaging process. 3D morphable model (3DMM) (Blanz & Vetter, 1999) learned from a collection of 3D face scans is often adopted as a strong prior assumption for this problem. 3DMM is a linear combination of bases to provide statistical parametric representation of 3D faces. Given a 2D image, the conventional approach is to search for the corresponding 3DMM parameters through analysis-by-synthesis optimization (Levine & Yu, 2009; Booth et al., 2018) . Specifically, a 3D face is generated through inverse rendering to match the 2D image by optimizing the shape, albedo (i.e., texture separated from illumination conditions), pose, and lighting parameters. However, such 3DMM optimization-based methods are usually timeconsuming due to high optimization complexity and suffer from local optima solutions. Regressing 3DMM parameters using convolution neural network (CNN) shows remarkable success in 3D face reconstruction (Richardson et al., 2016; Zhu et al., 2019; Genova et al., 2018; . However, these methods cannot go beyond but only search for a solution in the restricted linear low-dimensional subspace of 3DMM. Linear statistical models have limitations to construct 3D face shapes and textures. First, facial variations are nonlinear in the real world, e.g., various ethnic groups, ages, facial expressions, and skin colors. Second, in order to model highly variable 3D face, a large amount of 3D face scans are needed for training. The most popular 3DMM (Xiangyu Zhu et al., 2015) was built by merging Basel Face Model (BFM) (Paysan et al., 2009 ) with only 200 subjects in neutral expressions and FaceWarehouse with 150 subjects in 20 different expressions, which is not able to fully capture the variability of human faces. A large scale facial model (LSFM) was constructed by Booth et al. (2016) from around 10,000 distinct facial identities but only in neutral expressions. Tewari et al. (2018) , , and Guo et al. (2019) further proposed 3D face models composed of two networks: a coarse-scale linear 3DMM network and a fine-scale corrective network. Even though the finle-scale corrective model can generate more details, 3D face reconstruction will fail if the foundation face shape generated by the linear 3DMM network is not good enough. Recently, Tran & Liu (2018) and Tran et al. (2019) proposed encoder-decoder networks to regress the face shape and texture directly. The nonlinear networks have higher representation power compared to a linear model and are able to reconstruct high-fidelity facial texture. However, the nonlinear models were only trained on the 300W-LP dataset (Zhu et al., 2016) that is generated from a linear 3DMM with a face profiling technique. The models were further fine-tuned in a self-supervised manner on the same dataset. However, since most of the face images were synthesised based on the linear 3DMM, self-supervised training to reconstruct high-fidelity texture using inverse rendering makes limited contributions to the face shape reconstruction. Besides, in these methods, the face albedo and face shape are decoded from a albedo parameter and shape parameter separately without considering the facial identity. In fact, across one's different face images, the face albedo and identity shape should only depend on the facial identity, i.e., sharing the same identity representation. Learning albedo and shape parameter separately is difficult to disentangle the face albedo from lightings and occlusions. Especially, when the albedo decoder network has high representation power, the albedo decoder may reconstruct high-fidelity face albedo but without aligning with the face shape and fails to contribute to the face shape reconstruction. At last, the identity and expression representations are entangled in these methods and many applications, such as face recognition, face animation, and face reenactment, are not feasible. In this paper, we propose a novel encoder-decoder architecture using inverse rendering that combines computer vision and computer graphics techniques. The vision system (i.e., encoder network) decomposes an input 2D face image into disentangled and sematic representations: identity code, expression code, pose code, and lighting code. The graphics system renders back a face image to match the input image based on the decoder networks that regress the 3D face shape and albedo from the extracted representations. Combining computer vision and computer graphics techniques provides a unique opportunity to leverage the vast amounts of readily available unlabelled face images from unconstrained photo collections through self-supervised learning. Since 3D face reconstruction from a 2D image is ambiguous and ill-posed, self-supervised learning with unlabelled data through inverse learning is not sufficient. In this paper, we train the network in a semi-supervised manner on hybrid batches of large amounts of unlabeled face images and relatively small amounts of labelled face images that are generated from a linear 3DMM with optimizationbased methods. Moreover, following the idea of generative adversarial networks (GAN) (Goodfellow et al., 2014) , a discriminator network is used to ensure the reconstructed face shape is not too far away from the distribution of human face. Semi-supervised adversarial training not only prevents our model from generating unrealistic 3D face shape but also fully exploits the value of unlabeled face images without being constrained by the pre-existing linear 3DMM. To reconstruct the 3D face shape, we use graph convolutional network (GCN) (Defferrard et al., 2016; Kipf & Welling, 2017) instead of fully connected layers with activation or CNN used in Tran & Liu (2018) and Tran et al. (2019) . A 3D face shape is usually modeled as a mesh that is defined by a collection of vertices, edges, and faces and is considered as an unstructured graph. Modeling graph convolutions on 3D meshes can be memory efficient and allows for processing high resolution 3D structures. GCN-based methods to reconstruct 3D face shapes outperforms other state-of-the-art methods (Ranjan et al., 2018; Bouritsas et al., 2019) . To recover the 3D face albedo, we first use a GCN network that has the same architecture with the shape decoder to learn an illumination-independent face albedo. Then we apply a CNN-based decoder network that has skip connections with the encoder network (Ronneberger et al., 2015) and a patchGAN (Shrivastava et al., 2017) to improve the details of the facial texture. We apply a face recognition loss and a center loss (Wen et al., 2016) to extract the identity representation (i.e., facial identity) from one's unconstrained multiple face images. The center loss is used to ensure the identity representation's compactness for each person and separability for different people, so that the identity representation is disentangled from the pose, lighting, and expression representations. In order to further disentangle the identity and expression representations, pairwise training approaches are adopted. Given a pair of labelled face data, we keep the identity codes and interchange the expression codes of 3DMM to generate new 3D shapes as supervision. Comprehensive evaluation experiments show that the proposed method achieves state-of-the-art performance in 3D face reconstruction and can easily be used for the applications of face recognition and facial expression transfer. The main contributions of this paper are summarized below: • We propose an efficient semi-supervised and adversarial training process to fully exploit the value of unlabelled face data and go beyond the limitation of a linear 3DMM. • We design a novel framework to exact nonlinear disentangled representations from a face image with the help of face recognition losses and shape pairwise loss. • Extensive experiments show that our model achieves state-of-the-art performance in face reconstruction. This paper proposes an encoder-decoder architecture to reconstruct 3D face from a single image with disentangled representations: identity, expression, pose, and lighting. We develop an effective semi-supervised training scheme to fully exploit the value of large amount of unlabeled face images from unconstrained photo collections. An adversarial loss is applied to prevent our model from generating unrealistic 3D faces. We evaluate our model quantitatively and qualitatively. Our model outperforms the state-of-the-art single-view reconstruction methods and can effectively disentangle identity, expression, pose, and lighting features.
We train our face reconstruction model with adversarial loss in semi-supervised manner on hybrid batches of unlabeled and labeled face images to exploit the value of large amounts of unlabeled face images from unconstrained photo collections. This paper proposes a semi-supervised and adversarial training process to exact nonlinear disentangled representations from a face image with loss functions, achieving state-of-the-art performance in face reconstruction.