This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations. To this end, we propose a multi-modal learning framework that links the inference stage and generation stage. First, the inference networks are trained to match the speaker identity between the two different modalities. Then the pre-trained inference networks cooperate with the generation network by giving conditional information about the voice. Utilizing audio-visual cues together to recognize a person's identity has been studied in various fields from neuroscience (Hasan et al., 2016; Tsantani et al., 2019) to practical machine learning applications (Nagrani et al., 2018b; a; Wen et al., 2019a; Shon et al., 2019) . For example, some neurological studies have found that in some cortical areas, humans recognize familiar individuals by combining signals from several modalities, such as faces and voices (Hasan et al., 2016) . In conjunction with the neurological studies, it is also a well known fact that a human speech production system is directly related to the shape of the vocal tract (Mermelstein, 1967; Teager & Teager, 1990) . Inspired by the aforementioned scientific evidence, we would like to ask three related questions from the perspective of machine learning: 1) Is it possible to match the identity of faces and voices? (inference) 2) If so, is it possible to generate a face image from a speech signal? (generation) 3) Can we find the relationship between the two modalities only using cross-modal self-supervision with the data "in-the-wild"? To answer these questions, we design a two-step approach where the inference and generation stages are trained sequentially. First, the two inference networks for each modality (speech encoder and face encoder) are trained to extract the useful features and to compute the cross-modal identity matching probability. Then the trained inference networks are transferred to the generation stage to pass the information about the speech, which helps the generation network to output the face image from the conditioned speech. We believe, however, that it is impossible to perfectly reconstruct all the attributes in the image of a person's face through the characteristics of the voice alone. This is due to factors that are clearly unrelated to one's voice, such as lighting, glasses, and orientation, that also exist in the natural face image. To reflect the diverse characteristics presented in the face images "in-the-wild", we therefore model the generation process by incorporating two latent factors into the neural network. More specifically, we adopted conditional generative adversarial networks (cGANs) (Mirza & Osindero, 2014; so that the generator network can produce a face image that is dependent not only on the paired speech condition, but also on the stochastic variable. This allows the latent factors that contribute to the overall facial attributes to be disentangled into two factors: one that is relevant to the voice and the other that is irrelevant. Adopting cGANs negligently still leaves a few problems. For example, the condition in a cGANs framework is typically provided as embedded conditional vectors through the embedding look-up table for one-hot encoded labels (Brock et al., 2019; . The raw signals such as speech, however, cannot be taken directly from the embedding look-up table, so an encoder module is required. Therefore, the trained speech encoder from the inference step is reused to output a pseudo conditional label that is used to extract meaningful information relevant to the corresponding face. Then the generator and the discriminator are trained in an adversarial way by utilizing the pseudo-embedded conditional vectors obtained from the trained speech encoder in the first step. Another problem with applying the conventional cGANs for generating faces from voice arises from the fact that the distinction between different speakers can be quite subtle, which calls for a need for a more effective conditioning method. To mitigate this problem, we propose a new loss function, relativistic identity cGANs (relidGANs) loss, with modification of the relativistic GANs (JolicoeurMartineau, 2019), allowing us to generate the face with a more distinct identity. Each step will be described in greater detail in Section 3. Our contributions can be summarized as follows: 1. We propose simple but effective end-to-end inference networks trained on audio-visual data without any labels in a self-supervised manner that perform a cross-modal identity matching task. 2. A cGANs-based generation framework is proposed to generate the face from speech, to be seamlessly integrated with the trained networks from inference stage. 3. A new loss function, so called a relidGANs loss, is designed to preserve a more consistent identity between the voices and the generated images. 4. An extensive analysis is conducted on both inference and generation tasks to validate our proposed approaches. In this work, we proposed a cross-modal inference and generation framework that can be trained in a fully self-supervised way. We trained cGANs by transferring the trained networks from the inference stage so that the speech could be successfully encoded as a pseudo conditional embedding. We also proposed relidGANs loss to train the discriminator to penalize negatively paired face and speech so that the generator could produce face images with more distinguished identity between different speakers. As a future work, we would like to address a data bias problem (e.g., ethnicity, gender, age, etc.) that exists in many datasets. This is a significant problem as many publicly available datasets have biased demographic statistics, consequently affecting the results of many algorithms (Buolamwini & Gebru, 2018) . We believe that this can be solved with the use of a better data sampling strategy in an unsupervised manner such as (Amini et al., 2019) . In addition, we would like to expand the proposed methods to various multi-modal datasets by generalizing the proposed concept to other modalities. Real Generated Top-5 nearest face images
This paper proposes a method of end-to-end multi-modal generation of human face from speech based on a self-supervised learning framework. This paper presents a multi-modal learning framework that links the inference stage and generation stage for seeking the possibility of generating the human face from voice solely. This work aims to build one conditional face image generation framework from the audio signal. 