How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations
Florian Blume Runfeng Qu Pia Bideau Martin Maier Rasha Abdel Rahman Olaf Hellwich







9


Main Track









How Do You Perceive My Face?






GCPR 2024 Submission . CONFIDENTIAL REVIEW COPY.
	GCPR 2024 Submission . CONFIDENTIAL REVIEW COPY.
	
	Paper ID 


	
	
	F. Blume et al.
	
	
	
	
        Technische Universität Berlin
        Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK
        Humboldt-Universität zu Berlin
        Science of Intelligence, Research Cluster of Excellence, Berlin
        
            
 envelope florian.blume@tu-berlin.de, runfeng.qu@tu-berlin.de
        
    


How Do You Perceive My Face?

Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations
    Florian Blumeequal contribution1,40000-0002-7557-1508envelope Runfeng Qu1,40009-0008-7885-8812 Pia Bideau20000-0001-8145-1732 Martin Maier3,40000-0003-4564-9834 Rasha Abdel Rahman3,40000-0002-8438-1570 Olaf Hellwich1,40000-0002-2871-9266
    September 9, 2024
=================================================================================================================================================================================================================================================



§ ABSTRACT 
Facial expression perception in humans inherently relies on prior knowledge and contextual cues, contributing to efficient and flexible processing. For instance, multi-modal emotional context (such as voice color, affective text, body pose, etc.) can prompt people to perceive emotional expressions in objectively neutral faces. Drawing inspiration from this, we introduce a novel approach for facial expression classification that goes beyond simple classification tasks. Our model accurately classifies a perceived face and synthesizes the corresponding mental representation perceived by a human when observing a face in context. With this, our model offers visual insights into its internal decision-making process. We achieve this by learning two independent representations of content and context using a VAE-GAN architecture. Subsequently, we propose a novel attention mechanism for context-dependent feature adaptation. The adapted representation is used for classification and to generate a context-augmented expression. We evaluate synthesized expressions in a human study, showing that our model effectively produces approximations of human mental representations. We achieve State-of-the-Art classification accuracies of % on the RAVDESS dataset and % on the MEAD dataset. We make our code publicly available[https://github.com/tub-cv-group/recognizing-by-modeling].











§ INTRODUCTION



Integrating multi-modal contextual information is crucial for generating adaptive behavior and enabling an agent to respond appropriately to its environment. More specifically, contextual information encompasses the multi-modal information that enhances the agent's perception and thus is a prerequisite for adaptive behavior. Latest work in cognitive psychology has shown that the human brain leverages contextual cues and prior knowledge to dynamically adjust future predictions about incoming sensory input <cit.>. Concurrent sound, textual cues, or prior knowledge offer additional information that shapes social perception <cit.>. For example, in the interpretation of facial expressions, the individual's voice plays a significant role in understanding their overall expression. As illustrated in <ref>, the same neutral face is perceived as displaying a more positive or negative expression when presented in the context of the respective prior beliefs about the person. In this work, we refer to the perceived facial expression as the synthesized mental representation.







Previous works in computer vision typically either only perform fer <cit.> or generate expressions <cit.>. 
Only few approaches exist that perform both tasks jointly <cit.>. 
Addressing recognition and generation jointly, however, poses an essential element in modelling human social interaction and creating effective communication between humans and artificial agents, enabling agents to mimic the expressions of conversational partners in a 
context-sensitive manner and in direct alignment with the recognized expression. 

We propose a novel mechanism of fusing expressions and multi-modal context information encoded in the latent space of a vae <cit.>. In particular, using an attention mechanism, our model dynamically adapts previously learned representations of facial expressions using context. Operating on lower-dimensional representations of facial expressions enables us to simultaneously produce well-aligned classifications and an approximation of the perceived expression. We verify the validity of these approximations in a rating study with 160 human observers and show SOTA classification accuracy on the ravdess and mead datasets. Our model performs a task that is similar to the studies in <cit.>, where human participants were presented with individual photographs paired with affective-semantic contexts. Our contributions are threefold: (1) We present a model that for the first time simultaneously classifies expressions and produces approximations of their mental representations, which are inherently well aligned with the predicted class. (2) We evaluate these approximations in a human study. They capture the fine-grained effects of emotional context on human perception. (3) Our model is explainable due to its ability to visualize the adapted features by generating a context-augmented expression.





§ RELATED WORKS



We discuss three types of works: (1) context-sensitive fer-only, (2) synthesizing expressions, and (3) performing both tasks jointly.


Multi-modal context-sensitive facial expression recognition. Multi-modal context-sensitivity in fer ranges from incorporating visual surroundings as additional information cues <cit.>, to drawing on audio <cit.>, text <cit.>, body pose <cit.> or combining multiple context sources <cit.>. Transformers have successfully been incorporated into fer in multiple approaches <cit.>. Contrastive learning schemes have shown to extract general features that ensure good classification performance on unseen data <cit.>.  <cit.> outperform sota methods on ravdess using an unsupervised contrastive learning scheme on four modalities.   <cit.> use the transformer architecture and action units to predict expressions on ravdess. In contrast to our work, generating context-augmented versions of the input face is not straightforward in these approaches.


Generating facial expressions in context. gan have been used for generating realistic looking face images <cit.>.  <cit.> combined them with a vae to allow smooth transitions between representations.  <cit.> employ a gan to generate a talking face from audio.  <cit.> generate a 3D talking head based on the audio of the ravdess dataset, which is capable of producing facial expressions. Using artificial characters in a rating study results in a different perceptual experience for humans <cit.>, which makes them inapplicable to our goal.  <cit.> train their network in a CLIP-like <cit.> fashion to generate sequences of talking faces.  <cit.> employ latent diffusion <cit.> for the same task. None of these works target joint generation of mental representations and classification.


Joint Facial Expression Generation and Recognition. Few works exist that perform the task of simultaneously generating facial expressions while also predicting expression classes.  <cit.> train two gan cooperatively to recognize facial expressions under large view angle changes.  <cit.> employ a gan <cit.> to overcome the lack of labeled training data in fer by jointly training it together with an expression recognition network. Context sensitivity is not part of their work.  <cit.> also draw on a gan-based architecture and argue that by generating expressions, they help solve the issue of appearance variance in fer and lack of training data. Their network processes only input images and does not take additional modalities into account. None of the disucssed works model mental representation through synthesized expressions.


§ METHOD




In this Section, we describe our multi-modal approach that adapts an expression image using affective audio. Our proposed adaption module allows us to classify a facial expression in light of context and at the same time synthesize a novel facial expression as it would have been perceived by a human. We employ a two-stream encoder-decoder backbone, based on a vae-gan combination <cit.>, and an attention mechanism to combine their latent spaces in a context-sensitive way. <ref> depicts an overview of this design: The representations learned by the face_reconst_net and the context_reconst_net are adapted in the context_att_net, by shifting the facial features using the context features. The shifted representation is visualized using the fixed decoder (D) of the face_reconst_net and classified with the classification head (E). Our model operates and is trained on individual frames together with audio context. To evaluate our model on videos, we perform majority voting over classes of the frames.



 §.§ Face and Context Reconstruction Network


The face and context reconstruction networks (called face_reconst_net and context_reconst_net respectively) both consist of a vae and gan discriminator. We follow <cit.> and add a gan discriminator for training to increase image quality.


VAE Module. Similarly to <cit.>, we add skip connections to the en- and decoder of the vae to speed up the training process and allow processing of larger image resolutions. Let enc_i be the expression encoder of (A1) and face_input∈ℝ^m × n × 3 an input expression image:

    (img_mean, img_variance) = enc_i(face_input)

where img_mean∈ℝ^latent_space_dim and img_variance∈ℝ^latent_space_dim denote the mean and variance of a Gaussian distribution, respectively. Mean ctx_mean∈ℝ^latent_space_dim and ctx_variance∈ℝ^latent_space_dim variance of the context Mel spectrogram context_input∈ℝ^u × v are computed using the encoder of (B1) and the following formulas are applied analogously.

The prior loss term keeps the latent distribution close to a Gaussian:

    prior_loss = d_kl( posterior_latent_input_image || prior_latent_image)

d_kl is the kl_divergence, posterior_latent_input_image is the posterior of the latent vector latent_rep_image∈ℝ^latent_space_dim under input face_input and prior_latent_image is the Gaussian prior over the latent vector.

The reconstruction loss term penalizes the feature map of the discriminator (A2 and B2 in <ref>) at a certain level, as proposed in <cit.>, using mse:


    disc_loss = mse(dis_i^l(dec_i(enc_i(face_input))), dis_i^l(face_input))

where face_input is the input image and dis_i^l the l-th feature map of the discriminator.


GAN Module. The gan discriminator's task is to distinguish between input images face_input from the dataset and reconstructions img_reconst. In addition, it is tasked to identify reconstructions from random noise sampled_noise∼𝒩(0, 1) to enforce generation capabilities in the vae. The overall loss is the following:


    gan_loss =    log (dis_i(face_input)) + log (1 - dis_i(dec_i(enc_i(face_input))))  
        + log (1 - dis_i(dec_i(sampled_noise)))




  
Joint Training. In the pretraining phase, face_reconst_net and context_reconst_net are trained unsupervised for input reconstruction and are fixed in subsequent training. We follow the training algorithm of <cit.> and compute the joint update as follows:


    θ_enc   +⟵ - ∇_θ_enc(βprior_loss + disc_loss) 
    θ_dec_i   +⟵ - ∇_θ_dec_i( disc_loss - gan_loss) 
    θ_dis_i   +⟵ - ∇_θ_dis_igan_loss

where β∈ℝ is a hyperparameter to weigh the prior loss.



 §.§ Context-Attention Network (CAN) and Classification Head


Our proposed attention mechanism shifts the facial expression distribution of the face_reconst_net based on the context distribution of the context_reconst_net. <ref> illustrates this fusion technique.


Context-Attention Net. The context_att_net computes attention maps based on mean and variance of the distributions of the context and the facial expression input. We use these maps to compute offsets mean_offset∈ℝ^latent_space_dim and variance_offset∈ℝ^latent_space_dim to shift the face mean and variance context-dependently. We compute the following parameters for the attention mechanism:

    query_mu = attention_weights_mu_qctx_mean
    key_mu = attention_weights_mu_kimg_mean
    value_mu = attention_weights_mu_vimg_mean

attention_weights_mu_q, attention_weights_mu_k,attention_weights_mu_v∈ℝ^latent_space_dim×latent_space_dim are the trainable parameters of linear layers (bias omitted for simplicity). We compute the attention map attention_map∈ℝ^latent_space_dim×latent_space_dim as


    attention_map = softmax(query_mukey_mu^T)

where the softmax function is applied row-wise. Note that we reverse the attention mechanism from <cit.> - we do not compute the dot products of the query with all keys, instead, we compute the dot product of a key to all queries. We do this to get attention on the facial mean based on the context mean. The offset mean_offset and the resulting shifted mean shifted_mean are then computed as 

    mean_offset    =  attention_mapvalue_mu
    shifted_mean    = mean_offset + img_mean

<ref> provides a visualization of these relationships. During inference we can vary the strength of the context influence by multiplying the offset with a weight, to allow smooth modulation of the offset:

    shifted_mean = context_weight·mean_offset+ img_mean

We compute the new (shifted) variance shifted_variance analogously, the only difference being that we operate in log scale.


Joint Context Attention Network and Classification Head Training. To train the context_att_net and classification head jointly, we first initialize the latter by training it directly on the facial features using the expression classes and cross-entropy loss. Next, we train the network together using the following loss:

    ℒ = cross_entropy(E(shifted_mean), y_true) + αd_kl(posterior_latent_rep_image||posterior_latent_rep_shifted_face_context)


where cross_entropy is the cross-entropy, E is the classification head and α∈ℝ is a hyperparameter to regularize the shift. Furthermore, during training, we propose a novel data augmentation approach for multi-modal settings, where we swap contexts for a specific actor within its expression class.[Note, this data augmentation is only possible for multi-modal datasets, where content comes with different context variations within their respective expression class.]


§ EXPERIMENTS

We evaluate our approach on publicly available datasets <cit.>. We discuss the results in Section <ref> and <ref>. An ablation study is shown to support the understanding of our proposed approach for fer in multi-modal context.



 §.§ Datasets


celeb_db (Pretraining). We pretrain the face_reconst_net unsupervisedly for face reconstruction on the large-scale dataset celeb_db <cit.>. celeb_db is a prevalent dataset for face attribute recognition and consists of roughly 200k images showing 10k different identities. The context_reconst_net is trained using the respective downstream datasets.


ravdess (Downstream). ravdess <cit.> consists of videos of 24 identities. Each video is labeled with one of the seven expression classes calm, happy, sad, angry, fearful, surprise, and disgust, with an additional binary label for the intensity. We extract 16 frames from each video at regular intervals and use the video's label for each. Following <cit.> the neutral class is omitted to reduce noise.


mead (Downstream). mead <cit.> is a large-scale dataset targeting talking-face generation, which also features fer labels. Similar to <cit.>, mead contains videos of 60 actors speaking with different emotions at different intensity levels. We use the frontal view recordings as proposed by <cit.> and apply the same frame extraction approach as for ravdess.



 §.§ Implementation Details


We set the batch size to 256, β (<ref>) and α (<ref>) to 0.00001 and context_weight=1.0 (<ref>). The dimension of the latent space of the face_reconst_net and context_reconst_net is latent_space_dim = 512. We use MTCNN <cit.> to detect faces. We then resize them to 128 × 128 pixels, which is also the size of the reconstructed and generated images. We apply random horizontal flipping as data augmentation. For generating the Mel spectrograms, we chose 128 Mel bins, a sample rate of 22050, a window and FFT length of 1310, and a hop length of 755. We use the Adam optimizer <cit.> with a learning rate of 0.00003 and a weight decay of 0.01.


Reconstruction Pretraining. We pretrain the face_reconst_net unsupervised for facial image reconstruction on celeb_db and the context_reconst_net for context reconstruction on the Mel spectrograms of the downstream datasets. Pretraining is run for 400 epochs. The learning rate is decreased by factor 10 after 150 and 300 epochs.


Downstream Classification Training.  During downstream training, the face_reconst_net is fixed and the last two layers of the context_reconst_net are fine-tuned. We first initialize the one-layer classification head by training it directly on the facial features of the face_reconst_net to obtain a suitable initialization for its weights. Next, we train the context_att_net and the single-layer classification head jointly together using the loss from <ref>. 
Note that the decoder from (A1), which we use to visualize the shifted expression, is fixed and not trained in this step.

For both ravdess and mead, we performed k-fold cross validation with k=10, similarly to other works <cit.>, splitting the folds along the identities (one identity can only occur in test, validation or train set).



 §.§ Facial Expression Recognition Performance







We provide classification results for ravdess in <ref>, and for mead in <ref>. All compared methods use a dataset split by identities, ensuring no identity seen during training appears during testing.

For our method, the final prediction for a test video is obtained by majority voting across frames. 
Our model achieves an accuracy of % on ravdess and % on mead, matching SOTA performance on classical fer. We largely outperform methods - by 17.75% on ravdess and 8.47% on mead, that tackle the dual problem of classification and the synthesize of the corresponding percept.  




The influence of context on the final fer accuracy is highlighted by visualizing the per-class accuracy in confusion matrices for two conditions: in <ref>, the context_att_net computed the offset for the facial features based on the facial (instead of the context) features to simulate missing context. In <ref>, it received the face together with the context features, as intended by our approach. The latter exhibits higher (or equal) probabilities on the diagonal for every class. This proves that the reported accuracy cannot be attributed solely to the computational capabilities of the context_att_net but is a result of the meaningful adaption procedure.

0.8
0.49
colormap/jet





The tsne plots in <ref> visualize the structure of the latent space in different conditions: (1) The face-only features as we receive them from the face_reconst_net do not cluster in any particular way (<ref>). (2) Employing our context_att_net but providing the face features twice instead of combining with the context leads to a more structured latent space (<ref>) and (3) taking the audio context into account leads to a clustered latent space that makes classification easier (<ref>).



Ablation Study. <ref> lists classification performance of our context_att_net and its components. We assess the quality of face and context features independently by classifying learned features directly using a single-layer classifier. Experiments for the context_att_net in a simulated uni-modal mode are run by providing face or context twice. If no context is provided, the context_att_net performs similarly to the single layer classifier using learned representations. However, when learned features are adapted to context, the final classification performance improves by 15.44% (strict audio). Additionally, our new data augmentation technique for multi-modal settings further enhances performance by 1.72%.




 §.§ Mental Representations: Context-Augmented Expressions





We show qualitative results of our approach on ravdess in different conditions (additional generations for ravdess and mead in the suppl. material). To visualize the adapted features obtained from the context_att_net, we use the decoder (D) from <ref>. <ref> depicts the effect of the two kinds of intensities provided in ravdess on neural input faces. The stronger the intensity, the stronger the facial expression in the generated image. The strength of the effect is subtle but resembles human perception <cit.>, which is our goal. 

To our knowledge, the only publications showing generations on ravdess are  <cit.>,  <cit.> and  <cit.>. They aim at rendering an input face that strongly resembles the emotion in the accompanying audio file. This defies our goal of capturing the subtle changes in appearance a human observer would have, nevertheless we provide a comparison in the suppl. material.

How influential is the context in which humans perceive facial expressions? 
To empirically evaluate the strength of contextual influence that best captures context effects in humans, we aimed to manually control the computed offset using context_weight of <ref>. As shown in <ref>, this allowed us to generate facial expressions with varying degrees of contextual influence, which makes them useful as experimental stimuli. The context weight context_weight is increased from 0 to 1 in 0.1 steps from left to right. Note that this weight is independent of the two intensities provided by ravdess (<ref>). The smooth transition demonstrates that our model creates continuous representations of the input data.





 §.§ Human Study: Verifying Synthesised Mental Representations


To assess our model's ability to replicate the contextual impact of emotional speech on facial expression perception in human observers, we conducted two experiments with a total of 160 participants. In the first experiment, 80 participants evaluated neutral facial expressions from the ravdess dataset while listening to the depicted actor's speech with either happy or angry prosody. We restricted the study to these two classes because they are of opposite valence. This allowed us to obtain more fine-grained ratings on a continuous Likert scale. We aimed to measure the impact of the audio's emotion on perceived facial expressions. In the second experiment, a different group of 80 participants rated facial expressions synthesized by our model under happy or angry contextual influences. Each face was presented with five different context weights context_weight. 

To test whether the model's generations can approximate human-like perception, we compared ratings with those obtained in Experiment 1. This comparison was done for each context weight, determining the parameter that best approximates human responses to contextual influences.



Results. Mean facial expression ratings for different emotional context conditions across both experiments are depicted in <ref>. 
In experiment 1, a linear mixed effects model was employed, with the factor emotional context (happy vs. angry audio). A significant effect of emotional context on facial expression ratings was observed (b = 1.00, p < .001), indicating that identical neutral faces were perceived as more negative when accompanied by angry speech compared to happy speech. For experiment 2, a linear mixed effects model was run with the factors emotional context (happy vs. angry generated expression) and weight (context_weight = 0, 0.25, 0.5, 0.75, and 1). To compare rating differences within each weight, the emotional context factor was nested within the weight factor. As shown in  <ref> (Experiment 2), expression ratings for faces generated in the context of angry vs. happy prosody did not differ at weights 0 and 0.25 (bs ≤ 0.19, ps > .269), but showed a significant and increasing impact of emotional context at weights 0.5 (b = 0.37, p = .033), 0.75 (b = 0.51, p = .004), and 1 (b = 0.73, p < .001).


Discussion. The results of our rating study reveal two key findings: 1) Our model effectively captures the impact of context, exemplified by emotional prosody, on human facial expression perception. Our generations reflect how a neutral face would subjectively appear to a human observer when associated with a positive or negative context.  2) The model's efficacy in shifting facial appearance towards contextual emotions does not require further adjustment post-training, as evidenced by the optimal context weight being context_weight = 1.



§ CONCLUSION



In this work, we introduced a novel approach that simultaneously enhances expression class predictions by taking affective context into account, and provides the means to generate an approximation of the expression a human would perceive. Our model achieves sota accuracy on ravdess and mead, and outperforms joint competitor methods. The implications of our rating study showcase that our model not only accurately quantifies human context-sensitive perception but also successfully mirrors the altered subjective experience back to human observers. This has significant potential, particularly for social artificial agents, that could leverage contextual information to adapt to human mental and emotional states, ensuring successful communication. Our model also addresses the dual nature of context-sensitivity of human perception: on the one hand, leveraging context enhances perceptual efficiency and flexibility <cit.>, while on the other hand, it bears the potential for adversely biased perception, e.g., when contextual information originates from untrustworthy sources <cit.>.



§ ACKNOWLEDGEMENTS


Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2002/1 “Science of Intelligence” – project number 390523135. In addition, this work has been partially supported by MIAI@Grenoble Alpes, (ANR-19-P3IA-0003)



§ DISCLOSURE OF INTERESTS


The authors have no competing interests to declare that are relevant to the content of this article.

splncs04






§ ABSTRACT 
In our work, we presented a model that encodes facial images and audio context using vae. We developed a context fusion network called context_att_net which shifts the latent facial distribution, to allow more accurate classifications by the classification head, as well as generate an approximation of the facial expression a human would perceive. Here, we provide additional details on the joint training of the context_att_net and classification head and provide a derivation of their loss function. We also list more detailed parameters of our rating study. Lastly, we provide more facial expression generations that show the capabilities of our model to purposefully fuse a facial image with affective audio context. The generations are based on neutral face images paired with different audio contexts to mimic the setting in our rating study. In addition, we release the code of our work.




§ ARCHITECTURE DETAILS


The number of layers of the components of our model are given in <ref>.





§ RATING STUDY


In this section we provide additional information about the procedure used for our rating study with human participants. The study was conducted according to the principles expressed in the Declaration of Helsinki and was approved by the ethics committee of Department of Psychology at Humboldt-Universität zu Berlin. All participants gave their informed consent.



 §.§ Materials

In Experiment 1, faces of 24 actors from the RAVDESS database with a neutral expression and audio files in which the actors said the sentence “Kids are playing by the door” either with angry or happy emotional prosody served as stimuli. The images were neutral as generated by our network with a weight parameter of 0. We chose a generated face instead of the original frame from the RAVDESS database to eliminate potential effects of low-level visual differences between generated images and images from the database.

In Experiment 2, neutral faces of the same 24 actors were shifted towards the model’s representation of the face in the context of either a happy audio or an angry audio, with five different weights: 0 (i.e. the neutral expression also presented in Experiment 1), 0.25, 0.5, 0.75, and 1 (i.e. the model’s originally trained weight parameter).

In both experiments, we used counterbalancing across participants, such that one participant saw each actor either in the happy or in the angry emotion condition and each face was shown equally as often in each emotion condition.



  §.§.§ Participants


The study adhered to the principles of the Declaration of Helsinki, approved by the ethics committee of Department of Psychology at Humboldt-Universität zu Berlin. Participants were recruited from Prolific.com and received monetary compensation. For each experiment, 80 participants were recruited. The final samples included 72 English native speakers aged 18–35 years (M = 29.01) in Experiment 1 and 77 English native speakers aged 18–35 years (M = 28.58) in Experiment 2.
The samples were balanced, with 50% of participants identifying as female and 50% identifying as male. We used the following criteria for inclusion in the final samples: Participants who reported not being highly distracted during the experiment, participants who reported not giving random ratings, participants who reported being able to hear all the audio files (Experiment 1), and participants who did not report rating only the audio files (Experiment 1).
Participants were pre-screened for the following criteria based on their data provided to Prolific.com

Age: 18–35;
Prison: No;
Approval Rate: 90–100%;
Units of alcohol per week: 0, 1-4, or 5-9;
Neurodiversity: No;
Dyslexia: No;
Vision: Yes;
Hearing difficulties: No;
Cochlear implant: No;
Colourblindness: No;
Head Injury—Knock out history: No;
Head Injury: No;
Mental health/illness/condition - ongoing: No;
Medication use: No;
Mild cognitive impairment/Dementia: No;
Autism Spectrum Disorder: No;
Depression: No;
Mental illness daily impact: No;
Anxiety: No;
ADD/ADHD: No;
Anxiety Severity: No;
Mental Health Diagnosis: No;
Mental Health Treatment: None;
First Language: English.



 §.§ Procedure


Both experiments followed a similar procedure: After providing informed consent, participants rated the facial expressions of stimuli described above in random order on a 9-point Likert scale ranging from “very negative” to “very positive” with “neutral” in the middle. 
In Experiment 1, ratings pertained to a neutral face presented with happy or angry audio, while in Experiment 2, faces modified by our model to depict contextual influence were rated. Post-rating, participants answered questions about their task experience (e.g., whether they were distracted, whether they gave their ratings randomly, and their potential awareness of the hypothesis tested in the study), were debriefed on the study's purpose, and directed back to Prolific.com.



 §.§ Statistical Analyses




For Experiment 1, we ran a linear mixed effects model with the factor emotion (angry vs. happy audio) coded as a sliding difference contrast (meaning that the estimated effect reflects the predicted mean difference between faces seen with an angry audio vs. a happy audio). 
For Experiment 2, we ran a linear mixed effects model with the factor emotion (generated face shifted towards an angry vs. a happy expression) and the factor weight (with five levels, 0, 0.25, 0.5, 0.75, and 1). To compare the differences in expression ratings between the happy and the angry condition within each of the weights, the factor emotion was nested within the factor weight. 
For both experiments, we modelled random intercepts for participants and items (i.e. depicted actors), as well as random slopes for the effect of emotion over participants and items <cit.>. The significance of fixed effects coefficients (p < 0.05) was tested by Satterthwaite approximation.



 §.§ Results


Separate models were run for each weight level, including the factors emotional context, experiment (2 vs. 1), and their interaction. With a weight of 1, no significant interaction between experiment and emotional context was found (b = -0.22, p = .084), whereas this interaction was significant for all other weights (bs < -0.42, ps < .001). <ref> shows the results including interaction significance.



§ RAVDESS COMPARISON TO SOTA GENERATIONS


In the main paper we omitted a comparison to SOTA generation methods <cit.> because they pursue a different purpose—making a neutral input images resemble strongly the accompanying audio context. Due to our objective of synthesizing mental representations, such heavy shifts in the facial expression would be too strong for a human observer. Changes in subjective appearance are rather subtle <cit.>. For the sake of completeness, we provide a comparison to SOTA generations in <ref>. The two SOTA competitors use a target ground-truth sequence they want to model, whereas we compute our adaption only based on the expression class and the model implicilty learns a sensible shift.





§ ADDITIONAL SYNTHESIZED FACIAL EXPRESSIONS


We provide additional generations of facial expressions on the ravdess dataset using our proposed model. <ref> shows that our model picks up the subtle differences in the mouth opening in the generations. <ref> show generations for a neutral input image (left column) paired with different contexts. The context weight context_weight is increased from 0 (second column) to 1 (rightmost column) in 0.1 steps. <ref> shows additional generations for variations of neutral input images with different contexts, to showcase that our model adapts the style of the input image. The leftmost column is the neutral input image, from left to right follows the reconstruction without context, calm audio context, happy audio context, sad audio context, fearful audio context, disgusted audio context, and surprised audio context. <ref> shows generations of a neutral input image (top row) paired with normal (middle row) and strong context intensity (bottom row). The ravdess dataset provides these two intensities as a binary label for every sample. The context expression classes are from left to right: happy,
sad, angry, fearful, disgusted, surprised.



1.0








0.75



0.55





§ MEAD GENERATIONS


Reconstructions on the mead dataset are shown in <ref>.



