Reversible Graph Neural Network-based Reaction Distribution Learning for
  Multiple Appropriate Facial Reactions Generation
Generating facial reactions in a human-human dyadic interaction is complex
and highly dependent on the context since more than one facial reactions can be
appropriate for the speaker's behaviour. This has challenged existing machine
learning (ML) methods, whose training strategies enforce models to reproduce a
specific (not multiple) facial reaction from each input speaker behaviour. This
paper proposes the first multiple appropriate facial reaction generation
framework that re-formulates the one-to-many mapping facial reaction generation
problem as a one-to-one mapping problem. This means that we approach this
problem by considering the generation of a distribution of the listener's
appropriate facial reactions instead of multiple different appropriate facial
reactions, i.e., 'many' appropriate facial reaction labels are summarised as
'one' distribution label during training. Our model consists of a perceptual
processor, a cognitive processor, and a motor processor. The motor processor is
implemented with a novel Reversible Multi-dimensional Edge Graph Neural Network
(REGNN). This allows us to obtain a distribution of appropriate real facial
reactions during the training process, enabling the cognitive processor to be
trained to predict the appropriate facial reaction distribution. At the
inference stage, the REGNN decodes an appropriate facial reaction by using this
distribution as input. Experimental results demonstrate that our approach
outperforms existing models in generating more appropriate, realistic, and
synchronized facial reactions. The improved performance is largely attributed
to the proposed appropriate facial reaction distribution learning strategy and
the use of a REGNN. The code is available at
https://github.com/TongXu-05/REGNN-Multiple-Appropriate-Facial-Reaction-Generation.
Tong Xu, Micol Spitale, Hao Tang, Lu Liu, Hatice Gunes, Siyang Song
Reversible Graph Neural Network-based Reaction Distribution Learning for Multiple Appropriate Facial Reactions Generation

###### Abstract

Generating facial reactions in a human-human dyadic interaction is complex and highly dependent on the context since more than one facial reactions can be appropriate for the speaker's behaviour. This has challenged existing machine learning (ML) methods, whose training strategies enforce models to reproduce a specific (not multiple) facial reaction from each input speaker behaviour. This paper proposes the first multiple appropriate facial reaction generation framework that re-formulates the _one-to-many mapping_ facial reaction generation problem as a _one-to-one mapping_ problem. This means that we approach this problem by considering the generation of a distribution of the listener's appropriate facial reactions instead of multiple different appropriate facial reactions, i.e.,'many' appropriate facial reaction labels are summarised as 'one' distribution label during training. Our model consists of a perceptual processor, a cognitive processor, and a motor processor. The motor processor is implemented with a novel Reversible Multi-dimensional Edge Graph Neural Network (REGNN). This allows us to obtain a distribution of appropriate real facial reactions during the training process, enabling the cognitive processor to be trained to predict the appropriate facial reaction distribution. At the inference stage, the REGNN decodes an appropriate facial reaction by using this distribution as input. Experimental results demonstrate that our approach outperforms existing models in generating more appropriate, realistic, and synchronized facial reactions. The improved performance is largely attributed to the proposed appropriate facial reaction distribution learning strategy and the use of a REGNN. The code is available at [https://github.com/TongXu-05/REGNN-Multiple-Appropriate-Facial-Reaction-Generation](https://github.com/TongXu-05/REGNN-Multiple-Appropriate-Facial-Reaction-Generation).

 Multiple appropriate facial reaction generation, Reversible Graph Neural Network, Facial reaction distribution learning

## I Introduction

Non-verbal behaviour interaction plays a key role in human-human communication [1], with facial reactions providing important cues for understanding each other's emotional states. In dyadic interactions, a facial reaction refers to the **listener**'s non-verbal facial behaviours in response to the **speaker**'s verbal and non-verbal behaviours (e.g., facial muscle movements) [2, 3]. Previous studies [4, 5] have shown that the generation of listener's facial reactions to a speaker's behaviour in dyadic interaction consists of three main stages: Firstly, the listener's perceptual system (e.g., ears and eyes) receives external signals expressed by the speaker, which are pre-processed before being transmitted to the brain for further analysis. Then, the cognitive processor processes the pre-processed signals by taking personalized perception bias into account, resulting in the generation of personalized reaction signals. Finally, the motor processor decodes these personalized signals to the facial muscles, producing corresponding facial reactions.

In contrast to most machine learning tasks, the generation of listener's facial reactions to a specific speaker behaviour are characterized by variability and uncertainty [6, 7], meaning that different facial reactions can be expressed by listeners in response to the same speaker behaviour. Existing machine learning (ML)-based Facial Reaction Generation (FRG) models aim to reproduce the real facial reaction expressed under a specific context (called "GT reaction" in this paper) in response to each given speaker behaviour. These models - including Generative Adversarial Networks (GAN) [8, 9], VQ-VAE [7], and person-specific FRG networks [10, 11] - are trained by minimizing L1 or L2 loss between the generated and GT facial reactions. However, this training strategy creates an ill-posed problem for existing FRG models where similar inputs (speaker behaviours) are paired with different labels (listener facial reactions), resulting in a "one-to-many mapping" problem in the training phase. This limitation makes

Fig. 1: Our approach predicts an distribution representing multiple different but appropriate facial reactions from each input speaker behaviour, based on which multiple different but appropriate, realistic, and synchronized human listener facial reactions could be generated.
it theoretically very challenging for existing approaches to learn good FRG models that can generate diverse, appropriate, and photo-realistic facial reactions in response to speaker behaviours.

In this paper, we propose the first deep learning framework to generate multiple appropriate facial reactions in response to each speaker behaviour. Rather than simply reproducing the GT facial reaction expressed by the corresponding listener, our approach generates multiple different but appropriate, realistic, and synchronized facial reactions from the given context. Inspired by the theoretical framework of the human model processor [12], our approach is designed to consist of three modules: (i) a **perceptual processor** that encodes the input speaker audio and facial signals; (ii) a **cognitive processor** that predicts an appropriate facial reaction distribution from the encoded speaker audio-facial representation, which represents multiple different but appropriate facial reactions; and (iii) a reversible Graph Neural Network (GNN)-based **motor processor** that decodes an appropriate facial reaction from the learned distribution.

To address the "one-to-many mapping" problem during the training phase, we propose a novel Reversible Multi-dimensional Edge Graph Neural Network (REGNN) as the motor processor. The REGNN is designed to summarize a distribution that represents all appropriate real facial reactions displayed by listeners in training set in response to each input speaker behaviour. The summarised distribution is then used to supervise the training of the cognitive processor by enforcing it to learn an appropriate facial reaction distribution from the input speaker behaviour. This distribution learning strategy transforms the ill-posed training problem, where one input speaker behaviour corresponds to multiple appropriate listener facial reactions, into a well-posed problem, where one input speaker behaviour corresponds to one distribution representing all appropriate facial reactions. We illustrate our approach in Fig. 1 and Fig. 2. The main contributions and novelties of this paper are summarised as follows:

* To the best of our knowledge, we present the first deep learning framework capable of generating multiple appropriate, realistic, and synchronized facial reactions in response to a speaker behaviour. Our framework introduces a novel appropriate facial reaction distribution learning (AFRDL) strategy that addresses the ill-posed "one-to-many mapping " problem.This strategy reformulates the problem as a "one-to-one mapping" problem, thus providing a well-defined learning objective.
* We propose a novel Reversible Multi-dimensional Edge Graph Neural Network (REGNN) that can forwardly summarise a distribution from multiple real appropriate facial reactions at the training stage, and reversely decode a facial reaction from the predicted facial reaction distribution at the inference stage.
* using the proposed approach
- appropriate, realistic, and synchronized facial reactions achieving better performances compared to other existing related solutions, and we provide the first open-source code for the multiple appropriate facial reaction generation task.

## II Related Work

### _Facial reaction theory_

During dyadic interactions, the facial reactions of a listener are shaped by a combination of facial muscle movements. These movements are controlled by person-specific cognitive processes that are primarily influenced by the behaviours expressed by the corresponding speaker [13]. Research conducted by Hess et al. [14] also found that the generation of facial reactions is predominantly influenced by individual-specific cognitive processes, which are not only influenced by the speaker's behaviour but also by the listener's personality [15] and emotional states [16]. For instance, individuals who frequently experience fear possess more sensitive and easily stimulated amygdala, rendering them more prone to displaying facial reactions indicative of fear. Similarly, experiencing pleasant emotions triggers the contraction of the zygomatic major muscle, resulting in a smiling facial reaction, while confusion enhances the activity of the corrugator muscle, leading to a furrowed brow expression. Therefore, as summarised in [6], in dyadic interactions, a broad spectrum of different facial reactions might be _appropriate_ in response to a speaker behaviour according to the internal states of the listener. This is because human behavioural responses are stimulated by the context the listener experiences [3], which lead to different but appropriate facial reactions expressed by not only different listeners but also the same listener under different contexts (e.g., external environments or internal states) [6, 17, 18]. A similar hypothesis has been mentioned in a recent facial reaction generation study [7].

### _Automatic facial reaction generation_

To the best of our knowledge, there have been few studies [19, 20, 21, 10, 7, 8, 9, 11] on automatic facial reaction generation. An early approach [9] proposed a two-stage conditional GAN to generate facial reaction sketches based on the speaker's facial action units (AUs). Their later works [8, 19] exploited more speaker emotion-related features (e.g., facial expression features) to reproduce better facial reactions expressed by listeners. To consider personalized factors in expressing facial reactions, Song et al. [10, 11] used Neural Architecture Search (NAS) to explore a person-specific network for each listener adaptively. Ng et al. [22] extended and combined the cross-attention transformer with VQ-variational auto-encoder (VQ-VAE) [23] model to generate a wide range of diverse facial reactions for each listener by leveraging multi-modal speaker behaviours. Besides generating facial reactions, some studies aiming to generate non-verbal head [21] or gesture reactions [24]. However, none of these have attempted to generate multiple appropriate facial reactions from a speaker behaviour [6].

Note that the approach proposed in this paper is different from previous facial expression/display generation methods [25, 26, 27, 28, 29, 30, 31, 32], where the facial images are generated based on manually defined conditions such pre-defined AUs, landmarks and audio behaviours without considering interaction scenarios (i.e., they do not predict reactions from speaker behaviours).
## III Generation Task definition

Given a speaker behaviour \(B_{S}^{t_{1},t_{2}}=\{A_{S}^{t_{1},t_{2}},F_{S}^{t_{1},t_{2}}\}\) at the time \([t_{1},t_{2}]\), the goal is to learn a ML model \(\mathcal{H}\) that can generate multiple different spatio-temporal human facial reactions \(P(F_{L}|B_{S}^{t_{1},t_{2}})=\{p(F_{L}|B_{S}^{t_{1},t_{2}}),\cdots,p(F_{L}|B_{ S}^{t_{1},t_{2}})_{N}\}\) that are **appropriate** for responding to \(B_{S}^{t_{1},t_{2}}\), which is formulated as:

\[P(F_{L}|B_{S}^{t_{1},t_{2}})=\mathcal{H}(B_{S}^{t_{1},t_{2}}), \tag{1}\]

where \(P(F_{L}|B_{S}^{t_{1},t_{2}})=\{p(F_{L}|B_{S}^{t_{1},t_{2}})_{1}\neq\cdots\neq p (F_{L}|B_{S}^{t_{1},t_{2}})_{N}\}\) and each generated facial reaction \(p(F_{L}|B_{S}^{t_{1},t_{2}})_{n}\) should be similar to at least one real facial reaction \(f_{L}(B_{S}^{t_{1},t_{2}})_{m}\) that is appropriate in response to \(B_{S}^{t_{1},t_{2}}\) in the training set:

\[p(F_{L}|B_{S}^{t_{1},t_{2}})_{n}\approx f_{L}(B_{S}^{t_{1},t_{2}})_{m}\in F_{L }(B_{S}^{t_{1},t_{2}}), \tag{2}\]

where \(F_{L}(B_{S}^{t_{1},t_{2}})=\{f_{L}(B_{S}^{t_{1},t_{2}})_{1},\cdots,f_{L}(B_{S} ^{t_{1},t_{2}})_{M}\}\) denotes a set of real facial reactions expressed by listeners in the training set, which are appropriate in response to the speaker behaviour \(B_{S}^{t_{1},t_{2}}\). The above definition corresponds to the offline facial reaction generation task defined by [6] (See [6] for details).

## IV The proposed approach

This section presents our novel FRG approach. We first introduce the pipeline in Sec. IV-A, and then detail the proposed appropriate facial reaction distribution learning (AFRDL) strategy in Sec. IV-B. Finally, we provide the details of the novel REGNN in Sec. IV-C, which represents one of the main components of the AFRDL strategy.

### _Facial reaction generation framework_

This section develops an appropriate FRG model, \(\mathcal{H}=\{\textbf{Enc},\textbf{Cog},\textbf{Mot}\}\), which can generate multiple diverse and appropriate human facial reactions in response to each speaker audio-facial behaviour. As shown in Fig. 2, our model consists of three main modules inspired by the Human Model Processor (HMP) [12]: (i) **Perceptual Processor** (\(\textbf{Enc}=\{\textbf{Enc}_{\text{A}},\textbf{Enc}_{\text{F}}\}\)) that encodes each raw speaker audio and facial behaviours into a pair of latent audio and facial representations; (ii) **Cognitive Processor (Cog)** that predicts a distribution representing all appropriate listener facial reactions based on the produced speaker audio and facial representations; and (iii) the REGNN-based **Motor Processor (Mot)** that samples and generates an appropriate facial reaction from the predicted distribution. We also illustrate the model pipeline in Fig. 2.

Specifically, the **Perceptual Processor** is a two branch encoder consisting of a facial encoder \(\textbf{Enc}_{\text{F}}\) (Swin-Transformer [33]) and an audio encoder \(\textbf{Enc}_{\text{A}}\) (VGGish [34]). It takes a speaker's audio and facial signals \(A_{S}^{t_{1},t_{2}}\) and \(F_{S}^{t_{1},t_{2}}\) expressed at the time interval \([t_{1},t_{2}]\) as the input, and generates a pair

Fig. 2: Overview of the proposed multiple appropriate facial reaction generation framework. **Step 1:** the **Perceptual Processor** first encodes facial and audio representations from the perceived audio-visual speaker behaviours. **Step 2:** the **Cognitive Processor** then predicts a distribution from the combined audio-visual representation, which represents all appropriate facial reactions in response to the input speaker behaviour. **Step 3:** the REGNN-based **Motor Processor** finally samples and reversely decodes multiple appropriate facial reactions from the learned distribution.
of latent audio and facial representations \(\bar{A}_{S}^{t_{1},t_{2}}\) and \(\bar{F}_{S}^{t_{1},t_{2}}\) as:

\[\bar{A}_{S}^{t_{1},t_{2}} =\textbf{Enc}_{\textbf{A}}(A_{S}^{t_{1},t_{2}}), \tag{3}\] \[\bar{F}_{S}^{t_{1},t_{2}} =\textbf{Enc}_{\textbf{F}}(F_{S}^{t_{1},t_{2}}).\]

Based on \(\bar{A}_{S}^{t_{1},t_{2}}\) and \(\bar{F}_{S}^{t_{1},t_{2}}\), the **Cognitive Processor** first aligns and combines them as a latent speaker audio-facial behaviour representation \(\bar{B}_{S}^{t_{1},t_{2}}\) based on the same attention-based strategy introduced in [35]. Rather than predicting a specific facial reaction from \(\bar{B}_{S}^{t_{1},t_{2}}\), we propose to predict an _appropriate facial reaction distribution graph representation_\(\bar{Z}_{p}(B_{S}^{t_{1},t_{2}})\) from \(\bar{B}_{S}^{t_{1},t_{2}}\), as multiple facial reactions may be appropriate for responding to each input speaker behaviour. This means that \(Z_{p}(B_{S}^{t_{1},t_{2}})\) represents the distribution of all appropriate facial reactions. This is achieved by \(I\) projection heads (i.e., \(I\) fully connected (FC) layers), where each head learns a \(D\)-dimensional vector that is specifically treated as a node feature for \(\bar{Z}_{p}(B_{S}^{t_{1},t_{2}})\). These processes can be formulated as:

\[\bar{Z}_{p}(B_{S}^{t_{1},t_{2}})=\textbf{COG}(\bar{B}_{S}^{t_{1},t_{2}}), \tag{4}\]

where \(\bar{Z}_{p}(B_{S}^{t_{1},t_{2}})\in\mathbb{R}^{I\times P}\) (\(I\) nodes). This way, _the "one-to-many mapping" problem occurring in the FRG task is addressed by re-formulating it into a 'one-to-one mapping' problem at the training stage (one speaker behaviour-to-one distribution representing multiple appropriate facial reactions)_. Then, we feed all node features to our multi-dimensional edge feature learning (MEFL) block that consists of \(D\) attention operations, where each attention operation generates an \(I\times I\) attention map describing a specific type of mutual relationship between a pair of nodes. Consequently, \(D\) attention maps describing \(D\) types of relationship cues would be produced. Thus, a pair of multi-dimensional directed edge features can be obtained to describe the relationship between each pair of nodes, i.e., each multi-dimensional edge feature \(e_{i,j}^{\bar{Z}_{p}}\) is a \(D\) dimensional vector produced by concatenating the values at the \(i_{\text{th}}\) row and \(j_{\text{th}}\) column of \(D\) attention maps. Here, we only keep \(\mathcal{K}\) directed edges starting from each node, which has largest norms.

Since human facial behaviours can be interpreted by various medium and high level primitives (e.g., facial Action Units (AUs) and affects) which are usually mutually correlated [36], we propose to describe each spatio-temporal facial reaction by \(I\) facial attribute time-series primitives, which are represented as a graph consisting of \(I\) nodes, where the presence of each directed edge and its multi-dimensional edge feature is also defined by a MEFL block. Consequently, the relationship between each pair of facial attribute time-series (nodes) can be explicitly described by a pair of multi-dimensional edge features. To allow such facial reaction graph representation to be generated from the distribution \(\bar{Z}_{p}(B_{S}^{t_{1},t_{2}})\), we propose a novel Reversible Multi-dimensional Edge Graph Neural Network (REGNN) as the **Motor Processor**. It first samples an appropriate facial reaction latent graph representation \(\hat{g}_{p}(B_{S}^{t_{1},t_{2}})_{n}\in\mathbb{R}^{I\times D}\) from the predicted distribution \(\bar{Z}_{p}(B_{S}^{t_{1},t_{2}})\), and then decides it as a facial reaction graph representation \(\hat{g}_{p}(B_{S}^{t_{1},t_{2}})_{n}\in\mathbb{R}^{I\times D}\). Here, different facial reaction latent graph representations can be sampled from \(\bar{Z}_{p}(B_{S}^{t_{1},t_{2}})\), and thus multiple different facial reactions \(\hat{G}_{p}(B_{S}^{t_{1},t_{2}})=\{\hat{g}_{p}(B_{S}^{t_{1},t_{2}})_{1},\cdots,\hat{g}_{p}(B_{S}^{t_{1},t_{2}})_{N}\}\) can be generated. This can be formulated as:

\[\hat{g}_{p}(B_{S}^{t_{1},t_{2}})_{n}=\textbf{Mot}^{-1}(Z_{p}(B_{S}^{t_{1},t_{2 }})), \tag{5}\]

where \(\text{Mot}^{-1}\) denotes that motor processes that reversely infers facial reactions from the predicted distribution \(Z_{p}(B_{S}^{t_{1},t_{2}})\).

Subsequently, a 2D facial reaction sequence \(p(F_{L}|B_{S}^{t_{1},t_{2}})_{n}\) can be produced from \(\hat{g}_{p}(B_{S}^{t_{1},t_{2}})_{n}\). To train the **Cognitive Processor** that can accurately predict the distribution of all appropriate facial reactions in response to each input speaker behaviour, we propose a novel Reversible Multi-dimensional Edge Graph Neural Network (REGNN) as the motor processor, which learns an appropriate real facial reaction distribution graph representation \(Z_{L}(B_{S}^{t_{1},t_{2}})\) for each speaker behaviour \(B_{S}^{t_{1},t_{2}}\), representing all listeners' real facial reactions that are appropriate as the corresponding facial reaction to \(B_{S}^{t_{1},t_{2}}\).

### _Appropriate facial reaction distribution learning_

Our facial reaction distribution generation strategy aims to address the "one-to-many mapping" problem occurring in FRG models' training (i.e., one input speaker behaviour corresponds to multiple appropriate facial reaction labels) by re-formulating it as a "one-to-one mapping" problem (i.e., one input speaker behaviour corresponds to one distribution representing multiple appropriate facial reactions).

Fig. 3: Illustration of the proposed AFRDL strategy (Pseudo-code is provided in the supplementary material). Given a speaker behaviour, REGNN first encodes all appropriate real facial reactions as a set of latent graph representations. These representations are then summarised as an appropriate real facial reaction distribution to supervise the Cognitive Processor’s training, where the summarised distribution is a graph representation consisting of multiple nodes, and each node is represented by a Gaussian Mixture Model (GMM) summarising multiple facial attribute time-series corresponding to multiple appropriate real facial reactions. Here, the MSE loss function is employed to enforce the distribution predicted by the Cognitive Processor to be similar to the summarised real facial reaction distribution.
As shown in Fig. 3, given an audio-visual speaker behaviour \(B_{S}^{t_{1},t_{2}}=\{A_{S}^{t_{1},t_{2}},F_{S}^{t_{1},t_{2}}\}\), and its corresponding multiple appropriate real facial reactions \(F_{L}(B_{S}^{t_{1},t_{2}})\) expressed by human listeners of the training set, we first construct a set of real facial reaction graph representations \(G_{L}(B_{S}^{t_{1},t_{2}})=\{g_{L}(B_{S}^{t_{1},t_{2}})_{1},\cdots,g_{L}(B_{S}^{ t_{1},t_{2}})_{M}\}\) to represent all appropriate real facial reactions defined by \(F_{L}(B_{S}^{t_{1},t_{2}})\), where each node in a graph representation describes a facial attribute time-series and each edge explicitly describes the relationship between a pair of nodes. At the same time, the presence of each directed edge and its multi-dimensional edge feature is also defined by a MEFL block. Since these representations have the same property, i.e., each describes an appropriate facial reaction in response to \(B_{S}^{t_{1},t_{2}}\), we hypothesize that they are drawn from the same distribution. Subsequently, we train the REGNN by enforcing it to map all appropriate real facial reaction graph representations in response to the same speaker behaviour onto a 'ground-truth' (GT) real appropriate facial reaction distribution \(Z_{L}(B_{S}^{t_{1},t_{2}})\) as:

\[\begin{split}\bar{g}_{L}(B_{S}^{t_{1},t_{2}})_{m}& =\text{Mot}(g_{L}(B_{S}^{t_{1},t_{2}})_{m}),\\ \bar{g}_{L}(B_{S}^{t_{1},t_{2}})_{m}&\sim Z_{L}(B_{ S}^{t_{1},t_{2}}),\quad m=1,2,\cdots M\\ \textbf{subject to}\;\;f_{L}(B_{S}^{t_{1},t_{2}})_{m}& \in F_{L}(B_{S}^{t_{1},t_{2}}),\end{split} \tag{6}\]

where \(\bar{g}_{L}(B_{S}^{t_{1},t_{2}})_{m}\) denotes a latent graph representation produced from \(g_{L}(B_{S}^{t_{1},t_{2}})_{m}\), and all latent graph representations are expected to follow the same distribution \(Z_{L}(B_{S}^{t_{1},t_{2}})\). This is achieved by minimizing the sum of L1 distances obtained from all the corresponding latent graph representation pairs in an unsupervised manner:

\[\mathcal{L}_{1}=\sum_{m_{1}=1}^{M-1}\sum_{m_{2}=m_{1}+1}^{M}L1(\hat{g}_{L}(B_{ S}^{t_{1},t_{2}})_{m_{2}},\hat{g}_{L}(B_{S}^{t_{1},t_{2}})_{m_{1}}). \tag{7}\]

Inspired by the fact that Guassian Mixture Model (GMM) is powerful to describe distributed subpopulations (e.g., individual appropriate facial reactions) within an overall population (e.g., all appropriate facial reactions), we propose a novel **Gaussian Mixture Graph Distribution (GMGD)** to represent \(Z_{L}(B_{S}^{t_{1},t_{2}})=\{v_{1}^{2},v_{2}^{2},\cdots,v_{I}^{2}\}\), where each node \(v_{i}^{2}\) in \(Z_{L}(B_{S}^{t_{1},t_{2}})\) is represented by a Gaussian Mixture Model (GMM) consisting of \(M\) Gaussian distributions (defined as \(\mathcal{N}(\{\mu_{i}^{1},\cdots,\mu_{i}^{M}\},\{\sigma_{i}^{1},\cdots,\sigma_ {i}^{M}\})\)). For each node, the mean values \(\mu_{i}^{1},\cdots,\mu_{i}^{M}\) of \(M\) Gaussian distributions are defined by the \(i_{\text{th}}\) node features \(\bar{v}(L)_{i}^{1},\cdots,\bar{v}(L)_{i}^{M}\) of the corresponding \(M\) latent graph representations \(\bar{g}_{L}(B_{S}^{t_{1},t_{2}})_{1},\cdots,\bar{g}_{L}(B_{S}^{t_{1},t_{2}})_ {M}\) produced by the motor processor, while standard deviations \(\{\sigma(L)_{i}^{1},\cdots,\sigma(L)_{i}^{M}\}\) are empirically defined. These can be formulated as:

\[\mu_{i}^{m}=v_{i}^{m}\in\bar{g}_{L}(B_{S}^{t_{1},t_{2}})_{m},\;\;m=1,2,\cdots,M \tag{8}\]

where \(v_{i}^{m}\) is the \(i_{\text{th}}\) node in the \(m_{\text{th}}\) latent graph representation \(\bar{g}_{L}(B_{S}^{t_{1},t_{2}})_{m}\).

As a result, the **Cognitive Processor** is trained under the supervision of the GT real facial reaction distribution \(Z_{L}(B_{S}^{t_{1},t_{2}})\), i.e., it is trained to predict an appropriate facial reaction distribution graph representation \(\bar{Z}_{p}(B_{S}^{t_{1},t_{2}})\) from \(B_{S}^{t_{1},t_{2}}\) (formulated in Eq. (4)). This training process is achieved by minimizing the L2 distance between \(Z_{L}(B_{S}^{t_{1},t_{2}})\) and \(\bar{Z}_{L}(B_{S}^{t_{1},t_{2}})\) as:

\[\mathcal{L}_{2}=\text{MSE}(\bar{Z}_{p}(B_{S}^{t_{1},t_{2}}),Z_{L}(B_{S}^{t_{1},t _{2}})) \tag{9}\]

where MSE denotes the Mean Square Error. At the inference stage, the well-trained motor processor first samples a facial reaction graph representation \(\bar{g}_{p}(B_{S}^{t_{1},t_{2}})_{n}\) from the distribution \(\bar{Z}_{p}(B_{S}^{t_{1},t_{2}})\) predicted by the Cognitive Processor, and then reversely decodes it as facial reaction.

### _Reversible Multi-dimensional Edge Graph Neural Network_

The REGNN can forwardly encode a GT facial reaction distribution to describe all appropriate real facial reactions in response to the input speaker behaviour at the training stage, which plays a key role in supervising the cognitive processor's training. As shown in Fig. 4, the REGNN network consists of \(N\) REGNN layers, which forwardly takes a graph \(\mathcal{G}^{0}(\mathcal{V}^{0},\mathcal{E}^{0})\) as the input and generates a graph \(\mathcal{G}^{N}(\mathcal{V}^{N},\mathcal{E}^{N})\) as the output, where \(\mathcal{V}^{0}=\{v_{1}^{0},v_{2}^{0},\cdots,v_{I}^{0}\}\) and \(\mathcal{E}=\{e_{i,j}^{0}|v_{i}^{0},v_{j}^{0}\in\mathcal{V}\;\;\&\;\; \mathcal{A}_{i,j}=1\}\) denote a set of node and edge features contained in the input graph \(\mathcal{G}^{0}(\mathcal{V}^{0},\mathcal{E}^{0})\), respectively, and \(\mathcal{A}\) is the adjacency matrix that defines the connectivity between nodes. Importantly, the REGNN can also reversely output \(\mathcal{G}^{0}(\mathcal{V}^{0},\mathcal{E}^{0})\) from \(\mathcal{G}^{N}(\mathcal{V}^{N},\mathcal{E}^{N})\). In this paper, since only node features represent the target facial attributes/distributions, the REGNN is designed for node feature reasoning. In the following, we present the details of both the forward propagation and reverse propagation mechanisms of the REGNN.

**Forward propagation:** The \(n_{\text{th}}\) REGNN layer takes: (i) the node feature set \(\mathcal{V}^{n-1}\) generated from the \(n-1_{th}\) layer as its input node features, which is pre-processed by a normalization layer and a Sigmoid activation;; and (ii) the initial edge feature set \(\mathcal{E}^{0}\) as its input edge features, and then outputs a graph \(\mathcal{G}^{n}(\mathcal{V}^{n},\mathcal{E}^{n})\). This setting ensures the reversibility of the REGNN (explained and derived in the Supplementary Material). Specifically, the \(n_{\text{th}}\) REGNN layer first learns a set of edge features \(\mathcal{E}^{n}\) based on not only \(\mathcal{E}^{0}\) but also \(\mathcal{V}^{n-1}\)

Fig. 4: The proposed Reversible Multi-dimensional Edge Graph Neural Network (REGNN) is made up of \(N\) REGNN layers, where each layer can forwardly and reversely propagate node representations. Pseudo-code of its propagation is provided in the supplementary material.
allowing \(\mathcal{V}^{n-1}\)-related edge features to be used for updating \(\mathcal{V}^{n-1}\) to \(\mathcal{V}^{n}\), i.e., it computes each directed edge feature \(e_{j,i}^{n}\in\mathcal{E}^{n}\) starting from the node \(v_{j}^{n-1}\) to \(v_{i}^{n-1}\) as:

\[e_{j,i}^{n}=\frac{a_{j,i}^{n}e_{j,i}^{0}}{\sum_{v_{k}^{n-1}\in\mathcal{N}_{v_{i }^{n-1}}}a_{k,i}^{n}e_{k,i}^{0}}, \tag{10}\]

where \(e_{j,i}^{0}\in\mathcal{E}_{0}\) denotes an initial directed edge feature. The employed \(a_{j,i}^{n}\) is a learnable relationship coefficient that captures correlations between the node \(v_{i}^{n-1}\) and its high-order neighboring nodes to define \(\mathcal{V}^{n-1}\)-related and context-aware (i.e., aware of the high-order neighbors) edge feature \(e_{j,i}^{n}\), which contributes to updating \(v_{j}^{n-1}\) to \(v_{j}^{n}\). The term \(\sum_{v_{k}^{n-1}\in\mathcal{N}_{v_{i}^{n-1}}}a_{k,i}^{n}e_{k,i}^{0}\) regularizes the obtained edge feature. Here, the \(a_{j,i}^{n}\) can be computed as:

\[a_{j,i}^{n}=\frac{\exp(v_{i}^{n-1}\mathbf{W}_{q}^{n})(v_{j}^{n-1}\mathbf{W}_{ m})^{\top}}{\sum_{v_{k}^{n-1}\in\mathcal{N}_{v_{i}^{n-1}}}\exp\left((v_{i}^{n-1} \mathbf{W}_{q}^{n})(v_{k}^{n-1}\mathbf{W}_{m}^{n})^{\top}\right)}, \tag{11}\]

where \(\mathbf{W}_{q}\) and \(\mathbf{W}_{m}\) are learnable weight vectors and \(\mathcal{N}_{i}\) denotes the adjacent node set of \(v_{i}^{n-1}\). This way, the learned \(a_{j,i}^{n}\) captures information from not only the relationship cues between the corresponding node features \(v_{j}^{n-1}\) and \(v_{i}^{n-1}\) but also the context of the node \(v_{i}^{n-1}\) (i.e., its high-order neighboring nodes \(\mathcal{N}_{v_{i}^{n-1}}\)).

Building upon these learned edge features, the REGNN layer then update each node feature \(v_{i}^{n}\) based on: (i) node feature \(v_{i}^{n-1}\in\mathcal{G}^{n-1}\) itself; and (ii) the message \(\hat{v}_{i}^{n-1}\) aggregated from all its adjacent node, which is decided by its adjacent node feature set (\(\mathcal{N}_{v_{i}^{n-1}}\)) and the corresponding updated directed edge features \(e_{j,i}^{n}\in\mathcal{E}^{n}\) that point to \(v_{i}^{n-1}\) as:

\[v_{i}^{n}=v_{i}^{n-1}+\hat{v}_{i}^{n-1}, \tag{12}\]

where the message \(\hat{v}_{i}^{n-1}\) is computed as:

\[\hat{v}_{i}^{n-1}=\mathbf{W}_{c}^{n}\sum_{v_{j}^{n-1}\in\mathcal{N}_{v_{i}^{n- 1}}}\left(e_{j,i}^{n}\circ v_{j}^{n-1}\right), \tag{13}\]

where \(\mathbf{W}_{c}^{n}\in\mathbb{R}^{1\times D}\) is a learnable weight vector that combines messages passed by all dimensions (\(D\) dimensions) of the multi-dimensional edge \(e_{j,i}^{n}\).

**Reverse propagation:** The proposed REGNN layer is able to reversely infer each input node feature \(v_{i}^{n-1}\) from its output node features \(v_{i}^{n}\), which can be formulated as:

\[\begin{split}& x_{k}=v_{i}^{n}-\varphi(x_{k-1}),\\ & v_{i}^{n-1}=x_{k},\qquad\textbf{subject to}\qquad x_{k}=x_{k-1},\end{split} \tag{14}\]

where the function \(\varphi\) is defined as the combination of the Sigmoid activation, edge updating (Eqa. 10) and message passing functions (Eqa. 13); \(x_{k}\) is computed iteratively until it is converged to \(x_{k}=x_{k-1}\). Here, the \(x_{0}\) is set as a non-zero random value. To achieve the aforementioned reversibility (i.e., converged at the \(x_{k}=x_{k-1}\)), which is the key for decoding an appropriate facial reaction from the predicted distribution, the \(\mathbf{W}_{c}^{n}\) defined in Eq. (13), as well as \(W_{q}^{n}\) and \(W_{k}^{n}\) are optimized to enable the function \(varphi\) used for both forward and reverse propagation to satisfy the following condition:

\[\varphi(v_{i}^{n-1})=\frac{\varphi(\text{Sig}(v_{i}^{n-1}))}{1+2\|\ \mathbf{W}_{q}^{n}\mathbf{W}_{k}^{n \top}\|_{2}}, \tag{15}\]

where Sig denotes the Sigmoid activation function. This setting allows the function \(\varphi\) to be a contraction mapping function that satisfies the Lipschitz continuous and Lipschitz constant less than \(1\)[37, 38], and thus adhering to the fixed point theorem, i.e., the function \(\varphi\) is learned to ensure a fixed point \(x\) to satisfy the equation \(x=v_{i}^{n}-\varphi(x)\). The proof and derivation for defining Eq. (15) for ensuring the reversibility of the proposed REGNN are provided in the supplementary material.

## V Experiments

This section first provides the details of experimental settings in Sec. V-A, then, Sec. V-B compares our approach with previous facial reaction generation solutions. Finally, we conduct a series of ablation studies to investigate the contributions of different modules.

### _Experimental settings_

**Dataset.** This paper evaluates the proposed approach based on the video conference clips recorded under various dyadic interaction settings, which are provided by two publicly available datasets: NoXI [39] and RECOLA [40]. As there are only three valid video pairs in RECOLA dataset, we follow a public facial reaction challenge 1 to combine these two datasets, resulting in 2962 pairs of audio-visual speaker-listener dyadic interaction clips, including 1594 pairs of training clips, 562 pairs of validation clips, and 806 pairs of test clips, where each clip is 30 seconds long. All appropriateness labels are obtained by the same strategy provided in [6]. Note that the employed dataset is different from the challenge, as the UDIVA dataset [41] is not included because it is recorded under in-person dyadic interactions (with a lateral camera view which captures both participants), where the profile of participants' faces are frequently recorded.

Footnote 1: [https://sites.google.com/cam.ac.uk/react2023/home](https://sites.google.com/cam.ac.uk/react2023/home)

**Implementation details.** In our experiments, the Perceptual Processor takes the facial image sequence of an entire video clip and log-mel spectrogram features of an entire audio clip as input. At the training stage, we employ the Swin-Transformer [42] pre-trained on FER2013 [43] and pre-trained Vggish model [34] as the initial facial and audio feature extraction models. Then, the Adam optimizer [44] with a stochastic optimization strategy was employed to train the entire framework in an end-to-end manner using an initial learning rate of \(10^{-4}\) and weight decay of \(5\times 10^{-4}\). The maximum number of epochs was set to \(100\), with learning rate decay performed at the \(20^{th}\) and \(50^{th}\) epochs, each with a decay rate of \(0.1\). During the inference stage, we empirically set the \(\sigma\) of all GMM in the GMGD as \(0.6\), based on which the REGNN reversely samples and deco
In this paper, we use the model proposed in [25] to generate facial images from all predicted AUs.

**Evaluation metrics.** We employed the four sets of metrics defined in [6] to evaluate the generated facial reactions, namely in terms of: (i) **Appropriateness**: the distance and correlation between generated facial reactions and their most similar appropriate facial reaction using FRDist and FRCorr. In addition, we also report the Pearson Correlation Coefficient (PCC) between predictions and their most similar appropriate real facial reaction; (ii) **Diversity**: the variation among 1) frames in each generated facial reaction (**FRVar**), 2) multiple facial reactions generated from the same speaker behaviour (**FRDiv**), and 3) facial reactions generated for different speaker behaviours (**FRDvs**); (iii) **Realism**: employing Frechet Inception Distance (FID) used in [6] (**FRRea**); and (iv) **Synchrony** is measured by Time Lagged Cross Correlation (TLCC) between the speaker facial behaviour and the corresponding generated facial reaction.

### _Comparison to related works_

As this represents the first study aiming to generate multiple appropriate facial reactions, we compare our approach with different reproduced baselines that have been previously used for generating facial reactions:

* **Huang et al. (S) [8]:** This method employs a two-stage conditional GAN. In the first stage, feature vectors composed of \(17\) different facial action units of the speaker's facial behaviour are used as the condition to predict the listener's facial landmarks. Then, in the second stage, the obtained facial landmarks are employed as the condition to generate the corresponding facial reaction sequence. Here, the speaker's facial behaviour from time \(t-9\) to \(t\) is used to predict the \(t_{\text{th}}\) facial reaction frame. Consequently,

Fig. 5: Visualisation of the facial reactions generated from different approaches, where early approaches [8, 9, 45] generated some very low-quality facial images, while the predictions of a recent approach [7] is quite different from the ground-truth (i.e., low appropriateness and synchrony). Our approach generated multiple diverse but appropriate, realistic, and synchronized facial reactions from the input speaker behaviour.

Fig. 6: Examples of the low-quality facial images generated from the facial reaction attributes predicted by competitors.
this method cannot predict the first ten facial reaction frames. Therefore, in our experiments, its input size is set to \((750,25)\) and the output size is set to \((740,25)\).
* **Huang et al. (C) [9]:** Building upon [8], this method utilizes a feature vector composed of \(8\) distinct facial expressions of the speaker behaviour as a conditioning input. The final facial reaction is generated using a two-stage GAN similar to [9]. As both [8] and [9] require the inputs from time \(t-9\) to \(t\) to predict the outputs at time \(t\), they cannot make predictions for the first ten frames of the listener's facial reaction. Therefore, in our experiments, their input size is set to \((750,25)\), and the output is set to \((740,25)\).
* **UNet [45]:** UNet is a widely-used model in generative tasks, consisting of the symmetric encoder and decoder components. In this experiment, both the encoder and decoder are set to four layers. The input and output of the network are both time series data with a shape of \((750,25)\).
* **Ng et al. [7]:** This approach trains a VQ-VAE model to predict a discrete encoding from the input speaker behaviour, representing the corresponding listener's facial reaction. At the inference stage, a transformer-based predictor module is utilized to perform a lookup on the discrete encoding to predict the listener's facial reaction. In our experiments, we customized the size of hidden layers in the model to accommodate the input sequence size of \((750,25)\) and output size of \((750,25)\).

The implementation details of these reproduced approaches are provided in the supplementary material.

Table I shares the results achieved in our experiments demonstrates that our approach outperforms all competitors in generating appropriate, realistic, and synchronized facial reactions, as indicated by lower FRDist distances, Realism, Synchrony values, and higher correlations with the most similar real facial reactions. Specifically, our approach achieves improvements of \(4.37\) and \(2.74\) in FRDist, \(0.071\) and \(0.024\) in FRCorr, and \(0.095\) and \(0.007\) in PCC over the condition GAN-based approach [9] and the recently proposed VQ-VAE based approach [7], respectively. We also compare our predictions with predictions achieved by [7] in Fig. 7. Furthermore, our approach generates diverse facial reactions in response to different speaker behaviours, as well as decent diversity among frames of each generated facial reaction. Our approach can also generate different facial reactions in response to each speaker behaviour (visualized in Fig. 5). These results demonstrate the effectiveness of our AFRDL strategy in generating multiple different but appropriate, realistic, and synchronized facial reactions. It should be noted that while our approach did not generate facial reactions with as much diversity as C-GAN based [8, 9] and VQ-VAE based approaches [7], such high diversity results are partially associated to the generation of abnormal facial reactions (i.e., facial behaviours that are not appropriate or cannot be properly expressed by humans (illustrated in Fig. 6)), which is reflected by their much worse appropriateness, realism, and synchrony performances.

### _Ablation studies_

In this section, we first conduct a series of ablation studies to evaluate the effectiveness/importance of (i) each modality of the speaker behaviour; (ii) the proposed appropriate facial reaction distribution learning (AFRDL) strategy; (iii) the proposed reversible graph model (REGNN); and (iv) the multi-dimensional edge feature learning (MEFL) module. We also provide the sensitivity analysis for two main variables: (i) \(\sigma\) used in defining the Gaussian Mixture Graph Distribution (GMGD) \(Z_{L}(B_{S}^{t_{1},t_{2}})\); and (ii) the dimension \(D\) of the multi-dimensional edge feature output from the MEFL module, in the Supplementary Material.

**Contributions of different modalities.** The experimental results presented in Table II reveal that both audio and facial modalities of the speaker behaviour offer valuable cues for generating appropriate facial reactions. In particular, the facial behaviours of the speaker exhibit a greater impact on the performance, with better performances in terms of appropriateness, diversity, and realism. Since feeding both audio and visual speaker behaviours results in the best performance, it suggests that audio and visual cues from the speaker behaviour are complementary and relevant for the generation of appropriate facial reactions. Importantly, the proposed approach outperforms several existing methods [7, 8, 9] even when either audio or visual modality of the speaker behaviour alone is utilized, further validating our proposed framework's effectiveness.

**Appropriate facial reaction distribution learning (AFRDL) strategy.** Table II reports a comparative analysis of the effectiveness of the proposed AFRDL strategy, where we developed a variant of our framework with the same architecture but different training strategy. The variant was trained using MSE loss, which minimizes the difference between the generated facial reaction and the GT real facial reaction of the input speaker behaviour. The results indicate that the proposed
AFRDL strategy is crucial in generating high-quality facial reactions, as the variant achieved significantly worse performance in terms of appropriateness, realism, and synchrony.

**REGNN vs. reversible CNN.** Table II and Fig. 8 also compare the performance achieved by the proposed reversible GNN (REGNN) with a widely-used reversible CNN (i-ResNet [46]) to investigate the effectiveness of our REGNN, while keeping the rest of the framework and training strategy unchanged. The results demonstrate that the REGNN-based systems (i.e., single-value edge graph-based system and multi-dimensional edge graph-based system) outperform the i-ResNet-based system with substantial improvements across all metrics of appropriateness, realism, and synchrony. As discussed in Sec. V-B, the i-ResNet-based system sometimes generates abnormal facial reactions, which may account for its better diversity performance (illustrated in Fig. 6). In other words, the proposed REGNN predicts more appropriate facial reactions and allows the generated facial image sequences to be more realistic. We hypothesis that this is because the REGNN-based system can explicitly represents the task-specific relationship between each pair of facial attributes in the form graph representation, and thereby they can be better modeled.

**MEFL module.** Finally, we found that the proposed MEFL module also provides clear improvement for facial reaction generation. As seen in Table II, the additional usage of the multi-dimensional edge features generated by our MEFL module provides large improvements in terms of all appropriateness metrics (i.e., \(15\%\), \(135\%\), and \(59\%\) relative improvements

Fig. 8: Visualisation of the learned distributions. It is clear that the distribution learned by the proposed REGNN (depicted in (b)) are more discriminative than i-ResNet (depicted in (a)).

Fig. 7: Visualisation of the facial attribute predictions. Although our approach is trained to generate appropriate facial reactions, the facial attributes predicted by our approach are still highly correlated with the GT real facial reaction.
in FRDist, FRCorr, and PCC, respectively) as well as two diversity metrics (i.e., FRVar and FRDvs), showing that the task-specific relationship between facial attributes is complex, and thus could be better modeled by multi-dimensional edge features rather than single-value edge features. In contrast, the multi-dimensional edge features only have small impacts on the generated facial reactions' realism and synchrony performances.

## VI Conclusion

In this paper, we propose the first deep learning-based framework that opens up a new avenue of research for predicting multiple appropriate human facial reactions to a speaker behaviour. It is the first work that reformulates the "one-to-many mapping" problem occurring when training FRG models as a 'one-to-one mapping' problem (one speaker behaviour-to-one distribution representing multiple appropriate facial reactions), where a novel reversible GNN (REGNN) and a multiple appropriate facial reaction distribution learning (AFRDL) strategy are proposed. As the first specifically designed multiple appropriate FRG model, experimental results show that: (i) our approach can generate multiple diverse but appropriate, realistic, and synchronized facial reactions in response to each speaker behaviour, and achieve greater performance in appropriateness, realism, and synchrony metrics as compared to all the reproduced existing works; (ii) the proposed REGNN-based facial reaction distribution learning contributes substantially to the promising appropriateness, realism, and synchrony performances achieved by our approach; (iii) both audio and facial speaker behaviours provide relevant and complementary information; (iv) the proposed REGNN is crucial for the success of the AFRDL strategy; and (v) the MEFL module is crucial in generating appropriate facial reactions, as multi-dimensional edge features can comprehensively model task-specific relationships among facial attributes.

**Limitations and future work:** As the first multiple appropriate FRG framework, this paper only predicted reactions based on speakers' non-verbal behaviours, which did not achieved very decent performance. Another limitation is that the facial reaction distribution of different speaker behaviours sometimes are similar and thus not discriminative. Our future work will focus on (i) developing more advanced generative algorithms; (ii) considering both verbal and non-verbal behaviours of speakers; and (iii) investigating better ways to represent appropriate facial reaction distributions.
