# DRepMRec: A Dual Representation Learning Framework for Multimodal
Recommendation

Kangning Zhang1, Yingjie Qin2, Ruilong Su2, Yifan Liu1, Jiarui Jin1,
Weinan Zhang1 Yong Yu1 1Shanghai Jiao Tong University, 2XiaohongshuChina

###### Abstract.

Multimodal Recommendation focuses mainly on how to effectively integrate
behavior and multimodal information in the recommendation task. Previous works
suffer from two major issues. Firstly, the training process tightly couples
the behavior module and multimodal module by jointly optimizing them using the
sharing model parameters, which leads to suboptimal performance since behavior
signals and modality signals often provide opposite guidance for the
parameters updates. Secondly, previous approaches fail to take into account
the significant distribution differences between behavior and modality when
they attempt to fuse behavior and modality information. This resulted in a
misalignment between the representations of behavior and modality. To address
these challenges, in this paper, we propose a novel Dual Representation
learning framework for Multimodal Recommendation called DRepMRec, which
introduce separate dual lines for coupling problem and Behavior-Modal
Alignment (BMA) for misalignment problem. Specifically, DRepMRec leverages two
independent lines of representation learning to calculate behavior and modal
representations. After obtaining separate behavior and modal representations,
we design a Behavior-Modal Alignment Module (BMA) to align and fuse the dual
representations to solve the misalignment problem. Furthermore, we integrate
the BMA into other recommendation models, resulting in consistent performance
improvements. To ensure dual representations maintain their semantic
independence during alignment, we introduce Similarity-Supervised Signal (SSS)
for representation learning. We conduct extensive experiments on three public
datasets and our method achieves state-of-the-art (SOTA) results. The source
code will be available upon acceptance.

The email for Kangning Zhang is<EMAIL_ADDRESS>
††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Personalized recommendation has become increasingly prevalent in multimedia
content-sharing platforms to help users discover items of interest. Many
recommendation approaches are based on Collaborative Filtering (CF) (Koren et
al., 2009; He et al., 2020; Wang et al., 2019), which efficiently leverages
user-item interactions. However, the CF-based recommendation model is
incapable of accurately capturing user preferences when the interaction data
are sparse. To alleviate the data sparsity problem, multimodal information has
been introduced into the recommendation system. We claim that the main
challenge in current multimodal recommendation models is How to effectively
integrate the behavior information from interaction data and multimodal
information from image and text data?

Figure 1.  (a) The gradient direction conflict between behavior and modal
signal in MGCN (Yu et al., 2023); (b)-(c) The representation distribution
without and with BMA in BM3 (Zhou et al., 2023b) and VBPR (He and McAuley,
2016b). The arrows means distribution transformation. The detailed result is
reported in Table 4.

Though making significant progress, previous multimodal recommendation models
suffer from two major issues. First, there is excessive coupling during the
training process. Typically, behavior information emphasizes similar
purchasing or clicking behaviors, while modal information highlights semantic
comprehension of items. Therefore, behavior information and modal information
often provide conflicting guidance for model parameter updates.
Experimentally, we visualized the gradient directions of parameter updates for
user and item embedding with the latest multimodal recommendation model: MGCN
(Yu et al., 2023), with respect to the behavior-related loss and modal-related
loss separately. Figure 1(a) illustrates the phenomenon of inconsistent
gradient updates during the model training. The optimization conflict issue
caused by training coupling remains consistent with the previous literature
(Guo et al., 2023).

Second, the lack of alignment between behavior representation and modality
representation is a significant issue. As a result, Behavior representation
and Modal representation naturally exist in separate latent spaces. Previous
work has attempted various methods to blend their information: concatenating
(He and McAuley, 2016b), utilizing grph neural networks (Wei et al., 2019,
2020; Wang et al., 2021), introducing self-supervised methods (Zhou et al.,
2023b; Tao et al., 2022). However, we claim that if representations from
different domains cannot be properly aligned before integrating, the learning
capability of the model might be reduced significantly.

To solve the coupling problem, we introduce DRepMRec, a dual representation
learning framework for multimodal recommendation, where one representation
only learns from the interaction data (behavior information) and another
representation fully focuses on the pre-extracted multimodal features. In our
model, there are two parallel lines to calculate dual representations. On the
Behavior Line, we design the Behavior Representation Encoder to generate
Behavior representations for users and items. In the encoder, we perform
LightGCN (He et al., 2020) in the normalized user-item interaction graph. On
the Modal Line, we use modal-specific encoders to calculate the modal
representations for users and items. In these encoders, we introduce modal-
specific filter gate and modal graphs, inspired by (Zhou, 2022; Yu et al.,
2023). In this way, the behavior information and multimodal information are
captured separately and we solve the issue of coupling.

To facilitate alignment between behavior representation and modality
representation, we designed the Behavior-Modal Align (BMA) module, which
utilizes InfoNCE (Oord et al., 2018) loss for Intra-Alignment and Inter-
Alignment. The Intra-Alignment refers to the alignment of both user and item
behavioral representations, as well as the alignment of user and item modality
representations. It aims to ensure that the representations within each domain
(behavioral and modality) are aligned properly. Inter-Alignment, in contrast
to Intra-Alignment, focuses on aligning the behavioral and modality
representations of users and items separately. It aims to ensure that the
behavioral representation of a user (item) is aligned with corresponding
modality representation of the same user (item). Through our alignment
process, the Behavior and Modal representations are brought into the same
latent space. Furthermore, Behavior-Modal Alignment can be easily incorporated
into other multimodal recommendation models (e.g. VBPR (He and McAuley,
2016b), BM3 (Zhou et al., 2023b), MGCN (Yu et al., 2023)) to further enhance
their recommendation performance, as the results exported in section 4.7.1.
Finally, we employed T-SNE (Van der Maaten and Hinton, 2008) to visualize the
behavioral and modal representations of these models, demonstrating the
distribution shift after integrating our alignment module in Figure 1 (b)-(c).

Meanwhile, dual representations might lose their distinct semantic information
during the alignment process. To address this concern, we introduce the
Similarity-Supervised Signal (SSS) to ensure the Modality representations
retain similarity information within the original modality features. We
conducted comprehensive comparative experiments to validate the effectiveness
of SSS. Finally, we only need to perform a simple summation of the Behavior
and Modal representations to merge the relevant information and complete
recommendation task.

We summarize our main contributions as follows:

  * •

We propose a novel dual representation learning framework called DRepMRec,
which concurrently computes the Behavior and Modal representations for
multimodal recommendation. This approach decouples the learning of behavior
and multimodal information, effectively addressing coupling issues.

  * •

We introduce the Behavior-Modal Alignment (BMA) module to tackle misalignment
problems through both Intra-Alignment and Inter-Alignment. Furthermore, the
BMA module can be seamlessly integrated into other multimodal recommendation
models, thereby enhancing their recommendation performance.

  * •

To ensure dual representations retain distinct semantic information during the
alignment process, we introduce Similarity-Supervised Signal (SSS) and examine
its impact on model performance and representation learning.

We conduct extensive experiments on three public datasets and our method
achieves state-of-the-art (SOTA) results, which demonstrates the effectiveness
of our proposed method.

##  2\. Related Works

###  2.1. Multi-modal Recommendation

Most early multimodal recommendation models utilize deep learning to explore
user preferences. VBPR (He and McAuley, 2016b) obtain item representation by
concatenating the latent visual features from pre-trained network (Simonyan
and Zisserman, 2014) and id embedding. Deepstyle (Liu et al., 2017) augments
the representations with both visual and style features. MAML (Liu et al.,
2019) utilize a two-layer neural network to capture preference on textual and
visual features. VECF (Chen et al., 2017) utilizes the VGG model (Simonyan and
Zisserman, 2014) to capture the user attention on different image regions.
Recently, there has been a line of works that introduce GNNs into multimodal
Recommendation (Wu et al., 2022). MMGCN (Wei et al., 2019) construct modality-
specific user-item bipartite graph to capture users’ preference for specific
modality. DualGNN (Wang et al., 2021) introduces a user co-occurrence graph
learned from modality-specific graphs. GRCN (Wei et al., 2020) scores the
affinity between users and items and identifies the false-positive edges.
LATTICE (Zhang et al., 2021) and MICRO (Zhang et al., 2022) create the item-
item graph for each modality and fuse them together to obtain a latent item
graph. FREEDOM(Zhou, 2022) further freezes the item-item graph for effective
and efficient recommendation. Self-supervised learning has achieved
significant performance in various CV and NLP tasks (Jing and Tian, 2020; Liu
et al., 2021) and also has shown its effectiveness in addressing data sparsity
(Chen et al., 2023). BUIR (Lee et al., 2021) learns the representations solely
from positive interactions. BM3 (Zhou et al., 2023b) utilizes the simple
dropout technique to generate contrastive views and use three contrastive loss
functions to optimize the result representations, which help reduce the cost
of computational resources. SLMRec (Tao et al., 2022) design several SSL tasks
(FD, FM, and FAC) to supplement the supervised tasks to uncover the hidden
signals from the data itself. However, the above works still suffer from the
couple problem between behavior and modality and can not effectively align the
information from different domain. In this paper, we propose a dual
representation framework to solve these problems with multi-view learning and
explore the potential of utilizing multimodal information from multiple
perspectives.

Figure 2.  The overview of DRepMRec: (a) Behavior Line (section 3.2) utilizes
ID embedding for behavior representation learning. (b) Modal Line (section
3.3) focuses on utilizing the multimodal features through modal-specific
Filter Gate (Fig. 3) and modal graphs (Eq. 5). (c) Behavior-Modal Alignment
module (section 3.4) consists of Intra-Alignment (Eq. 10 ) and Inter-Alignment
(Eq. 11) for representation alignment. (d) Similarity-Supervised Signal (Fig.
4) retains semantic information of Modal representations.

###  2.2. Multi-View Representation Learning

Recently, multi-view representation learning has become a rapidly growing
direction in data mining (Li et al., 2019). The core challenge in Multi-View
Learning includes multi-view representation alignment and fusion. Canonical
correlation analysis (CCA) and its kernel extensions (Hotelling, 1992; Hardoon
et al., 2004) are representative techniques. Inspired by the success of deep
neural networks, deep CCAs (Andrew et al., 2013) have been proposed to learn a
joint representation that is coupled between multiple views. In Multimodal
Recommendation, DRAGON (Zhou et al., 2023a) learns the dual representation of
users and items through constructing homogeneous graphs and heterogeneous
user-item interaction graphs. MGCN (Yu et al., 2023) propose a multiview graph
network and design a behavior-guided purifier to purify modality information.
Although these papers introduce Multi-View Learning into Multimodal
Recommendation, they lack a systematic utilization and analysis of behavior
information and modal information. In contrast to the aforementioned work,
DRepMRec primarily focuses on the issues of excessive coupling between
behavior and modal information, as well as representation misalignment.

##  3\. Dual Representation Model

###  3.1. Problem Definition

Given a set of M users $\mathcal{U}=\\{u_{i}\\}_{i=1}^{M}$, a set of N items
$\mathcal{I}=\\{i_{t}\\}_{t=1}^{N}$. Each item is associated with multimodal
features $e_{m}^{i}\in\mathbb{R}^{d_{m}}$, where $m\in\\{V,T\\}$, $V$ and $T$
represent visual and textual modality respectively (The multimodal features
are pre-extracted from the corresponding Vision network and Language model).
User historical behaviour data is denoted as $R\in\mathbb{R}^{M\times N}$,
where $R_{ui}=1$ means user $u$ has interacted with item $i$, otherwise
$R_{ui}=0$. Naturally, the historical interaction data $R$ can be regarded as
a sparse behavior graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where
$\mathcal{V}=\mathcal{U}\cup\mathcal{I}$ denotes the set of nodes and
$\mathcal{E}=\\{(u,i)|u\in\mathcal{U},i\in\mathcal{I},R_{ui}=1\\}$ denotes the
set of edges. We use $\mathcal{I}_{u}=\\{i|(u,i)\in\mathcal{E}\\}$ to denote
the set of items which a user $u$ has interacted with. The purpose of the
multimodal recommendation is to accurately predict users’ preferences by
ranking items for each user according to predicted preferences scores
$y_{ui}=f_{\theta}(e_{B}^{u},e_{B}^{i},e_{m}^{i},e_{m}^{u})$, where $\theta$
means model parameters, $e_{B}^{u}$, $e_{B}^{i}$ mean ID embedding and
$e_{m}^{i}$, $e_{m}^{u}$ mean modal features for user and item. In this work,
the user modal feature $e_{m}^{u}$ can be easily calculated by
$e_{m}^{u}=\frac{1}{|\mathcal{I}_{u}|}\sum_{i\in\mathcal{I}_{u}}e_{m}^{i}$.
111Specifically, we use $e$ to represent the embedding of an individual
instance, and $E$ represent the concatenated embedding table. Superscripts $u$
and $i$ represent users or items, while subscripts $B$ and $m\in\\{V,T\\}$
represent behavior or multimodal domains.

###  3.2. Behavior-Line Representation Learning

####  3.2.1. Behavior Representation Encoder

In our work, we use the Behavior representation to fully capture the behavior
signal (e.g. click or purchase feedback). We construct a symmetric adjacency
matrix $A\in\mathbb{R}^{|\mathcal{V}|\times|\mathcal{V}|}$ from the user-item
interaction matrix $R\in\mathbb{R}^{M\times N}$:

(1) $A=\begin{pmatrix}0&R\\\ R^{T}&0\end{pmatrix}$

and each entry $A_{ui}$ of $A$ is set to 1 if user $u$ has interacted with
item $i$, otherwise 0. The diagonal degree matrix of $A$ is denoted by $D$ and
$D_{ii}=\sum_{j}A_{ij}$. Let $E_{B}^{u}$ denote the behavior embedding table
of users by stacking the embedding of all users. The behavior embedding of
user $u$ can be denoted as $e_{B}^{u}$. Similarly, we use $E_{B}^{i}$ to
denote the behavior embedding table of items. Specifically, we use
$E_{B}^{l}\in\mathbb{R}^{|\mathcal{V}|\times d}$ to denote the behavior
embedding at the $l$-th layer by stacking all the embedding of users and items
at layer $l$. $E_{B}^{0}$ is the concatenation of $E_{B}^{u}$ and $E_{B}^{i}$.
We use simplified graph convolutional layers in LightGCN (He et al., 2020) to
perform forward propagation:

(2) $E_{B}^{l+1}=(D^{-\frac{1}{2}}AD^{-\frac{1}{2}})E_{B}^{l}$

We use a readout function to aggregate all representations in hidden layers
for Behavior user representation and Behavior item representation:

(3) $\displaystyle\overline{E}_{B}$
$\displaystyle=\textbf{Readout}(E_{B}^{0},E_{B}^{1},...,E_{B}^{L})$

where the Readout function can be any differentiable function. We use the mean
function of LightGCN for the final Behavior representation calculation.
Finally, $\overline{E}_{B}^{u}$ and $\overline{E}_{B}^{i}$ can be obtained by
respectively taking the first M rows and the last N rows of
$\overline{E}_{B}$, denoting the Behaviour Representation for users and items.

###  3.3. Modal-Line Representation Learning

####  3.3.1. Modal-specific Encoder

Figure 3.  The structure of Image-specific Filter Gate in Vision modality. The
Text-specific Filter Gate is similar.

Let $E_{V}^{i}\in\mathbb{R}^{N\times d_{v}}$ and
$E_{T}^{i}\in\mathbb{R}^{N\times d_{t}}$ denote the item raw feature table in
the vision and text modalities. In the same way,
$E_{V}^{u}\in\mathbb{R}^{M\times d_{v}}$ and $E_{T}^{u}\in\mathbb{R}^{M\times
d_{t}}$ denote the user modal feature tables.

The Modal-specific Encoder includes the Modal-specific Filter Gate and Modal-
specific relation graphs. We herein only illustrate the process for the vision
modality, and text modality is similar.

Inspired by (Yu et al., 2023), we design the modal-specific Filter Gate for
denoising the initial multimodal features, which is showed in Figure 3. The
Gate network filters the behavior-unrelated noise from raw modal features to
help improve model performance (Yu et al., 2023). The Vision-specific Filter
Gate takes $E_{V}^{i}$, $E_{V}^{u}$ and $E_{B}^{i}$, $E_{B}^{u}$ as inputs and
output the filtered vision representation $\hat{E}_{V}^{i}$ and
$\hat{E}_{V}^{u}$:

(4) $\displaystyle\begin{aligned}
\hat{E}_{V}^{i}&=\text{Gate}_{v}(E_{V}^{i},E_{B}^{i})=\sigma(E_{V}^{i}W_{2}+b_{2})\odot
E_{B}^{i}\\\
\hat{E}_{V}^{u}&=\text{Gate}_{v}(E_{V}^{u},E_{B}^{u})=\sigma(E_{V}^{u}W_{2}+b_{2})\odot
E_{B}^{u}\end{aligned}$

where $W_{2}\in\mathbb{R}^{d_{v}\times d}$ and $b_{2}\in\mathbb{R}^{d}$ are
learnable parameters, $\odot$ denotes the element-wise product and $\sigma$ is
the sigmoid function.

Previous works (Zhang et al., 2021; Zhou, 2022; Yu et al., 2023) have focused
on exploring the modal item-item graphs, but they neglect the impact of the
modal user-user graphs. Therefore, we propose modal-specific relation graphs
that contain not only item-item graphs but user-user graphs, which help to
better capture modal semantic information.

Inspired by FREEDOM (Zhou, 2022), we first calculate the similarity score
between item or user pair on their raw modal features. To make the relation
graph sparse, we only keep the top-$k$ similar items (users) of every item
(user) and convert the weighted graph into an unweighted graph. Finally, we
normalize the discretized graph. The calculation procedure about item-item
graph in vision modality is as follows:

(5) $\displaystyle S_{V}^{ij}$
$\displaystyle=\frac{(e_{V}^{i})^{T}e_{V}^{j}}{||{e_{V}^{i}}||||e_{V}^{j}||}$
$\displaystyle\hat{S}_{V}^{ij}$
$\displaystyle=\begin{cases}1&S_{V}^{ij}\in\text{topk}(S_{V}^{i})\\\
0&\text{otherwise}\end{cases}$ $\displaystyle\tilde{S}_{V}$
$\displaystyle=(D^{v})^{-\frac{1}{2}}\hat{S}_{V}(D^{v})^{-\frac{1}{2}}$

$\tilde{S}_{V}$ is the final item-item relation graph in modality $V$. We use
a similar way to construct user-user relation graph $\tilde{J}_{V}$ in
modality $V$.

The graph convolution over the item-item relation graph and user-user relation
graph in modality $V$ is:

(6)
$\displaystyle\overline{E}_{V}^{i}=\tilde{S}_{V}\hat{E}_{V}^{i}\quad\overline{E}_{V}^{u}=\tilde{J}_{V}\hat{E}_{V}^{u}$

.

Finally, we fuse vision and text modality representation with modality-
specific weight to obtain Modal Representation:

(7) $\displaystyle\overline{E}_{M}^{u}$
$\displaystyle=\alpha_{V}\overline{E}_{V}^{u}+\alpha_{T}\overline{E}_{T}^{u}$
$\displaystyle\overline{E}_{M}^{i}$
$\displaystyle=\alpha_{V}\overline{E}_{V}^{i}+\alpha_{T}\overline{E}_{T}^{i}$

where $\alpha_{V}\in[0,1]$ and $\alpha_{T}\in[0,1]$ are hyper-parameters and
$\alpha_{V}+\alpha_{T}=1$.

###  3.4. Behavior-Modal Alignment

####  3.4.1. Intra-Alignment

The Intra-Alignment includes Behavior Intra-Alignment (BIA) and Modal Intra-
Alignment (MIA). After the encoder outputs the Behavior representation
$\overline{E}_{B}^{u}$ and $\overline{E}_{B}^{i}$, we use InfoNCE to maximize
the mutual information between the user representation and item representation
in behavior domain:

(8)
$\mathcal{L}_{BIA}=\sum_{(u,i)\in\mathcal{B}}-\text{log}\frac{exp(\overline{e}_{B}^{u}\cdot\overline{e}_{B}^{i}/\tau)}{\sum_{i^{\prime}\in\mathcal{B}}exp(\overline{e}_{B}^{u}\cdot\overline{e}_{B}^{i^{\prime}}/\tau)}$

where $\tau$ is the temperature hyper-parameter and $\mathcal{B}$ is sampled
batched interaction data, and $(u,i)$ pair denotes that user $u$ has
interacted with item $i$. Similarly, we also utilize InfoNCE loss to align the
Modal user representation and item representation in Modal Intra-Alignment:

(9)
$\mathcal{L}_{MIA}=\sum_{(u,i)\in\mathcal{B}}-\text{log}\frac{exp(\overline{e}_{M}^{u}\cdot\overline{e}_{M}^{i}/\tau)}{\sum_{i^{\prime}\in\mathcal{B}}exp(\overline{e}_{M}^{u}\overline{e}_{M}^{i^{\prime}}/\tau)}$

The Intra-Alignment Loss is the combination of Behavior Intra and Modal Intra
Alignment:

(10) $\displaystyle\mathcal{L}_{Intra}=\mathcal{L}_{BIA}+\mathcal{L}_{MIA}$

Experimentally, we have proved that Intra-Alignment is crucial for model
performance in section 4.7.1

####  3.4.2. Inter-Alignment

Until now, we get the Behavior representation $\overline{E}_{B}^{u}$,
$\overline{E}_{B}^{i}$ and Modal representation $\overline{E}_{M}^{u}$,
$\overline{E}_{M}^{i}$. However, representations of two lines do not share the
same latent space. Therefore, in order to align behavior and modal
information, we utilize the InfoNCE to align dual user/item representations
obtained from Behavior Line and Modal Line:

(11) $\displaystyle\mathcal{L}_{Inter}$
$\displaystyle=\sum_{u\in\mathcal{B}}-\text{log}\frac{exp(\overline{e}_{B}^{u}\cdot\overline{e}_{M}^{u}/\tau)}{\sum_{u^{\prime}\in\mathcal{B}}exp(\overline{e}_{B}^{u}\cdot\overline{e}_{M}^{u^{\prime}}/\tau)}$
$\displaystyle+\sum_{i\in\mathcal{B}}-\text{log}\frac{exp(\overline{e}_{B}^{i}\cdot\overline{e}_{M}^{i}/\tau)}{\sum_{i^{\prime}\in\mathcal{B}}exp(\overline{e}_{B}^{i}\cdot\overline{e}_{M}^{i^{\prime}}/\tau)}$

It is worth mentioning that in spite of the alignments between user and item
representations from different lines, Behavior and Modal representations still
retain their distinctiveness in terms of semantic information, which is
further proven in section 4.6.

####  3.4.3. Similarity-Supervised Signal

It is worth mentioning that in spite of the alignments between user and item
representations from different lines, preserving the independent semantics of
Behavioral and Modal representations remains crucial. We introduce the
Similarity-Supervised Signal (SSS) to help the Modal representations retain
the similarity information that exists in their original modal features:

(12) $\displaystyle e_{M}^{i}=\alpha_{V}e_{V}^{i}$
$\displaystyle+\alpha_{T}e_{T}^{i}$ $\displaystyle
e_{M}^{u}=\alpha_{V}e_{V}^{u}$ $\displaystyle+\alpha_{T}e_{T}^{u}$
$\displaystyle\mathcal{L}_{SSS}=\|\overline{e}_{M}^{i}\cdot\overline{e}_{M}^{i^{\prime}}-sg(e_{M}^{i}\cdot
e_{M}^{i^{\prime}})\|_{2}$
$\displaystyle+\|\overline{e}_{M}^{u}\cdot\overline{e}_{M}^{u^{\prime}}-sg(e_{M}^{u}\cdot
e_{M}^{u^{\prime}})\|_{2}$

where $\overline{e}_{M}^{i}$,$\overline{e}_{M}^{u}$ is from Eq. (7).
$(i,i^{\prime})$ and $(u,u^{\prime})$ are item/user pair from the same batch,
and $sg(\cdot)$ means stop gradients. In the concrete implementation, we
compute the loss $\mathcal{L}_{SSS}$ at the batch matrix level, as illustrated
in Figure 4. In Section 4, we conducted comprehensive experiments to
demonstrate the effectiveness of Similarity-Supervised Signal (SSS) and
highlight the differences between using SSS and without it in the learning of
dual representation.

Figure 4.  The Similarity Supervised Signal (SSS) module in Behavior-Modal
Alignment (BMA).

####  3.4.4. General Optimazation

After we conduct Intra-Alignment and Inter-Alignment, we can get the general
representations by just adding dual representations from Behavior (Eq. (3))
and Modal lines (Eq. (7):

(13) $\displaystyle E_{G}^{u}$
$\displaystyle=\overline{E}_{B}^{u}+\overline{E}_{M}^{u}$ $\displaystyle
E_{G}^{i}$ $\displaystyle=\overline{E}_{B}^{i}+\overline{E}_{M}^{i}$

Finally, we adopt the Bayesian Personalized Ranking (BPR) Loss (Rendle et al.,
2012) as general optimazation:

(14)
$\mathcal{L}_{general}=\sum_{(u,i,i^{\prime})\in\mathcal{B}}-log(\sigma(e_{G}^{u}\cdot
e_{G}^{i}-e_{G}^{u}\cdot e_{G}^{i^{\prime}}))$

where each triple $(u,i,i^{\prime})$ satisfies $A_{ui}=1$ and
$A_{ui^{\prime}}=0$, $\sigma(\cdot)$ is the sigmoid function. Our final loss
is:

(15) $\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{general}+\alpha\mathcal{L}_{Intra}+\beta\mathcal{L}_{Inter}+\gamma\mathcal{L}_{SSS}$

###  3.5. Top-$k$ Recommendation

In collaborative filtering (CF) methods, the top-$k$ items are recommended by
ranking the scores obtained from the inner product of a user’s embedding with
the embedding of all candidate items. In our work, we use $e_{G}^{u}$ and
$e_{G}^{i}$ to predict the preference scores between the user and candidate
items. That is:

(16) $y_{ui}=e_{G}^{u}\cdot e_{G}^{i}$

Subsequently, we rank the candidate items in descending order based on the
predicted interaction scores. We then select the top-$k$ ranked items as
recommendations for the user.

##  4\. Experiment

We conduct comprehensive experiments to evaluate the effectiveness of DRepMRec
and answer the following questions:

  * •

RQ1: How does DRepMRec perform compared with the state-of-the-art multimodal
recommendation methods and other collaborative filtering methods?

  * •

RQ2: What is the relationship between the dual representations and how do they
impact the model’s performance?

  * •

RQ3: How do the different modules (e.g. Intra-Alignment, Inter-Alignment, SSS)
influence the performance of the DRepMRec?

  * •

RQ4: How do different hyper-parameter settings impact the results of our
model?

Table 1. Statistics of the experimental datasets. Datasets | # Users | # Items | # Interactions | Sparsity
---|---|---|---|---
Baby | 19, 445 | 7, 050 | 160, 792 | 99.88%
Sports | 35, 598 | 18, 357 | 296, 337 | 99.95%
Clothing | 39, 387 | 23, 033 | 278, 677 | 99.97%
Table 2. Overall performance achieved by different recommendation methods in
terms of Recall and NDCG. We mark the global best results on each dataset
under each metric in boldface and the second best is underlined.

| General Model | Multi-modal Model | Ours
---|---|---|---
Dataset | Metric | BPR | LightGCN | VBPR | MMGCN | DualGNN | BM3 | FREEDOM | MGCN | DRepMRec
Baby | R@10 | 0.0357 | 0.0479 | 0.0423 | 0.0378 | 0.0448 | 0.0564 | 0.0627 | 0.0620 | 0.0655
R@20 | 0.0575 | 0.0754 | 0.0663 | 0.0615 | 0.0716 | 0.0883 | 0.092 | 0.0964 | 0.1040
N@10 | 0.0192 | 0.0257 | 0.0223 | 0.0200 | 0.0240 | 0.0301 | 0.0330 | 0.0339 | 0.0360
N@20 | 0.0249 | 0.0328 | 0.0284 | 0.0261 | 0.0309 | 0.0383 | 0.0424 | 0.0427 | 0.0459
Sports | R@10 | 0.0432 | 0.0569 | 0.0558 | 0.0370 | 0.0568 | 0.0656 | 0.0717 | 0.0729 | 0.0776
R@20 | 0.0653 | 0.0864 | 0.0857 | 0.0605 | 0.0859 | 0.0980 | 0.1089 | 0.1106 | 0.1182
N@10 | 0.0241 | 0.0311 | 0.0307 | 0.0193 | 0.0310 | 0.0355 | 0.0385 | 0.0397 | 0.0425
N@20 | 0.0298 | 0.0387 | 0.0383 | 0.0254 | 0.0385 | 0.0438 | 0.0481 | 0.0496 | 0.0529
Clothing | R@10 | 0.0187 | 0.0340 | 0.0280 | 0.0197 | 0.0454 | 0.0437 | 0.0629 | 0.0641 | 0.0683
R@20 | 0.0279 | 0.0526 | 0.0414 | 0.0328 | 0.0683 | 0.0648 | 0.0941 | 0.0945 | 0.0988
N@10 | 0.0103 | 0.0188 | 0.0159 | 0.0101 | 0.0241 | 0.0247 | 0.0341 | 0.0347 | 0.0367
N@20 | 0.0126 | 0.0236 | 0.0193 | 0.0135 | 0.0299 | 0.0302 | 0.0420 | 0.0428 | 0.0448

###  4.1. Experimental Datasets

Following (Zhou et al., 2023c, b), we use the Amazon review dataset (He and
McAuley, 2016a) for experimental evaluation. We choose three per-category
datasets, i.e., Baby, Sports, and Clothing. The raw data of each dataset are
pre-processed with a 5-core setting on both items and users, and their 5-core
filtered results are presented in Table 1. We use the pre-extracted
4,096-dimensional visual features and 384-dimensional text features, which
have been published in (Zhou et al., 2023c).

###  4.2. Compared Methods

To evaluate the effectiveness of DRepMRec, we compare it with several
representative recommendation methods, which can be divided into two groups:
General CF recommendation models: BPR (Rendle et al., 2012), LightGCN (He et
al., 2020). Multimodal recommendation model: VBPR (He and McAuley, 2016b), BM3
(Zhou et al., 2023b), MMGCN (Wei et al., 2019), DualGNN (Wang et al., 2021),
FREEDOM (Zhang et al., 2022), MGCN (Yu et al., 2023).

###  4.3. Setup and Evaluation Metrics

For a fair comparison, we follow the same evaluation setting of (Zhou et al.,
2023c, b) with a random data splitting 8:1:1 on the interaction history of
each user for training, validation, and testing. Further, we use Recall@K and
NDCG@K to evaluate the top-K recommendation performance of different
recommendation methods. In experiments, we empirically report the results of K
at 10, 20 and abbreviate the metrics of Recall@K and NDCG@K as R@K and N@K,
respectively.

###  4.4. Implementation Details

We implement the proposed model and all the baselines with MMRec (Zhou et al.,
2023c). To ensure fair compareness, we fix the embedding size of both users
and items to 64 for all models, initialize the embedding parameters with the
Xavier method(Glorot and Bengio, 2010), and use Adam (Kingma and Ba, 2014) as
the optimizer with a learning rate of 0.001. Specifically, the GCN layer in
Behavior Line is fixed as 2, the layer in Modal Line is fixed as 1, the
regularization coefficient $\lambda$ is set as $10^{-4}$, the batch size is
set to $B=2048$ and the temperature $\tau=0.2$. For convergence consideration,
the early stopping and total epochs are fixed at 20 and 1000, respectively.
Following (Zhang et al., 2021), we use Recall@20 on the validation data as the
training-stopping indicator.

###  4.5. Effectiveness of DRepMRec (RQ1)

The performances achieved by different methods are summarized in Table 2. From
the table, we have these observations:

DRepMRec significantly outperforms both general recommendation models and
multimodal recommendation models on multiple indicators. Specifically,
DRepMRec improves the best baselines by 4.47%, 6.45%, 6.55% in terms of
Recall@10, 6.19%, 7.05%, 5.76% in terms of NDCG@10 on Baby, Sports and
Clothing.

DRepMRec inherits some advantages from latest SOTA models (e.g. utilizes
freezed item-item graph from FREEDOM (Zhou, 2022) and introduces modal-
specific filter gate inspried by MGCN (Yu et al., 2023)). However, DRepMRec
successfully extends these module to the user perspective. Furthermore, our
model places greater emphasis on separate extraction and alignment of behavior
representation and modality representation. First, DRepMRec utilizes the
Behavior Line and Modal Line to calculate Behavior representation and Modal
representation respectively. Then, by introducing the Behavior-Modal Align
Module, the dual representations are aligned in the same representation space.
By addressing the issues of training coupling and misalignment in multimodal
recommendation models, DRepMRec achieves SOTA performance across all datasets.

###  4.6. In-Depth Analysis on Dual Representations (RQ2)

####  4.6.1. The Recommendation Performace of Dual Representation

In our work, we adopt the dual representation learning method for multimodal
recommendation. To investigate the individual impacts of the dual
representations learned by the model, we attempted to directly use Behavior
representation and Modal representation for recommendation evaluation, with
the result shown in Figure 5.

Figure 5. The Recall@20 and NDCG@20 results of Behavior, Modal, and General
representations on three datasets.

. Some insights can be inferred from Figure 5: (1) The preformance of Behavior
representation is generally better than Modal representation. This can be
attributed to the significance of behavior information, which is deemed
crucial in recommendation systems. On the contrary, multimodal information
generally assumes a supplementary role in enhancing the effectiveness of
recommendations. (2) Both the Behavior and Modal representations effectively
extract domain-specific information and successfully fuse each other’s
information through the BMA module. Although we introduce Alignment module to
map the Behavior and Modal representations into same latent space, the results
show that this does not affect their respective information distinction. After
fusing modal information, the Behavior representation alone exhibits excellent
recommendation performance. At the same time, Modal representation still has a
certain recall ability, which verifies that multimodal information can capture
additional interests from the semantic perspective. By leveraging both Modal
and Behavior representations together, General representation can capture a
more comprehensive understanding of user preferences and item characteristics,
leading to best recommendation performance.

####  4.6.2. The Cosine Distance between Dual Representations

To further investigate the relationship between the dual representations
during the model learning process, we present cosine distance between Behavior
representation and Modal representation at each epoch in Figure 6. From Figure
6, we can observe that in the early stages of training (epoch 0-10), the
cosine distance between Behavior Representation and Modal representation
rapidly decreases. This phenomenon can be attributed to the pronounced
disparity in the distributions of dual representations at the outset, which
engenders a substantial alignment loss. Such a loss predominates in guiding
the updates of model parameters, thereby facilitating a swift convergence of
the dual representations toward each other. Nevertheless, this dynamic
concurrently impedes the dual representations’ ability to encapsulate distinct
semantic significances independently. As the training progresses (After epoch
10), the Behavior Line and Modal Line calculate dual representations with
better semantic independence. The cosine distance gradually stabilizes,
indicating that the model is converging. We also compared the impact of
introducing the Similarity-Supervised Signal (SSS) on the cosine distance. By
incorporating SSS, we attempt to let the Modal representation capture the
similarity signals in the original multimodal features, thereby further
preserving the semantics of the dual representations. As a result, the cosine
distance becomes larger and the model performance improves significantly.

Figure 6. The cosine distance of Behavior and Modal representations on Baby
and Sports.

####  4.6.3. The Visualization of Dual Representation Distribution

We randomly selected 1000 instances (items and users) from the Sports and
Clothing dataset and visualized the two-dimensional distributions of the
Behavior representation and Modal representation generated by T-SNE (Van der
Maaten and Hinton, 2008). The results are presented in Figure 7, which shows
that the dual representations have converged to similar overall distributions.
The dashed line connects the behavior representation (square) with the
modality representation (circle) of the same item, indicating that there is
still a significant distance between the dual representations of the same
item. These observations demonstrate the capacity of DRepMRec to effectively
align distinct features in dual domains while also preserving the independent
semantics of each representation.

(a) Sports

(b) Clothing

Figure 7. The 2D distributions of Behavior representation and Modal
representation in DRepMRec.

###  4.7. Ablation Study (RQ3)

In our work, Behavior-Modal Alignment includes Intra-Alignment, Inter-
Alignment, and Similarity-Supervised Signal. We design ablation experiments to
investigate the influence of these module on model performance. We use W/O XX
to represent “without” the specific module.

Table 3. The Recall@20 and NDCG@20 metric of Ablation Study, where W/O
Alignment means removing both Inter-Alignment and Intra-Alignment.

| Baby | Sports | Clothing
---|---|---|---
Metric | R@20 | N@20 | R@20 | N@20 | R@20 | N@20
DRepMRec | 0.1040 | 0.0459 | 0.1182 | 0.0529 | 0.0991 | 0.0448
W/O Intra-Alignment | 0.0971 | 0.0433 | 0.1114 | 0.0497 | 0.0912 | 0.0406
W/O Inter-Aligment | 0.0985 | 0.0442 | 0.1125 | 0.0507 | 0.0951 | 0.0431
W/O Alignment | 0.0899 | 0.0405 | 0.0965 | 0.0427 | 0.0806 | 0.0360
W/O SSS | 0.0976 | 0.0434 | 0.1103 | 0.0491 | 0.0987 | 0.0448

####  4.7.1. The influence of Intra-Alignment and Inter-Alignment

From Table 3, we can observe that the Intra-Alignment and Inter-Alignment both
are crucial for model performance, because without any of these components,
the model performance significantly decreases. Without the alignment process,
it becomes challenging to directly fuse behavior and modal information, which
greatly impacts the model performance. Further, we try to integrate the
Behvaior-Modal Alignment module into other multimodal models (e.g. VBPR (He
and McAuley, 2016b), BM3 (Zhou et al., 2023b), MGCN222The original MGCN
framework has already incorporated a module similar to Inter-Alignment.
Therefore, we solely introduce an additional Intra-Alignment module. (Yu et
al., 2023)). With model structure and hyperparameters kept unchanged, we
observed a significant improvement in the performance of these models in Table
4. This once again emphasizes the importance of aligning multimodal
information with behavioral information. Furthermore, we visualize the
transformation in the representation distributions of these models after
incorporating BMA, as demonstrated in Figure 1 (b)-(c).

Table 4.  The performance improvement of integrating BMA into other models,
while keeping their hyper-parameters and model structure unchanged.

| Baby | Sports | Clothing
---|---|---|---
Models | R@20 | N@20 | R@20 | N@20 | R@20 | N@20
VBPR (He and McAuley, 2016b) | 0.0664 | 0.0285 | 0.0857 | 0.0383 | 0.0414 | 0.0193
\+ BMA | 0.0682 | 0.0292 | 0.0874 | 0.0388 | 0.0638 | 0.0282
BM3 (Zhou et al., 2023b) | 0.0883 | 0.0383 | 0.0980 | 0.0438 | 0.0648 | 0.0302
\+ BMA | 0.0907 | 0.0387 | 0.1078 | 0.0494 | 0.0909 | 0.0414
MGCN (Yu et al., 2023) | 0.0964 | 0.0427 | 0.1106 | 0.0496 | 0.0945 | 0.0428
\+ BMA | 0.1013 | 0.0435 | 0.1161 | 0.0513 | 0.0949 | 0.0431

####  4.7.2. The influence of Similarity Supervised Signal(SSS)

We claim that in multimodal recommendation, unifying the behavior information
and modal information into the same latent space is beneficial for information
fusion. However, it is important to maintain the semantic independence of the
behavior and modal information. Based on this, we propose the Similarity-
Supervised Signal (SSS), which aims to preserve the similarity information in
the original multimodal features within the aligned modality representation.
The last row of Table 3 reflects the contribution of SSS to the model
performance, demonstrating its effectiveness in enhancing the recommendation
quality. Additionally, Figure 6 further illustrates the effectiveness of SSS
in preserving the semantic information of dual representations.

###  4.8. Sensitivity Analysis (RQ4)

Figure 8. The Recall@20 results of different SSS weight and image weight.

####  4.8.1. The weight of SSS

Based on the analysis of Figure 8 (a), we observed that the optimal
performance of the SSS weight varies across different datasets. Interestingly,
the model’s performance initially improves and then deteriorates as the weight
of SSS increases. This suggests that the SSS signal is indeed beneficial for
enhancing the model’s performance. However, excessively high weights assigned
to SSS result in an overemphasis on capturing similarity information within
modality representation. Consequently, the model’s ability to capture behavior
information might be compromised, leading to a decline in overall performance.

####  4.8.2. The weight of modal weight

From Figure 8 (b), we find that both textual and visual features can improve
performance. By combining both the textual and visual features, DRepMRec
achieves best recommendation performance. In all three datasets, best vision
weight is $0.3$. It means text modality is more important than vision
modality, which is consistent with BM3 (Zhou et al., 2023b).

Figure 9. The Recall@20 results of different Intra-alignment and Inter-
alignment weight.

####  4.8.3. The weight of Intra-alignment and Inter-alignment weight

In our study, we propose the Behavior-Modal Alignment module for achieving
both Intra-Alignment and Inter-Alignment. In this section, we examine the
impact of weights, denoted as $\alpha$ for Intra-Alignment and $\beta$ for
Inter-Alignment. The results obtained from three datasets are presented in
Figure 9.

##  5\. Conclusion

In this study, we propose a novel framework for multimodal recommendation,
named DRepMRec. DRepMRec leverages two independent lines of representation
learning to calculate behavior and multimodal representations, effectively
mitigating the coupling problem. To address the issue of misalignment, we
introduce the Behavior-Modal Alignment (BMA) module. Importantly, the BMA can
be seamlessly integrated into existing multimodal models, leading to improved
recommendation performance. To preserve the semantic independence of the dual
representations during alignment, we also design a Similarity-Supervised
Signal (SSS). This signal ensures that the behavior and multimodal
representations maintain their distinct semantics while being aligned.
Experimental results demonstrate that DRepMRec achieves state-of-the-art
performance, based on our current knowledge. As future research directions, we
aim to explore more effective approaches to align and integrate behavioral
information and multimodal information in the context of recommendation
systems.

## References

  * (1)
  * Andrew et al. (2013) Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013.  Deep canonical correlation analysis. In _International conference on machine learning_. PMLR, 1247–1255.
  * Chen et al. (2017) Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017.  Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In _Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval_. 335–344.
  * Chen et al. (2023) Mengru Chen, Chao Huang, Lianghao Xia, Wei Wei, Yong Xu, and Ronghua Luo. 2023.  Heterogeneous graph contrastive learning for recommendation. In _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining_. 544–552.
  * Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010.  Understanding the difficulty of training deep feedforward neural networks. In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 249–256.
  * Guo et al. (2023) Zhiqiang Guo, Jianjun Li, Guohui Li, Chaoyang Wang, Si Shi, and Bin Ruan. 2023.  LGMRec: Local and Global Graph Learning for Multimodal Recommendation.  arXiv:2312.16400 [cs.IR]
  * Hardoon et al. (2004) David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. 2004.  Canonical correlation analysis: An overview with application to learning methods.  _Neural computation_ 16, 12 (2004), 2639–2664.
  * He and McAuley (2016a) Ruining He and Julian McAuley. 2016a.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In _proceedings of the 25th international conference on world wide web_. 507–517.
  * He and McAuley (2016b) Ruining He and Julian McAuley. 2016b.  VBPR: visual bayesian personalized ranking from implicit feedback. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 30.
  * He et al. (2020) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020.  Lightgcn: Simplifying and powering graph convolution network for recommendation. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. 639–648.
  * Hotelling (1992) Harold Hotelling. 1992.  Relations between two sets of variates.  In _Breakthroughs in statistics: methodology and distribution_. Springer, 162–190.
  * Jing and Tian (2020) Longlong Jing and Yingli Tian. 2020.  Self-supervised visual feature learning with deep neural networks: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ 43, 11 (2020), 4037–4058.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009.  Matrix factorization techniques for recommender systems.  _Computer_ 42, 8 (2009), 30–37.
  * Lee et al. (2021) Dongha Lee, SeongKu Kang, Hyunjun Ju, Chanyoung Park, and Hwanjo Yu. 2021.  Bootstrapping user and item representations for one-class collaborative filtering. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 317–326.
  * Li et al. (2019) Yingming Li, Ming Yang, and Zhongfei Zhang. 2019.  A Survey of Multi-View Representation Learning.  _IEEE Transactions on Knowledge and Data Engineering_ 31, 10 (oct 2019), 1863–1883.  https://doi.org/10.1109/tkde.2018.2872063
  * Liu et al. (2019) Fan Liu, Zhiyong Cheng, Changchang Sun, Yinglong Wang, Liqiang Nie, and Mohan Kankanhalli. 2019.  User Diverse Preference Modeling by Multimodal Attentive Metric Learning. In _Proceedings of the 27th ACM International Conference on Multimedia_. ACM.  https://doi.org/10.1145/3343031.3350953
  * Liu et al. (2017) Qiang Liu, Shu Wu, and Liang Wang. 2017.  Deepstyle: Learning user preferences for visual recommendation. In _Proceedings of the 40th international acm sigir conference on research and development in information retrieval_. 841–844.
  * Liu et al. (2021) Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021.  Self-supervised learning: Generative or contrastive.  _IEEE Transactions on Knowledge and Data Engineering_ 35, 1 (2021), 857–876.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Rendle et al. (2012) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012.  BPR: Bayesian personalized ranking from implicit feedback.  _arXiv preprint arXiv:1205.2618_ (2012).
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Tao et al. (2022) Zhulin Tao, Xiaohao Liu, Yewei Xia, Xiang Wang, Lifang Yang, Xianglin Huang, and Tat-Seng Chua. 2022.  Self-supervised learning for multimedia recommendation.  _IEEE Transactions on Multimedia_ (2022).
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Wang et al. (2021) Qifan Wang, Yinwei Wei, Jianhua Yin, Jianlong Wu, Xuemeng Song, and Liqiang Nie. 2021.  Dualgnn: Dual graph neural network for multimedia recommendation.  _IEEE Transactions on Multimedia_ (2021).
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019.  Neural Graph Collaborative Filtering. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM.  https://doi.org/10.1145/3331184.3331267
  * Wei et al. (2020) Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, and Tat-Seng Chua. 2020.  Graph-refined convolutional network for multimedia recommendation with implicit feedback. In _Proceedings of the 28th ACM international conference on multimedia_. 3541–3549.
  * Wei et al. (2019) Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019.  MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video. In _Proceedings of the 27th ACM international conference on multimedia_. 1437–1445.
  * Wu et al. (2022) Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. 2022.  Graph neural networks in recommender systems: a survey.  _Comput. Surveys_ 55, 5 (2022), 1–37.
  * Yu et al. (2023) Penghang Yu, Zhiyi Tan, Guanming Lu, and Bing-Kun Bao. 2023.  Multi-View Graph Convolutional Network for Multimedia Recommendation.  _arXiv preprint arXiv:2308.03588_ (2023).
  * Zhang et al. (2021) Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, and Liang Wang. 2021.  Mining latent structures for multimedia recommendation. In _Proceedings of the 29th ACM International Conference on Multimedia_. 3872–3880.
  * Zhang et al. (2022) Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Mengqi Zhang, Shu Wu, and Liang Wang. 2022.  Latent structure mining with contrastive modality fusion for multimedia recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ (2022).
  * Zhou et al. (2023a) Hongyu Zhou, Xin Zhou, and Zhiqi Shen. 2023a.  Enhancing Dyadic Relations with Homogeneous Graphs for Multimodal Recommendation.  _arXiv preprint arXiv:2301.12097_ (2023).
  * Zhou et al. (2023c) Hongyu Zhou, Xin Zhou, Zhiwei Zeng, Lingzi Zhang, and Zhiqi Shen. 2023c.  A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions.  _arXiv preprint arXiv:2302.04473_ (2023).
  * Zhou (2022) Xin Zhou. 2022.  A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation.  _arXiv preprint arXiv:2211.06924_ (2022).
  * Zhou et al. (2023b) Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, and Feijun Jiang. 2023b.  Bootstrap latent representations for multi-modal recommendation. In _Proceedings of the ACM Web Conference 2023_. 845–854.


#  Ab initio calculations of spin-nonconserving exciton-phonon scattering in
monolayer transition metal dichalcogenides

Xiao-Wei Zhang  Department of Materials Science and Engineering, University of
Washington, Seattle, WA 98195, USA  Ting Cao  Department of Materials Science
and Engineering, University of Washington, Seattle, WA 98195, USA
<EMAIL_ADDRESS>
###### Abstract

We investigate the spin-nonconserving relaxation channel of excitons by their
couplings with phonons in two-dimensional transition metal dichalcogenides
using ab initio approaches. Combining GW-Bethe-Salpeter equation method and
density functional perturbation theory, we calculate the electron-phonon and
exciton-phonon coupling matrix elements for the spin-flip scattering in
monolayer WSe${}_{\text{2}}$, and further analyze the microscopic mechanisms
influencing these scattering strengths. We find that phonons could produce
effective in-plane magnetic fields which flip spin of excitons, giving rise to
relaxation channels complimentary to the spin-conserving relaxation. Finally,
we calculate temperature-dependent spin-flip exciton-phonon relaxation times.
Our method and analysis can be generalized to study other two-dimensional
materials and would stimulate experimental measurements of spin-flip exciton
relaxation dynamics.

  * December 2021

##  1 Introduction

The ability to manipulate, maintain, and detect the spin of electronic
quasiparticles not only has the potential to revolutionize traditional
electronics, but also holds great promise in many novel applications, such as
quantum computing and quantum information science [1, 2, 3, 4, 5]. One central
focus of spintronics research is to understand the interplay between the spin
degree of freedom and the environment where spin-carrying quasiparticles live
in, which may result in shorter spin lifetimes and dephasing period [1]. The
interactions between the spin-carrying quasiparticles and phonon bath, namely
the spin-phonon scattering, is one contributing factor for the spin-
environment coupling. The spin-phonon scattering, an intrinsic process in
materials, plays a vital role in spin relaxation, especially dominant at
ambient temperatures [1].

In recent years, two-dimensional (2D) materials such as monolayer transition
metal dichalcogenides (TMD) are emerging as a fertile platform for studies of
spin- and valley-tronics thanks to the existence of strong spin-orbit coupling
(SOC) that locks to the emergent valley degree of freedom [6, 7, 8, 9, 10, 11,
12, 13, 14, 15]. Furthermore, in contrast to conventional bulk semiconductors,
these 2D semiconductors have large exciton binding energies in the order of
hundreds of meV due to the reduced dielectric screening and quantum
confinement effects that are absent in the bulk [10]. As such, these excitons
have been considered to host robust valley-spin indices that can encode
information. Nevertheless, spin-flip processes are often found in exciton
relaxation and recombination dynamics for monolayer TMD [16, 17]. It is
therefore intriguing to study spin-flip exciton relaxation dynamics in
monolayer TMD, which would quantitatively assess the robustness of the
excitonic valley-spin indices.

Theoretical calculations of spin-phonon scattering from ab initio have made
important progress in recent years [18, 19, 20, 21, 22]. However,
incorporating the two-body excitonic interactions in phonon-mediated spin
relaxation calculations is largely unexplored and challenging [23]. The reason
lies in the difficulties to accurately calculate the exciton states and the
electron-phonon coupling (EPC) matrix elements. The former problem can be
solved by the Bethe-Salpeter equation (BSE) approach [24] with very dense
sampling of the Brillouin zone (BZ), especially for 2D materials [25, 26].
Since phonons can contribute a large quasi-momentum, the finite-momentum BSE
is also needed to describe the final exciton states, despite that the initial
states, i.e., the optically excited excitons, have zero momentum. In addition,
SOC is crucial to correctly describe phonon-assisted spin-flip scattering.
These factors further increase the complexity of the problem. Conceptually, it
is also important to understand the microscopic mechanisms of spin-flip
relaxation since phonon doesn’t explicitly carry a spin index.

In this article, we apply the theoretical method implemented previously by the
authors [22] to investigate the spin-flip exciton relaxation dynamics in 2D
semiconductors. We calculate the EPC and exciton-phonon matrix element (ExPC)
matrix elements fully based on ab inito method, in which SOC has been
incorporated in the calculations to allow phonon flipping of electron spin. By
analyzing the phonon-mode dependence of EPC matrix elements, we obtain the
conditions of spin-flip scattering. Finally, we give the intra-valley and
inter-valley exciton-phonon scattering times.

##  2 Theory and calculation details

Figure 1: (a) Schematic crystal structure of monolayer WSe${}_{\text{2}}$ in
top view. (b) The first Brillouin zone corresponding to the unit cell in (a).
(c)-(d) Low-energy intra-valley and inter-valley excitons, having the hole in
the valence band valleys and the bound electron in the conduction band
valleys. The orange and blue colors denote the spin-up and spin-down bands,
respectively.

For the completeness of this work, we summarize the general formalism of
exciton-phonon coupling in the current section. Adopting the second-
quantization representation, the exciton-phonon coupled Hamiltonian to the
lowest order is written as [27]:

$\displaystyle H=$
$\displaystyle\sum_{S\bm{Q}}\Omega_{S\bm{Q}}a_{S\bm{Q}}^{\dagger}a_{S\bm{Q}}+\sum_{\bm{q}\nu}\left(b_{\bm{q}\nu}^{\dagger}b_{\bm{q}\nu}+\frac{1}{2}\right)\hbar\omega_{\bm{q}\nu}$
$\displaystyle+\sum_{SS^{\prime}\bm{Q}}\sum_{\bm{q}\nu}G_{S^{\prime}S}\left(\bm{Q},\bm{q}\nu\right)a^{\dagger}_{S^{\prime}\bm{Q}+\bm{q}}a_{S\bm{Q}}\left(b_{\bm{q}\nu}+b_{-\bm{q}\nu}^{\dagger}\right).$
(1)

On the right-hand side, the first term approximate excitonic subsystem as a
non-interacting bosonic system using the exciton creation and annihilation
operators, $a_{S\bm{Q}}^{\dagger}$ and $a_{S\bm{Q}}$. $\Omega_{S\bm{Q}}$ is
the energy eigenvalue corresponding to the exicton state with the center-of-
mass momentum being $\bm{Q}$ and band index being $S$. $\Omega_{S\bm{Q}}$ can
be obtained by solving the following BSE [28]:

$\displaystyle(E_{c\bm{k}+\bm{Q}}-E_{v\bm{k}})A^{S}_{v\bm{k},c\bm{k}+\bm{Q}}$
$\displaystyle+\sum_{v^{\prime}c^{\prime}\bm{k}^{\prime}}K_{vc\bm{k},v^{\prime}c^{\prime}\bm{k}^{\prime}}(\bm{Q})$
$\displaystyle\times
A^{S}_{v^{\prime}\bm{k}^{\prime},c^{\prime}\bm{k}^{\prime}+\bm{Q}}=\Omega_{S\bm{Q}}A^{S}_{v\bm{k},c\bm{k}+\bm{Q}}.$
(2)

The eigen function, exciton envelope function, of Eq. (2) is the expansion
coefficient of exciton wave function in the electron-hole basis,

$\chi_{S}^{\bm{Q}}(\bm{r}_{e},\bm{r}_{h})=\sum_{vc\bm{k}}A_{v\bm{k},c\bm{k}+\bm{Q}}^{S}\psi_{c\bm{k}+\bm{Q}}(\bm{r}_{e})\phi_{v\bm{k}}^{\ast}(\bm{r}_{h}),$
(3)

where $\psi_{c\bm{k}+\bm{Q}}$ ($\phi_{v\bm{k}}$) is the Bloch wavefunction of
the electron (hole). The second term of Eq. (1) treats the nuclear subsystem
as a non-interacting phonon system under harmonic approximation, where
$b_{\bm{q}\nu}$ ($b_{\bm{q}\nu}^{\dagger}$) is the phonon annihilation
(creation) operator and $\omega_{\bm{q}\nu}$ is the vibration frequency.

The third term of Eq. (1) is the first-order ExPC Hamiltonian. In the
perturbative limit, the ExPC coupling matrix element can be written as [27]:

$\displaystyle G_{S^{\prime}S}(\bm{Q},\bm{q}\nu)$
$\displaystyle=\sum_{\bm{k}vcc^{\prime}}A_{v\bm{k},c^{\prime}\bm{k}+\bm{Q}+\bm{q}}^{S^{\prime}\ast}A_{v\bm{k},c\bm{k}+\bm{Q}}^{S}g_{c^{\prime}c}(\bm{k}+\bm{Q},\bm{q}\nu)$
$\displaystyle-\sum_{\bm{k}cvv^{\prime}}$ $\displaystyle
A_{v\bm{k}-\bm{q},c\bm{k}+\bm{Q}}^{S^{\prime}\ast}A_{v^{\prime}\bm{k},c\bm{k}+\bm{Q}}^{S}g_{v^{\prime}v}(\bm{k}-\bm{q},\bm{q}\nu).$
(4)

Here, $g_{mn}(\bm{k},\bm{q}\nu)=\langle
m\bm{k}+\bm{q}|\delta_{\bm{q}\nu}V^{\text{scf}}|n\bm{k}\rangle$ is the EPC
matrix element for the electron or hole of the exciton. To capture the spin-
flip scattering, it is crucial to include SOC when calculating the EPC matrix
elements. Specifically, the initial and final Bloch states should both be two-
component spinors, and the self-consistent potential of density functional
theory, $V^{\text{scf}}$, should be a two-by-two matrix in the spin subspace
[29, 30],

$V_{\alpha\beta}^{\text{scf}}=V^{\text{ext}}_{\alpha\beta}+V^{\text{H}}_{\alpha\beta}\delta_{\alpha\beta}+V^{\text{xc}}_{\alpha\beta}.$
(5)

Here, $\alpha$ and $\beta$ denotes spin coordinates. The external potential
includes the usual electron-nucleus attractive Coulomb interaction and SOC,

$V^{\text{ext}}=-\sum_{J}\frac{Z_{J}e^{2}}{|\bm{r}-\bm{R}_{J}|}\cdot\bm{I}+\mu_{B}\bm{B}^{\text{eff}}\cdot\bm{\sigma},$
(6)

where $\sigma$ is the Pauli matrix and $\bm{B}_{\text{eff}}$ is the magnetic
field operator produced by the relative motion between the electron and
nucleus, $\bm{B}^{\text{eff}}=\frac{\bm{E}\times\bm{p}}{2mc^{2}}$. For non-
magnetic materials and collinear magnetization, the exchange-correlation
potential is diagonal in the spin space. So for our system, i.e. monolayer
TMD, the external potential is the source for spin-flip EPC and ExPC. More
specifically, since the first term of Eq. 6 is diagonal in spin, changes to
the off-diagonal terms of the second term by nuclear vibrations is crucial to
make spin-flip scattering happen. In pseudopotential-based density functional
theory calculations, SOC is usually included by using fully-relativistic
pseudopotentials, which give equivalent descriptions to Eq. 6.

After defining the ExPC matrix elements, the phonon-mediated exciton
transition rate from an exciton with band index $S$ and center-of-mass
momentum $\bm{Q}$ to other exciton states is calculated as [27],

$\displaystyle\Gamma_{S\bm{Q}}=\frac{2\pi}{\hbar}\frac{1}{N_{q}}\sum_{S^{\prime}\bm{q}\nu}$
$\displaystyle|G_{S^{\prime}S}(\bm{Q},\bm{q}\nu)|^{2}[(n_{\bm{q}\nu}+1)$
$\displaystyle\times\delta(\Omega_{S\bm{Q}}-\Omega_{S^{\prime}\bm{Q}+\bm{q}}-\hbar\omega_{\bm{q}\nu})$
$\displaystyle+n_{\bm{q}\nu}\delta(\Omega_{S\bm{Q}}-\Omega_{S^{\prime}\bm{Q}+\bm{q}}+\hbar\omega_{\bm{q}\nu})],$
(7)

where $n_{\bm{q}\nu}$ is the Bose distribution function of phonons. The first
term in the square bracket is the phonon-emission process and the second term
is the phonon-absorption process. The scattering time is the inverse of the
transition rate.

We choose monolayer WSe${}_{\text{2}}$, a typical monolayer TMD, to
investigate the spin-flip exciton relaxation dynamics. The absence of
inversion symmetry and the presence of three-fold rotation symmetry endow
monolayer WSe${}_{\text{2}}$ with valley-selective optical properties,
allowing for optical access of the valley-spin polarized excitons [6, 7]. In
addition, this material has multiple nearly degenerate valleys at K and
$\Lambda$ that hold rich spin physics [31, 32, 33, 34]. All density-functional
theory calculations in this work are performed using the Quantum Espresso
package [35]. The exchange-correlation potential is treated by the Perdew-
Burke-Ernzerhof functional revised for solids (PBEsol) [36]. The ionic
potential is treated with the optimized norm-conserving Vanderbilt
pseudopotential (ONCVPSP) [37]. The fully-relativistic ONCVPSP pseudopotential
is used to describe SOC. The phonon band structures and EPC matrix elements
are calculated based on the density-functional perturbation theory [38] using
the Quantum Espresso package [35]. SOC is included for the calculations of EPC
matrix elements. The $GW$-BSE calculations are performed to obtain exciton
eigenvalues and wave functions using the BerkeleyGW package [28, 39]. We adopt
the effective mass approximation for the exciton energies in Eq. 7 to
calculate the phonon-mediated exciton relaxation times. Interested readers may
refer to Ref. [22] for more details of the method and calculations.

##  3 Results and discussions

Figure 2: (a) The modulus square of calculated ExPC matrix elements,
$|G|^{2}$, for the scattering from KK$\uparrow\downarrow$ 1$s$ to
K$\Lambda\uparrow\uparrow$ 1$s$ and the modulus squares of the EPC matrix
elements ($|g|^{2}$) for the scattering between conduction-band edges of
K$\downarrow$ and $\Lambda\uparrow$. The conventions for the exciton valley-
spin labels are included in the main text. (b) Top-view schematics of a
vibration snapshot for phonon mode 4 . (c) and (d) are top-view and side-view
schematics of a vibration snapshot for phonon mode 5, respectively. W atom (Se
atom) is illustrated as a blue (red or orange) ball.

Due to the presence of in-plane reflection symmetry of the material, the
crystal electric field that generates the effective magnetic field
($\bm{B}_{\text{eff}}$ in Eq. 6) in the reference frame of the electron mainly
aligns in the 2D plane. Since the electron’s momentum also lies in the 2D
plane, the effective internal magnetic field for a Bloch electron is out-of-
plane and opposite at its time-reversal k-pair. As seen in Figs. 1c-d, a K (or
$\Lambda$) electron has the opposite spin polarization to that of a degenerate
electron at K′ (or $\Lambda^{\prime}$). Therefore, a prerequisite to spin-flip
events among these valleys is to generate effective in-plane magnetic field by
phonons. In fact, external in-plane magnetic field has been used to brighten
the dark exciton in monolayer WSe${}_{\text{2}}$ by coupling of the dark and
bright excitons [40]. Using the formula of the effective magnetic field, we
show that those phonon modes break the in-plane reflection and induce out-of-
plane crystal electric field that plays important roles in the spin-flip EPC
and ExPC.

Fig. 2a shows the phonon-mode dependent spin-flip scattering strengths from K
to $\Lambda$ valleys, i.e., KK$\uparrow\downarrow\rightarrow$
K$\Lambda\uparrow\uparrow$ exciton scattering and K$\downarrow\rightarrow$
$\Lambda\uparrow$ electron scattering, both corresponding to the electron spin
flipping events but ignoring excitonic effects in the latter. Here we label
the excitons by the valley-spin configuration of their constituent electron
and hole, with that of the hole on the left. We denote the nine phonon modes
by their energy-ascending order for the subsequent discussions. We find that,
the phonon mode 5 (Fig. 2c,d) leads to strong scattering in both electron-
phonon and exciton-phonon levels of calculations, whereas other phonon modes
have relatively weak scattering strength. By comparing the ExPC and EPC matrix
elements, we conclude that the two scattering strengths are similar in size,
which is consistent with the ExPC selection rules developed in Ref. [22]—
under the conditions that the initial and final exciton states are both 1$s$
states, and the EPC matrix elements of a particular phonon mode is large, the
ExPC strengths should be similar to the EPC one.

The variation of EPC matrix elements among different phonon modes can be first
analyzed from group theory. The group of wave vector at K is $C_{3h}$ and at
$\Lambda$ is $C_{1h}$. Considering the transformation of out-of-plane spin
under in-plane reflection $\hat{\sigma}_{h}$, the allowed phonon modes for
spin-flip scattering should be anti-symmetric under $\hat{\sigma}_{h}$, such
as modes 4 and 5 schematically shown in Figs. 2b-d. This mode dependence can
also be understood by considering the physics picture that symmetric phonon
modes under $\hat{\sigma}_{h}$ don’t produce a net out-of-plane electric
field.

Another striking feature in Fig. 2 is that very different scattering matrix
elements appear even among the symmetry-allowed phonon modes. Taking the
close-in-energy modes 4 and 5 for example, although they are both symmetry-
allowed based on group theory analyses, the scattering strengths of mode 4 are
nearly zero while the strengths of mode 5 are $\sim$ 600 meV2, showing a two
orders of magnitude variation. We note that the physical origin behind this
observation is complicated, involving the phonon-mode dependent variation of
potential (i.e., $\delta V^{\text{scf}}$), as well as the orbitals of the
initial and final Bloch states. To account for large spin-flip EPC matrix
elements, the variation of potential should strongly couple the electron’s
orbitals of the initial states and those of the final states. At K (K′), the
conduction bands are mainly contributed by the $d_{0}$ orbital of W atom [33,
41, 42, 43]. In comparison, at $\Lambda$ ($\Lambda^{\prime}$), the conduction
bands are mainly contributed by the $d_{+2}$($d_{-2}$) orbital of W atom [33,
41, 42, 43]. As such, the potential variation arising from mode 4 generates
weaker coupling between the initial states at K and final states at $\Lambda$,
compared with that from mode 5. Besides, comparing Fig. 2b with Fig. 2c, mode
5 can produce larger out-of-plane electric field changes than mode 4. In Fig.
2b, although the top Se atoms vibrate in opposition to the bottom Se atoms,
the electric field variation felt by the electron around the W atom is largely
canceled out. In contrast, in Fig. 2c, although the electric field variation
produced by the top Se atoms is also canceled out, the bottom Se atoms can
produce sizable out-of-plane electric field variation.

Figure 3: a) The modulus square of calculated ExPC matrix elements, $|G|^{2}$,
for the scattering from KK$\uparrow\uparrow$ 1$s$ to
K$\Lambda^{\prime}\uparrow\downarrow$ 1$s$ and the modulus squares of the EPC
matrix elements ($|g|^{2}$) for the scattering between conduction-band edges
of K$\uparrow$ and $\Lambda^{\prime}\downarrow$. (b) Top-view schematics of a
vibration snapshot for mode 4. (c) and (d) are top-view and side-view
schematics of a vibration snapshot for mode 5, respectively.

Fig. 3a shows the phonon-mode dependent spin-flip ExPC matrix elements for the
scattering from KK$\uparrow\uparrow$ 1$s$ to
K$\Lambda^{\prime}\uparrow\downarrow$ 1$s$ and spin-flip EPC matrix elements
of K$\uparrow\rightarrow\Lambda^{\prime}\downarrow$ electron scattering.
Similar to the scattering between K and $\Lambda$ valleys, the allowed phonon
modes need to be anti-symmetric under $\hat{\sigma}_{h}$, such as modes 4 and
5 schematically shown in Figs. 3b-d. At first glance, we would expect the
K$\uparrow\rightarrow\Lambda^{\prime}\downarrow$ electron scattering to hold
similar patterns as the K$\downarrow\rightarrow\Lambda\uparrow$ ones. However,
compared with K$\downarrow\rightarrow\Lambda\uparrow$, the spin-flip
scattering strengths for K$\uparrow\rightarrow\Lambda^{\prime}\downarrow$ are
significantly smaller, due to the different vibrational patterns and induced
potential variation. For example, the strengths of modes 4 and 5 are $\sim$ 60
meV2, much smaller than that of mode 5 in Fig. 2.

Figure 4: The modulus square of calculated ExPC matrix elements, $|G|^{2}$,
for the scattering from KK$\uparrow\uparrow$ 1$s$ to KK$\uparrow\downarrow$
1$s$ and the modulus squares of the EPC matrix elements ($|g|^{2}$) for the
scattering between the two conduction-band edges at K valley. The top-view
vibration schematic of $E^{\prime\prime}$ gamma-point phonon mode is shown in
the inset.  Figure 5:  Schematics of vibration snapshots for two K phonon
modes which are allowed for K$\rightarrow$K′ intervalley spin-flip scattering
based on group theory analysis:(a) ${}^{2}E^{\prime\prime}$ mode scatters
K$\uparrow$ to K${}^{\prime}\downarrow$;(b)$A^{\prime\prime}$ mode scatters
K$\downarrow$ to K${}^{\prime}\uparrow$.  Figure 6: Calculated temperature-
dependent scattering times of KK 1$s$ excitons for the (a) spin-conserving and
(b) spin-flip relaxation pathways. The experimental value from Ref. [32] at 90
K is marked by a star.

Fig. 4 shows the phonon-mode dependence of like-to-unlike spin exciton
scattering strength, KK$\uparrow\uparrow$ 1$s$ to KK$\uparrow\downarrow$ 1$s$,
and spin-flip electron scattering strengths, K$\uparrow$ to K$\downarrow$. The
allowed phonon modes are $E^{\prime\prime}$ modes under group theory analysis.
$E^{\prime\prime}$ modes have two degenerate phonon eigenvectors, which means
there exists a two-component subspace that allows for arbitrary linear
combinations of vibrational modes. Under the linearly-polarized phonon
vibrations, the spin-flip scattering strengths of these two modes are near 90
meV2, which is similar to the previously calculated frozen-phonon result,
$\sim$ 83 meV2 [16]. Under the chiral vibrations, these two modes can be
transformed to one right-circularly polarized mode and one left-circularly
polarized mode. As a result, the presence of three-fold rotation symmetry
requires that the right-circularly polarized can scatter K$\uparrow$ to
K$\downarrow$ while the left-circularly polarized mode can scatter
K$\downarrow$ to K$\uparrow$ [16, 44]. Specifically, the scattering strengths
of these two chiral phonon modes are near 180 meV2 (twice the case of each
linear polarization) or zero. As shown in the inset of Fig. 4, the circular
vibration of phonon can shift the center of the Se-atom triangle and produce
out-of-plane electric field. Note that although $E^{\prime\prime}$ phonon
modes only vibrate in the plane, the spin-flip scattering strength is sizable,
which is different from the case of spin-flip scattering between K and
$\Lambda$ or $\Lambda^{\prime}$ valleys. This is likely due to the fact that
the initial and final states share almost the same spatial part of
wavefunction, making it easier to have larger EPC matrix elements under
phonon-induced potential variation [16]. Noticing that the spin-flip
scattering between K and K′ valleys seems to be similar to the intra-valley
scattering at K, because K phonon modes also have three-fold rotation symmetry
and K′ valley is also composed of $d_{0}$-orbital. According to group theory
analyses, among K phonon modes, only ${}^{2}E^{\prime\prime}$ mode can scatter
K$\uparrow$ to K${}^{\prime}\downarrow$ and only $A^{\prime\prime}$ mode can
scatter K$\downarrow$ to K${}^{\prime}\uparrow$. Fig. 5 shows the vibration
schematics of these two modes. However, the scattering strengths of these two
modes are almost zero. This seems to be surprising since $A^{\prime\prime}$ K
phonon mode, for example, also has a similar circular vibration as
$E^{\prime\prime}$ $\Gamma$ phonon mode. The reason behind this difference
lies in the destructive interference of the potential generated by K phonon
modes. For $A^{\prime\prime}$ K phonon mode, due to the 120-degree phase
difference between the motions of the three Se atoms around the W atom, the
center of the Se-atom triangle actually doesn’t change with time, which
implies the absence of net out-of-plane electric field. Similarly, for
${}^{2}E^{\prime\prime}$ K phonon mode, although the top Se atoms vibrate
along the the same direction as the bottom Se atoms, the out-of-plane electric
field variation is almost canceled out.

Finally, we give spin-flip exciton-phonon relaxation times as shown in Fig.
6b. The temperature dependence is caused by the phonon occupation number (see
Eq. 7). These spin-flip relaxation pathways are crucial to understanding the
exciton relaxation dynamics of monolayer WSe2. Taking the KK$\uparrow\uparrow$
1$s$ exciton as the initial state, there are mainly five inter-valley
relaxation pathways, three spin-conserving: (1)
KK$\uparrow\uparrow\rightarrow$K$\Lambda$ $\uparrow\uparrow$; (2)
KK$\uparrow\uparrow\rightarrow$K$\Lambda^{\prime}$ $\uparrow\downarrow$; (3)
KK$\uparrow\uparrow\rightarrow$KK′ $\uparrow\uparrow$; and two spin-flip: (4)
KK$\uparrow\uparrow\rightarrow$KK$\uparrow\downarrow\rightarrow$K$\Lambda\uparrow\uparrow$;
(5) KK$\uparrow\uparrow\rightarrow$
KK$\uparrow\downarrow\rightarrow$K$\Lambda^{\prime}\uparrow\downarrow$. Since
the two spin-flip channels both involve the intra-valley spin-flip process,
the comparison of scattering times between this process,
KK$\uparrow\uparrow\rightarrow$KK$\uparrow\downarrow$, and the three spin-
conserving ones is important for complete understanding of the scattering
mechanism of the experiment in Ref. [32]. Therefore, we also show the spin-
conserving exciton-phonon relaxation times in Fig. 6a, which has been partly
discussed in Ref. [22]. Unfortunately, there have been no direct experimental
measurements of spin-flip exciton relaxation times for monolayer WSe2. We hope
our full ab initio study can stimulate more experimental studies of exciton
relaxation dynamics for TMD.

##  4 Conclusions

In conclusion, we performed systematic ab initio calculations of EPC and ExPC
of spin-flip scattering in monolayer WSe${}_{\text{2}}$. Based on the
mechanism of Ising-type SOC in the valleys, we propose that the effective in-
plane internal magnetic field induced by phonon vibration is a prerequisite to
flip spin. In addition, we find sensitive phonon-mode dependence for the spin-
flip scattering rate at both EPC and ExPC levels of calculations. Finally, the
calculated temperature-dependent exciton relaxation times can be directly
compared with future experiments. Considering the similarity of electronic
structure and phonon vibration among the TMD family, these numerical results
can shed light on the microscopic origin for spin-flip exciton relaxation
dynamics of other TMD systems.

## Acknowledgments

This research was primarily supported by NSF through the University of
Washington Materials Research Science and Engineering Center DMR-1719797.

## References

## References

  * [1] Žutić I, Fabian J and Sarma S D 2004 Rev. Mod. Phys. 76 323
  * [2] Wolf S A, Chtchelkanova A Y and Treger D M 2006 IBM J. Res. Dev. 50 101–110
  * [3] Hirohata A, Yamada K, Nakatani Y, Prejbeanu I L, Diény B, Pirro P and Hillebrands B 2020 J. Magn. Magn. Mater. 509 166711
  * [4] Loss D and DiVincenzo D P 1998 Phys. Rev. A 57 120
  * [5] Awschalom D D, Hanson R, Wrachtrup J and Zhou B B 2018 Nat. Photonics 12 516–527
  * [6] Xiao D, Liu G B, Feng W, Xu X and Yao W 2012 Phys. Rev. Lett. 108 196802
  * [7] Cao T, Wang G, Han W, Ye H, Zhu C, Shi J, Niu Q, Tan P, Wang E, Liu B et al. 2012 Nat. Commun. 3 887
  * [8] Xiao J, Zhao M, Wang Y and Zhang X 2017 Nanophotonics 6 1309–1328
  * [9] Manzeli S, Ovchinnikov D, Pasquier D, Yazyev O V and Kis A 2017 Nat. Rev. Mater. 2 17033
  * [10] Wang G, Chernikov A, Glazov M M, Heinz T F, Marie X, Amand T and Urbaszek B 2018 Rev. Mod. Phys. 90 021001
  * [11] Xu X, Yao W, Xiao D and Heinz T F 2014 Nat. Phys. 10 343–350
  * [12] Mak K F, Xiao D and Shan J 2018 Nat. Photonics 12 451–460
  * [13] Ahn E C 2020 npj 2D Mater. Appl. 4 1–14
  * [14] Schaibley J R, Yu H, Clark G, Rivera P, Ross J S, Seyler K L, Yao W and Xu X 2016 Nat. Rev. Mat. 1 1–15
  * [15] Avsar A, Ochoa H, Guinea F, Özyilmaz B, Van Wees B and Vera-Marun I J 2020 Rev. Mod. Phys. 92 021003
  * [16] Li Z, Wang T, Jin C, Lu Z, Lian Z, Meng Y, Blei M, Gao S, Taniguchi T, Watanabe K et al. 2019 Nat. Commun. 10 2469
  * [17] Tang Y, Mak K F and Shan J 2019 Nat. Commun. 10 4047
  * [18] Heid R, Bohnen K P, Sklyadneva I Y and Chulkov E 2010 Phys. Rev. B 81 174527
  * [19] Restrepo O D and Windl W 2012 Phys. Rev. Lett. 109 166604
  * [20] Giustino F 2017 Rev. Mod. Phys. 89 015003
  * [21] Xu J, Habib A, Kumar S, Wu F, Sundararaman R and Ping Y 2020 Nat. Commun. 11 2780
  * [22] Zhang X W, Xie K, Wang E G, Cao T and Li X Z 2021 arXiv:2110.08873
  * [23] Antonius G and Louie S G 2017 arXiv:1705.04245
  * [24] Strinati G 1988 Riv. Nuovo Cimento 11 1–86
  * [25] Qiu D Y, Felipe H and Louie S G 2013 Phys. Rev. Lett. 111 216805
  * [26] Qiu D Y, Felipe H and Louie S G 2016 Phys. Rev. B 93 235435
  * [27] Knox R S 1963 Theory of excitons (Academic Press)
  * [28] Rohlfing M and Louie S G 2000 Phys. Rev. B 62 4927
  * [29] Von Barth U and Hedin L 1972 J. Phys. C: Solid State Phys 5 1629
  * [30] Fernández-Seivane L, Oliveira M A, Sanvito S and Ferrer J 2006 J. Phys.: Condens. Matter 18 7999
  * [31] Hsu W T, Lu L S, Wang D, Huang J K, Li M Y, Chang T R, Chou Y C, Juang Z Y, Jeng H T, Li L J et al. 2017 Nat. Commun. 8 929
  * [32] Madéo J, Man M K, Sahoo C, Campbell M, Pareek V, Wong E L, Al-Mahboob A, Chan N S, Karmakar A, Mariserla B M K et al. 2020 Science 370 1199–1204
  * [33] Zhu Z Y, Cheng Y C and Schwingenschlögl U 2011 Phys. Rev. B 84 153402
  * [34] Nguyen P V, Teutsch N C, Wilson N P, Kahn J, Xia X, Graham A J, Kandyba V, Giampietri A, Barinov A, Constantinescu G C et al. 2019 Nature 572 220–223
  * [35] Giannozzi P, Baroni S, Bonini N, Calandra M, Car R, Cavazzoni C, Ceresoli D, Chiarotti G L, Cococcioni M, Dabo I et al. 2009 J. Phys.: Condens. Matter 21 395502
  * [36] Perdew J P, Ruzsinszky A, Csonka G I, Vydrov O A, Scuseria G E, Constantin L A, Zhou X and Burke K 2008 Phys. Rev. Lett. 100 136406
  * [37] Hamann D 2013 Phys. Rev. B 88 085117
  * [38] Baroni S, De Gironcoli S, Dal Corso A and Giannozzi P 2001 Rev. Mod. Phys. 73 515
  * [39] Deslippe J, Samsonidze G, Strubbe D A, Jain M, Cohen M L and Louie S G 2012 Comput. Phys. Commun. 183 1269–1289
  * [40] Zhang X X, Cao T, Lu Z, Lin Y C, Zhang F, Wang Y, Li Z, Hone J C, Robinson J A, Smirnov D et al. 2017 Nat. Nanotechnol. 12 883–888
  * [41] Liu G B, Xiao D, Yao Y, Xu X and Yao W 2015 Chem. Soc. Rev. 44 2643–2663
  * [42] Feng W, Yao Y, Zhu W, Zhou J, Yao W and Xiao D 2012 Phys. Rev. B 86 165108
  * [43] Kośmider K, González J W and Fernández-Rossier J 2013 Phys. Rev. B 88 245436
  * [44] Liu E, van Baren J, Taniguchi T, Watanabe K, Chang Y C and Lui C H 2019 Phys. Rev. Res. 1 032007