Paper Title: MIMNet: Multi-Interest Meta Network with Multi-Granularity Target-Guided Attention for Cross-domain Recommendation.

















[correspondingauthor]Corresponding author 

1]Xiaofei Zhu
zxf@cqut.edu.cn 

1]Yabo Yin  
yinyabo@stu.cqut.edu.cn

2]Li Wangcorrespondingauthor
wangli@tyut.edu.cn 



 

[1]College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, China
[2]College of Data Science, Taiyuan University of Technology, 
Shanxi Jinzhong 030600, China 










§ ABSTRACT 
Cross-domain recommendation (CDR) plays a critical role in alleviating the sparsity and cold-start problem and substantially boosting the performance of recommender systems. Existing CDR methods prefer to either learn a common preference bridge shared by all users or  a personalized preference bridge tailored for each user to transfer user preference from the source domain to the target domain. Although these methods significantly improve the recommendation performance, there are still some limitations. First, these methods usually assume a user only has  a unique interest, while ignoring the fact that a user may interact with items with different interest preferences. Second,  they learn transformed preference representation mainly relies on the source domain signals, while neglecting the rich information available in the target domain. To handle these issues, in this paper, we propose a novel method named Multi-interest Meta Network with Multi-granularity Target-guided Attention (MIMNet) for cross-domain recommendation. To be specific, we employ  the capsule network to learn user multiple interests in the source domain, which will be fed into a meta network to generate multiple interest-level preference bridges. Then, we transfer user representations from the source domain to the target domain based on these multi-interest bridges. In addition, we introduce both fine-grained and coarse-grained target signals to aggregate user transformed interest-level representations by incorporating a novel multi-granularity target-guided attention network. 
We conduct extensive experimental results on three real-world CDR tasks, and the results show that our proposed approach MIMNet consistently outperforms all baseline methods. The source code  of MIMNet is released at https://github.com/marqu22/MIMNet.





















Cross-Domain Recommendation Capsule Network Multi-Interest Meta Network Multi-Granularity












MIMNet: Multi-Interest Meta Network with Multi-Granularity Target-Guided Attention for Cross-domain Recommendation 
    [
    August 5, 2024
===================================================================================================================





§ INTRODUCTION









































Recommender systems <cit.> have become indispensable in shaping the modern user experience in a wide range of domains, deeply influencing user choices in short videos <cit.>, e-commerce <cit.>, online recruitment <cit.>, and etc.

Existing methods have been proven to be effective when rich user-item interaction information is available. However, in real application scenarios, some users would have few interaction information, especially for newly joined users (cold-start users),  which significantly hinders the application of a recommender system. 
To solve the problem, cross-domain recommendation (CDR) has been proposed, which attempts to alleviate the sparsity and cold-start issue in the target domain by exploiting knowledge from the source domain. 
Learning a common bridge to transfer user preference from the source domain to the target domain is a  general CDR strategy <cit.>. 
For example, EMCDR <cit.> employs a two-phase strategy to enhance the recommendation performance in the target domain. Specifically, it first encodes user embeddings in the source and target domains, respectively.  Then, a common bridge  between the source and target domains is learned based on overlapping users for aligning user embeddings in the two domains. 
Since the number of overlapping users would be inadequate for learning a high quality common bridge, some research works <cit.>  have been proposed to handle this issue. SSCDR <cit.> applies a semi-supervised framework to learn a common bridge between different domains and capture the neighboring information of users. 
TMCDR <cit.> employs the meta learning technique to learn a common bridge between the source and target domains. However, user preference usually varies from one to one, applying a common bridge to transfer user preference would lead to inferior performance. To the end, PTUPCDR <cit.> proposes to transfer user preference by using personalized bridges which take users' characteristic embeddings in the source domain as input. 


Despite the significant successes achieved by these methods, they suffer from two limitations. First, these methods neglect the complexity of user interests where a user may have diverse interests rather than a unique interest. As depicted in Figure <ref>, a user has different interest preferences on movie styles, ranging from Comedy and Anime to Action & Adventure and Science Fiction. However, the rich information of these interest-level preferences  are largely ignored by existing methods, as they either assume all users share a common preference transference or each user takes a specific preference transference.
Although there are a few efforts <cit.> trying to model users' multiple interests, they heavily rely on external knowledge such as item category and brand, which
may not available in real applications. 
Second, existing state-of-the-art methods learn transformed preference representation mainly based on information from the source domain. However, the rich information from the target domain is not well explored to guide the learning process.  
 
 
 
  
  

To handle the above-mentioned issues, in this paper, we propose a multi-interest meta network with multi-granularity target-guided attention for cross-domain recommendation, termed MIMNet. To be specific, we first leverage the capsule network with dynamic routing to decouple user multiple interest representations. Then, the learned representations will be fed into a multi-interest meta network to generate a multi-interest preference bridge, which attempts to transfer user  preference from the source domain to the target domain in an interest-level manner. 
In addition, we   develop a multi-granularity target-guided attention network, which aims to incorporate fine-grained and coarse-grained target guidance to facilitate the aggregation of user representations with diverse interest preferences.
To guide the adaptive aggregation process of user preference in the target domain, the fine-grained guidance leverages the target item-level signal, while the coarse-grained guidance relies on the target prototype-level signal. 

We carry out extensive experiments on three real-world different CDR tasks to evaluate the effectiveness of our proposed approach. The results demonstrate that MIMNet is consistently superior to existing state-of-the-art baselines. The relative performance improvements of MIMNet  over  the best-performing baseline, REMIT <cit.>, on Task1 are 14.45%, 16.36%, 12.86% when β equals to 20%, 50%, 80%  in terms of the MAE metric. Similar performance improvements can be observed on the other two Tasks. The main contributions of our work are summarized as:

    
  * We propose a novel multi-interest meta network  to  decouple users' multiple interests, and generating multi-interest bridges to transfer user embeddings from the source domain to the target domain.
    
  * By exploring both fine-grained and coarse-grained target signals, we develop a multi-granularity target-guided attention network to adaptively guide the aggregation process of user representations with different interest preferences in the target domain. 
    
  * We conduct extensive experiments on  three CDR tasks to validate the effectiveness of MIMNet. Experimental results show that our proposed  method outperforms state-of-the-art baseline methods.  








  




§ RELATED WORK




















   






























Cross-domain recommendation (CDR) attempts to alleviate the data sparsity and cold-start issue in the recommendation system by transferring user preference from the source domain to the target domain. It has substantially boosted the performance of recommendation and attracted the increasing attention of researchers. 


In early years, CMF <cit.> conducts cross-domain information transfer by applying matrix factorization across multiple domains to mitigate the problem of data sparsity in the target domain. 

CST <cit.> utilizes coordinate systems for knowledge transfer between the source and target domains based on a principled matrix-based transfer learning framework.   

Recently, some researchers leverage deep learning-based CDR methods to model the collaborative relationships between different domains. 

CoNet <cit.> assumes the hidden layers between two domains are connected, and proposes to apply dual knowledge transfer instead of one direction knowledge transfer.

MINDTL <cit.> facilitates  the modeling of user preference in the target domain by extracting and transferring rating patterns from the source domain. 

DDTCDR <cit.> develops a latent orthogonal mapping to preserve  relations  between users among different domains, and transfers knowledge between these domains in an iterative way. 

  

Other researchers attempt to learn a common bridge to transfer user preference between the two domains.

EMCDR <cit.>   models user preference in the source and target domains respectively, and then  learns a common bridge between the two domains based on the overlapping users. 
DCDCSR <cit.> extends EMCDR by employing MF to learn user and item embeddings, and incorporates the fine-grained sparsity degrees of users and items to combine the learned embeddings. Due to the limited number of overlapping users, the two methods may result in unsatisfying performance. 

To handle this issue, SSCDR <cit.> employs a semi-supervised framework to learn a common bridge between different domains and capture the neighboring information of users. 

TMCDR <cit.> applies the meta learning technique to handle the above-mentioned issue based on its strong generalization ability. It develops a transfer-meta framework for CDR by learning a common bridge, which consists of two stages, i.e., a transfer stage and a meta stage. The former stage trains a source model and a target model on the source and target domains respectively, and the latter stage transforms user preference from the source domain to the target domain based on the common bridge.


Since the preference transition patterns of different users  between the source and target domains may vary considerably, PTUPCDR <cit.>  utilizes a personalized bridge  rather than a single common bridge to transfer each user's specific preference. It generates  a personalized bridge for each user by taking their characteristic representations in the source domain as input of the bridge.
REMIT <cit.> proposes to extract users' multiple interests in the source domain based on different meta-path based aggregations, and utilizes a reinforcement learning framework to aggregate transformed interests. 
The main difference between our method and REMIT are three aspects. 
First, REMIT relies on external knowledge such as item category and brand, which may not available in real applications, to construct different meta-paths. In contrast, our method only utilizes user interaction data which is more efficient and practical. 

Second,  REMIT learns transformed preference representation by mainly leveraging the information in the source domain while neglecting the rich information from the target domain.  
However, our method proposes to introduce informative target signal to guide the aggregation of user transformed representations. 

Third, REMIT transfers user preference based on  a single granularity which overlooks the inherent multiple granularity property of user preference. Different from REMIT, our method  develops both fine-grained and coarse-grained target guidance to facilitate the
multi-granularity transformation of user diverse preferences. 

 

  



§ PROBLEM FORMULATION

In the task of cross-domain recommendation (CDR), there are two domains, i.e., a source domain 𝒟_s and a target domain 𝒟_t. In each domain, we have a user set 𝒰={u_1, u_2, ⋯, u_m},  an item set 𝒱={v_1, v_2, ⋯, v_n}  and a rating matrix ℛ, where r_ij∈ℛ indicates  there is an interaction between user u_i and item v_j, m and n are the number of users and items, respectively.
We use 𝒰^s, 𝒱^s, and ℛ^s to denote the user set, the item set, and the rating matrix in the source domain. Similarly, we use 𝒰^t, 𝒱^t and ℛ^t for the target domain. 
The overlapping users between the source and target domains are defined as 𝒰^o = 𝒰^s ∩𝒰^t. It is worth noting that there are no shared items between the two domains, i.e., 𝒱^s ∩𝒱^t = ∅. The cold-start users denoted by 𝒰^c={u|u∈𝒰^s  u ∉𝒰^t} are those who have interactions with items in the source domain while having no interactions with items in the target domain. 
For each user u_i^s∈𝒰^s in the source domain, we denote S_u_i={v_1^s,v_2^s 
…,v_n_i^s}  as her corresponding  interacted items, where n_i and v_j^s are the number of interacted items and the j-th  interacted item  of u_i^s. 
We can transform the users and items into dense vectors, also called embeddings, with the latent factor model <cit.>. In this paper, we use u_i^*∈ℝ^d  and v_j^*∈ℝ^d to denote the embeddings of the user u_i^* and the item v_j^* respectively, where d denotes the dimensionality of the embedding and *∈{s,t} represents the label of domain. Specifically, we use  S_u_i={v_1^s,v_2^s,…,v_n_i^s} to denote the embeddings of interacted items of  u_i^s in the source domain, where  v_j^s∈ℝ^d is the  embedding of the interacted item v_j^s.   


The goal of CDR is to improve the performance of recommendations in the target domain by leveraging rich information from the source domain. 




§ OUR PROPOSED MODEL
 
The architecture of our proposed MIMNet model is illustrated in Figure <ref>, which contains three components, i.e., interest representation learning,  multi-interest meta network, and multi-granularity target-guided attention network.


 
 


 §.§ Interest Representation Learning
 

In this sub-section, we attempt to learn the multiple interest representations of a user based on her sequential interaction items.
Inspired by <cit.>, we utilize dynamic routing in capsule network to  extract users' multiple interests. To make this paper self-contained, we briefly revisit the key basics of dynamic routing.
In dynamic routing, there are two layers of capsules, including  low-level capsules and high-level capsules. The two layers of capsules are updated in an iterative way, and the final high-level capsules are considered as user extracted interests.

Given the embeddings of sequential interaction items for a user u in source domain, i.e., S_u = {v^s_1,v^s_2,⋯,v^s_n}. Our goal is to extract K interests e_k ∈ℝ^h, k ∈{1,⋯,K}.
In each iterative process of dynamic routing, we first compute the routing logit score b_jk as:

    b_jk =(e_k)^TM v^s_j,

where the M∈ℝ^h × d denotes the transformation matrix to be learned.
With the routing logit scores calculated, the candidate vector  z_k for the k-th high-level capsule  is computed as a weighted sum of all low-level capsules. The calculation of z_k is as follows:

    z_k = ∑_j=1^nw_jkM v^s_j,


    w_jk = exp(b_jk)/∑_m=1^Kexp(b_jm),

where w_jk denotes the contribution weight score of the j-th low-level capsule to the k-th high-level capsule. 

Then, a non-linear “squash” function is used to ensure high-level capsule vectors are in an appropriate range:

    𝐞_k=squash(𝐳_k)=z_k^2/1+z_k^2z_k/z_k,

where e_k is the k-th interest we want to extract from the sequential interaction items in the source domain. For each user u_i, K interest vectors can be extracted from the interaction sequence in the source domain through the dynamic routing method. We combine these K interest vectors into a matrix E_u_i = [e_1,⋯ ,e_K] ∈ℝ^K × d to serve the downstream task. 




 §.§ Multi-Interest Meta Network





The core of cross-domain recommendation task is to transfer user  preference from the source domain to the target domain. To the end, existing methods either learn a common preference bridge shared by all users <cit.> or leverage a personalized preference bridge tailored for each user <cit.>. Although these methods have achieved promising performance, they neglect the fact that a user may interact with items with multiple interest preferences. To solve this issue, we resort to leverage interest-level preference bridges. Specifically, we first employ a meta network that takes the multiple interests E_u_i  of a user in the source domain as input to generate multi-interest bridges, and then we transfer her embeddings from the source domain  to the target domain based on the multi-interest bridges. 
Formally, we have:

    W_u_i = g(E_u_i ; ϕ),

where g(·) is a two layer feed-forward neural network, ϕ are learnable parameters, and W_u_i = [ w_1,u_i,⋯,w_K,u_i ]∈ℝ^K× d^2  with w_k,u_i∈ℝ^d^2 be the generated parameter vector for the k-th interest bridge. 
We reshape w_k,u_i into a matrix w̃_k,u_i∈ℝ^d × d, and formulate the k-th interest bridge for user u_i as:

    f(· ; w̃_k,u_i).

, where f(·) is a linear layer. With the  learned multi-interest bridges for u_i, we can obtain her corresponding transformed representations in the target domain. Formally,  we have:

    u̅^t_k,i = f(u^s_i ; w_k,u_i),

where u^s_i ∈ℝ^d denotes the embedding of user u_i in the source domain, and u̅^t_k,i∈ℝ^d denotes the transformed embedding of user u_i in the target domain based on the k-th interest bridge. As there are K transformed embeddings for u_i, i.e., {u̅^t_k,i}_k=1^K, we combine these transformed embeddings into a matrix U̅^t_i =[u̅^t_1,i,...,u̅^t_K,i]∈ℝ^d × K, which will be utilized to obtain the final representation for u_i in the target domain.



 §.§ Multi-Granularity Target-Guided Attention Network

Since the transformed embeddings U̅^t_i of u_i  solely rely on information from the perspective of source domain, while the informative signals from the target domain are ignored. 
As information from the target domain also plays a critical role in guiding the transfer process, we further propose to take into account the target signal to learn better transformed user representation in the target domain. Specifically, we develop a novel multi-granularity target-guided attention network which leverages both fine-grained and coarse-grained target signals to aggregate a user's different interest preferences.



  §.§.§ Fine-grained Target-Guided Attention


In this module, we attempt to aggregate the learned multiple interest preferences U̅^t_i into a transformed user embedding û^t_i with the guidance of fine-grained target signals, i.e., the information from each candidate item v^t_j in the target domain. Let v^t_j be the corresponding embedding of v^t_j. We employ attention mechanism to aggregate the multiple interest preferences  U̅^t_i, where we take v^t_j as the query, and U̅^t_i as both keys and values. The fine-grained target-guided aggregation is defined as follows:

    û^t_i    = Attention(v^t_j, U̅^t_i,U̅^t_i)
 
where û^t_i ∈ℝ^d is the user's transformed embedding in the target domain from a fine-grained perspective.



  §.§.§ Coarse-grained Target-Guided Attention

 
To improve the generalization ability of our model, we propose to extract the prototype of each candidate item in the target domain and leverage it a coarse  signal to aggregate the learned multiple interest preferences U̅^t_i.  To be specific, we employ K-Means[https://github.com/facebookresearch/faiss] to cluster items in the target domain  and utilize the center of each cluster as the  prototype of  items in the cluster. Denote p^t_j ∈ℝ^d as the corresponding prototype of v^t_j, and we define the process of the coarse-grained target-guided aggregation as follows: 

    ǔ^t_i = Attention(p^t_j, U̅^t_i,U̅^t_i),
 
where ǔ^t_i ∈ℝ^d denotes the user's embeddings generated by the prototype of the candidate item guide.

After we obtain û^t_i and ǔ^t_i, we introduce an adaptive fusion module to generate the final  user's embeddings ũ^t_i in the target domain, which is defined as:

    ũ^t_i    = α·û^t_i + (1-α)  ·ǔ^t_i,
    α   = σ(MLP([v^t_j ; p^t_j])),
 
where [;] denotes the concatenation operator and σ is a sigmoid(·) function.



 §.§ Prediction and Model Optimization



  §.§.§ Prediction.
 Given a candidate item v_j, we first utilize the target-guided adaptive fusion module to derive the user's embedding ũ^t.
Then, we apply the inner product to calculate the user u_i ratings for the candidate item v_j as follows:

    ŷ_ij = (ũ^t_i)^T  v_j,

where ŷ_ij denotes predicted rating of user u_i for candidate item v_j.


  §.§.§ Model Optimization.
 
The overall model consists of two stages of training, including a pre-training stage and a cross-domain training stage.

Pre-training stage:    
The goal of the pre-training step is to learn embeddings of   users and items in the source domain and target domain, respectively. As interactions of the overlapping users in the target domain are not visible in this stage, we use 
The loss function in the pre-training stage is formulated as:

    min_u,v1/| ℛ |∑ _r_ij∈  ℛ(r_ij-u_i^T v_j)^2 .


Cross-domain training stage:  
In the cross-domain recommendation stage, we employ the task-oriented training strategy, which utilizes the final rates  as the optimization goal rather than leverages user embeddings in the target domain for optimization.
Specifically, the loss function in the cross-domain training stage is formulated as:

    min _1/|ℛ_o^t|∑_r_i j∈ℛ_o^t(r_i j- ŷ_ij)^2.










§ EXPERIMENTS



In this section, we first conduct extensive experiments on three cross-domain tasks under different cold-start settings to evaluate the performance of our proposed method.
Then, we analyze the effect of using different base models for our method and investigate the effectiveness of the different numbers of interests on our model performance.
Finally, we analyze  the magnitude of the contribution of different modules in the model to the overall effect of the model by using ablation experiments and visualized the analysis.


 §.§ Experimental Settings










  §.§.§ Datasets
 
Amazon review dataset[http://jmcauley.ucsd.edu/data/amazon/] is a real-world public dataset which has been widely used in the task of CDR <cit.>.
Following most existing works, we utilize the Amazon dataset, in which users rated items in a range of 1-5.
Specifically, the dataset has 24 categories, in which 3 popular categories, i.e., movies_and_tv (Movie), cds_and_vinyl (Music), and books (Book), are selected for the experiments. Table <ref> shows the detailed statistics of the dataset. 




  §.§.§ Baselines
 
To evaluate the performance of our proposed approach, we choose the following baselines for comparison.
 
    
  * TGT <cit.>. TGT is a method which is trained by only leveraging information from the target domain.
    
  * CMF <cit.>. CMF is an extended version of MF, which enables cross-domain recommendation for users by sharing an embeddings for the same user in the source domain and target domain.
    
  * EMCDR <cit.>. EMCDR is a widely used cross-domain recommendation method. It applies a common bridge to transfer user preference from the source domain to the target domain. Specifically, it first utilizes  matrix factorization (MF) to learn user and item embeddings on each domain, and then adopts a common bridge to capture the underlying relationship between the two domains.
    
  * DCDCSR <cit.>. Similar to EMCDR, this method also employs MF to learn user and item embeddings. However, it combines the learned embeddings by introducing the fine-grained sparsity degrees of users and items which can utilize more rating data in both domains.
    
  * SSCDR <cit.>. It first models user-user similarities and user-item interactions, and then learns a common bridge between different domains with a semi-supervised framework. To infer the latent representations of the cold-start users, it further takes into account the neighbors of these users.
    
  * PTUPCDR <cit.>. Compared with previous works, PUTPCDR attempts to learn a personalized bridge rather than a common bridge  to transfer user preference.  To generate the personalized bridge, it learns a meta network based on users' characteristics in the source domain.
    
  * REMIT <cit.>. This method employs a heterogeneous information network and different meta-path based aggregations to extract user interests in the source domain. 
    A reinforcement learning-based strategy is then used to transfer and aggregate users' interests to the target domain. Different from existing methods, REMIT relies on additional knowledge such as categories and brands of items.
    






  §.§.§ Evaluation Metrics
 In the experiments, we adopt two metrics: Mean absolute Error (MAE) and Root Square Error (RMSE) to measure the performance of all methods following  <cit.>, which are widely used in the task of CDR.





  §.§.§ Implementation Details
 
We set the learning rate for the Adam optimizer to 0.01, the size of mini-batch to 512, and  the embedding dimension d to 10. The iteration number of dynamic route is fixed to 3, and the number of prototypes in the target domain is set to 100. 
To evaluate the performance of our proposed method, we randomly select some overlapping users and remove their ratings in the target domain, and regard these users as test users. The remaining overlapping users are utilized for training multi-interest bridges.  
Similar to <cit.>, we set the ratio of test users β to 20%, 50%, 80% of all overlapping users, respectively. 
For each task, the averaged results over five random runs are reported.




 §.§ Performance Comparison






  §.§.§ Overall
 In this section, we analyze the performance of our proposed MIMNet method  in different cold-start scenarios, and  the overall results of all comparing methods are shown in Table <ref>. We have the following key observations:


  * The performance of TGT  is the worst among all baseline methods. This is because  TGT solely relies on information from the target domain to conduct the recommendation while ignoring the rich information from the source domain. 
CMF demonstrates a superior performance as compared with TGT since it can leverage information from both source and target domains.  

  * Although CMF achieves a better performance, it simply combines information of different domains into a single domain which inevitably ignores the potential domain drift.
DCDCSR, SSCDR, and EMCDR outperform CMF as they further capture the potential domain drift by introducing a common bridge to transfer user embeddings from the source domain into the target domain. 

  * By comparing PTUPCDR with these common bridge based methods, we can observe a considerable performance improvement brought by learning personalized bridges for each user.
Among baseline methods, REMIT obtains the best performance since it employs multiple personalized bridges together with a RL-based bridge sector to select transformed interests. 

  * Our proposed method MIMNet yields the best performance  on all three tasks under all different cold-start settings. For example, the relative performance improvements with β=20% of MIMNet over the best performing baseline method REMIT on Task1  in terms of MAE and RMSE are 14.45% and 9.44%, respectively. 
The main reason are that MIMNet can  extract better user multiple interests based on the capsule network and the meta network. In addition, MIMNet incorporates a multi-granularity target-guided  aggregation module to effectively aggregate the learned multiple interests.





















 




















  §.§.§ Generalization Experiments 





Most of the previous  CDR methods <cit.> have predominantly focused on the design of the bridge for cross-domain mapping, utilizing relatively simple non-neural model (i.e., MF) as the base models.

However, considering the diversity of recommendation models, this simplification raises concerns about the  generalization capability of bridge-based CDR methods.

Following <cit.>, we introduce two additional neural network models, GMF <cit.> and YouTube DNN <cit.>, as foundational domain models to validate the effectiveness of our proposed MIMNet method.
We compare MIMNet with two bridge-based approaches, i.e., EMCDR and PTUPCDR, upon all the three base models. 

It is worth noting that we neglect the baseline REMIT as it relies on additional information such as item categories and brand names.
The experimental results are shown in Figure <ref> , and we can have the following observations: 



  * By replacing the base model MF with GMF, all three methods have shown considerable performance improvements. In addition, when we  employ  YouTube DNN as the base model, it will further boost the performance of all three methods. 


  * PTUPCDR generally outperforms EMCDR across the three different base models, indicating the superiority of using the personalized bridge over the  common bridge to transfer user preference.



  * Additionally, it can be observed that our proposed method MIMNet consistently exhibits the best performance across all base models, which reflects the generalization capability of MIMNet.


 

 §.§ Model Analysis







  §.§.§ Ablation Study 

To verify the effectiveness of the main components in MIMNet, we conduct an  additional ablation study in this section. Specifically, we consider the following variants of MIMNet for experiments:

    
  * MIMNet w/o multi: we utilize a single-interest meta network instead of a multi-interest meta network to  transfer user preference.
    
  * MIMNet w/o target: we remove the fine-grained target-guided aggregation module, where   
    the fine-grained signal based on each candidate items will be ignored.  
    
  * MIMNet w/o proto:  we remove the  coarse-grained target-guided aggregation module, where the coarse-grained signal based on the prototypes of each candidate item will be overlooked.
    
  * MIMNet w/o adapt: we replace the adaptive gating fusion module with a mean-pooling.    


As shown in Table <ref>, our proposed method MIMNet   obtains the best performance  compared to other variants. 

Specifically, replacing the multiple interests with only a single interest leads to a significant performance degradation. 
This experimental result illustrates that by decoupling multiple interests within user interaction sequences in source domain, a more enriched representation of user preference can be obtained compared to modeling methods based on a single interest.
Additionally, the fine-grained signal based on each candidate item in the target domain is beneficial for guiding the aggregation process of multiple transformed interest representations. Moreover, discarding the  coarse-grained target-guided aggregation module results in a considerable performance decay, indicating the necessity to introducing coarse-grained guided signals from the target domain.


At last, adaptively fusing the fine-grained and coarse-grained transformed embeddings of a user will obtain better performance  as compared to  the strategy of applying the mean-pooling.    
 






  §.§.§ Performance with Different Interaction Sparsity Degrees 








To investigate the performance of our proposed model under different interaction sparsity degrees, we group users in the source domain into four categories, i.e., (0,5], [6,10], [11,15], [16,20]. Figure <ref> illustrates the performance comparison between our proposed method (i.e., MIMNet) and two competitive baseline methods (i.e., EMCDR and PTUPCDR).   


We observe that our proposed method MIMNet consistently outperforms  the two baseline methods across all interaction sparsity degrees, which reveals the robustness of our model under different interaction sparsity degrees. 





In addition, the two baselines have their own performance advantages in different tasks, e.g., EMCDR outperforms PTUPCDR in Task1 and Task3, while it is inferior to PTUPCDR in Task2. This result is attributed to that Task2 have relatively larger number of overlapping users as compared to the other two tasks. 
In contrast, our model performs better than both EMCDR and PTUPCDR in all tasks, which indicates the superiority of MIMNet among different task scenarios.



Moreover, the results also show that the performance of our method improves gradually when the number of user interactions increases. This reflects that our model can effectively capture user preference  when more interaction behaviors are available. 
  







  §.§.§ Impact of Interest Numbers K
 
We investigate how the number of interests, denoted as K, affects the performance of MIMNet.
Specifically, we vary the interest number from 1 to 16 with a step size of 3. The results are presented in Figure <ref>. 
On the Task1, we can observe that the performance of MIMNet rises up first with the increment of K, and reaches a peak when K =7.  The performance keeps stable until K =13 and then it starts to decline  if K becomes larger. Similar trends can also be observed on  Task2 and Task3. 
This results demonstrate that our proposed model prefer a relative large number of interest, which means employing multiple interest bridges to transfer user preference will significantly boost model performance compared to adopting a single interest bridge.
Moreover, a larger K facilitates the extraction of effective user preference from different perspectives, leading to a more precise user modeling. However, a too high value of K may introduce unnecessary interests, potentially degrading the performance of our model.

  


  §.§.§ Visualization of Multiple Interest Weight Distributions
 




To verify the effectiveness of the learned multiple user interests, we visualize the weight distribution of different interests extracted via the capsule network. 
To simplify the analysis, we only extract three interests for a sampled user, whose interacted items together their corresponding categories are “Nuns on the Run (Comedy), The Ghost of Frankenstein (Horror), One In The Chamber (Action & Adventure), The Heiress (Drama), Mr Palfrey of Westminster (Drama), Homicide Life on the Street (A&E Home Video), .Hack//SIGN (Animation), Sports Night  (Comedy) and The Last Godfather  (Comedy)”. 
Figure <ref> illustrates the learned weight distribution of each item along these interests, where each row corresponds  an interest and each column corresponds an interacted item. 
We can observe that the items with same or close categories will have similar  interest weight distribution. For example, the first, eighth and ninth items which belong to the same category (i.e., “Comedy”), all assign most of their weights on the interest-1. Similar phenomenon can also be observed on the second and third items which have  close categories (e.g., “Horror" and “Action & Adventure”). 
The results indicate that our proposed model can effectively learn expressive   and decoupled multiple interests for a user, which subsequently benefits for guiding the transfer process.









  §.§.§ Convergence Study


We conduct experiments to compare the convergence speed of our proposed  model MIMNet with two baselines, i.e.,  PTUPCDR and EMCDR. Figure <ref> shows the convergence speed of different models on all three tasks with β= 20%. 


We observe that the personalized bridge-based model PTUPCDR converges faster than the common bridge-based model EMCDR, and it also consistently  yields better performance than EMCDR at each epoch. The results demonstrate the superiority of learning a specific bridge function for each user to transfer her preference.

In addition, among all three comparing models, our proposed model demonstrates the best convergence speed. It  requires  only 10 epochs to achieve the best performance, while the two baselines generally take around 20 epochs to converge. Moreover,  the performance of MIMNet is consistently better than that of the two baselines at each epoch. The main reason is that MIMNet facilitates  the transfer of user preference in a more fine-grained manner, which leverages multi-interest bridges to transfer user embeddings from the source domain to the target domain.









  §.§.§ Case Study.





To further illustrate why our proposed method outperforms the  state-of-the-art baselines, we present a case study of a real user from Task2 in Figure <ref>. 
We can observe that for the two items (i.e., Low Impact Step  and  The Gravedancers)  interacted by the user in the target domain (i.e., Movie), both EMCDR and PTUPCDR make incorrect predictions. In contrast, our proposed method MIMNet predicts the same ratings as the real ratings of the two items, respectively. The result is attributed to that the two baselines EMCDR and PTUPCDR both transfer user preference from the source domain to the target domain in a coarse-grained manner, while ignoring the fine-grained interest-level user preference. 
To be specific, in the source domain (i.e., Book), the user interacts with five items, including  50 Delicious and Nutritious Snacks (Healthy Diet),  Vegan Baking Classics (Healthy Diet), Without You (Thrillers & Suspense), Overcoming Overeating (Fitness & Dieting), Ripper (Horror & Thriller) . Both EMCDR and PTUPCDR consider all these items together with a common bridge or a personalized bridge to transfer the user preference. However, we can see that the first, second and fourth items in Figure <ref> indicate an interest in Healthy, while the remaining two items relate to an interest in Thriller. Therefore, it is more reasonable to transfer user preference  separately as conducted in MIMNet, which utilizes a multi-interest bridge  to transfer user diverse interests effectively.

 

 

 
 









§ CONCLUSION


This paper introduces MIMNet, a novel framework for solving the cold-start problem in cross-domain recommendation. 
To be specific, we propose a novel multi-interest meta network by utilizing dynamic routing mechanism to obtain users’ multiple interests, and generating
multi-interest bridges to transfer user embeddings from the source do-
main to the target domain. Moreover, we develop a multi-granularity target-guided attention network via exploring both fine-grained and coarse-grained target signal as a guidance of learning better transformed user representation in the target domain. 
Extensive experiments on three CDR tasks demonstrate that our proposed method can considerably outperform state-of-the-art baselines in cross-domain recommendation. Further studies verify the effectiveness of the compatibility and robustness of our model as well as the importance of each model component.


 



Acknowledgments: 
This work was supported by the National Natural Science Foundation of China [grant number 62141201];  the Natural Science Foundation of Chongqing, China [grant number CSTB2022NSCQ-MSX1672]; the Major Project of Science and Technology Research Program of Chongqing Education Commission of China [grant number KJZD-M202201102].


elsarticle-num 




 Paper Title: Adaptive Retrieval-Augmented Generation for Conversational Systems.


Confining modeling of quark propagator
    X.L. Shang
    August 5, 2024
======================================

§ ABSTRACT 
Despite the success of integrating large language models into the development of conversational systems, many studies have shown the effectiveness of retrieving and augmenting external knowledge for informative responses. Hence, many existing studies commonly assume the always need for Retrieval Augmented Generation (RAG) in a conversational system without explicit control. This raises a research question about such a necessity. In this study, we propose to investigate the need for each turn of system response to be augmented with external knowledge. In particular, by leveraging human judgements on the binary choice of adaptive augmentation, we develop RAGate, a gating model, which models conversation context and relevant inputs to predict if a conversational system requires RAG for improved responses. We conduct extensive experiments on devising and applying RAGate to conversational models and well-rounded analyses of different conversational scenarios. Our experimental results and analysis indicate the effective application of RAGate in RAG-based conversational systems in identifying system responses for appropriate RAG with high-quality responses and a high generation confidence. This study also identifies the correlation between the generation's confidence level and the relevance of the augmented knowledge.









§ INTRODUCTION




Recently, the advancement of Large Language Models (LLMs) has significantly improved conversational systems, enabling the generation of natural and high-quality responses <cit.>. Despite these advancements, recent studies have identified several limitations on the simple use of LLMs to address conversational tasks <cit.>. These limitations include the lack of up-to-date knowledge <cit.>, the generation of non-factual or hallucinated content <cit.>, and restricted domain adaptability <cit.>. These issues can hinder the development of conversational agents with satisfactory user experience. To address these identified challenges, a common approach is to retrieve and augment LLMs with external knowledge to enhance the conversational response, making them more accurate, reliable, and adaptable to different domains <cit.>. For example, <cit.> demonstrated that using a dense retrieval model (DPR) <cit.> to retrieve relevant knowledge for augmentation can significantly reduce the hallucination rate, according to a corresponding human evaluation. Similarly, <cit.> showed that leveraging a graph-structured knowledge base can boost the reasoning ability and domain generalisability of task-oriented conversational agents. These achievements of knowledge-augmented techniques highlight a promising direction for enhancing conversational agents and address the current limitations. 

However, while implementing retrieval augmentation to a conversational system for improved response, we question the necessity of knowledge augmentation for every turn of system responses. To develop effective human-computer conversations, it is essential to provide factual and relevant responses, offer appropriate amount of information, and not unnaturally drive and shift the conversation to non-relevant topics <cit.>. We argue that overusing external knowledge could result in system responses against these core criteria. Figure <ref> presents a conversation example that shows how the system response to a generic user utterance about suggesting activities can vary with and without augmented knowledge. The knowledge-augmented system response is being information conditioned with limited diversity and assuming specific user preferences.

In contrast, without the addition of external knowledge, the system response is more diverse and natural in this early stage of a conversation. This indicates that misusing external knowledge can lead to problematic system responses and a negative user experience. 

To address this, we investigate an adaptive retrieval-augmented generation solution for effective conversational systems. In particular, motivated by the gate function in long-short term memory models <cit.>, which explicitly controls the use of input and memory, we propose a binary knowledge gate mechanism, called RAGate, to manipulate the use of external knowledge for a conversational system. To model the conversation context and accurately estimate the need for augmentation, we leverage the human labels as ground truth and develop RAGate by exploring the use of recent advanced language models or constructing attention neural gate models. To validate the effectiveness of RAGate, we conduct extensive experiments on an annotated Task-Oriented Dialogue (TOD) system dataset, KETOD, that builds upon the SGD dataset with TOD-spanning
16 domains, such as Restaurant and Weather. The experimental results show that RAGate enables conversational systems to efficiently use external knowledge at appropriate conversation turns, producing high-quality system responses. In particular, by modelling the uncertainty and confidence level of the system – which correlates with the likelihood of hallucinated output <cit.> – we show that the "always" augmentation of external knowledge can significantly increase generation uncertainty and the risk of hallucination. After applying RAGate, we can effectively control the conversation system to make confident and informative responses. In addition, by varying the use of knowledge snippets in different relevance levels, we also observe the positive correlation between the calculated confidence score and the relevance of augmented knowledge, which can be valuable for many future studies. 





§ RELATED WORK

In the pipeline of knowledge-augmented generation for a conversation system, two main components are identified: the knowledge retriever and the response generator. Existing studies have improved conversational responses to different extents by improving one or both components <cit.>. 

Knowledge Retrieval: Several studies have explored the use of dense passage retrieval techniques <cit.> and public search service for effective retrievers <cit.>. For example, <cit.> retrieved Wikipedia passages through a database interface and then ranked them according to statistical relevance, calculated by TF-IDF, or semantic relevance as per cosine similarity. Similarly,  used a search engine API to retrieve relevant knowledge but first transformed the dialogue context into a natural search query using an encoder-decoder model before searching. 

Joint Optimisation of Retriever and Generator: On the other hand, another thread of research studies has explored joint optimisation approaches. For instance, <cit.> introduced a retriever-generator architecture that aims to improve the performance of Task-Oriented Dialogue (TOD) systems by using a dual-feedback mechanism. The retriever identifies relevant knowledge from a database, while the generator uses this information to create appropriate system responses. The feedback from the generator is further used as pseudo-labels to train the retriever to select pertinent information. <cit.> introduced a training method based on maximal marginal likelihood. This method jointly optimise a perceptive retriever and the response generation in a feedback loop. The proposed approach incorporates meta-knowledge, which guides the generator to improve the utilisation of knowledge and, consequently, the quality of the generated responses.  <cit.> further advance the retriever by proposing SUbgraph Retrieval-augmented GEneration (SURGE), which employed a graph neural network (GNN)-based context-relevant subgraph retriever. SURGE incorporates contrastive learning to optimise the latent representation space, ensuring that generated texts closely resemble the retrieved subgraphs. 


Despite the richness of existing retrieval-augmented generation techniques for conversational systems, they commonly hypothesise that every conversation turn needs external knowledge. However, the necessity of augmenting every turn of the conversation with external knowledge remains questionable. A relevant thread of work that aims to answer this question is the introduction of the knowledge-seeking turn detection task using the DSTC-9 dataset <cit.>, and the follow-up studies, such as <cit.>. However, this task is to identify the turns in conversations injected by human workers about knowledge enquiry instead of identifying the system responses that require knowledge augmentation for improvements. This research gap highlights the value and novelty of this study, which investigates the adaptive use of retrieval-augmented generation for advanced conversational systems.




















§ METHODOLOGY




 §.§ Problem Formulation

This study addresses the challenge of effectively identifying conversation turns that require augmentation of external knowledge. In particular, we aim to develop a gate mechanism that dynamically determines when to search for external knowledge to ensure natural, relevant and contextually appropriate responses. First, we define the task of user-system conversation. Let D = {d_1, d_2, ...,d_|D|} be a set of user-system dialogues, and each dialogue d comprises a sequence of interactions between users and systems (i.e., d = {u_0, s_0, u_1, s_1, ..., u_T, s_T}) with varying lengths. Here, u_t and s_t denote the user utterance and system response at the t-th turn, respectively. The conversational context up to turn t can be formulated by aggregating the previous user-system interactions, i.e., c_t = u_0, s_0, .., u_t. With this context information c_t, the conversation system can augment it with a list of retrieved external knowledge, e_t,k, where k represents the ranking cutoff for the retrieved knowledge. Hence, the binary gate mechanism proposed in this study, deciding the knowledge augmentation, can be formulated as f(c_t) = {0,1} or f(c_t, e_t,k) = {0,1} if the external knowledge e_t,k is considered. Then, the follow-up response generation function g(·) can be formulated as follows:

    g(·) = 
            g(c_t, e_t,k)    if f(c_t) or f(c_t, e_t,k)
    
            g(c_t)    otherwise.

Hence, by evaluating and estimating the necessity of augmenting with external knowledge, we dynamically update the conversational response generation accordingly.





 §.§ RAGate Gate Mechanism

To effectively estimate the need to use external knowledge and implement adaptive retrieval augmented generation for a conversation system, we introduce our proposed gate mechanism, RAGate, that uses the conversational context and, optionally, the retrieved external knowledge to predict the binary choice of using external knowledge. In particular, we explore three RAGate variants that are implemented by the use of Large Language Models (LLMs) with devised prompts, with parameter efficient fine-tuning (e.g., QLoRA <cit.>) and the construction of an end-to-end multi-head attention encoder. This exploration is motivated by the recent advancement of transformer-structured neural models in natural language processing. In Figure <ref>, we illustrate the application of RAGate and its three variants. We describe each of these three variants to clarify the use of RAGate:

RAGate-Prompt:  As denoted by <cit.>, a language model can effectively adapt to new tasks by using a natural language prompt that explains the process to address the tasks without extra training. Hence, we can formulate 
a gate function f(·) as f(y|c_t) = f(y|Θ, c_t, p), where Θ denotes the used language model with its pre-trained weights and p is the devised natural language prompt. Alternatively, if the retrieved knowledge is also involved in prediction, we have f(y|c_t) = f(y|Θ, c_t, e_t,k, p). Specifically, we explore two types of prompts: zero-shot and in-context learning. Zero-shot prompts describe the task that uses the conversational context and, optionally, the retrieved knowledge to generate a response with binary feedback. As for the in-context learning prompts, we augment the zero-shot prompts with illustrative examples. We show the set of prompts in Appendix <ref>. 

RAGate-PEFT:  Despite the high adaptability of the language model with devised prompts, we further explored the use of instruction tuning on language models with a parameter-efficient fine-tuning method (i.e., QLoRA <cit.>) to meet the goal of an effective gate function. QLoRA is built upon the known Low-rank Adapter (LoRA) <cit.>, which keeps the pre-trained weight matrix W_0 frozen and addresses the gradient updates of the weight matrix Δ W through low-rank approximation (i.e., Δ W = BA, where B and A are the result of lower-rank decomposition on Δ W). Hence, the forward pass during the model training can be updated from h = W_0x+Δ Wx to h = W_0x + BAx. QLoRA <cit.>, which is used in this study, further quantises the language model into a 4-bit NormalFloat data type and leverages the page-to-page transfer between the CPU and GPU to further avoid memory spikes. To implement RAGate-PEFT, we format the train data with devised instructions, joined with paired inputs and outputs for developing parameter-efficient fine-tuned large language models. In particular, we provide a set of instruction-input-output triples for model training. The input can vary with the provision of a set of available features. Apart from the use of the conversational context (contx), we also include the system response (resp), synthetic responses generated by the language model (syn-resp) due to the missing responses as input in the practical scenario, the name entities within the incoming responses (ner), retrieved knowledge (know) and the description of the knowledge source, e.g., the WikiHow website (source). By using various combinations of inputs and customising the corresponding instructions, we explore the effectiveness of the resulting learned language models that implement the RAGate-PEFT. 






RAGate-MHA:  Apart from the use of pre-trained language models and further fine-tuned language models, we also explore the introduction of a multi-head attention neural encoder to model the context as input and estimate the augmentation necessity (i.e., RAGate-MHA). Here, we describe the model structure of RAGate-MHA. At first, as denoted by <cit.>, the attention mechanism is formulated as the interaction between three objects, queries Q, keys K, and values V: Attention(Q,K,V) = softmax(QK^T/√(d_k))V. To estimate the necessity of augmentation, we fit the context and the retrieved knowledge into the roles of these three objects. Specifically, we include the setups of (1) using context only (contx) or (2) using the concatenated context and retrieved knowledge (contx ⊕ know) as queries, keys, and values, and (3) using the context as queries and interact with the retrieved knowledge as keys and values (contx × know).

Next, following <cit.> in the encoder construction of a transformer model, we encode the inputs via an input embedding layer into latent vectors and a position encoding layer to encode the order of tokens in the sequence. After that, we leverage the multi-head attention to learn attention weights on the inputs and then followed by a feed-forward network: 

    FFN(x) = max(0, xW_1 + b_1)W_2 + b_2

where W_1 and W_2 are two learned parameter matrics with two bias terms (b_1 and b_2). Both multi-head attention and feed-forward neural modules are followed by residual connection <cit.> and layer normalisation <cit.>. Unlike the introduction of another decoder module that addresses the sequence-to-sequence generation in <cit.>, we followed the encoder output with a linear projection module and a softmax function for our binary classification task.



§ MODEL TRAINING AND EVALUATION SETUPS

We evaluate the performance of introducing RAGate according to its binary classification performance and the effectiveness of the resulting response generation. Specifically, we use the KETOD dataset <cit.>, which has fully annotated 5,324 dialogues and 52,063 turns of conversations. In particular, it is associated with 33,761 knowledge snippets to be retrieved and augmented. In addition, KETOD was developed with human labels on turns of conversations (around 12.1% of turns) about the need for augmenting with retrieved knowledge snippets for a natural and informative system response. Hence, we use these human labels as natural ground truths when evaluating RAGate. It is worth indicating that many current knowledge-augmented conversational datasets often ground their conversations on the knowledge snippet, such as Wizard of Wikipedia <cit.> and CMU_DoG <cit.>, which makes them not a natural fit to be investigated in this study.



Due to the limited computational resource availability, we explore the use of Llama-v2-7B and Llama-v2-13B to implement RAGate-prompt and fine-tune Llama-v2-7B for RAGate-PEFT. We implement QLoRA using the PEFT library <cit.> and set the lower rank to 16. As discussed in Section <ref>, we have various input features to be combined for performance optimisation. We begin with the use of context only, then concatenate the context with the real response (contx-resp), with the synthetic response and recognised entities (contx-syn-resp-ner) and further extend with the use of retrieved knowledge (contx-syn-resp-ner-know) or the source of knowledge (contx-syn-resp-ner-source). Specifically, we retrieve the relevant knowledge by exploring the use of TF-IDF and a learned BERT ranker. We evaluate their performance with the classic Recall@1 and Recall@3 on the test collection. We use a shallow cutoff because we only use top-relevant knowledge snippets for augmentation. Table <ref> shows their retrieval performance. According to the leading performance of BERT-Ranker, we augment knowledge with its retrieved top 3 relevant knowledge snippets (i.e., k=3). Regarding the development of RAGate-MHA, we explore the combinations of 2 to 8 layers, 2 or 4 heads and the embedding size in [64, 128, 256] for the best classification accuracy. We report the precision, recall, F1, Area Under Curve (AUC) and the False Discovery Rate (FDR) as the main measures to show the classification effectiveness. 

Next, we further deploy the best-performing RAGate gate function to update the KETOD dialogue system <cit.>, which uses GPT-2 <cit.> as the backbone model. To highlight the effect of various augmentation setups, we use the context with the gold action without extra prediction as input to KETOD. Then, we compare the resulting performance to the KETOD model without knowledge augmentation and augmenting every system response as baselines. To report the response generation effectiveness, we report how close the response is to the ground truth via BLEU, ROUGE-1/2/L and BERTScores and the confidence score calculated by the minimum probabilities of individual tokens that compose the response.
As argued by <cit.>, this calculated confidence score can highly correlate with a language model's likelihood of generating hallucinated responses.  We trained our models and conducted the evaluations on one machine with one NVIDIA 4090 GPU. 




§ RESULTS AND ANALYSIS


























































 §.§ Augmentation Need Classification

First, we evaluate the classification accuracy of our developed RAGate gate methods for addressing the adaptive RAG to system responses. Table <ref> presents the classification performance of RAGate baselines while evaluated on the test collection of the KETOD dataset, which includes rich human labels on the use of RAG for response generation. As discussed in Section <ref>, we explore the development of RAGate with three variants: the use of LLM prompting (RAGate-Prompt), parameter-efficient fine-tuned LLMs (RAGate-PEFT), and a neural classifier with Multi-Head Attention structure (RAGate-MHA). 

RAGate performance with LLM prompting versus fine-tuning. By comparing the corresponding performance reported in Table <ref>, we observe that, on average, fine-tuning a Llama-2-7B with QLoRA (i.e., RAGate-PEFT) can significantly outperform RAGate-Prompt. For example, by looking at the RAG-PEFT with context-only input, without using extra input features and instruction updates, it can outperform all RAG-Prompt approaches by a big margin (e.g., 0.4082 versus the highest 0.1230 F1 scores). This reflects the difficulty of this adaptive knowledge augmentation task, which can not be properly addressed by prompting a general pre-trained language model. In particular, the use of larger language models and the in-context learning setup, which often result in improved performance <cit.>, can not guarantee the enhancement of models' classification accuracy regarding this classification task.

Regarding the performance of RAGate-PEFT approaches, by first examining the effect of using synthetic response and recognised name entities, we observe significantly improved precision (0.5203 to 0.6818) but with the cost of lower recall (0.3359 to 0.2321). In addition, when we add the retrieved knowledge to the input features for prediction, we observe a significant performance drop across all evaluated aspects.  This can be caused by the additional complexity introduced by the included retrieved knowledge snippets. Furthermore, we also explored the performance impact of naming the source of the knowledge snippet. We use wikiHow[https://www.wikihow.com] in this study, which can provide rich task instructions for offering informative task-oriented system response <cit.>. However, the fine-tuned model cannot reasonably connect the promised rich resource from the knowledge source and the prediction of augmentation necessity.  


RAGate Performance between fine-tuned LLM and MHA classifier. Next, by comparing the experimental results of RAGate-MHA and RAGate-PEFT in Table <ref>, we observe a wide-margin recall improvement using RAGate-MHA, reaching a minimum recall of 0.52, but with significantly lower precision accuracy. In Table <ref>, we also include the use of both the context and the initial system responses (i.e., MHA([contx, resp])) for additional insights. We can observe that a higher precision can be achieved but the use of response does not improve the recall performance. These results are consistent with the observed performance of RAGate-PEFT, which further encourages the use of a synthetic response due to the unavailability of a system response in a practical scenario. In addition, we also observe a similar performance drop when including the retrieved knowledge snippets for classification. Even though the RAGate-MHA model, using the interaction between context and retrieved knowledge snippets, can achieve the highest recall of 0.5835, it can not outperform the RAGate-MHA using context-only on other metrics. Hence, considering the similar F1 and AUC performance levels of RAGate-PEFT and RAGate-MHA leads to a trade-off balance between precision and recall for the two groups of approaches. To further evaluate the classification effectiveness of RAGate, in Appendix <ref>, we provide a detailed discussion of a conducted user study that explores whether RAGate can also assess the potential contribution of retrieved snippets when predicting the decision for retrieval augmentation.







 §.§ Adaptive Augmentation Analysis

In addition to the classification accuracy, we also compare the choice of human workers and RAGate approaches in augmenting specific turns. Specifically, we analyse the frequency of augmentation in different positions of conversations and different domains covered in the KETOD dataset. We use the RAGate-PEFT (contx-(syn-resp)-ner) with the highest precision and RAGate-MHA (MHA(contx)) with the best overall performance in the above analysis as representatives for comparison. Figure <ref> presents the frequency in different positions. Due to the unequal number of conversational turns, we use the ratio to indicate the relative position. According to the reported results in Figure <ref>, most human augmentation selections happen at the beginning of a conversation. This trend is also effectively captured by both RAGate approaches, especially RAGate-MHA. This can be caused by the reason that a conversation is semantically coherent, and once sufficient additional information is provided at the early stage, the value of knowledge augmentation to the later turns is naturally lower. 


On the other hand, Figure <ref> presents the augmentation frequency over different domains. We observe that system responses about certain domains are selected more often by humans than other domains, such as travel, hotels, trains, flights, service and rental cars, which require access to additional information to assist the suggestion-making, and the domains, like movies, music, media, events that often include entities require enriched description. By looking into the performance of RAGate-PEFT and RAGate-MHA, RAGate-MHA can make aligned selections for humans. However, the RAGate-PEFT does not guarantee the identification of appropriate augmentation use and often presents fewer augmentations, apart from the travel domain. Hence, by considering both position and domain augmentation frequency, we conclude that RAGate-MHA can outperform RAGate-MHA and effectively capture the trend of augmentation needs.  



















     	



































 §.§ RAGate for Response Generation

To evaluate the effect of adaptive RAG for a conversational system, we use RAGate-PEFT (contx-(syn-resp)-ner) with the highest precision and RAGate-MHA (MHA(contx)) with the best overall performance in the above analysis, to support the adaptive retrieval augmented conversational response generation. Table <ref> presents the results of applying RAGAte to the KETOD model for adaptive knowledge augmentation when evaluated on the KETOD dataset. We include four types of adaptive augmentation, namely the use of RAGate and comparison to the random selection with equal numbers of selections, human choice, and the commonly used "all" augmentation. 
In addition, to explore the effect of varied quality of knowledge snippets, we also extend the evaluation of using the top-3 knowledge snippets ranked by different retrievers (i.e., BERT-ranker and TF-IDF) and the use of knowledge snippets at the 1st and 5th rank according to the BERT-ranker. Due to the space limit, we first present the results of using BERT-ranker retrieved and top-1 relevant knowledge and top-1 relevant in Table <ref> and show the full results in the Appendix <ref>. 

At first, without adaptive knowledge augmentation, we compare the choice of response generation without augmentation and with "always" augmentation (i.e., No-Aug versus Aug-All). In Table <ref>, we observe that by augmenting a total of 4,964 system responses in the test collection, the conversational model can generate more informative and effective responses according to the reported scores of BLEU, ROUGE and BERTscore. This aligns with the reported effectiveness of RAG in many existing studies. However, we also identify a significant drop in the model's generation confidence level. As denoted by <cit.>, a lower confidence level can correlate with a higher chance of generating hallucinated responses, which could be caused by the unnecessary use of external knowledge. Hence, to investigate the effectiveness of adaptive knowledge augmentation, we examine the impact of using RAGate. According to the reported experimental results in Table <ref>, the adaptive augmented response generation with fewer knowledge snippets can indeed result in a higher confidence level than Aug-All. 

Moreover, comparing the performance between RAGate and random selections shows that, considering equal numbers (230 or 787 according to the classification with RAGate) of system responses for augmentation, RAGate can further result in a higher quality of generated response. RAGate-MHA even enables results that are comparable to Aug-All's response quality, with only 787 turn augmentations instead of all 4964 turns. Specifically, the use of RAGate-PEFT, which identifies 230 turns of system responses for knowledge augmentation, can even outperform the random baseline that augments 787 system response turns with improved response quality. 
Apart from the improved response quality, RAGate also enables the conversational model to maintain a high confidence level and ensure faithful responses. Indeed, using RAGate-MHA, which augments 787 system responses, only lowers the average confidence score by 0.36%, instead of the 1.65% when randomly selecting an equal number of turns to augment. 

In addition, considering the use of different quality and amount of knowledge snippets for augmentation, we also include the use of the most relevant knowledge snippet according to BERT-ranker in Table <ref>. We observe that the use of different amounts of knowledge snippets in different relevance levels has a marginal effect on this learned dialogue system. However, we observe a significant difference in the confidence level. We observe that using only the most relevant knowledge snippet enables the Aug-All to suffer less from a lower confidence level. In particular, the application of RAGate can even increase the confidence level of the conversation system in response generation. This indicates that the confidence score can also correlate with the quality of the augmented knowledge snippets. This observation is further validated using knowledge snippets with fifth-ranking positions by BERT-ranker and the use of TF-IDF ranker.  
We include the full experimental results in Table <ref> and attached in the Appendix. 
These observations indicate the value of adaptive system response augmentation via RAGate in generating high-quality outputs, ensuring faithful responses, and potentially saving retrieval costs. We also show the value of using confidence scores to reflect the contribution of RAG.



§ CONCLUSIONS

Our study investigates a core research question about whether retrieval-augmented generation is always useful to a conversational system. To answer this research question, we propose adaptive retrieval-augmented generation for conversational systems and introduce corresponding gate functions, RAGate, for explicit control. A comprehensive set of experiments and results show the RAGate approaches can effectively identify augmentation needs. In addition, RAGate can capture human preference by augmenting the beginning turns of conversations, and RAGate can further identify knowledge augmentation for assisting suggestion-making and enriching description. When applying RAGate to conversational systems, we observe that it can ensure comparable quality of generated responses and enable the system to increase generation confidence for faithful outputs, especially with the appropriate use of relevant knowledge snippets.













§ LIMITATIONS

There are three limitations of this study. At first, due to the main focus of examining the adaptive retrieval-augmented generation for a conversation system. We only consider a few examples of retrieval techniques (TF-IDF and BERT-ranker), which can be further extended to recent retrieval techniques, such as dense passage retrieval for additional insights. The second limitation is the missing use of larger language models, such as GPT-4, due to the shortage of computational resources. Including larger language models for conversational systems could introduce additional experimental insights. The third limitation is the shortage of appropriate conversational data for extensive evaluations. This is mainly caused by the recent development of the retrieval augmented generation technique and its application to conversational systems. Future research is encouraged to address this limitation. 












§ ETHICS STATEMENT

All experiments in this study were conducted using publicly available datasets and open-released language models, which do not contain any private information that could raise ethical concerns. 





























§ PROMPTS FOR RAGATE-PROMPT


In this section, we list the used prompts for the RAGate-Prompt gate mechanism. 

Zero-Shot Prompt: 

Below is an instruction that describes a task. Please respond with `True' or `False' only that appropriately completes the request. 

### Instruction: Analyse the conversational context so far. Generate an appropriate response. Consider the invovled entites. Estimate if augmenting the response with external knowledge is helpful with an output of `True' or `False' only.

### Input: [Converstion Context Input]

### Response: 




In-Context Learning Prompt: 

Below is an instruction that describes a task. Please respond with `True' or `False' only that appropriately completes the request. 

### Instruction: Analyse the conversational context so far. Generate an appropriate response. Consider the invovled entites. Estimate if augmenting the response with external knowledge is helpful with an output of `True' or `False' only.

### Example 1: USER: I'm planning a trip, can you help me look for a flight? SYSTEM: Which day are you planning to return and from which city? USER: I want to go from NYC the day after tomorrow and return on the 13th of this month. SYSTEM: Where would you like to go? USER: I want to go to Vancouver, BC. Can you look for a Premium Economy class ticket. SYSTEM: I found 1 flight for you. It is a Delta Airlines flight that takes off at 6 am and returns at 2:50 am. The price is $505. USER: What is the departure airport, and how many stops does the flight have?

### Response: True

### Example 2: USER: Get me bus tickets to a Cher event on March 6th. SYSTEM: How many to buy? USER: only one, please.

### Response: False

### Input: [Converstion Context Input]

### Response: 



§ IMPACT OF RETRIEVAL QUALITY ON ADAPTIVE RAG

To have a successful conversation model with a retrieval-augmented system, two main criteria must be met. One is identifying insufficient context, and the other is the quality of retrieved information <cit.>. A conversational model performs better when both criteria are satisfied. In our proposed approach, as shown in Table 2, we have already assessed whether our adaptive retrieval method can detect insufficient context. We further explored to determine whether our model can inherently estimate the quality of the retrieved snippets to address such insufficiency and, based on that, decide on the retrieval. Although we do not explicitly provide retrieved snippets to our model, retrieval comes with a corpus that includes potentially relevant knowledge snippets. Consequently, given a query and a retrieval collection, it can be estimated whether useful information for the query exists in the corpus to address the insufficient context. To investigate by following this direction, we randomly selected 50 samples from instances where our proposed approach (RAGate-MHA, the best-performing gate model) predicted using retrieval augmentation. We asked domain experts (co-authors) to score whether they thought the retrieved snippets in those scenarios could be useful to response generation. Users rated the snippets on a scale of 0-4, with scores of 3 or 4 indicating `useful' or `highly useful'. We found that in 54% of cases where the prediction was for augmentation, users also found the snippets useful. This indicates that our proposed approach can implicitly capture the potential for obtaining high-quality retrieval snippets. 




§ ADDITIONAL EXPERIMENTAL RESULTS ABOUT RAGATE FOR RESPONSE GENERATION
 
In Table <ref>, we include the complete experimental results of applying RAGate for adaptive retrieval-augmented system response generation. Specifically, explore the use of retrieved knowledge snippets to different extents of relevance. We include top-3 knowledge snippets retrieved by BERT-ranker and TF-IDF. In addition, we also explore the use of knowledge snippets in different ranking positions (rank 1 and 5) according to the BERT-ranker retriever. The experimental result shows that precisely using a suitable amount of relevant knowledge can generate a response with higher confidence (i.e., less is more). In addition, this observation also indicates the potential use of confidence levels to evaluate the quality of the augmented knowledge.




 Paper Title: RDP: Ranked Differential Privacy for Facial Feature Protection in Multiscale Sparsified Subspace.








RDP: Ranked Differential Privacy for Facial Feature Protection in Multiscale Sparsified Subspace

    Lu Ou
Hunan University

Changsha, China 

oulu9676@gmail.com
Shaolin Liao
IIT

Chicago, USA 

sliao5@iit.edu
Shihui Gao
Hunan University

Changsha, China 

gaoqaq@hnu.edu.cn
Guandong Huang
State Grid Hunan Materials Company

Changsha, China 

huanggd@hnu.edu.cn
Zheng Qin
Hunan University

Changsha, Country 

zqin@hnu.edu.cn







    
================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
With the widespread sharing of personal face images in applications' public databases, face recognition systems faces real threat of being breached by potential adversaries who are able to access users' face images and use them to intrude the face recognition systems. Although large amounts of work on facial features privacy preserving methods have been proposed, it is difficult to provide accurate and lightweight protection without degrading the visualization quality of face images, or data utility, under the constraint of given privacy budget for effective facial features protection. In this paper, we propose a novel privacy protection method in the multiscale sparsified feature subspaces to protect sensitive facial features, by taking care of the influence or weight ranked feature coefficients on the privacy budget, named “Ranked Differential Privacy (RDP)”. After the multiscale feature decomposition, the lightweight Laplacian noise is added to the dimension-reduced sparsified feature coefficients according to the geometric superposition method. Then, we  rigorously prove that the RDP satisfies ε_0-Differential Privacy. After that, the nonlinear Lagrange Multiplier (LM) method is formulated for the constraint optimization problem of maximizing the utility of the visualization quality protected face images with sanitizing noise, under a given facial features privacy budget ε_0. Then, two methods are proposed to solve the nonlinear LM problem and obtain the optimal noise scale parameters: 1) the analytical Normalization Approximation (NA) method where all Laplacian noise scale parameters are normalized to the average noise scale parameter, which is good for real-time online applications; and 2) the LM optimization Gradient Descent (LMGD) numerical method to obtain the nonlinear solution through iterative updating, which is better for more accurate offline applications. Experimental results on two real-world datasets show that our proposed RDP outperforms other state-of-the-art methods: at a privacy budget of ε_0 = 0.2, the PSNR (Peak Signal-to-Noise Ratio) of the RDP is about ∼ 10 dB higher than (10 times as high as) the highest PSNR of all compared methods.



data privacy, multiscale feature space, differential privacy, optimization, face image databases.




§ INTRODUCTION

In the current digital age, it has become a common phenomenon that individuals' static face images are shared in the public domain, such as Facebook, TikTok, Bing, <cit.>. Moreover, there are universal and unique facial features with rich biometric information in such images. Existing research shows that facial features provide a solid foundation for face recognition techniques, which plays a vital role in applications such as identity authentication and access control. For example, the face recognition payment function launched by Alipay allows a user to verify his/her identity through scanning his/her face <cit.>; and it is a common identity authentication application to filter permissions using face recognition techniques <cit.>. Unfortunately, adversaries could breach face recognition systems by public face images <cit.>. Then they can use smart devices without authorization <cit.>, access private systems <cit.>, and even obtain personal sensitive information <cit.>, . As shown in Figure <ref>, due to the sensitive facial features in face images, once adversaries have obtained original face images in public databases, they could breach face recognition-based systems. From that, it means that facial features in public face images not only threaten personal privacy, but also may lead to a series of security threats, including identity theft, fraud, and even political and economic espionage, which has been brought up by <cit.>. What's more, the privacy leakage even could be caused by only one face image <cit.>. All of that said, it is necessary to develop an effective facial feature protection method for public face images.


In existing research, cryptography  <cit.>, deep learning,   <cit.> and Differential Privacy (DP) <cit.> are adopted to protect facial features effectively. Although cryptography-based facial feature privacy protection methods can provide strong security, they usually focus on defending against untrusted models or servers, and have expensive computation and communication overhead. Besides, deep learning<cit.> needs to be trained on large-scale data sets to avoid privacy risks, which is more used for preserving face recognition accuracy while protecting facial features. Different from the previous two kinds of methods, DP <cit.> is a solid mathematical method that can adds sanitizing noise for protecting facial features while preserving not only the recognition accuracy but also the visual quality of images with released demand. What's more important is that DP can be optimized to maximize the data utility of face images after the sanitizing noise is added, under the constraint of a given privacy budget. Existing methods based on DP <cit.> may be more likely to add perturbation during the training process, which comes with a large training cost.



In this paper, the data utility is the visualization quality of the noisy face images after the sanitizing noise is added. And the sensitive information is the facial features in face images that could be used to breach some face recognition-based systems. For hiding sensitive information of each individual's face image and conserving the face image's data utility as far as possible, the perturbation on the facial features should be minimized while maximizing each noisy face image's visualization quality with a given privacy budget of this image's facial features. 

And then, we propose such an optimized facial features protection method based on Ranked Differential Privacy (RDP): first, we perform the dimension-reduced sparsification of each original face image for extracting the original facial details into the multiscale feature space and obtaning sparse feature coefficients; then, the influence or weight of each sparse feature coefficient on the total privacy budget ε_0 is evaluated; finally, sanitizing noise of the optimal scale parameters is added to the ranked significant sparse feature coefficients by taking care of the influence or weight of each coefficient. The main contributions of this paper are as follows,


  * First, after reducing the dimension of every face image and obtaining the sparse feature coefficients, the weights of each coefficient with respect to the facial feature vector and thus the total privacy budget ε_0 are evaluated and coefficients are ranked in accordance with such weights. Then, Lagrange Multiplier method (LM) constraint optimization problem is formulated: optimizing the Laplace scale parameters of all feature coefficients with significant influence or weights to  maximize the visualization quality of the privacy-preserving noisy face image, under the constraint of the total privacy budget ε_0 of the face image's facial features. 

  * What's more important, two methods are proposed to solve the nonlinear LM optimization problem: the Normalized Approximation (NA) analytical method by approximating all weighted Laplacian noise scale parameters to be identical, which is suitable for real-time online applications; and the iteratively updating LM optimization Gradient Descent (LMGD) numerical method, which is first proposed here to achieve more accurate offline applications.

  * Finally, experiments on two real-world datasets show that the facial features have been effectively protected, while the face image's visualization quality is only affected minimally, compared to other state-of-the-art DP methods.






§ RELATED WORK
































Due to facial features that existed in face images, the images could be used to breach face recognition-based systems <cit.>. There is the related research on facial features protections. Then, we will review the research in recent years. 

A lot of research focus on the problem of how to reserve the efficiency of face recognition while preventing facial features in face images from being leaking for individuals' sensitivity information or face images reconstruction: Chamikara   <cit.> proposed a face recognition privacy protection scheme based on Local Differential Privacy (LDP), which calculates by perturbing the eigenface and storing the perturbed data in a third-party server;  
Ding <cit.> proposed a model with dual channels and multiple monitors to prevent DeepFake anti-forensic attacks for protecting facial features; Meng   <cit.> proposed a face privacy framework, which extracts purified clusters locally through Differentially Private Local Clustering (DPLC) method and then encourages global consensus among clients to generate more discriminative features; Wang <cit.> considered that facial features used by face recognition system for authentication may be attempted to reconstruct the original face image and proposed a privacy protection method based on disturbing the mapping from facial features to the image; Lin <cit.> proposed a privacy-preserving facial features based on a new facial recognition paradigm with a feature compensation mechanism for the challenge between the public demand for powerful face recognition and the individuals' privacy concerns; Mi <cit.> proposed the MinusFace facial features privacy protection framework for enhancing facial feature protection by random channel shuffling and achieving high accuracy in face recognition system. Others are more interested in preventing adversaries from attacking face recognition models or servers storing the relevant data to obtain facial features: Wang and Leng <cit.> proposed an image encryption algorithm that combines face recognition and bitonal sequence technologies, and designed a row-column scrambling algorithm to scramble facial features through the characteristics of bitonal sequences; Zhuo <cit.> proposed an AdaBoost-based framework POR to protect the privacy of users' facial features and the service provider's key learning parameters in face recognition integrated deep learning algorithms. Additionally, there are some studies worrying about the facial features leaking of face images released to the public domain: Ji <cit.> proposed a frequency-domain DP protection method DCT-DP for facial features, with an optimized privacy budget allocation method based on the loss of the face recognition network; Croft <cit.> proposed a face obfuscation DP method for generating machine learning models, adding noise to the image pixel intensities for protecting facial features.

All above, such research effectively protects individuals' facial features in face images. However, most studies are more concerned with the trade-off between face recognition accuracy and faial features protection, or consider the unreliability of face recognition models and servers storing related data. In this paper we intend to solve how to release face images with high-quality visual effects on the premise of protecting facial features. Different from <cit.>, we attempt to propose a lightweight perturbation method with adding less noises than <cit.> in the face images not in training model.







§ PRELIMINARY WORK

In this section, we will formally define the notations involved in this paper and systematically introduce models and problem statement. 


At beginning, notations are shown in Table <ref>.







 §.§ Models and Problem Statement

Next, we will give the system and adversary models, and state the problem. 



  §.§.§ System Model
 
With a certain individual 𝔸, there exist a large amount of 𝔸's static face images represented as a dataset Ω in public, which can be unrestricted observed and obtained. F̅^( A̅) is the standard feature vector retrieved from 𝔸's standard face A̅, used for authentication in some systems with face recognition techniques. For any face image A visually similar to the individual 𝔸, its feature vector is F̅ in the same face recognition with F̅^( A̅). The system identifies that individual 𝔸 is passing and can pass through A, if F̅ and F̅^( A̅) meet the following matching criterion

    | F̅_̅i̅ - F̅_̅i̅^( A̅)| ≤R̅_i;  i = 1,2, ⋯ ,M_F,

where F̅_̅i̅ and F̅_̅i̅^( A̅)  are the i-th element of F̅ and F̅^( A̅) with M_F elements respectively, and R̅_i is the i-th element of the feature identification radius vector R̅ for this authentication system.











  §.§.§ Adversary and Problem






























In this paper, adversaries are considered to be interest to and have the ability to download the face image dataset Ω of the certain individual 𝔸 from the public domain, and they could obtain information about authentication devices equipped with the face recognition system, which are commonly used by individual 𝔸. Adversaries may attempt to breach these face recognition system by using such face image in Ω, and pass the authentication to obtain 𝔸's privacy information.

For the face image dataset Ω of 𝔸, our goal is to develop an accurate and lightweight facial feature vector privacy-preserving method f' that sanitizes these images by adding noise, A' = f'( A ) for any face image A  ∈Ω. Such method can ensure that the noisy face image A' can prevent adversaries from breaching face recognition-based systems while having extremely high visual quality, meaning that we aim to optimize the utility of the sanitized images by maximizing images' visualization quality while providing privacy protection for facial features in face images.












 §.§ Basic Concepts

In this section, we give some preliminary definitions for DP. 



First, in the terminology of DP, we define a pair of neighboring face images, expressed in Definition <ref>.















[Neighboring Face Images]
If a face image A belongs to the individual 𝔸 and a face image B does not belong to the individual 𝔸, The face images A and B are a pair of neighboring face images.



Then, the sensitivity of facial features is defined as Definition <ref>.

[The Sensitivity of Facial Features]
If ℳ is a measurement of face images' facial features, its sensitivity is 

    Δ_i = max_A,B||ℳ(A)-ℳ(B)||_1,

where A and B are neighboring face images. 




































Finally, the definition of DP is as follows.

[Differential Privacy]
There is a measurement ℳ' and a pair of neighboring face images A and B. ℳ' is ε_0-differentially private, if the following is satisfied



    |[  ℳ'(A)=F̅^']/[  ℳ'(B)=F̅^']|≤exp( ε _0).































§ RANKED DIFFERENTIAL PRIVACY

In this section, we introduce our proposed Ranked Differential Privacy (RDP) for facial features with weighted sparsified subspace. Overall, before releasing the original face images, the Laplacian noise is added to the face images' facial features by geometric superposition <cit.>, in order to hide the facial features which are used in the process of face recognition of the systems. 























 §.§ Perturbation in Multiscale Feature Space














As well known, the face images' facial features play a key role in the implementation of face recognition. So that they should be protected to avoid adversaries' breaching face recognition-based systems. From this, before the perturbation, facial features should be extracted in the multiscale feature space. Because of its efficiency and rigorous mathematical expression, Multi-level Haar Wavelet Transform (HWT) <cit.> is adopted to reduce the dimensionality of the face image's pixels and obtain accurate information about the face image's facial features, , the wavelet coefficient vector constructed by the sparsified wavelet coefficients is obtained. After the perturbation on the wavelet coefficient vectors, the noisy face image could be generated by the noisy pixels matrix P' obtained by the noisy coefficients vector C̅' through Multi-level Inverse Wavelet Transform (MIWT).

During the extraction and dimensionality reduction of facial features from face images, we use the first-order Taylor expansion to describe the liner mapping relationship between the noise of the sparse wavelet space and the noisy reduced dimensionality facial features of the noisy face image. After adding Laplacian noise to the elements of the original wavelet coefficient vectors C̅, we obtain the noisy feature vector F̅' by the noisy face image, which comes from the noisy wavelet coefficient vectors C̅' with MIWT.  According to the first-order Taylor formula for the i-th element F̅_i^' of the noisy facial feature vector F̅', we have

    F̅_i^'    = F̅_i + ∑_j = 1^M_P∂F̅_i^'/∂C̅_j^'·( C̅_j^'  - C̅_j)  + o( C̅'),

where 1 ≤ i ≤M_F; C̅_j^' is the j-th element of C̅', where M_P is the number of the elements of the face image's pixel matrix and 1 ≤ j ≤ M_P; and o( C̅') is the minimal term, , o( C̅') ∼ 0. Thus, we think

    F̅_i^'  = F̅_i + ∑_j = 1^M_P∂F̅_i^'/∂C̅_j^'·( C̅_j^'  - C̅_j)  .

Obviously, C̅_j^'  - C̅_j represents the noise of C̅_j^', and the noise of F̅_i^' can be considered as the weighted sum of elements' noises of C̅', with that the first-order partial derivative of F̅_i^' on C̅_j^' is regarded as the weight for the noise of C̅_j^'. 

Then, we define the weight w_ik for the i-th element F̅_i^' of the noisy feature vector F̅' and the k-th element C̅_k^ * ^' of the sorted noisy wavelet coefficient vector C̅^ * ^' as w_ik = ∂F̅_i^'/∂C̅_k^ * ^'. 

After this, with the perturbation provides by weighted sum, we can select the wavelet coefficients more significant in facial features from the sorted wavelet coefficients basis of such weights for adding Laplacian noises with appropriate scale to wavelet coefficients, thereby providing stricter privacy preservation. The definition of the noise mechanism based on such ranked wavelet coefficients is shown in Definition <ref>.

[Ranked Differential Privacy]
Given the geometric number K between 1 and M_P with mean 1/p, 0 < p < 1. For obtaining  the noisy wavelet coefficients vector C̅', K Laplacian noises with mean 0 are added to the top K of the wavelet coefficients vector C̅'s ranked elements of a face image, with ranking elements of C̅' by the corresponding weights between the noisy feature vector's elements and such elements. After the inverse transformation with MIWT of C̅', the noisy pixel matrix P' is obtained from C̅', and the noisy face image is generated by P'. Such method,  providing perturbation to the elements of the feature vector F̅ by the weighted sum of ranked wavelet coefficients' Laplacian noises for protecting facial features, is called Ranked Differential Privacy, referred as "RDP".



















In RDP, after perturbation of wavelet coefficients, the noisy pixels matrix P' is obtained by C̅^ * ^' through MIWT, and the noisy feature vector of the noisy face image generated by the noisy face image from P' is F̅' with M_F elements. Let the noise of the i-th element F̅_i^' in F̅' be ζ _i, F̅_i^'  = F̅_i + ζ _i, with  1 ≤ i ≤M_F. Then we have Theorem <ref> of RDP about F̅'.


The noise ζ _i of the i-th element F̅_i^' in F̅',which is  obtained by RDP, satisfies Laplace distribution, , [ ζ _i] = Lap( 0,Δ _i/ε _i), where ε _i is the privacy budget for the i-th element F̅_i^' of F̅', and 1 ≤ i ≤M_F.






Let C̅^*^' and C̅^*  be the noisy ranked wavelet coefficient vector and the ranked wavelet coefficient vector, separately. According to Eq. (<ref>), we have the following first-order Taylor expansion for F̅_i^',

    F̅_i^' = F̅_i + ∑_k = 1^M_P∂F̅_i^'/∂C̅_k^ * ^'·( C̅_k^ * ^' - C̅_k^ * )  + o( C̅^ * ^'),

where C̅_k^*^' and C̅_k^* are the k-th element of C̅^*^' and C̅^*, separately, and o(C̅^ *^') are the minimal term, , o( C̅^*^') ∼ 0. Thus, we think

    F̅_i^'  = F̅_i + ∑_k = 1^M_P∂F̅_i^'/∂C̅_k^ * ^'·( C̅_k^ * ^' - C̅_k^ * ) .


The weight for the i-th element F̅_i^' of the noisy feature vector F̅' and the k-th element C̅_k^ * ^' of the sorted noisy wavelet coefficient vector C̅^ * ^' is w_ik, w_ik = ∂F̅_i^'/∂C̅_k^ * ^'. K is the given geometric number between 1 and M_P with mean 1/p, 0 < p < 1. After we add Laplacian noises to the top K elements of the ranked wavelet coefficient vector C̅^ * according to RDP, the first-order Taylor expansion of F̅_i^' is further

    F̅_i^'  = F̅_i + ∑_k = 1^M_P∂F_i^'/∂C̅_k^ * ^'( C̅_k^ * ^' - C̅_k^ * )  = F̅_i + ∑_k = 1^M_Pw_ikξ _k  .

Then due to there are K noises, ξ _k = {[ ξ _k,   1 ≤ k ≤ K;   0,   k ≤ k ≤M_P ]., and F̅_i^'  = F̅_i + ∑_k = 1^K w_ikξ _k. From that, we have ζ _i = ∑_k = 1^K w_ikξ _k, and can get the probability of ζ _i, [ ζ _i] = [ ∑_k = 1^K w_ikξ _k]. And based on the geometric superposition, ζ _i satisfies Laplace distribution, , ζ _i∼ Lap( 0,Δ _i/ε _i) with sensitivity Δ _i and privacy budget ε _i of F̅_i^'. Especially, the scale parameter b_k of the noise ξ _k is related to the privacy budget ε _i of F̅_i^', 1 ≤ k ≤ M_P.

This theorem has been proved.




Next, the distribution of the noisy feature vector F̅' can be derived from elements of F̅'. Based on Theorem <ref>, we can obtain the relationship between the total privacy budget of F̅' and the scale parameters of Laplacian noises in C̅^ * ^'.




 §.§ Privacy Budget Analysis

In face recognition, the facial feature vector is obtained by projecting the face image onto the eigenfaces, which are constructed by a set of orthogonal eigenvectors through SVD decomposition and dimensionality reduction from the original face images dataset <cit.>. Such eigenvectors capture the most significant variations of features and represent the main change directions in the feature space, and their corresponding singular values represent the importance of eigenfaces. Due to their orthogonality, these eigenvectors are independent and can serve as the basis vectors of the feature space, enabling the representation of the original face image. Consequently, the feature vector's elements, which are projections of the original image onto these eigenvectors, exhibit independence. According to Theorem <ref>, ζ _i = ∑_k = 1^K w_ikξ _k, where K is the given geometric number with mean 1/p, 0 < p < 1, and w_ik is the weight between F̅_i^' and C̅_k^ * ^', w_ik = ∂F̅_i^'/∂C̅_k^ * ^', 1 ≤ k ≤M_P. Let the sensitivity Δ _i and the privacy budget ε _i of F̅_i^', we have [ ζ _i] = Lap( 0,Δ _i/ε _i). According to the variance operation, the variance of ζ _i is 

    Var[ ζ _i] = 2( Δ _i/ε _i)^2.

and the variance of ∑_k = 1^K w_ikξ _k is

    Var[ ∑_k = 1^K w_ikξ _k]    = ∑_K = 1^M_Pp( 1 - p)^K∑_k = 1^K Var[ w_ikξ _k]
        = 2∑_K = 1^M_Pp( 1 - p)^K∑_k = 1^K ( w_ikb_k)^2
        = 2∑_k = 1^M_P( w_ikb_k)^2∑_K ≥ k^M_Pp( 1 - p)^K
        = 2∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1].


Because of Var[ ζ _i] = Var[ ∑_k = 1^K w_ikξ _k] from [ ζ _i] = [ ∑_k = 1^K w_ikξ _k], we get the expression of ε _i with respect to b_k as 

    ε _i( b_k) = Δ _i/√(∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]).

And based on that the elements of F̅' are independent, the total privacy budget of F̅' is ∑_i = 1^M_Fε _i( b_k), then we have the total privacy budget's expression ε( b_k) of F̅' with respect to b_k as

    ε( b_k) = ∑_i = 1^M_FΔ _i/√(∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^1+M_P]).


















 §.§ Optimal Data Utility under Privacy Budget


Our optimization goal is to maximize the noisy face image's visualization quality under the constraint of the privacy budget, , optimizing the noise scale parameters of all wavelet coefficients, which is a constraint optimization problem.

Since the variance of the noise quantifies the negative impact of the noise on the visualization quality and a constant does not have influence on the variance, the cost function for the visualization quality of the face image can be defined by deriving the variance of the noisy pixel matrix, of which value is smaller with the smaller loss of visualization quality. 

Further, the constrained optimization problem can be converted into minimizing the cost function under the constraint of privacy budget. Then, we define the cost function as Definition <ref>.

[Cost Function]






After multi-level HWT for the original face image, C̅ is the one-dimensional vector that the wavelet coefficient matrix expansion to. C̅^ * is a one-dimensional vector after sorting the elements of C̅. According to RDP (Definition <ref>), the Laplacian noises are added  to C̅^ * and get the ranked noisy one-dimensional wavelet coefficient vector C̅^ * ^'. Define that b_k is the scale parameter of the Laplacian noise of the k-th element of  C̅^ * ^', 1 ≤ k ≤M_P, then the expression of the cost function f( b_k) of the noisy face image with respect to b_k is 

    f( b_k) = ∑_k = 1^M_P( b_k)^2[ ( 1 - p)^k - ( 1 - p)^1+M_P].













Given the pixel matrix P of a face image, and it undergoes a multi-level wavelet transform to obtain a matrix of wavelet coefficients C. Assume that the transformation matrix of wavelet transform is H̃ <cit.>, which is a √(M_P)×√(M_P) orthogonal matrix. After N-level wavelet transform, let H = H̃^N, and H^T = ( H̃^T)^N. Then 

    C = H̃^NP = HP, and P = ( H^T)^NC = H^TC,

where H^TH = I, and I is an identity matrix.

Define the Laplacian noise matrix of wavelet coefficient matrix C is N, and the noisy wavelet coefficient matrix is C' = C + N. Then for the noisy pixel matrix P',

    P' = ( H̃^T)^NC' = H^T( C + N) = P + H^TN.










Let B be the Laplacian scale parameter matrix of N, N_mn∼ Lap( 0,B_mn), 1 ≤ m ≤M_P, 1 ≤ n ≤M_P. Thus,

    [                                                              Var[ P'] = 2∑_m = 1^√(M_P)∑_n = 1^√(M_P)∑_t = 1^√(M_P)[ ( H^T)_mt]^2·( B_tn)^2;                                                                           = 2∑_n = 1^√(M_P)∑_t = 1^√(M_P)( B_tn)^2∑_m = 1^M_P[ ( H^T)_mt]^2;                                                                                                    = 2∑_n = 1^√(M_P)∑_t = 1^√(M_P)( B_tn)^2;                                                                                                                                = Var[ C'] . ]


For the convenience of calculation, we flatten the pixel matrix P and the wavelet coefficient matrix C into the one-dimensional pixel vector P̅ and the wavelet coefficient vector C̅. C̅^ * is a one-dimensional vector after sorting the elements of C̅. Then the variance of the noisy pixel vector P̅' and the noisy wavelet coefficient vector C̅' is equal, 
    [                                                Var[ P̅']                                         = Var[ C̅^ * ^'];                                                                       = 2∑_K = 1^M_Pp( 1 - p)^K∑_k = 1^K ( b_k)^2;                                                          = 2∑_k = 1^M_P( b_k)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1] , ]

where C̅^ * ^' is the ranked noisy one-dimensional wavelet coefficient vector.

From that, the expression of the cost function of P̅' with respect to b_k is as follows 

    f( b_k) = ∑_k = 1^M_P( b_k)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]   ,

which is the cost function of the noisy face image.







From that, a constant ε_0 is given as the total privacy budget of the noisy feature vector F̅', and the constraint is ε( b_k) = ε_0. Thus, the optimization problem is as follows,

    b_k = argmin_b_k{f( b_k)|ε( b_k) = ε _0}.


We solve this optimization problem according to the Lagrangian Multiplier method (LM) to better balance the need to sanitize the facial features and preserve the visualization quality of the face image. According to LM, the Lagrangian function with respect to b _k is

    L( b_k,λ) = f( b_k) + λ( ε( b_k) - ε _0),

which can be solved by setting the first derivatives with respect to λ and b_k to zeros,

    {[   ∂ L( b_k)/∂λ = 0;; ∂ L( b_k)/∂b_k = 0, ].




where the first equation reduces to the following,

    ∑_i = 1^M_FλΔ_i( w_ik)^2/{∑_k = 1^M_P( w_ikb_k)^2[ (1-p)^k-(1-p )^1+M_P]}^3/2 = 2.


It is clear that Eq. (<ref>) is nonlinear. Here we propose two ways to solve the nonlinear LM  equations for two different applications, , 1) the Normalization Approximation (NA) for the real-time online applications; and 2) the LM optimization Gradient Descent (LMGD) for the offline applications that need more accurate result.





  §.§.§ Optimization with Normalization Approximation

It is clear that Eq. (<ref>) is nonlinear,  which makes it difficult to find the analytical solution. Therefore, we propose the optimization with Normalization Approximation (NA), which assumes that all wavelet coefficients of interest contribute equally to the total privacy budget ε_0. 

To achieve such goal, it would be intuitive to choose the weighted Laplacian noise scale parameters b_k of Eq. (<ref>) to be the average value, , w_ikb_k≈b̅ = ∑_i = 1^M_Fw_ikb_k/M_F, where 1 ≤ i ≤M_F and 1 ≤ k ≤M_P. And substituting b̅ into Eq. (<ref>), we get

    ∑_i = 1^M_FλΔ _i( w_ik)^2/b̅^3{∑_k = 1^M_P( w_ik)^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2  = 2.

It can be substituted by w_ikb_k to replace b̅ for obtaining the equations about every b_k and λ, which can be used to solve ∂ L( b_k)/∂b_k = 0. And then, Eq. (<ref>) can be solved as follows,

    b_k = ∑_i = 1^M_FΔ_i/w_ik/ε _0[1 - p - ( 1 - p)^1+M_P/p - M_P( 1 - p)^1+M_P]^1/2.




  §.§.§ LM Optimization via Gradient Descent

Although NA can solve the system of nonlinear equations in this optimization problem in real time, the assumption as its basis is are very idealistic, which may cause errors far from the best result. For the case allowing solving this problem offline, we propose solving the LM optimization problem via Gradient Descent (LMGD), which continuously tunes the initial value of b_k until an optimal solution is obtained through iterative gradient descent updating. 

It aims to find the best solution for b_k and λ by solving following equations,

    {[   ∂ L( b_k)/∂λ = 0;; ∂ L( b_k)/∂b_k = 0, ].
 which can be solved with two alternating gradient descent updates: 

1) gradient descent update with respect to b_k; 

2) gradient descent update with respect to λ. 

However, simple alternating gradient descent updates have the problem of finding the optimal step length or learning rate, which is difficult for non-convex optimization problems. Also, the nonlinear operation is required to achieve the minimum b_k, which motivates us to develop the LMGD to solve the nonlinear LM problem.  

According to LM, the purpose of LMGD is to obtain the optimal solution of b_k and λ,  through solving the following equations, 

    {[           ∑_i = 1^M_FΔ _i/√(∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P])  - ε _0 = 0,; ∑_i = 1^M_FλΔ _i( w_ik)^2/{∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2  - 2 = 0. ].

We define the error functions of two updating sub-tasks in LMGD as follows, 

    F_1( b_k^( h ))
    
     =    | ∑_i = 1^M_FΔ _i/√(∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P])  - ε _0|;
      
       F_2( b_k^( h ),λ ^( h ))
    
     =    | ∑_i = 1^M_Fλ ^( h )Δ _i( w_ik)^2/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2  - 2|,

where h represents the h-th parameter update. From that, the total loss function is

    F( b_k^( h ),λ ^( h )) = F_1( b_k^( h )) + F_2( b_k^( h ),λ ^( h )).


When F_1( b_k^(h)) → 0 with F_2( b_k^( h ),λ^( h )) → 0, the closer b_k and λ are to the optimal solution. 

The LMGD takes an initial solution b_k^( 0 ) of b_k and λ ^( 0 ) of λ as input, where ε( b_k^( 0 )) ≈ε _0, however ε( b_k^( 0 )) ε _0. After the initial values are inputted, the optimal solution of b_k and λ is obtained through the iterative updating process updated by two alternating gradient descent until the loss function F( b_k^( h ),λ ^( h )) is less than a minimum value δ, which is very close to 0. 

Given the step length or learning rate of η. Now the iterative updating process takes two alternating updates as follows,

(1) Step 1: firstly, b_k^(h) is updated iteratively by minimizing the loss function F_1( b_k^( h )) using the gradient descent, while b_k^(h) and λ ^(h) are updated iteratively by minimizing the loss function F_2( b_k^(h),λ ^( h )) using gradient descent, ,

    [                                                                                                   b_k^( h ) = b_k^( h ) - η[ ∂F_1( b_k^( h ))/∂ b_k^( h ) + ∂F_2( b_k^( h ),λ ^( h ))/∂ b_k^( h )];;                                                                                                                                         λ ^( h ) = λ ^( h ) - η∂F_2( b_k^( h ),λ ^( h ))/∂λ ^( h ), ]

where the gradient ∂F_1( b_k^( h ))/∂ b_k^( h ) can be obtained as follows

    [ ∂F_1( b_k^( h ))/∂ b_k^( h ) = | ∑_i = 1^M_FΔ _ib_k^( h )( w_ik)^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2|. ]


(2) Step 2: then, the gradients of ∂F_2( b_k^( h ), λ ^( h ))/∂ b_k^( h ) and ∂F_2( b_k^( h ),λ ^( h ))/∂λ ^( h ) are calculated by

    ∂F_2( b_k^( h ),λ ^( h ))/∂ b_k^( h )
    
     =    | 3∑_i = 1^M_Fλ ^( h )b_k^( h )Δ _i( w_ik)^4[ ( 1 - p)^k - ( 1 - p)^1 + M_P]/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^5/2|;
       ∂F_2( b_k^( h ),λ ^( h ))/∂λ ^( h )
    
     =    | ∑_i = 1^M_FΔ _i( w_ik)^2/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2|.


(3) Step 3: finally, after alternatively repeating Step 1 and Step 2 above, the optimal solution of b_k and λ are obtained until F( b_k^( h ),λ ^( h )) < δ. 

The process by Eq. (<ref>) is proved as follow:


If the noise's scale parameter of the i-th element F_i^' of the noisy feature vector F̅' is b_i^( F̅), then the expression of b_i^( F̅) with respect to b^( h )_k is

    b_i^( F̅)( b_k^( h )) = √(∑_k = 1^M_P( w_ik b_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1])  ,

and the expression of the total privacy budget ε with respect to b^( h )_k is 

    ε( b_k^( h )) = ∑_i = 1^M_FΔ _i/√(∑_k = 1^M_P( w_ik b_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1])  ,

where  Δ_i is the sensitivity of the i-th element F̅_i^' of F̅', 1 ≤ i ≤M_F, w_ik is the weights between F̅_i^' and C̅_k^ * ^',  1 ≤ k ≤M_P, and p is the probability parameter greater than 0 and less than 1.

According to the chain rule, 

    ∂ε( b_k^( h ))/∂ b_k^( h ) = ∂ε( b_k^( h ))/∂ b_i^( F̅)( b_k^( h ))·∂ b_i^( F̅)( b_k^( h ))/∂ b_k^( h ) .

 Due to the independence of the elements of the feature vector,

    ∂ε( b_k^( h ))/∂ b_i^( F̅) =  - Δ _i/( b_i^( F̅))^2,

and

    ∂ b_i^( F̅)/∂ b_k^( h ) = b_k^( h )( w_ik)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]/√(∑_k = 1^M_P( w_ik b_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]).


Next, we substitute the above expression into the derivative ∂ε( b_k^( h ))/∂ b_k^( h ), and obtain it as follows

    ∂F_1( b_k^( h ))/∂ b_k^( h )
    
    =    | ∑_i = 1^M_FΔ _ib_k^( h )( w_ik)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]}^3/2|.
    
  






§ ANALYSIS OF PRIVACY AND UTILITY




Privacy. Now let's look at the privacy settings of the RDP for the feature vectors F̅' of the noisy face image according to Definition <ref>. In this paper, the elements of the feature vector are independent of each other with the given total privacy budget ε _0. 


RDP satisfies ε _0-differential privacy.






The face images A and B are a pair of neighboring face images. According to Definition <ref>, proving that RDP satisfies ε _0-differentially privacy is equivalent to prove 

    | [ F̅^( A )^']/[ F̅^( B )^']| ≤exp( ε _0),

where F̅^( A)^' and F̅^( B )^' is the noisy feature vectors of A and B. 




























Let F̅^( A̅) be the standard feature vector of a certain individual 𝔸's standard face A̅. Then, Δ _i is the sensitivity fot the i-th element of the noisy feature vector, and the Laplace scale parameter of that is b_i^( F̅). Define the privacy budget forthe i-th element of the noisy feature vector as ε _i, ε _i = Δ _i/b_i^( F̅), 1 ≤ i ≤M_F. Let ε _0 = ∑_i = 1^M_Fε_i, 

    | [ F̅^( A )^']/[ F̅^( B )^']| =    ∏_i = 1^M_F| [ F̅_i^( A )^']|/| [ F̅_i^( B )^']|
    
     =    ∏_i = 1^M_F| exp(  - | F̅_i^( A )^' - F̅_i^( A̅)|/b_i^( F̅))|/| exp(  - | F̅_i^( B )^' - F̅_i^( A̅)|/b_i^( F̅))|
    
     =    ∏_i = 1^M_F| exp( | F̅_i^( B )^' - F̅_i^( A̅)| - | F̅_i^( A )^' - F̅_i^( A̅)|/b_i^( F̅))|
    ≤   | ∏_i = 1^M_Fexp( | F̅_i^( B )^' - F̅_i^( A )^'|/b_i^( F̅))|
    ≤   | ∏_i = 1^M_Fexp( Δ _i/b_i^( F̅))|
    
     =    exp( ε _0)


Now we have

    | [ F̅^( A )^']/[ F̅^( B )^']| ≤exp( ε _0),

from which Theorem <ref> is proved.




Data utility. In this paper, we use the face image's PSNR (Peak Signal-to-Noise Ratio) value to represent the data utility ( visualization quality) of the face image. PSNR is a measure to evaluate image quality. The higher the value, the less the face image distortion. PSNR of the noisy face image is

    PSNR = 10 ·log _10( ( max{P̅'})^2/σ _r^2),

where P̅' is the one-dimensional pixel vector after the noisy pixel matrix P' flattened, max{P̅'} is the maximum element of P̅', and σ _r^2 is the real variance of P̅'. The higher PSNR is, the more image quality of the noisy face image retains, , the smaller the difference with the original face image. 






§ THE OPTIMIZED RDP ALGORITHM


Algorithm <ref> summarizes the optimized RDP: the inputs include the original face image pixel matrix P, the total privacy budget  ε _0, and the geometric superposition parameter p ∈ [0, 1]; and the outputs are the noise sanitized face image pixel matrix P' and the noisy feature vector F̅'.

First, we obtain the original sparse feature coefficient matrix C by multiscale feature dimensionality reduction of the original face image, which can be cast into the one-dimensional feature coefficient vector C̅. Then, the influence or weight w_ik the k-th element C̅_k^ * ^' of the sorted noisy wavelet coefficient vector C̅^ * ^', with respect to the i-th element F̅_i^' of the noisy feature vector F̅', is calculated, with 1 ≤ i ≤M_F, 1 ≤ k ≤M_P. After that, we obtain the optimized scale parameter b_k of the Laplacian noise for the k-th element C̅_k^ * ^' of the sorted noisy feature coefficient vector C̅^ * ^' according to the NA or the LMGD, with 1 ≤ k ≤M_P. Finally, the sorted noisy feature coefficient vector C̅^ * ^' is obtained through the Definition <ref>, from which we obtain the noisy feature coefficient matrix C' and the noisy feature vector F̅' of the noisy face image generated from C'. 




















Now let's look at the time complexity and storage complexity of Algorithm <ref>, which is mainly determined by two steps: 1) the step of calculating the weights between the noisy feature vector F̅' and the sorted noisy feature coefficient vector C̅^ * ^'; and 2) the step of obtaining the optimized Laplacian noise scale parameters of C̅^ * ^'.

Step 1) calculating the weights between the noisy feature vector F̅' and the sorted noisy feature coefficient vector C̅^ * ^' (lines 4-8): because F̅' has M_F element and C̅^ * ^' has M_P elements, the number of weights is M_F×M_P. Then the time complexity is 𝒪( M_FM_P) and the storage complexity of this stage is 𝒪( M_FM_P). 

Step 2) obtaining the optimized Laplacian noise scale parameters of C̅^ * ^' (lines 9-11): the number of elements in C̅^ * ^' is M_P. When the RDP is optimized by NA,  the temporal complexity is 𝒪( 1 ) and the spatial complexity is 𝒪( M_P). When the optimization is LMGD, the time complexity is 𝒪( M_P) and the spatial complexity is 𝒪( M_P). 
  
Since these two steps are sequentially composed, the time complexity is 𝒪( M_FM_P) and the spatial complexity is 𝒪( M_FM_P + M_P) if the scale parameters are optimized by NA; and the time complexity is 𝒪( M_FM_P + M_P) and the spatial complexity is 𝒪( M_FM_P + M_P) if the scale parameters are optimized by the LMGD. 


§ EXPERIMENTAL EVALUATIONS


We implement our simulations with Python 3.10 on a laptop with Intel Core i7-13700KF, 3.40GHz and Windows 10 system equipped with 64GB main memory.



 §.§ Experimental Setup




  §.§.§ Datasets


We use the following datasets to conduct experiments:
Labeled Faces in the Wild (LFW) <cit.> contains more than 13000 images of 5749 identities and provides a standard benchmark for face verification, composed of 6000 face pairs with 3000 matched and the other 3000 non-matched.
PubFig83 <cit.> contains 8300 cropped face images, made up of 100 images for each of 83 public figures.


  §.§.§ Evaluation Metrics


We use the following metrics in the experiment for performance comparisons: 

Variance. This metric measures the visualization quality of the noise sanitized face images. Because noise will have a negative impact on the quality of the face image, the smaller the variance of the face image's noise, the higher the retained quality of the face image. In this paper, we compare the theoretical variance and the true variance of the noisy image of five methods over the given range from ε_0 = 0.2 to ε_0 = 1.0 of the total privacy budget with probability parameter p=0.02. Since the noise follows the Laplace distribution, the theoretical variance σ _t^2= 2∑_k = 1^K ( b_k)^2, 




where b_k is the scale parameter of Laplacian noise of the k-th element C̅_k^ * ^' of the sorted feature coefficient vector C̅^ * ^' and K is the geometric number with mean 1/p.  

The real or actual variance during the numerical experiment, denoted as σ _r^2, is given by σ _r^2 = ∑_k = 1^M_P( P̅_k^'  - P̅_k)^2,



where P̅_k^' is the k-th element of the noisy one-dimensional pixel vector P̅' of the face image, and P̅_k is the k-th element of the original one-dimensional pixel vector P̅ of the face image.

Structural Similarity Index (SSIM) <cit.>. We conduct SSIM between the noise sanitized face images and the original face images. It imitates the human visual system and quantifies the attributes of images from brightness, contrast and structure. It uses the mean to estimate brightness, the variance to estimate contrast, and the covariance to estimate structural similarity.

Peak Signal-to-Noise Ratio (PSNR). This metric evaluates the visualization quality of the sanitized face images. It is calculated by Eq. (<ref>). The higher the PSNR value, the better the visualization quality of the sanitized face images or the less the face image distortion. 

False Negative Rate (FNR). This metric measures the effect of protection against the bypassing attack. It refers to the ratio of face images that should be identified as this individual but not. The larger the FNR is, the smaller the probability that the real facial features in this face image can be obtained, and the stronger protection for the method provides. 

In particular, for the accuracy of face recognition after adding noise, Ω ^( 𝔸) represents the face image dataset of the individual 𝔸, and F_𝔸 is a function that outputs the face recognition results, representing the feature vector of any face image A and the standard of 𝔸 whether the feature vector of the face satisfies Eq. (<ref>). If it does, the output is 1, , F_𝔸( A ) = 1, indicating that the result of face recognition is correct, otherwise the output is 0. f' represents the differential privacy mechanism with noise. Then, before adding noise, the accuracy is 1, that is [ . F_𝔸( A ) = 1|A ∈Ω ^( 𝔸)] = 1. Let [ . F_A( f'( A )) = 0|A ∈Ω ^( A )] = p̂, then 

    p̂ = ∏_i = 1^M_Fε _i/2Δ _i·exp(  - ∑_i = 1^M_Fε _i) =    ∏_i = 1^M_Fε _i/2Δ _i·exp(  - ε _0)
    ≤   ( ε _0/2)^M_Fexp(  - ε _0)/∏_i = 1^M_FΔ _i,

where ε _i^( F̅) is the privacy budget of the i-th element of the noisy feature vector. Thus, we can get 

    M_Fln( ε _0) - ε _0≥ln( p̂) + ∑_i = 1^M_Fln( Δ _i)  + M_Fln 2 .

From that, the total privacy budget ε _0 of the noisy feature vector has the ability to represent the accuracy of face recognition, , equal total privacy budget ε _0 can be considered as equal face recognition accuracy. Therefore, we will no longer separately show the comparison of the face recognition accuracy of the five methods on the two data in this paper. 



  §.§.§ Evaluated Methods

In experiments, we compare the proposed method with the existing advanced methods. And the methods are depicted as follows. 

DCT-DP. <cit.> proposed a privacy-preserving face recognition method based on frequency-domain Discrete Cosine Transform (DCT) differential privacy, , DCT-DP.

Pixel-DP. This method adds noises to all pixels of the face images. According to the geometric superposition, the Laplacian noises with the same scale parameter b̅_P are added to the selected geometric number of pixels of the face image, and the sum of the privacy budgets of the facial feature vector elements of the noisy image is the given total privacy budget.

RDP. This method is to add Laplacian noises with the same scale parameter b̅_W, as shown in Definition <ref>. 

RDP-NA. This method is the optimized RDP with NA proposed in Sec. <ref>). And its Laplacian noise with the optimized scale parameter b_k is calculated by Eq. (<ref>).

RDP-LMGD. The method is the optimized RDP through the LMGD proposed in Sec. <ref>). It calculates the optimized Laplacian noise scale parameter b_k by LMGD, with the step length or learning rate set to η∈ [0.01, 0.1].




 §.§ Evaluation on Privacy Budget and Utility



  §.§.§ Variance







	 












Now, let's look at the normalized difference or deviation between the real variance of numerical experiment σ_r^2 and the theoretical variance σ _t^2, which are shown in Figure <ref> and Figure <ref>. As the privacy budget ε_0 increased, for LFW data, the differences of the methods other than RDP (as shown in Definition <ref>) do not undergo drastic changes, but fluctuate within a narrow range; while the differences of the five methods consistently fluctuate within a small value range for PubFig83 data. Specifically, RDP-NA (RDP Optimized with Normalization Approximation, as seen in Sec.<ref>) and RDP-LMGD fluctuate slightly near the minimum value of 0 and do not exceed 0.05 for LFW data and PubFig83 data. This demonstrates the effectiveness of our defined cost function for the privacy budget allocation optimization problem. In addition, among the normalized differences between the other three methods, RDP fluctuates greatly, but is never higher than 0.15, while DCT-DP and Pixel-DP will never exceed ( 0.05, 0.15) range. Obviously, our experiment is reasonable and valid

























 






Finally, let's look at the real variance vs. the privacy budget ε_0. Figure <ref> and Figure <ref> show the fitted curves of the logarithm log(σ _r^2) of the real variance σ _r^2 of the five methods for the above two datasets under the given privacy budget. When the privacy budget ε_0=0.2,  the log(σ _r^2) of RDP-NA and RDP-LMGD on both data is less than 4.0. Meanwhile, log(σ _r^2)of DCT-DP is the smallest of the other three methods, around 4.75, yet it is significantly higher than that of RDP-NA and RDP-LMGD, which shows that the real variance σ _r^2 of RDP-NA and RDP-LMGD on the two data is significantly smaller than that of other methods when ε_0=0.2. Furthermore, as the privacy budget ε_0 increases, although the variance of all methods decreases significantly, no significant changes are found in this case, and log(σ _r^2)of RDP-LMGD is always slightly less than RDP-NA for LFW data, meaning that the real variance of RDP-LMGD sometimes is slightly less than RDP-NA. These findings suggest that RDP-NA and RDP-LMGD have less visual damage to the original face images than other methods.









  §.§.§ SSIM


In this paper, we conduct simulation experiments of SSIM on two real datasets for five methods within a given privacy budget range, as shown in Table <ref>. It can be seen that with the increase of privacy budget ε_0, RDP-NA and RDP-LMGD always have higher SSIM than other methods. This indicates that the scale parameters of Laplacian noise optimized by NA or LMGD can make the noisy image more similar to the original face image. In particular, the SSIM of RDP-LMGD is always slightly higher than that of RDP-NA in most cases, except that SSIM of RDP-NA in LFW data is slightly higher than that of RDP-LMGD when ε _0 = 0.2, and SSIM of RDP-NA and RDP-LMGD are equal when ε _0 = 1.0 on both data. 



  §.§.§ PSNR

Finally, let's look at the visualization quality of the sanitized face images after the sanitizing noise is added. 

In this paper, we compared the PSNR of the noise sanitized face images produced by the tested methods within a given range of total privacy budgets from ε_0 = 0.2 to ε_0 = 1.0 with the geometric superposition of parameter p=0.02. We obtained face images from two datasets within the given privacy budgets. The fitting curves of PSNR vs. face images are shown in Figure <ref>, from which we can observe that as the privacy budget increases, the PSNR becomes higher. As shown in Figure <ref>, the PSNR of RDP-NA and RDP-LMGD for LFW data higher than all compared methods. In particular, they are very close. Also, as the privacy budget increases, the advantages of RDP-NA and RDP-LMGD become more pronounced. Similarly, Figure <ref> shows the result For PubFig83 data, from which it is clear that the PSNRs of both RDP-NA and RDP-LMGD are much higher than those of compared  methods. For both data at high level of noise of ε_0 = 0.2, the PSNRs of both RDP-NA and RDP-LMGD significantly exceeded 50dB, while the best PSNR of the compared methods is only around 42dB, which is ∼10dB less than that of RDP-NA. In sum, both RDP-NA and RDP-LMGD achieve better visualization quality of the sanitized face images. 


 
Now, let's look at the visualization effect. As shown in Figure <ref> and Figure <ref>, we show the sanitized face images of five methods obtained  with a privacy budget ε_0 = 0.2 and the geometric superposition parameter of p=0.02 for LFW data and FubFig83 data. It can be shown that the Pixel-DP obtains more local mosaics on images than other wavelet transform based methods, which means that adding noise to selective wavelet coefficients of large significance on the privacy budget is better than adding noise directly to image pixels. Also, compared to the other four methods, the DCT-DP method contains deviation cross the whole face image, which is due to the global effect of the cosine basis used. On the contrary, no significant distortion nor visable mosaic effect can be seen in the sanitized face images of both RDP-NA and RDP-LMGD,  which means better data utility of visualization quality of the sanitized face images has been achieved.






 §.§ Evaluation of Bypass Attacks Against




We use the open-source facial recognition library face-recognition to calculate the Euclidean distance of the encoding vectors of the noisy facial image and the original face image. If the Euclidean distance is less than the set threshold 0.975, it is considered that the noisy facial image belongs to the certain individual to whom the original facial image corresponds. Otherwise, we consider it not the same face as the original facial image. 

Table <ref> shows the false negative rate of the face recognition corresponding to the five methods under the same utility settings on LFW data.





§ CONCLUSION

In this paper, a novel and efficient RDP is proposed to protect the facial features privacy for face image data, which can be formulated as the constraint optimization problem of maximizing the data utility or visualization quality of the noise sanitized face images, for a given privacy budget ε_0. First, the RDP calculates the influence or weights of the noisy feature coefficients in the multiscale transform subspaces with respect to the total privacy budget ε_0; then, the RDP adds Laplacian noises to the weight-ranked feature coefficients by geometric superposition, followed by the inverse transform to obtain the noise sanitized face image. It is rigorously proved that the noisy facial feature vector obtained with RDP satisfies ε_0-differential privacy. After that, the constraint optimization problem is formulated by the LM method by making the first-order Taylor expansion approximation on the influence or weight of the feature coefficients on the facial features. Furthermore, to solve the nonlinear LM, two methods are proposed, one for the real-time online applications and the other for the accurate offline applications: 1) the analytical NA method  for the real-time online applications: by assuming that all Laplacian noise scale parameters to be identical to their average value;  and 2) the LMGD method for the accurate offline applications: by computing the optimal scale parameters of Laplacian noise through iterative updating. Experiments on two real-world datasets are performed, which show that the RDP does achieve better data utility for a given facial feature privacy budget ε_0, compared to other state-of-the-art DP methods. 






IEEEtran


















 Paper Title: Personalized Multi-task Training for Recommender System.














The work is finished during an internship at Salesforce AI Research

  University of Illinois Chicago
  Chicago
  USA
lyang84@uic.edu



  Salesforce AI Research
  Palo Alto
  USA
zhiweiliu@salesforce.com



  Salesforce AI Research
  Palo Alto
  USA
jianguozhang@salesforce.com



  Salesforce AI Research
  Palo Alto
  USA
rithesh.murthy@salesforce.com



  Salesforce AI Research
  Palo Alto
  USA
shelby.heinecke@salesforce.com



  Salesforce AI Research
  Palo Alto
  USA
huan.wang@salesforce.com



  Salesforce AI Research
  Palo Alto
  USA
cxiong@salesforce.com



  University of Illinois Chicago
  Chicago
  USA
psyu@uic.edu














§ ABSTRACT 
In the vast landscape of internet information, recommender systems (RecSys) have become essential for guiding users through a sea of choices aligned with their preferences. These systems have applications in diverse domains, such as news feeds, game suggestions, and shopping recommendations. Personalization is a key technique in RecSys, where modern methods leverage representation learning to encode user/item interactions into embeddings, forming the foundation for personalized recommendations. However, integrating information from multiple sources to enhance recommendation performance remains challenging.
This paper introduces a novel approach named , the first personalized multi-task learning algorithm to obtain comprehensive user/item embeddings from various information sources. Addressing challenges specific to personalized RecSys, we develop modules to handle personalized task weights, diverse task orientations, and variations in gradient magnitudes across tasks. dynamically adjusts task weights based on gradient norms for each user/item, employs a Task Focusing module to align gradient combinations with the main recommendation task, and uses a Gradient Magnitude Balancing module to ensure balanced training across tasks.
Through extensive experiments on three real-world datasets with different scales, we demonstrate that significantly outperforms existing multi-task learning methods, showcasing its effectiveness in achieving enhanced recommendation accuracy by leveraging multiple tasks simultaneously. Our contributions open new avenues for advancing personalized multi-task training in recommender systems. 




<ccs2012>
   <concept>
       <concept_id>10002951.10003317.10003331.10003271</concept_id>
       <concept_desc>Information systems Personalization</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10002951.10003317.10003347.10003350</concept_id>
       <concept_desc>Information systems Recommender systems</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010257.10010258.10010262</concept_id>
       <concept_desc>Computing methodologies Multi-task learning</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Information systems Personalization
[500]Information systems Recommender systems
[500]Computing methodologies Multi-task learning














Personalized Multi-task Training for Recommender System
    Philip S. Yu
    
=======================================================





§ INTRODUCTION

Regarding the pervasive landscape of information available on the Internet <cit.>, recommender systems (RecSys) <cit.> play a pivotal role in facilitating the exploration of items for users.


RecSys improves the efficiency of information digestion 

and has permeated into every corner of our lives, such as news feeds <cit.>, game suggestions <cit.>, and shopping recommendations <cit.>. The great commercial value of RecSys also establishes its indispensability in web services and drives the success of many Internet companies <cit.>.

Personalization is the key technique for an effective RecSys <cit.>. 
Personalized RecSys suggests specific candidate items for each user based on his/her own historical interactions. It offers each user a distinct experience with the web service tailored to their interests. Due to the success of representation learning <cit.>, current state-of-the-art methods <cit.> in RecSys encode user/item interactions into dense vector representations, referred to as embeddings. The embedding of each user/item contains his/her/its unique information, which is the bedrock for providing personalized recommendations. Based on the encoded embedding, these methods recommend items with the highest similarities for each user. Besides user-item interactions, more information can be found in web services and applications such as users' social networks <cit.> and item features <cit.>. Then, a natural question arises: Can we obtain a more comprehensive embedding from multiple sources of information to improve the recommendation performance?




In this paper, we seek to answer this question with multi-task learning <cit.>, which has been shown effective in various application domains such as computer vision <cit.> and natural language processing <cit.>. The core idea behind it is to obtain a comprehensive user/item embedding that can perform multiple tasks at the same time. As illustrated in Figure <ref>, we show three tasks associated with distinct information. 
The black edges denote the recommendation task to predict users' preference for items, which is the main task for optimization. 
The blue edge indicates social information, where the social link prediction task between users characterizes social information into embedding of users. 
The red line represents two items in the same “Electronics” category. 
Predicting whether two items share the same category encodes item category similarity into item embedding. 
Ideally, training user/item embedding with all these tasks endows incorporating information from multiple sources, leading to a more comprehensive vector representation.

Several key challenges are still under-explored when developing a multi-task learning framework for personalized RecSys, which are threefold as follows: 
(1) Personalized task weights. For each user, the importance of different tasks should be distinct.
As illustrated in Figure <ref>, the suitable task weights for u_1 and u_2 are different.
u_1 has sufficient item interactions to obtain an accurate gradient for updating embedding, which suggests a higher RecSys task weight for u_1. However, u_2 does not interact with any item before, in which case the RecSys task fails to provide informative information, and it is beneficial for u_2 to have a higher weight on the Social Prediction task. Supporting specific task weights for each user is the first challenge, and it is essential for personalized training with multiple tasks.
(2) Varied task orientations. The purpose of different tasks is varied, leading to diverged embedding updates during training, which may conflict with each other or even be meaningless for improving the main RecSys task. 
Handling the varied updating orientations from multiple tasks to improve the RecSys task poses the second challenge.
(3) Difference in gradient magnitude across tasks. The loss functions for multiple tasks are inconsistent, and thus the gradient magnitude could be magnificently different. 
Directly averaging all those gradients from multiple tasks may overlook tasks with smaller gradients.
However, balancing the gradient magnitude difference and making the small gradient counts are an important yet under-explored challenge.

This paper proposes the first Personalized Multi-task Training algorithm for Recommender System () to obtain a more comprehensive user/item embedding from multiple information sources. Different modules are designed accordingly to deal with specific challenges. 
We dive into the gradient level during the backward propagation to enable personalized task weights. first collects the gradient from each task during backpropagation and then combines the gradient on each user embedding separately. conducts meticulous user/item-specific gradient level operations rather than the existing task-wise algorithms <cit.>.
For a specific gradient of one user from one task, firstly calculates the norm to measure the importance. 
A gradient with a higher magnitude signifies a greater impact on optimizing the corresponding task. 
then utilizes the gradient norm to calculate task weight for gradient combination. The gradient norm on each task for different users is distinct, leading to personalized gradient weights.
We also designed a Task Focusing module to tackle the varied task orientation challenge. 
We enforce the learning process gradually, focusing on the main RecSys task during the training stage, achieved by increasing the corresponding gradient combination weight. 
After initial exploration of other tasks, the Task Focusing module leads to the combined gradient gradually aligning well with the RecSys task, leaving the varied task orientations imperceptible at the final stage.
Different tasks generate gradients of different magnitudes. Multi-task training algorithms overlook gradients with smaller magnitudes without considering the difference.
Thus, we further propose the Gradient Magnitude Balancing module to deal with this challenge and balance the training of all tasks. Our contributions can be summarised as follows:

    
  * Conceptually, for personalized multi-task learning, we first illustrate the problem, identify the challenges, and propose feasible solutions for recommender system.
    
  * Methodologically, we propose the novel algorithm that operates on the gradient level during backpropagation to provide personalized gradient weights.
    
  * Experimentally, we conduct extensive experiments on three real-world datasets with varied scales to justify the effectiveness of compared with current multi-task training algorithms.




§ PRELIMINARIES

This section illustrates the preliminaries for , including a problem statement and multi-task learning.


 §.§ Problem Statement

Given a set of users 𝒰 = {u_1,u_2,...,u_ | 𝒰|} and a set of items ℐ = {i_1,i_2,...,i_ | ℐ|}, the personalized RecSys task aims to generate a list of items for a given user u.
The most important information for training a recommendation model is the currently observed historical interactions, which can be represented as a user-item bipartite graph 𝒢=(𝒱, ℰ), where 𝒱= 𝒰∪ℐ and there is an edge (u,i) ∈ℰ between u and i if u has interacted with i historically. The adjacency matrix of 𝒢 is represented by R∈ℝ^|𝒱| ×|𝒱|. For implicit feedback <cit.> such as clicks, views, and purchases, R_u,i=1 if user u has interacted with item i, and R_u,i=0 otherwise. For explicit feedback, the user provides, such as rating score, R_u,i is the explicit rating score from u to i. Besides the direct historical interactions, auxiliary information can also be observed from both the user and item side, which is termed by profile 𝒫=(𝒫^u,𝒫^i). 𝒫^u contains all the auxiliary information from the user side, such as features and social friends. 𝒫^i represents the item side auxiliary information such as category and co-view information. Based on all the above information, personalized RecSys aims to predict the preference score in the adjacency matrix of 𝒢 and recommends a list of k items with the top-k highest score.



 §.§ Multi-task Learning

Multi-task learning (MTL) aims to learn a comprehensive model to accomplish multiple tasks by training them simultaneously. This learning paradigm assumes different tasks can enhance each other and perform better on all tasks. Without loss of generality, we integrate and illustrate the current multi-task learning algorithms <cit.> from the gradient level. Each task is trained and can be categorized by a loss function ℒ, which directly leads to a gradient ▽ℒ after backpropagation <cit.>. Assuming we have T tasks, then a list of the gradient for different tasks is obtained (▽ℒ_1, ▽ℒ_2, ..., ▽ℒ_T). An MTL algorithm can be seen as a pooling function over the gradient list to obtain the combined gradient for descent ▽=Pool_MTL(▽ℒ_1, ▽ℒ_2, ..., ▽ℒ_T). Different MTL algorithms differ in the design of the pooling function, which can be parameterized or heuristic.



§ METHOD

This section demonstrates the proposed , shown in Figure <ref>. It includes a forward propagation stage to gather gradients from multiple tasks and a backward stage to aggregate gradients from different tasks individually. Our essential contribution lies in the backward stage, which contains the task focusing and gradient magnitude balancing modules.






 §.§ Forward Propagation

In the forward propagation stage, computes the losses for different tasks. It consists of an encoder to obtain the user/item representation and several task modules for loss computation.



  §.§.§ Encoder

Current state-of-the-art personalized RecSys <cit.> learns an encoder to embed historical interactions 𝒢 into dense embedding representations as:

    𝐄 = Encoder(𝒢),

where 𝐄=(𝐄^u, 𝐄^i). 𝐄^u∈ℝ^|𝒰|× d and 𝐄^i ∈ℝ^|ℐ|× d represent the user/item embedding separately. 
𝐞_u∈𝐄^u is user u's embedding, which encodes the specific information related to u's historical interaction. 𝐞_u is the bedrock to provide personalized recommendation for u, and d is the dimension size. In , we use matrix factorization <cit.> as the encoder to directly embed interactions into dense user/item vectors. aims to encode more information into those vector representation with auxiliary tasks.





  §.§.§ Multi-task Loss computation

For RecSys, tasks 𝒯 are classified into the recommendation task t_RecSys and auxiliary tasks 𝒯_Aux = {t_1, t_2, ..., t_|𝒯_Aux|}. Auxiliary tasks can be built in different ways based on the available information. As illustrated in Figure <ref>, Task 1 is built purely from item embedding 𝐄^i and item profile 𝒫^i. It aims to embed the item's profile information to the corresponding item embedding. Task 2 is built in a similar way from the user side. Tasks should be designed based on the available information for each dataset. For example, we demonstrate the task design in detail for the three experimental datasets in Section <ref>. Feeding the embedding table 𝐄 encoded from Section <ref> into the loss computation module of different tasks, we can obtain a list of losses:

    (ℒ_RecSys, ℒ_1, ℒ_2, ...,ℒ_|ℒ_Aux|)




 §.§ Backward Propagation

During the backward propagation stage, combines the gradients in a personalized fashion, i.e., computes specific gradient weights for each user. The backward stage consists of gradient collection, task focusing and gradient magnitude balancing modules.



  §.§.§ Gradient Collection

For each user/item, each loss ℒ in Equation <ref> leads to one gradient on the corresponding embedding vector, denoted as ▽_𝐞ℒ. Then a collection of gradients from different tasks on the specific user/item is represented by:

    (▽_𝐞ℒ_RecSys, ▽_𝐞ℒ_1, ▽_𝐞ℒ_2, ...,▽_𝐞ℒ_|ℒ_Aux|),

where 𝐞 is the embedding for one user/item.
Then measures the impactfulness of the gradient by l_2 norm as n_t=|▽_𝐞ℒ_t|_2. A larger n_t indicates the current user/item is more impactful to optimize task t. Then a list of norms is computed from Equation <ref> as (n_RecSys, n_1, n_2, ..., n_|ℒ_Aux|).





  §.§.§ Task Focusing

For personalized multi-task training, treats the RecSys task as the main task and all the other tasks as auxiliary ones for further improvement. Auxiliary tasks are built differently, leading to varied task orientations and may even contradict the main task. Thus we design the Task Focusing module to focus the training on the main RecSys task gradually. To this end, a hyper-parameter α>1 is introduced as the increasing base. Before transforming the gradient norm from Section <ref> into gradient weights, we multiply the weight on the main RecSys task as:

    n_RecSys *= n_RecSys * α^s,

where s is the current epoch number. In this design, will explore all the tasks first and gradually focus the training on the main RecSys task. The training will be aligned with the main RecSys task at the final stage. A larger α will fasten the focusing speed. 
Though in exponential design, the following Softmax computation with built-in log-sum-exp trick ensures the number stability.



  §.§.§ Gradient Magnitude Balancing

The loss functions within Equation <ref> are different due to the varied task designs, making the gradient magnitude essentially different (more than 10 times). The gradient with a small magnitude will be overlooked if we directly transform the gradient norm into gradient weight. Thus, we propose the gradient magnitude balancing module to balance the impact of different gradient magnitudes. In detail, the gradient weight is obtained by a Softmax with temperature:

    w_t = exp^n_t / τ/∑_t^*∈𝒯exp^n_t^* / τ,

where τ is the hyper-parameter to balance gradient magnitude. For example, all task weights will be the same without being impacted by different gradient magnitudes when τ→ +∞. Conversely, will only count the gradient with the largest norm when τ→ -∞. A suitable τ balance should be selected for the best gradient magnitude. With Equation <ref>, we can obtain the gradient weights from different tasks as (w_RecSys, w_1, w_2, w_|ℒ_Aux|). Then obtains the gradient on the corresponding user/item by weighted sum on the gradients:

    ▽_𝐞ℒ=∑_t∈𝒯w_t▽_𝐞ℒ_t,

where ▽_𝐞 is the combined gradient for a specific use/item and is used to update corresponding embedding 𝐞. The variance in gradient norms observed across distinct tasks for different users/items culminates in the derivation of personalized gradient combination weights. This distinctive attribute positions as a pioneering instance of personalized multi-task training algorithms.



 §.§ Discussion

RS naturally faces the data sparsity issue <cit.> as the candidate item pool is too large for each user to explore. Information from external knowledge such as social networks <cit.> and knowledge graph <cit.> has proven to provide extra information to enrich user/item representation. However, previous methods model external knowledge as part of the encoder, such as adding auxiliary edges besides the user-item bipartite graph, which needs sophisticated design within the encoder. proposes another solution, i.e., embed the external knowledge into user/item representation with multi-task training. It can keep the model intact and embed that information with more tasks. 


We further analyze the overhead time complexity caused by multi-task training. For each user/item, firstly computes the norm of an embedding gradient in Section <ref>, which takes (O(d)). The Task Focusing module in Section <ref> will only cost O(1) as it only computes on the RecSys task. Then the Gradient Magnitude Balancing module in Section <ref> takes O(|𝒯|). To sum up, the overall time complexity of is O((|𝒰| + |ℐ|)×(d + |𝒯|)), where 𝒰, ℐ, 𝒯 is the user/item/task set respectively and d is the embedding dimension. It is noted that the (|𝒰| + |ℐ|) dimension can be easily batched up for parallel computation, which makes the wall clock time even much faster. The batched version of is shown in Algorithm <ref> with all the modules marked correspondingly, and ▽_𝐄 is the aggregated gradient on embedding 𝐄.








§ EXPERIMENT

This section empirically evaluates the proposed on three real-world datasets. The goal is to answer the four following research questions (RQs). 

    
  * RQ1: Is effective compared with other multi-task learning algorithms? 
    
  * RQ2: Does each designed task help in improving the recommendation performance?
    
  * RQ3: Does the designed task focusing and gradient magnitude balancing module plays a role in ? 
    
  * RQ4: What is the impact of hyper-parameter α and τ in ?
    
  * RQ5: How does the training curve of the newly proposed multi-task training algorithm look like?






 §.§ Experimental Setup

In this section, we illustrate our experiment setting to justify the effectiveness of , including datasets, baselines, evaluation method, and detailed experiment setting.



  §.§.§ Datasets
 
We compare against baselines on three real-world datasets with different sizes. The detailed data statistics of each dataset are shown in Table <ref>. Epinion [<https://www.cse.msu.edu/ tangjili/datasetcode/truststudy.htm>] collects user's reviews on items with user's social friends on e-commerce website. Based on the available information, the Social Prediction task (predict whether two users are social friends) and the Category Prediction task (predict whether two items belong to the same category) are constructed as auxiliary tasks.
Video Game and Office <cit.> are both from the Amazon platform. For the Video Game dataset, we build a co-view and co-buy prediction task to predict whether two items are co-viewed/co-buyed together frequently. As for the Office dataset, we construct the Alignment/Uniformity task <cit.> to directly regularize on the embedding table as well as a rating prediction task to predict the user's explicit rating score towards items. Following previous researches <cit.>, a 5-core setting is applied to all datasets, i.e., we remove users/items with less than 5 interactions to keep the dataset quality.







  §.§.§ Baselines

We compare with 7 baselines in multi-task learning to test the effectiveness of .

    
  * EW: It is the most basic method by directly adding all losses together to optimize user/item representation.
    
  * GradDrop <cit.>: It detects the gradient conflict on each parameter and drops the conflict signal with a scale-measured probability. A smaller scale is easier to be dropped.
    
  * PCGrad <cit.>: Project Conflicting Gradients (PCGrad) changes one of the optimization directions of conflict gradients to make the gradients conflict-free.
    
  * GradVac <cit.>: Gradient Vaccine (GradVac) is the updated version of PCGrad to further align the unconflict gradients.
    
  * CAGrad <cit.>: Conflict-averse gradient descent (CAGrad) minimizes the average loss function by leveraging the worst local improvement of individual tasks to regularize the trajectory.
    
  * Aligned MTL <cit.>: It proposes to use a condition number of a linear system of gradients as a stability criterion to guide the multi-task learning optimization.
    
  * Nash MTL <cit.>: It achieves state-of-the-art performance on several MTL benchmarks. Nash MTL suggests conceptualizing the process of combining gradients as akin to a bargaining game, wherein individual tasks engage in negotiation to mutually agree upon a collective direction for parameter updates.
    
  * RLW <cit.>: Random Loss Weight (RLW) assigns loss weight randomly in each epoch to explore the different tasks.









  §.§.§ Evaluation Method
 
We randomly split the dataset into a training set (60%), validation set (20%), and test set (20%). Models are trained on the training set, hyper-parameters are tuned based on the performance on the validation set, and the reported results are from the test set.
We test the effectiveness of on Top-K personalized recommendation task with three widely adopted evaluation metrics: Recall@K (R@K), Hit Ratio@K (HR@K), and Normalized Discounted Cumulative Gain@K (N@K). To comprehensively assess our model, we present evaluation results for Top-20 and Top-40 recommendations in our experiments.
Our evaluation approach calculates average metrics across all users within the test set. These metrics are computed based on the rankings of items that users have not interacted with. To align with established research practices <cit.>, we adopt the complete full ranking for evaluation. This method entails ranking all items that a user has not yet engaged with, thereby providing a comprehensive evaluation. 




  §.§.§ Experiment Settings
 
To make a fair comparison, we test and all the other baselines under the same setting. User/Item embedding size d is fixed to 32 with Xavier <cit.> initialization. To keep simplicity, we directly use the embedding table as the Encoder. It is to be noted that also supports other encoders. We fix the batch size as 2048 and utilize Adam optimizer <cit.> to optimize the parameters. For the remaining hyper-parameters, we used the grid search method to find the optimal settings for : the learning rate is searched in {0.1,0.05,0.01,0.005,0.001}, coefficient of weight decay is tuned in {1e^-2, 1e^-4, 1e^-6, 1e^-8}. We search the task focusing base α from 1.0 to 1.2 with a step size of 0.02, and the temperature τ in gradient magnitude balancing module in {10,1,1e^-1, 1e^-2, 1e^-3, 1e^-4,1e^-5,1e^-6}. We stopped the training early without improvement in successive 10 epochs on the main recommendation task for all the experiments and reported the results on the test set.




 §.§ RQ1: Performance Comparison

We compare on 3 real-world datasets with 8 baselines in multi-task learning. The experiment results are shown in Table <ref>. The best performance is bolded, and the second best is underlined, respectively. We can have the following observations:

    
  * consistently outperforms the second-best method on all three datasets. On the Epinion dataset, even surpasses the second best with 9.8% improvement in N@5. It justifies the effectiveness of compared with all previous multi-task training algorithms for RecSys.
    
  * The improvement of for different datasets varies. achieves remarkable improvement on Epinion and Office datasets while only achieving about 2% improvement on the Video Game dataset. We assume it is because the designed tasks do not provide much extra information. The co-view, co-buy item prediction task can be easily inferred from user-item interactions.
    
  * RLW nearly always ranks the second best across all three datasets. RLW randomly assigns task weights in each epoch during training. It is a quite simple method to add more randomness for gradient exploration. It surpasses the sophisticated designed gradient manipulation methods such as PCGrad and GradVac by a large margin. The previous gradient manipulation methods directly consider all the parameter sets without personalized consideration, which hinders their performance.
    
  * EW is still a strong baseline for the personalized RecSys task. EW directly adds the losses from different tasks together, which is the simplest multi-task learning method. On the Video Game dataset, it even ranked the second best on R@40 and HR@40. It indicates we do not need a more complicated scheme like previous algorithms. At the same time, surpasses EW by a large margin, which is also simple with heuristic designs.




To show the compatibility of with the current widely used RecSys, we further conduct experiments on different encoders, including MF <cit.>, NGCF <cit.> and LightGCN <cit.>. Experiment results are shown in Figure <ref>. We can observe that always achieves the best performance regardless of the encoder. It shows that is compatible with current RecSys algorithms. Besides, we can also observe the performance improvement with a more advanced encoder. It shows the improvement brought by multi-task training is orthogonal to the encoder, and we can integrate it within current RecSys algorithms to achieve better performance.







 §.§ RQ2: Task Investigation


This section aims to investigate whether each designed task helps improve the recommendation performance. Towards this end, we experiment with individual tasks, and Table <ref> shows the influence of tasks on recommendation performance across different datasets. To have a thorough investigation on the effectiveness of multi-task learning in RecSys, we conduct these experiments with different encoders, including MF <cit.>, NGCF <cit.> and LightGCN <cit.>. We can have the following observations in this experiments:

    
  * We observe the improvements across all three datasets comparing pure BPR performance and the one combining BPR, T1, and T2. On the Epinion dataset, improves the BPR performance from 0.0117 to 0.0170 on R@5 with two extra auxiliary tasks, a 45.2% improvement. The huge improvement indicates the necessity of combining knowledge from extra tasks to alleviate the data sparsity problem in RecSys. The experiment results also validate can effectively fuse the knowledge from auxiliary tasks to improve the RecSys performance.
    
  * There is a general trend of performance improvements when additional tasks are integrated. For example, the pure RecSys task on the Epinion dataset achieves an N@5 of 0.0118. Adding T1 increases to 0.0158, and adding T2 increases to 0.0166. Jointly combining the two tasks achieves the best performance of 0.0168. 
    
  * The impact of individual tasks varies between datasets. On Epinion and Video Game datasets, RecSys performance gradually improves with more tasks. However, the addition of T2 on the Office dataset results in a performance drop on all three backbones, which indicates the auxiliary tasks can also have a negative impact on the RecSys performance, and it should be designed carefully to align it with the main RecSys task.
    
  * We can observe the effectiveness of on all the three backbones. Nearly always performs best with both T1 and T2 on all three backbones. It shows the can be easily integrated into current RecSys algorithms.









 §.§ RQ3: Ablation Study




To investigate the significance of each module, we conducted an ablation study on the task focusing and gradient magnitude balancing modules, and figure <ref> displays the corresponding results. From the figure, we can have the following observations:

    
  * We observe performance deterioration across all three datasets upon omitting any module. It shows each module contributes to the effectiveness of .
    
  * Excluding the task-focusing module leads to a slight decline in the Epinion and Video Game datasets but substantially decreases the Office dataset. Notably, Recall@40 and NDCG@40 drop by 5.7% and 4.7% on Office, respectively. It reveals that gradually focusing the training on the main RecSys task can improve the main RecSys task, which is overlooked by all previous multi-task training algorithms.
    
  * The omission of the gradient magnitude balancing module has a marked impact on the Epinion and Video Game datasets. Specifically, the Recall@40 and NDCG@40 metrics are decreased by 4.4% and 7.6% on Epinion, and by 14.7% and 15.1% on Video Game, highlighting the module's critical role in model performance.





 §.§ RQ4: Hyper-Parameter Sensitivity

This section aims to study the Hyper-parameter sensitivity of . It introduces two hyper-parameters in different modules. α is the multiplying base in the task-focusing module. With a larger α, the training of is faster focused on the main RecSys task. τ is the temperature in the gradient magnitude balancing module. All task weights will be the same without being impacted by different gradient magnitudes when τ→ +∞. Conversely, will only count the gradient with the most prominent norm when τ→ -∞. We show the experiment results on all three datasets in Figure <ref>. From the results, we can observe that:

    
  * With the increase of α, the performance of firstly increases to the peak point and then quickly decreases. At the initial stage, increasing α will focus the training of on the RecSys task at the final stage, showing the advantages of task focusing. When α is large, focuses the training on RecSys quickly without enough exploration of auxiliary tasks. It reveals the importance of learning procedures on auxiliary tasks.
    
  * As τ increases, distinct curve patterns emerge across various datasets. For the Epinion dataset, performance initially reaches a peak before declining. Conversely, optimal performance in the Video Game dataset is achieved with the smallest τ, while the Office dataset attains its highest performance with the largest τ. Auxiliary tasks are constructed in different ways for different datasets. It shows uniqueness of datasets and task construction.
    
  * In most cases, does not perform best when α=1 or τ=1. When the hyper-parameter is equal to 1, the corresponding module does not play a role, leading to inferior performance. It re-validates the effectiveness of each designed module in .








 §.§ RQ5: Training Curve Comparison



As a new multi-task training algorithm, we further show the training curve of in Figure <ref>. All the baselines are also included for comparison. We can have the following observations:

    
  * On the Epinion dataset, other baselines improve quickly at the first few epochs and reach stable on inferior performance. The improvement of is more stable and achieves the highest performance at the final stage. It is because, at the initial stage, tends to explore auxiliary tasks first without fast improvement on the RecSys task. With more epochs, the task-focusing module gradually plays a role in enforcing model train toward the RecSys task. Thus, we observe a gradual improvement of on the RecSys task.
    
  * Similar trend as Epinion dataset is also observed on the Video Game dataset. explores the auxiliary task first and improves slowly and steadily at the beginning stage. After sufficient exploration, it achieves the best performance at the final stage.
    
  * On Office dataset, also improves stability and achieves the highest performance compared to other multi-task training methods. The training curve on the Office dataset justifies the effectiveness of .
    
  * Another interesting finding is seems more stable across datasets. For all three datasets, improves at similar paces. They all improve fastest on 25 epochs and reach the best performance on about 60 epochs. However, other baselines do not seem to have a similar property. All the baselines on the Epinion dataset improve fast at the initial stage, while no similar trend is observed on the Office dataset. We assume personalized updates in lead to this observation as it captures the personalization character of the RecSys task.







§ RELATED WORK

This section introduces the most related literature with this paper and the difference between current methods and our newly proposed . It includes Multi-task Learning and Representation Learning in Recommender Systems.


 §.§ Multi-task Learning

Multi-Task Learning (MTL) <cit.> is an approach that focuses on collectively training multiple interconnected tasks. The goal is to enhance their generalization capabilities by capitalizing on shared insights across these tasks. There has been a fast development of MTL in recent years. 
Current methods can be classified into two categories: conflict resolution-based methods and randomness-based methods.
Conflict resolution-based methods aim to identify and solve the gradient conflict/imbalance from different tasks.
MGDA-UB <cit.> utilizes multiple gradient descent algorithms to find a common descending direction among all gradients to avoid the gradient conflict by solving a quadratic programming problem.
GradNorm <cit.> aims to solve the gradient imbalance problem by constraining the gradient magnitudes to be similar.
PCGrad <cit.> modifies the optimization direction of conflicting gradients, redirecting them to achieve gradient conflict resolution and promote coherence among the gradients.
GradVac <cit.> represents an enhanced iteration of PCGrad, designed to align non-conflicting gradients in similar directions more effectively.
ATTITTUD <cit.> and Forkmerge <cit.> proposes auxiliary task update to judge the effectiveness of primary task and auxiliary tasks.
CAGrad <cit.> employs the concept of worst local improvement from individual tasks, and it minimizes the average loss function, thereby guiding the algorithm's trajectory through regularization.
Aligned MTL <cit.> suggests utilizing a conditioned number of linear gradient systems as a stability criterion, guiding optimizing multi-task learning to avoid conflict.
TAWT <cit.> proposes target aware cross task weighted training algorithm.
IMTL-G <cit.> strives to identify an aggregated gradient characterized by uniform-length projections onto the gradients of each individual task.
RotoGrad <cit.> addresses conflict issues by aligning gradient magnitudes and directions concurrently. It achieves this by introducing a trainable rotation matrix to modify the gradient direction for each task, followed by the computation of weights that enforce uniform gradient magnitudes across all tasks.
On the other hand, randomness-based methods aim to add more gradient exploration based on multiple tasks.
UW <cit.> employs consistent uncertainties as loss weights for individual tasks, which are adaptively adjusted through backpropagation.
GradDrop <cit.> posits that conflict arises from discrepancies in gradient signs across various tasks. To mitigate this conflict, GradDrop drops gradient values based on magnitude-measured probability.
RLW <cit.> introduces variability by assigning loss weights randomly in each epoch, facilitating the exploration of task optimization from diverse perspectives.
Nash MTL <cit.> attains top-notch performance across various Multi-Task Learning (MTL) benchmarks. The approach advocates for interpreting the gradient combination process as a form of bargaining game, where individual tasks negotiate to reach a consensus on a shared direction for parameter updates.


All the previous methods do not consider personalized gradient combination, which stands out from previous methods. It is the first multi-task training algorithm designed for RecSys that supports assigning different task weights for each user/item.




 §.§ Representation Learning for RecSys

The core of existing recommender systems is learning user/item representations. 
BPR-MF <cit.> is one of the most popular methods to characterize user-item interactions and train user/item representations. 
FM <cit.> further incorporates real-value attributes of users and items to learn representations.
Additionally, if given a user-item rating matrix, NMF <cit.> is widely adopted to learn user/item embeddings. 
As deep learning technology develops, NCF <cit.> is proposed to encode user/item IDs with neural networks as embeddings. And NFM <cit.> enables the feature incorporation in representation learning with neural networks.
Recently, the successes of graph neural network GNN <cit.> prompt the graph-based collaborative filtering, such as NGCF <cit.> and LightGCN <cit.>. 
Those graph-based collaborative filtering methods learn user/item representation via aggregating neighbor information, which characterizes the high-order signals in the user-item graph.
Learning user/item representation based on graph embedding also opens more potential to harness additional information. 
GraphRec <cit.> and ConsisRec <cit.> devise additional graph propagation on social graphs, which enables the joint learning of social and recommendation information.
Besides social information between users, those multiple relations between items <cit.>, such as co-view and co-buy, also benefit the representation learning for the recommendation.
The success of contrastive learning in representation learning <cit.> motivates the novel DirectAU <cit.> loss for the recommendation.
DirectAU optimizes the user/item representation by minimizing uniformity and alignment losses, eliminating the inefficient negative sampling process in learning representation.
GraphAU <cit.> further considers the graph-related high-order alignment signal during optimizing user/item representation.
3MN <cit.> builds different meta networks for multi-scenario task learning.
With the emerging ability of large language models (LLM), most recent researches also investigate improving user/item representation with pre-trained large language models for the RecSys task.
P5 <cit.> and OpenP5 <cit.> train different tasks within large language models by next-token-prediction task.
LlmRec <cit.> augments user-item bipartite graph with LLMs to learn a more informative user/item representation.
RLMRec <cit.> proposes contrastive alignment and generative alignment to align user/item embedding to the large language model encoded space.
CARec <cit.> directly aligns the user/item's ID embedding with semantic embedding in the iterative alignment phase.
Previous research shows that a more discriminative, comprehensive, and informative user/item representation is the foundation for effective RecSys.

This paper proposes to directly learn user/item representation under a multi-task learning paradigm and proposes the first personalized multi-task training method . 
It can encode information from multiple tasks to obtain a more comprehensive and informative user/item representation.



§ CONCLUSION



This paper introduces , a novel multi-task training algorithm specifically designed for recommender systems. This algorithm distinctively supports personalized gradient combinations for individual user/item. It intricately navigates the backpropagation process, manipulating gradients directly at their level. Within , two principal modules are constructed: a task-focusing and a gradient magnitude balancing module. The former progressively concentrates training efforts on the primary recommender system (RecSys) task, whereas the latter equilibrates gradients of varying magnitudes. Empirical evaluations conducted on real-world datasets substantiate 's efficacy. As the first multi-task training algorithm designed explicitly for the recommender system, we also open-sourced for further research.




ACM-Reference-Format









 Paper Title: GenRec: Generative Personalized Sequential Recommendation.















panfengc@umich.edu

  University of Michigan
  Ann Arbor
  USA


pl219@cam.ac.uk

















§ ABSTRACT 
Sequential recommendation is a task to capture hidden user preferences from historical user item interaction data. Significant progress has been made in this domain by leveraging classification based learning methods. Inspired by the recent paradigm of “pretrain, prompt and predict” in NLP, we consider sequential recommendation as a sequence to sequence generation task and propose a novel model named Generative Recommendation (GenRec). Unlike classification based models that learn explicit user and item representations, GenRec utilizes the sequence modeling capability of Transformer and adopts the masked item prediction objective to effectively learn the hidden bidirectional sequential patterns. Different from existing generative sequential recommendation models, GenRec does not rely on manually designed hard prompts. The input to GenRec is textual user item sequence and the output is top ranked next items. Moreover, GenRec is lightweight and requires only a few hours to train effectively in low-resource settings, making it highly applicable to real-world scenarios and helping to democratize large language models in the sequential recommendation domain. Our extensive experiments have demonstrated that GenRec generalizes on various public real-world datasets and achieves state-of-the-art results. Our experiments also validate the effectiveness of the the proposed masked item prediction objective that improves the model performance by a large margin.







<ccs2012>
 <concept>
  <concept_id>10010520.10010553.10010562</concept_id>
  <concept_desc>Information systems Recommender systems</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>10010520.10010553.10010563</concept_id>
  <concept_desc>Information systems Users and interactive retrieval</concept_desc>
  <concept_significance>400</concept_significance>
 </concept>
 <concept>
  <concept_id>10010520.10010575.10010755</concept_id>
  <concept_desc>Computing methodologies Natural language generation</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
</ccs2012>


[500]Information systems Recommender systems
[400]Information systems Users and interactive retrieval
[300]Computing methodologies Natural language generation
















10 May 2024






GenRec: Generative Personalized Sequential Recommendation
    Pietro Liò
    August 5, 2024
=========================================================





§ INTRODUCTION

Recent years have witnessed the great success of recommender systems in online platforms such as Amazon and Yelp, which help people make micro decisions and fulfil their demands in daily life. Within online platforms, historical sequential user interaction data is utilized to capture the evolved and dynamic user behaviors and make appropriate recommendations of next items for the user. Different from traditional recommender systems that treat each user behavior as a sample and directly model the user preference on a single item, sequential recommendation learns timestamp-aware sequential patterns to recommend the next item <cit.>. For example, one might purchase peripheral accessories after buying a desktop. Various methods have been proposed to accurately model sequential user behaviors. <cit.> employ Recurrent Neural Networks (RNN) to encode the left-to-right context. <cit.> utilizes Convolutional Neural Networks (CNN) to learn sequential patterns as local features. <cit.> leverages Graph Neural Networks (GNN) to construct the item-item interest graphs and user interest sequences. <cit.> designs contrastive loss to maximize the mutual information between different views of the sequential data. Contextual information such as item attributes is incorporated as self-supervised signals to learn user and item representations. 

The recent success of Transformer <cit.> based large language models (LLMs) in various NLP tasks has inspired the research of utilizing LLMs in recommender systems <cit.>. <cit.> utilizes the left-to-right self-attention mechanism to identify next relevant items from the sequential interaction history and <cit.> further improves the performance by employing the bidirectional self-attention mechanism. Although both methods are effective, they do not consider personalization and the model performance is potentially limited. Generative LLMs based recommender system is considered a promising approach <cit.>. <cit.> is the first model to unify different recommendation tasks in the same framework to facilitate the knowledge transfer and it is pretrained across task-specific datasets to capture the deep semantics for personalization and recommendation. However, different modality information is encoded in the textual format of natural language, which might cause suboptimal modality representation. Moreover, <cit.> relies on manually designed task-specific hard prompts to formulate the problem as question answering, which requires additional data processing and prompt search.



To address the mentioned limitations, we propose GenRec, a novel generative framework for personalized sequential recommendation. GenRec utilizes the encoder-decoder Transformer as the backbone and formulates sequential recommendation as a sequence-to-sequence generation task. We utilize the cloze task <cit.> as the training objective and pretrain the model on the corpus to learn the bidirectional sequential patterns. Inspired by <cit.>, to make the downstream sequential recommendation task consistent with the pretraining cloze task, we append the  token at the end of the input sequence to predict the next item. Extensive experiments on public real-world datasets demonstrate that GenRec generalizes effectively and achieves state-of-the-art performance across all datasets.
In conclusion, our contributions include: 


  * We propose GenRec, a novel generative model for personalized sequential recommendation that can generate next items auto-regressively without prompts.

  * As far as our knowledge, it is the first generative sequential recommendation method that adopts the cloze objectives in both pretraining and finetuning. Our method only utilizes task specific datasets and does not rely on additional large pretraining corpus [Our source code, datasets and pretrained models are publicly available at <https://github.com/caop-kie/GenRec>.].

  * We conduct experiments on three public real-world datasets, demonstrating consistent improvements on multiple evaluation metrics compared with baseline methods.




§ RELATED WORK

Transformer based LLMs have achieved remarkable success on various research fields such as text summarization, question answering, document understanding, etc. Typically, LLMs are trained on a vast amount of textual corpus from diverse sources including wikipedia, news, articles and books. LLMs can have emergent zero-shot learning capability as the model parameter size scales up with large training datasets <cit.>, making LLMs better generalize to unseen domains. Recent efforts have been made to explore the potential of LLMs as recommender systems. <cit.> employs T5 as the model backbone and achieves competitive performance on a variety of recommendation tasks, demonstrating the generalization capability of LLMs based recommender systems. <cit.> develops a unified foundation model based on M6 and reduces the model size and training time by utilizing prompt tuning. Similar to <cit.>, user interaction data is represented as plain texts and recommendation tasks are formulated as either natural language understanding or generation. <cit.> leverages ChatGPT to build a conversational recommender system to improve the interactivity and explainability of the recommendation process. <cit.> finetunes T5 with user-personalized instruction data, which is generated from manually designed templates, enabling users to communicate with the system with natural language instructions. <cit.> proposes an efficient framework to finetune LLMs for recommendation tasks and demonstrates significant improvements in the domains of movie and book recommendations. <cit.> employs a funneling approach where it first retrieves the candidates utilizing the user item interactions and then generates recommended items from candidates with a LLMs based framework.



§ TASK FORMULATION

The goal of sequential recommendation is to predict the next item that the user is most likely to interact with given the historical item interaction sequence of the user. Formally, i_th user u_i has item interaction sequence t_i = {t^i_1, ..., t^i_n_i} ordered chronologically. t^i_j and n_i denotes the j_th item in the sequence and the length of t_i respectively. t^i_j:k denotes the subsequence, i.e., t^i_j:k = {t^i_j, t^i_j+1, ..., t^i_k}, where 1 ≤ j < k ≤ n_i. Given the user item sequence {u_i, t^i_1:n_i}, the task of sequential recommendation is to predict the next item that u_i is most likely to interact with at the n_i+1 timestep. 





§ METHODOLOGY

The overall architecture of  is shown in Figure <ref>. Our  is established upon a sequence-to-sequence Transformer encoder-decoder framework. The encoder of  embeds cross modal user and item features from the input sequence and the decoder generates next items auto-regressively. In the following sections, we elaborate the feature embedding and learning objectives of pretraining and finetuning.


 §.§ Model

The feature embedding consists of token embedding, positional embedding, user ID embedding and item ID embedding. The input sequence to the model {u_i, t^i_1:n_i} is tokenized into a sequence of tokens, which is wrapped around with the start indicator token [BEG] and the end indicator token [END]. Extra [PAD] tokens are appended to the end to unify the sequence length in the batch. The token sequence S is represented as: 

    S = [BEG], Tok(u_i), Tok(t^i_1), ..., Tok(t^i_n_i), [END], ..., [PAD],

where Tok is the tokenizer function.
For the j_th token in S, we apply textual token embedding, positional embedding and user or item ID embedding. The textual token embedding is formulated as Emb(S_j), where Emb is the token embedding function. The positional information of tokens in the sequence is captured by the positional embedding PosEmb(j), where PosEmb is the 1D positional embedding function. And we utilize the user ID and item ID embedding to add personalized information of users and items to the sequence. Personalized information helps reveal the sequential patterns of user behaviors. For example, a user that previously viewed a Samsung tablet is more likely to continue viewing electronic products from Samsung or other brands. Different from P5 <cit.>, which apply whole-word embeddings shared by both user and item tokens. Our ID embedding layers are not shared to better capture modality specific features. The user ID embedding is represented by UIDEmb(UID(j)) and the item ID embedding is represented by TIDEmb(TID(j)). UIDEmb and TIDEmb are the ID embedding functions for users and items respectively. Since a word in the sequence can be split into multiple tokens after tokenization, we utilize UID(j) or TID(j) to retrieve the user ID or item ID for the j_th token. For example, item_1234 is split into 4 tokens i.e. item, _, 12 and 34. Those tokens share the same item ID 1234 and embedding. Note only user tokens have user ID embeddings and only item tokens have item ID embeddings. All the aforementioned cross modal embeddings are added element-wisely to produce the final embedding X. Specifically the j_th token cross modal embedding is formulated as: 


    X_j =    Emb(S_j) + PosEmb(j) + IDEmb(ID(j)), j ∈ [0, n_i),
       IDEmb ∈{UIDEmb, TIDEmb},
       ID ∈{UID, TID}



The decoder input is the next item that serves as the learning target. The same tokenizer and textual embedding in the encoder are utilized. Following baseline methods, the model generates top 20 items with beam search during inference to calculate the evaluation metrics.




 §.§ Pretraining


We pretrain  with the masked sequence modeling task to deeply fuse the personalized sequential patterns. Given a user item sequence, we randomly sample an item, which is not necessarily the last item, from the sequence and replace it with the  token. The model is trained to generate the masked item in the decoder output. Cross-entropy loss is computed between the generated item and the masked item and minimized in pretraining. The masking process is also illustrated in Figure <ref>. Different from <cit.>, where a number of items are masked, we only mask a single item in the sequence to unify the learning objectives of pretraining and finetuning. Our bidirectional masked sequence modeling task is based on the observation that the item sequence is not strictly ordered. For example, given similar items item_1 and item_2, user_1 might interact with item_1 and then item_2 while user_2 interacts with item_2 and then item_1. It is crucial to incorporate both left and right contexts to encode the user behavior <cit.>. To avoid the data leakage, we pretrain the model on the training split of datasets (See <ref> for details about the training split).


 §.§ Finetuning

In the finetuning stage, the learnt user and item sequential patterns in pretraining are utilized for the next item prediction task. As mentioned in section <ref>, finetuning has the same learning objective with pretraining for knowledge transfer. Following the masking mechanisms in Figure <ref>, we prepare the dataset in the same format as in pretraining by appending the  token to the end of the token sequence. The input to the model is the masked user item sequence and the output is the predicted next item.


§ EXPERIMENTS




§ DATASETS


Amazon Sports and Amazon Beauty datasets are obtained from Amazon review datasets <cit.>, which are collected from Amazon.com online platform. 
Yelp dataset is a large business recommendation dataset. Following <cit.>, we use transactions between January 1, 2019 and December 31, 2019. For all datasets, item sequences are extracted per user and sorted ascendingly by the interaction timestamps. Following the same data preprocessing in the baseline methods, unpopular items and inactive users with less than five interaction records are filtered. The statistics of the datasets after preprocessing are summarized in Table <ref>.

To generate the training, validation and test splits of datasets, we apply the leave-one-out strategy following <cit.>. In each user item sequence, we use the last item as the test data, the item before the last item as the validation data, and the remaining item sequence as the training data.





§ BASELINES


We compare the performance of GenRec with several competitive sequential recommendation baselines.

Caser <cit.> is a CNN based method, which utilizes both horizontal and vertical convolutional filters to capture high-order Markov chains for sequential recommendation.

FDSA <cit.> models the feature sequence with the self attention module and captures the feature transition patterns.

GRU4Rec <cit.> applies GRU to model the user click history sequence for session based recommendation.

HGN <cit.> adopts hierarchical gating networks to learn both long-term and short-term user behaviors.

Bert4Rec <cit.> learns the bidirectional item representation by masked language modeling for sequential recommendation.

SASRec <cit.> applies the self attention mechanism to model the user sequential behaviors.

S3Rec <cit.> designs self-supervised learning objectives to capture the correlations of items and attributes for sequential recommendation.

P5 <cit.> is a pretrained language model for recommendation and unifies different recommendation tasks by formulating them in natural language with prompts.

All the baselines except P5 are classification based.



 §.§ Implementation Details

The model weights of  is initialized from the pretrained BART <cit.> base model, which consists of a 6-layer Transformer encoder and a 6-layer Transformer decoder. There are 12 attention heads for both encoder and decoder and the model dimension is 768. The total number of model parameters is 184 million. We use the pretrained BART tokenizer for tokenization.  is trained with an AdamW optimizer <cit.>, in which 1e-5 is set as the learning rate and weight decay. The maximum length of the input tokens is set to 512. First 5% of iterations are used as the warmup stage. The beam size is set to 20 in inference.  is pretrained for about an hour and finetuned for 25 epochs on one NVIDIA RTX 3090 GPU.



 §.§ Evaluation Metrics

To evaluate the model performance, we employ top-k Hit Ratio (HR@k) and Normalized Discounted Cumulative Gain (NDCG@k) and report the the results of HR@{1, 5, 10} and NDCG@{5, 10} under the all-item setting, i.e. all items are possible candidates to be recommended as the next item. Our model is evaluated on Amazon Sports, Amazon Beauty and Yelp datasets. In the tables, bold numbers refer to the best scores, while underlined numbers refer to the second best scores.



 §.§ Results and Analysis

This section provides experimental results and analyses of our model's effectiveness. The performance metrics of the baselines are obtained from the public results in <cit.>. See Appendix <ref> for more details about baselines. As is presented in Table <ref>, GenRec outperforms other baselines on all metrics on Sports and Yelp datasets. Although it is outperformed by P5 on the HR@10 metrics of the Beauty dataset, GenRec still achieves comparable performance with other baselines. 

We speculate due to the smaller size of Beauty dataset, the sequential patterns are not learnt thoroughly in pretraining. Increasing the number of pretraining epochs could potentially improve the performance. Since we focus on low-resource efficient training in this work, we leave the improvement for future work. Since P5 is a unified model and trained using different types of recommendation tasks, it learns better user item representation with the knowledge transfer between tasks on smaller datasets. The advantage of our method compared with P5 is that our model is lightweight and the training only takes a few hours to complete. We do not leverage prompt engineering and no prompt search is required to find the best prompt. 





§ ABLATION STUDY


To verify the effectiveness of the proposed masked sequence modeling task, we conduct the ablation study on Amazon Sports, Amazon Beauty and Yelp datasets. As is shown in Table <ref>, the model performance drops across all metrics without the masked sequence modeling task. It indicates the pretraining objective helps the model effectively capture the user behavior patterns and learn the user item representations. It is worth mentioning even without pretraining, GenRec still achieves comparable performance with P5 and outperforms other baselines, which proves the effectiveness of our method.



§ CONCLUSION

In this work, we propose GenRec, a novel sequence to sequence framework that generates personalized sequential recommendation. Compared with existing generative models, GenRec is lightweight and efficient and does not rely on prompt engineering to find the best prompt. By leveraging the Transformer architecture as the backbone, GenRec effectively learns the bidirectional sequential patterns with the attention mechanism and achieves state-of-the-art performance on various public datasets. Besides, our proposed method is also flexible to integrate with other sequence to sequence language models and can be applied in other recommendation domains such as direct recommendation, which we leave for future work.





ACM-Reference-Format



 Paper Title: Breaking the Hourglass Phenomenon of Residual Quantization: Enhancing the Upper Bound of Generative Retrieval.


Explainable and Controllable Motion Curve Guided Cardiac Ultrasound Video Generation
    Junxuan Yu1,2,3Junxuan Yu and Rusi Chen contribute equally to this work. Rusi Chen1,2,3⋆ Yongsong Zhou1,2,3 Yanlin Chen1,2,3 Yaofei Duan4 Yuhao Huang1,2,3 Han Zhou1,2,3,5 Tan Tao4 Xin Yang1,2,3() Dong Ni1,2,3()
    August 5, 2024
======================================================================================================================================================================================================================

§ ABSTRACT 







Generative retrieval (GR) has emerged as a transformative paradigm in search and recommender systems, leveraging numeric-based identifier representations to enhance efficiency and generalization. Notably, methods like TIGER employing Residual Quantization-based Semantic Identifiers (RQ-SID), have shown significant promise in e-commerce scenarios by effectively managing item IDs. However, a critical issue termed the "Hourglass" phenomenon, occurs in RQ-SID, where intermediate codebook tokens become overly concentrated, hindering the full utilization of generative retrieval methods. This paper analyses and addresses this problem by identifying data sparsity and long-tailed distribution as the primary causes. Through comprehensive experiments and detailed ablation studies, we analyze the impact of these factors on codebook utilization and data distribution. Our findings reveal that the "Hourglass" phenomenon substantially impacts the performance of RQ-SID in generative retrieval. We propose effective solutions to mitigate this issue, thereby significantly enhancing the effectiveness of generative retrieval in real-world E-commerce applications.




§ INTRODUCTION

In recent years, GR has surfaced as a groundbreaking retrieval paradigm, marking significant advancements in search and recommendation environments including recommender systems <cit.>, search question answering <cit.>, and E-commerce retrieval <cit.>. In this paradigm, target items are initially represented as identifiers (e.g., numbers, subwords, n-grams, token IDs, URLs, semantic codes). Subsequently, leveraging input information such as queries and user details, large models are employed to output the final items in an end-to-end manner. This approach not only enhances retrieval efficiency but also improves the model’s generalization capability.

In generative retrieval, numeric-based identifier representation methods are widely adopted in the industry due to their simplicity, efficiency, and strong generalization, especially in long behavior sequence recommendations. These methods significantly reduce sequence lengths and accelerate the inference process. Notable methods include DSI <cit.>, NCI <cit.>, TIGER <cit.>, GDR <cit.>, and GenRet <cit.>. Among these, the TIGER method generates Semantic Identifiers (SID) through Residual Quantization (RQ) <cit.>, effectively capturing both semantic information and hierarchical structures. This approach is particularly advantageous in item-dominated e-commerce scenarios, where it accurately reflects the complex hierarchical relationships and semantic features inherent in e-commerce data, thereby significantly enhancing recommendation performance.


It is important to highlight that the performance upper bound of RQ-based methods critically depends on the generation of SID. However, we have identified a significant "hourglass" phenomenon in SID produced via RQ, as illustrated in Figure <ref>. Specifically, the codebook tokens in the intermediate layers are excessively concentrated, leading to a one-to-many and many-to-one mapping structure. This concentration results in path sparsity, where the matching paths for the item constitute a minimal fraction of the total path space and a long-tail distribution of intermediate layer tokens with a majority of SID concentrated in a few head tokens. This hourglass effect is particularly exacerbated in datasets with long-tail characteristics, which substantially constrains the representational capacity of GR methods. The underlying cause of this issue stems from the intrinsic nature of progressively quantizing high-dimensional vector residuals.














Furthermore, we analyzed the process of generating SID from residuals, demonstrating that sparsity and long-tail distributions are inevitable. To assess the general impact of SID on downstream GR tasks, we trained models of different scales (such as 0.8B, 7B) and types (Qwen1.5 <cit.>, Baichuan2 <cit.>, LLaMA2 <cit.>) based on RQ-SID. Through a series of experiments, including altering the distribution of Semantic IDs by interacting with the first and second layers and swapping tokens between the first and second layers, we not only confirmed the existence of the Hourglass effect but also detailed its specific impact on model performance. This analysis provides a robust foundation for future model optimization.

To alleviate the hourglass effect, we propose two straightforward yet effective methods: the heuristic approach and the adaptive variable-length token strategy. The heuristic method involves directly removing the second layer, while curtailing the long-tail impact, it may lead to insufficient spatial capacity. The second method implements an adaptive token distribution adjustment to remove the top tokens from the second layer, thereby transforming the semantic ID into a variable-length structure. This strategy ensures that the overall distribution remains consistent while effectively mitigating the hourglass effect by selectively token removal. Extensive experimental results reveal that although both methods are straightforward, they successfully alleviate the impact of the hourglass effect to varying extents. Notably, the adaptive variable-length token strategy method emerges as the most effective.


The contributions of this paper can be summarized as follows:

    
  * To our knowledge, this is the first study to systematically investigate the deficiencies of residual quantization-based semantic identifiers in generative retrieval, specifically identifying the "hourglass" phenomenon where intermediate layer codebook tokens are overly concentrated.
    
    
  * We conduct thorough experiments and ablation studies that reveal data sparsity and long-tail distributions as the primary causes of the "hourglass" effect, limiting the representation and performance capabilities of generative models.
    
    
    
    
  * We propose and validate a novel method to alleviate the "hourglass" effect, which significantly enhances model performance by improving codebook utilization and addressing token long-tail distributions.
    
    





§ PRELIMINARY






 §.§ Residual Quantization

Residual-quantized is a multi-level vector quantizer that applies quantization on residuals to generate a tuple of codewords (i.e., Semantic IDs). Residual-quantized variational AutoEncoder (RQ-VAE) <cit.> is jointly trained by updating the quantization codebook and the encoder-decoder reconstruction parameters. 

Support that there is a vector x∈ℛ^D, we aim to quantize it using  L codebooks (L layer) of M elements each, where codebook could be denoted as C∈ℛ^L × M × D, D is the dimension of vector.
When l=1, the initial residual is simply defined as r_1=x. 
Then, r_l is quantized by mapping it to the nearest embedding from that layer’s codebook C_l ∈ℛ^M × D. The index of the closest embedding at this layer could be computed as follows:
 
    c_l = min_m ∈ M∥r_l - C_l,m∥_2^2

where c_l represents the l-th codeword(semantic ID).
Note that, at the l-th layer (l > 1), the residual is: 

 
    r_l=r_l-1 - C_l,c_l-1

The above process is repeated recursively L times to get a tuple of L codewords that represent the Semantic ID for the given x, denoted as (c_1, c_2, …,c_L).









To reconstruct the raw vector, we sum the corresponding codebook elements as:

    x̂ = ∑_l=0^LC_l,c_l


This method could approximate the raw vector from a coarse-to-fine granularity by the norm of residuals decreasing, i.e., ‖x̂ - x‖^2 < ϵ,  ϵ≪ 0.001.



 §.§ Generative Retrieval

Generative retrieval <cit.>, has been proposed in the recommendation field, search field and question-answer field.
These models advocate generating identifiers of target passages/items directly through the autoregressive language models. 










In personalized search scenarios, a core task is to provide the most relevant candidates that the user is likely to purchase based on their given query and historical interaction behaviors. In this paper, we re-frame this task as a Next Token Prediction (NTP) problem utilizing LLM and Semantic ID. Specifically, given user u, query q, and  the user's historical item sequence,  we first convert the sequence into a Semantic ID sequence, denoted as Seq :={(c_1,1,·,c_1,M)_item_1; (c_2,1,·,c_2,M)_item_2;…; (c_t,1,·,c_t,M)_item_t}
where (c_i,1, ·, c_i,M) denotes the M-length Semantic ID for item_i. The LLM is then trained to predict the Semantic ID of item_t+1, represented as (c_t+1,1, ·, c_t+1,M). 
The generation objective could be formulated as,

    ℒ_sft = - ∑_i^Mlogp_θ(i|q,u, Seq, I_<i)

where I_<i = {c_t+1,1,⋯,c_t+1,i},  p_θ is the supervised fine-tuning (SFT) model.

 










§ PROBLEM OF GR BASED ON RQ




 §.§ Hourglass Phenomenon


To generate the semantic IDs used RQ, we first leverage the query-item data from billions of search logs within the company to train dual-tower models such as DSSM and BERT <cit.>. Subsequently, we obtain the embeddings for hundreds of millions of items using the item tower. Finally, we employ RQ to generate semantic IDs for all items.

Upon the successful generation of semantic IDs, we proceed to aggregate and compute the three-layer distribution maps for all items. As illustrated in Figure <ref>, it is evident that the second layer of the Semantic ID architecture is concentrated with a substantial number of routing nodes. The overall distribution of the three-layer code exhibits an hourglass phenomenon.  


To investigate the generalizability of this phenomenon, we conducted multiple visualization experiments under various parameter combinations, e.g., code table size and number of layers. As shown in Figure <ref> in the appendix, the results indicate that the hourglass effect is highly pronounced, and the path distribution among the tokens across the three layers of the code table is relatively sparse.


Additionally, based on the aforementioned experiments, we conducted statistical analyses of the token distribution in the second layer using three metrics: entropy <cit.>, Gini coefficient <cit.>, and standard deviation <cit.>, as shown in the Figure <ref>. The results indicate that the token distribution in the second layer exhibits low entropy, high Gini coefficient, and large standard deviation, suggesting that the distribution is highly skewed and exhibits a long-tail effect.


Overall, this hourglass phenomenon is statistically evidenced in the code table by path sparsity and a long-tail distribution of tokens. 1) Path sparsity, resulting from the Semantic ID structure, leads to low code table utilization. 2) The long-tail distribution indicates that in the intermediate layer, a predominant number of routes converge on a single token.









 §.§ Analysis of Residual Quantization




To explore the causes of the hourglass phenomenon, we will conduct an in-depth analysis and discussion based on the operating mechanism of the RQ.
Without loss of generality, we consider two distributions of raw embedding:  un-uniform and uniform, denoted as X = {x|x∈X}∈ℛ^N × M, N is the size of the dataset.
Now, we use the RQ to produce the semantic ID for X.  



In the first layer, all candidate's points are divided into M different cluster buckets. Each cluster bucket contains n_m data points and has a radius of e_m.
For the uniform distribution, n_m = N/M, and e_1 = e_2 = … =e_m. Therefore, the in-degree of all tokens in this layer are equal.   


In the second layer,  all input embedding is X^', the residual of the first layer. Due to the difference in the magnitude of residual values, the input distribution in this layer is non-uniform. There are a large number of points with smaller magnitudes (points near the cluster centers in each bucket from the previous layer), which is equal to n_m * M * ρ = N *ρ, ρ is the ratio. At the same time, there are small points with larger magnitudes, which are considered as outliers. To reduce the clustering loss, the clustering process in this layer focuses on these outliers. As a result, the points with smaller magnitudes will occupy fewer cluster centers, while the outliers will either occupy individual cluster centers or multiple cluster centers. Therefore, this layer's semantic IDs will form large routing nodes, exhibiting a long-tail phenomenon, which is also demonstrated in the second layer of Figure <ref>.

In the third layer, all input point magnitudes become consistent again and relatively uniform. Therefore, the code distribution in this layer is similar to the first layer, with a uniform distribution. As a result, it can be directly observed that the large routing nodes from the second layer diverge into multiple smaller nodes in the third layer, creating a one-to-many situation, as shown in the third layer of Figure <ref>. At the same time, if the residuals in the second layer tend towards zero, there will still be some clustering in the third layer. However, since all magnitudes are very small at this point, the impact of the clustering effect is limited.

As we continue to iterate through the layers, this phenomenon of non-uniform distribution and long-tail clustering followed by uniform distribution will alternate. However, as the number of layers increases, the residuals become smaller (refer to layer 4 of Figure <ref>), and the clustering effect weakens, so it can be ignored. Ultimately, this leads to the formation of an hourglass-like structure, where the input data is first compressed into a smaller number of clusters, then expands back out into a larger number of clusters, and finally converges to a uniform distribution. Upon the completion of SID construction, the influence of the RQ quantization method, coupled with the dominance of head tokens in the intermediate layer, naturally leads to the sparsity of paths.



Similarly, for the un-uniform distribution, such as long-tail distribution, the residual distribution becomes even more uneven, resulting in a more severe phenomenon.



 §.§ Impact on the GR



In the above section, we have discussed the long-tail distribution in the second layer of Semantic ID, indicating a one-to-many and many-to-one structure. We argue that this phenomenon significantly impacts the generation of downstream tasks, especially for generative retrieval task.

To measure this impact, we conducted various experiments. First, we altered the distribution of Semantic ID by interacting with the first and second layers. On this basis, we only predicted the tokens of the second and third layers while keeping the tokens of the first layer fixed. 

During the evaluation process, we divide the test set into two groups according to the distribution of second-layer tokens: the head token test set and the tail token test set. As shown in Table <ref>, the performance of the head token test set significantly improved, whereas the performance of the tail token test set was notably poorer. This performance disparity can be attributed to the previously analyzed path sparsity and long-tail distribution of tokens, leading to biased results. This phenomenon has been observed across models of different scales (LLaMA2, Baichuan2, and Qwen1.5) and different parameters of RQ, highlighting the widespread impact of long-tail token distribution and path sparsity on model performance. 

To further investigate the impact of the hourglass phenomenon on model performance, we conduct two critical experiments: 1) exchange the tokens of the first and second layers, and 2) give the first token of the swapped sequence as input.
These approaches aim to mitigate the effects of the long-tail distribution, and results verify a significant improvement. This finding indicates that the hourglass phenomenon has a substantial negative impact on model performance. Through the above experiments, we not only confirmed the existence of the hourglass effect but also elucidated its specific impact on model performance, thereby providing a robust basis for future optimization.





§ METHODS AND EXPERIMENTS

To alleviate the hourglass effect, we propose two simple yet effective methods.


 §.§ Heuristic Method

One heuristic approach is to directly remove the second layer, eliminating the impact of the long tail. However, it can lead to insufficient spatial capacity,  i.e., M^L → M^L-1.
Note that, here needs first to generate an L-layer SID and then remove the second layer, which differs from directly generating a two-layer SID, where large routing nodes may still exist. 





 §.§ Variable Length of SID

Another simple method is to adaptively remove the top tokens of the second layer, making the semantic ID a variable-length structure. Here, a top@K strategy is used, with p as a threshold. This approach ensures that the distribution remains unchanged while reducing the impact of the hourglass effect selectively. What's more, the spatial capacity is sufficient, i.e., M^L →  M^L + K(M^L-2 - M^L-1).
 




 §.§ Experiments

To further validate the effectiveness of the method, experiments are conducted on the LLaMA model. Results indicate that by applying the adaptive token removal strategy, the performance of the model is improved while maintaining a similar computational cost compared to the base model, and several objective optimizations, such as Focal Loss <cit.> and Mile Loss <cit.>.

Specifically,  experimental results showed that the model with top@400 token removal outperforms the baseline model in terms of most evaluation metrics. This suggests that the method effectively reduces the impact of the long-tail effect. As the number of tokens removed increases, the performance improvement of the model encounters a bottleneck. Especially when all tokens are removed, this limitation is particularly pronounced, which is presumed to be due to the absence of long-tail tokens, resulting in a loss of recall. At the same time, removing the second layer directly will cause one SID to correspond to multiple items.  

This fine-grained analysis provides strong evidence for the effectiveness of the proposed method, which selectively removes less important tokens while retaining the most informative ones, leading to improved model performance even when a substantial amount of data is removed.




 §.§ Valid Ratio

During the autoregressive decoding process, as the model decodes the next token of the target SID, it may predict invalid SIDs, SIDs that are not in the SID's vocabulary, or do not correspond to any item in the full dataset. Therefore, we have calculated the proportion of invalid SIDs on the LLaMA2-0.8B model with RQ3x12. As shown in Figure <ref>,  we can see the base model, the invalid ratio of the proposed method is lower than the base model, indicating that the higher-quality generation items with a lower ratio of hallucination. Furthermore, when the number of recalls is less than 10, the invalid ratio is below 5%. Thus, the effectiveness of generation is to meet practical needs. In other situations, where a higher number of recalls is required (k=50), the invalid ratio is higher. Therefore, it is necessary to employ the retrieval augmented generation (RAG) <cit.> for processing during the inference process, such as prefix-tree <cit.>, and FM_Index <cit.>.





























§ CONCLUSION

This study systematically explores the limitations of RQ-SID in GR, particularly identifying the "hourglass" phenomenon in the intermediate layer where codebook tokens are overly concentrated, leading to data sparsity and long-tail distribution. Through extensive experiments and ablation studies, we have demonstrated the existence of this phenomenon and conducted an in-depth analysis attributing its root cause to the characteristics of residuals. To alleviate this issue, we propose two methods: a heuristic approach that removes the second layer and a variable-length token strategy that adaptively adjusts token distribution. Experimental results show both methods effectively mitigate the bottleneck effect, with the adaptive token distribution adjustment yielding the best results. To the best of our knowledge, this is the first systematic exploration of the deficiencies of RQ-SID in GR, providing a solid foundation for future model optimizations and significantly enhancing model performance by improving codebook utilization.









    



    

    

    
    















§ DISTRIBUTION AND CONNECTIONS OF DIFFERENT RQ-SEMANTIC IDS







 Paper Title: Exploiting Preferences in Loss Functions for Sequential Recommendation via Weak Transitivity.














  Omnious AI
  Seoul
  South Korea

hyunsoo.chung@omnious.com



  University of Pittsburgh
  Pittsburgh
  Pennsylvania
  United States

jungtaek.kim@pitt.edu



  Omnious AI
  Seoul
  South Korea

hyungeun.jo@omnious.com



  Omnious AI
  Seoul
  South Korea

hyungwon.choi@omnious.com


§ ABSTRACT 
A choice of optimization objective is immensely pivotal in the design of a recommender system as it affects the general modeling process of a user's intent from previous interactions. 
Existing approaches mainly adhere to three categories of loss functions: pairwise, pointwise, and setwise loss functions.
Despite their effectiveness, a critical and common drawback of such objectives is viewing the next observed item as a unique positive while considering all remaining items equally negative.
Such a binary label assignment is generally limited to assuring a higher recommendation score of the positive item, neglecting potential structures induced by varying preferences between other unobserved items.
To alleviate this issue, we propose a novel method that extends original objectives to explicitly leverage the different levels of preferences as relative orders between their scores.
Finally, we demonstrate the superior performance of our method compared to baseline objectives.








<ccs2012>
 <concept>
  <concept_id>10002951.10003317.10003347.10003350</concept_id>
  <concept_desc>Information systems Recommender systems</concept_desc>
  <concept_significance>500</concept_significance>
  </concept>
</ccs2012>


[500]Information systems Recommender systems














Exploiting Preferences in Loss Functions for Sequential Recommendation via Weak Transitivity
    Hyungwon Choi
    August 5, 2024
============================================================================================









§ INTRODUCTION




Distinguishing items by relevance to an each user's previous interaction history is the essence of most applied sequential recommender systems.
Conventional approaches <cit.> frame this task as learning the feature representations of a chronologically-ordered list of user's interacted items and a candidate item.
By computing the inner product between two vectors, we then obtain a recommendation score that quantitatively indicates the relevance.
On that basis, a prevalent training process reformulates the problem into a supervised learning framework with binary labels, where the next interacted item of each user becomes a unique positive to its sequence of previous items.
Such a procedure often accompanies negative sampling <cit.> among the rest of items since the score computation of all items during each training step inevitably results in severe inefficiency <cit.>.


In this line of research, numerous approaches have adopted advanced neural networks as feature encoders <cit.> to capture more complex correlations.
Apart from vast architectural improvements, the majority of models yet utilize one of three types of loss functions as an optimization objective:
pairwise, e.g., BPR <cit.>,
pointwise, e.g., BCE <cit.>,
and setwise, e.g., SSM <cit.> functions.
While the formulation of each function varies, they all share the desired result of increasing the score of the unique positive than the scores of the other items.
Notably, most recent studies on loss functions and negative sampling strategies <cit.> aim for improved training efficiency and robustness to false negatives within three core objectives.
Despite their effectiveness, the model trained with such objectives consequently learns to regard unobserved items as equally negative with their labels simply set to zero.
In real-world recommendation, however, a subset of negative items is occasionally more favorable than the other due to various side factors, e.g., item popularity.
Unfortunately, the current scheme of binary label assignments hinders from fully taking advantage of diverse levels of preferences among unobserved items.


To address the aforementioned issue, we first derive an inductive relation, dubbed weak transitivity, which represents preference-driven orders of item scores.
We then propose novel extensions of original loss functions that directly leverage this weak transitivity in their forms.
Consequently, the recommendation scores of unobserved items are aligned with respect to their preferences.
Figure <ref> highlights such a property of resulting recommendation policy trained with our proposed family of objectives.
Opposed to the results of BPR <cit.> and BCE <cit.>, item scores from our transitive extensions TransBPR_pop and TransBCE_pop (horizontal hatch) are generally proportional to their popularity.
Meanwhile, scores from TransBPR_niche and TransBCE_niche (diagonal hatch) are inversely proportional to their popularity, favoring more niche (i.e., unique) items.
To cap it all, we serve the predefined preferences of items as an additional supervisory factor for their recommendation scores.


It is noteworthy that in this work we do not propose any new distributions for negative sampling but instead introduce the modification of original objectives.
Hence, we solely utilize a combination of item popularity and uniform distributions for negative sampling, easily accessible in implicit settings.
We validate the effectiveness of our proposed extensions compared to the original loss functions and their renowned variants on four sequential recommendation benchmarks.
In all settings, our approaches substantially improve the recommendation performance compared to baseline methods.






§ BACKGROUND



In this section, we describe a sequential recommendation task and the details of representative loss functions for recommendation.




 §.§ Problem Statement


Let 𝒰 = {u_1, …, u_M} and ℐ = {i_1, …, i_N} denote a set of users and items where M and N are maximum numbers of users and items, respectively.
Given a chronologically ordered history h_u = {i_1^u, …, i_t^u} of observed items for a user u,
a goal of sequential recommendation is to recommend the most relevant next item i_t+1^u.
We first embed the history h_u and a candidate item i onto vectors h'_u and i', respectively.
Accordingly, a recommendation score ŝ_ui is calculated via an inner product between two embeddings.




 §.§ Training Objectives





  
Pairwise Objective 
Bayesian personalized ranking (BPR) <cit.> models personalized ranking of items.
It forces the score of the positive (i.e., next interacted) item to be higher than the scores of the rest unobserved items.
With a dataset 𝒟_s composed of a triplet (u,i,j) of which item i as a positive and item j as a sampled negative to a user u,
the corresponding loss is formulated as follows:

    ℒ_BPR = -
        ∑_(u,i,j) ∈𝒟_slogσ(ŝ_ui - ŝ_uj),

where σ(·) is a sigmoid function.
A resulting term σ(ŝ_ui - ŝ_uj) is the probability of a user u preferring an item i more than an item j. 




  
Pointwise Objective
Similar to the pairwise loss form, binary cross-entropy (BCE) <cit.> casts the recommendation problem into binary classification with a single sampled negative.
Hence with a dataset 𝒟_s of a triplet (u,i,j), the loss is defined as below:

    ℒ_BCE = -
        ∑_(u,i,j) ∈𝒟_slog(σ(ŝ_ui)) + 
        log(1 - σ(ŝ_uj)).





  
Setwise Objective
The sampled softmax loss (SSM) <cit.> turns the problem into a multi-class classification with a finite number of sampled negatives.
With a dataset 𝒟_m of a triplet (u,i, 𝒩_j) consisting of the positive i and a set of multiple negative samples 𝒩_j to the user u, the loss optimizes the probability of the positive as:

    ℒ_SSM = -
        ∑_(u,i,𝒩_j) ∈𝒟_mlogexp(ŝ_ui)/exp(ŝ_ui) + 
        ∑_j ∈𝒩_jexp(ŝ_uj).











§ OUR METHOD



Here, we introduce weak transitivity between unobserved items,
and then describe how to integrate it to original objectives.




 §.§ Weak Transitivity



Inducing orders on recommendation scores of unobserved items requires a model to sample two or more negatives differing in preferences.
In a basic form, let p_1 and p_2 represent two different sampling distributions for negatives.
Given a user u and its positive i,
we sample a negative j from p_1 and k from p_2 where i ≠ j ≠ k.
Since the positive i is the most preferred item to the user u,
a transitive relation ŝ_ui > ŝ_uj > ŝ_uk holds true when j is more preferred than k.
Strict transitivity then corresponds to a scheme where any j from p_1 is guaranteed to be more preferred than any k from p_2.
However, when the sampled negative j is actually less preferable than k, the relation ŝ_ui > ŝ_uj > ŝ_uk is violated.
We refer this scheme as weak transitivity which allows such occasional violations.


The concept of transitivity with multiple negatives is straightforward as well.
Instead of sampling a single negative from each distribution, we sample a set of negatives 𝒩_j = {j_1, …, j_n} from p_1 and 𝒩_k = {k_1, …, k_n} from p_2.
Consequently, with ŝ_𝒩_j = {ŝ_uj_1, …, ŝ_uj_n} and ŝ_𝒩_k = {ŝ_uk_1, …, ŝ_uk_n},
a transitive relation is modified as ŝ_ui > max(ŝ_𝒩_j) > min(ŝ_𝒩_j) > max(ŝ_𝒩_k) with the same criteria for weak and strict cases.




 §.§ Extensions with Weak Transitivity



We propose novel extensions of original training objectives that resolves the limitation of binary label assignments by incorporating the derived transitive relation to the loss formulation.
We first introduce two sampling schemes by utilizing an item popularity distribution p_pop and a uniform distribution p_unif:

    𝒟'_s(pop)    = {(u,i,j,k) | j ∼ p_pop, k ∼ p_unif},
        
    𝒟'_s(niche)    = {(u,i,j,k) | j ∼ p_unif, k ∼ p_pop},

where j is the more preferred negative and k is the less preferred one. 
The mini-batch 𝒟'_s(pop) assumes that a user generally prefers popular items whereas 𝒟'_s(niche) regards niche items more favored.
Our extension of the BPR objective with transitivity is then given by the following:

    ℒ_TransBPR
        = -
        ∑_(u,i,j,k)∈𝒟'_slogσ(ŝ_ui - ŝ_uj)_original + 
        γlogσ(ŝ_uj - ŝ_uk)_preference,

where γ is a balancing coefficient for two terms.
In essence, the preference term encourages the score of j to be higher than that of k while the original term assures it to be smaller than that of i.
As a consequence, our objective explicitly imposes the transitive relation ŝ_ui > ŝ_uj > ŝ_uk to recommendation scores. 
Combining the proposed formulation with the previously introduced sampling schemes, we obtain two distinct training objectives as follows:

    TransBPR_pop   =
        ℒ_TransBPR(𝒟'_s(pop); θ),
    TransBPR_niche   =
        ℒ_TransBPR(𝒟'_s(niche); θ),

where θ is learnable parameters of the model.
Similarly, the extension of the BCE function is formulated as below:

    ℒ_TransBCE =     -
        ∑_(u,i,j,k) ∈𝒟'_s[ log(σ(ŝ_ui)) + 
        log(1 - σ(ŝ_uj))
        
        + γ(
        log(σ(ŝ_uj)) + 
        log(1 - σ(ŝ_uk)) )].

Naturally, TransBCE_pop and TransBCE_niche are two consequent training schemes with 𝒟'_s(pop) and 𝒟'_s(niche), respectively.
For the setwise loss function, we sample a set of items 𝒩_j and 𝒩_k from each distribution instead of sampling a single item j or k:

    𝒟'_m(pop)    = {(u,i,𝒩_j,𝒩_k) |𝒩_j ∼ p_pop, 𝒩_k ∼ p_unif},
        
    𝒟'_m(niche)    = {(u,i,𝒩_j,𝒩_k) |𝒩_j ∼ p_unif, 𝒩_k ∼ p_pop}.

A corresponding extension for the SSM loss function is given by:

    ℒ_TransSSM =     -
        ∑_(u,i,𝒩_j,𝒩_k) ∈𝒟'_m[
        logexp(ŝ_ui)/exp(ŝ_ui) + 
        ∑_j ∈𝒩_jexp(ŝ_uj)
       + γ1/|𝒩_j|∑_j ∈𝒩_jlogexp(ŝ_uj)/exp(ŝ_uj) + 
        ∑_k ∈𝒩_kexp(ŝ_uk)].

As similar to previous extensions, TransSSM_pop and TransSSM_niche employs either 𝒟'_m(pop) or 𝒟'_m(niche) for negative sampling.


One thing to note is that our introduced sampling strategies 𝒟'_s and 𝒟'_m are both weak transitive.
Nonetheless, modifying them to a strict setting can be readily accomplished if we can quantify the preference of each item.
For instance, we can reformulate the mini-batch construction for pairwise and pointwise objectives as:

    𝒟^†_s(pop)    = {(u,i,j,k) | f(j) > f(k), j ∼ p_pop, k ∼ p_unif},
        
    𝒟^†_s(niche)    = {(u,i,j,k) | f(j) < f(k), j ∼ p_unif, k ∼ p_pop},

where f(·) denotes a function that measures the popularity of an item.
However, we argue that such strict transitivity rather hurts the quality of the resulting recommendation policy.







§ EXPERIMENTS




In this section, we conduct experiments to compare our proposed extensions to original objectives and their variants.




 §.§ Experimental Setup





  
Datasets
We employ public sequential recommendation tasks from different domains:
Beauty, Toys, and Sports,
which are product review datasets introduced by Amazon.com <cit.>,
and Yelp, which is a widely tested business recommendation dataset. 
Detailed statistics of preprocessed datasets are reported in Table <ref>.




  
Evaluation Settings and Metrics
For dataset partitioning, we adopt the conventional leave-one-out strategy <cit.> to assure the quality of the trained recommender system.
Then, we recommend K items with the highest recommendation scores from the entire item pool.
For evaluation, we adopt two common top-K metrics, HR@K and NDCG@K with K fixed to 10.




  
Baselines
We fix a model architecture to SASRec <cit.>
and switch only a training objective. 
For baselines, we compare our method against BPR <cit.>,
GBPR <cit.>,
BCE <cit.>,
SSM <cit.>,
InfoNCE <cit.>,
BPR-DNS <cit.>,
and gBCE <cit.>.
Here, gBCE_1 and gBCE_𝒩 denote gBCE with a single negative and multiple negatives, respectively.




  
Hyperparameters
For all objectives, We train with a fixed batch size of 256, a learning rate of 0.0003, and a maximum sequence length of 50.
The SASRec model is with 2 layers and 1 attention head with an embedding dimension of 256. 
For setwise objectives, we sample 100 negatives in total.
Our proposed TransSSM sets cardinality of 𝒩_j and 𝒩_k to 50 such that they sum to 100.
A balancing coefficient γ is selected from {0.5, 1.0, 1.5}.




 §.§ Performance Comparison



Table <ref> summarizes the performance of models trained with different optimization objectives. 
In general, our proposed objectives outperform three original objectives and their notable variants in both metrics within all benchmarks.
Particularly, we recognize TransSSM_pop obtain the highest performance among all baselines in all datasets.
Such results demonstrate that inducing preference order through weak transitivity consistently improves the performance regardless of base loss functions.
Typically, we observe our extensions with popularity preference (e.g., TransSSM_pop) achieve commonly improved metrics compared to niche preference (e.g., TransSSM_niche).
Hence, exploiting popularity as the preference indicator is particularly effective regardless of datasets.
Though, we find recommendations with niche preference performs on par with best-performing baselines or often even better.







 §.§ Transitivity Analysis



As seen in Table <ref>,
we observe comparatively higher performance with weak transitivity.
We hypothesize that the number of uninformative (i.e., easy) negatives <cit.> increases more within the strict transitivity as training proceeds.
To validate the claim, we report how average value of each term in the BPR extension,
logσ(x̂_uij) and logσ(x̂_ujk), changes throughout the training with different schemes.
Here, we compare three classes of transitivity:
Weak, i.e., (<ref>) and (<ref>),
Strict, i.e., (<ref>) and (<ref>),
and Disjoint, i.e., a specific case of Strict that sets 50% of total items with high preference as a support for p_pop and the rest for p_niche.
Results in Figure <ref> illustrate more steep decline of the preference term, logσ(x̂_ujk), as the transitivity becomes more strict.
Thus, weak transitivity provides the most informative gradients, increasing the training effectiveness.




§ CONCLUSION


In this work, we identified the common and crucial disadvantage of binary label assignments within conventional recommendation objectives.
To overcome this limitation, we have proposed novel extensions that directly exploit the preference differences of unobserved items.
Through extensive experiments, we demonstrated the effectiveness of our method with the thorough analysis.
As future work,
the uniformity of normalized embeddings <cit.> can be applied to our framework to improve recommendation performance more.


ACM-Reference-Format




 Paper Title: Temporal Subspace Clustering for Molecular Dynamics Data.






Both authors contributed equally to this research.

  University of Vienna
  Vienna
  Austria
anna.beer@univie.ac.at


[1]

  LMU Munich
  Munich
  Germany

heinrigs-martin@t-online.de



  University of Vienna
  Vienna
  Austria
plant@univie.ac.at




  Aarhus University
  Aarhus
  Denmark
ira@cs.au.dk



§ ABSTRACT 
We introduce MOSCITO (MOlecular Dynamics Subspace Clustering with Temporal Observance), a subspace clustering for molecular dynamics data. 
MOSCITO groups those timesteps of a molecular dynamics trajectory together into clusters in which the molecule has similar conformations.
In contrast to state-of-the-art methods, MOSCITO takes advantage of sequential relationships found in time series data. Unlike existing work, MOSCITO does not need a two-step procedure with tedious post-processing, but directly models essential properties of the data. 
Interpreting clusters as Markov states allows us to evaluate the clustering performance based on the resulting Markov state models. 
In experiments on 60 trajectories and 4 different proteins, we show that the performance of MOSCITO achieves state-of-the-art performance in a novel single-step method.
Moreover, by modeling temporal aspects, MOSCITO obtains better segmentation of trajectories, especially for small numbers of clusters.





Temporal Subspace Clustering for Molecular Dynamics Data
    Ira Assent
    August 5, 2024
========================================================





§ INTRODUCTION

Proteins play an important role in every living organism.
Analyzing the transition between different shapes of a protein, also known as folding, is important for understanding the structure and function of proteins, e.g., for drug design. 
Many diseases like Alzheimer, Parkinson, and different types of cancer are connected to misfolding of certain proteins <cit.>. Thus, understanding such folding processes may help to prevent or cure diseases. 

Molecular dynamics data contains trajectories of proteins that change their shape over time. It is usually very high-dimensional, with thousands of time steps and hundreds of atoms for a single protein. 
Analyzing molecular dynamics data with traditional clustering is typically not meaningful due to the curse of dimensionality. To mitigate issues with high dimensionality, subspace clustering methods detect clusters in lower-dimensional subspaces <cit.>.
However, existing subspace clustering methods do not cater for the characteristics of molecular dynamics data.

In this paper, we introduce MOSCITO (Molecular Dynamics Subspace Clustering with Temporal Observance), a subspace clustering algorithm that uses temporal regularization to exploit sequential relationships found in molecular dynamics trajectories that describe folding activities of single molecules. MOSCITO is the first one-step approach to handle this complex data type by identifying the best subspace projection. It finds features tailored specifically to molecular dynamics data. Leveraging methods from video analytics, we show that MOSCITO finds temporally contiguous groups of of protein conformations.  

In trajectories, data points that are near in time are usually more closely related than those that are further apart. 
MOSCITO allows users to choose different sizes of time windows and different weighting methods for temporal regularization: binary, Gaussian, exponential, or logarithmic. 
Interestingly, the resulting clusters can be interpreted as states in a Markov State Model (MSM), which we use for comparing MOSCITO to approaches like SSC, PCA + k-Means, and TICA + k-Means.
As we demonstrate in thorough empirical evaluation, MOSCITO successfully clusters molecular dynamics data in a model that directly captures the essential features and temporal relationships.
Our main contributions are as follows:

    
  * We introduce MOSCITO, a subspace clustering method for molecular dynamics data that makes use of temporal relations between time steps that are close
    
  * MOSCITO finds temporally contiguous groups of protein conformations that are similar 
    
  * In contrast to currently used methods that require several steps, the clusters found by MOSCITO can directly be used as states of a Markov State Model describing the trajectory, yielding state-of-the-art quality with a one-in-all holistic approach



§ CLUSTERING TRAJECTORIES OF MOLECULAR DYNAMICS DATA
 

In this paper, we study the trajectories of a single molecule over time. A trajectory is given by the three-dimensional Cartesian coordinates of all atoms of the molecule for a number of time steps with fixed intervals.
Where most traditional clustering methods target numerical data in ℝ^d, clustering MD data brings some unique challenges as we outline in the following. 

Due to the size of MD data, automated methods for analyzing them are required. These often involve clustering methods to automatically group similar groups of atoms or timesteps.

In this paper, we focus on simulations of very long molecular dynamics trajectories that show folding processes of single molecules <cit.>. 
Based on the starting positions of a molecule's atoms, physical forces on all other atoms can be calculated, yielding simulated trajectories with a temporal resolution that usually lies in the femtosecond range.
While the overall shape of the molecules is decisive for its function, several features are shown to be relevant, e.g., the torsions between C^α atoms that form the backbone of the protein, distances between specific atoms, or the solvent accessible surface area.


There are different use cases for which a clustering of MD trajectories is required. 
E.g., clusters of atoms that move similarly over time can be used to find dynamic domains, i.e., fragments of a protein that are internally rigid <cit.>. 
Clustering time steps in which the conformation of the protein, i.e., the arrangement of its atoms in space, is similar is used for detecting relevant states of the protein. 
Those states can be linked to (local) valleys in the energy landscape of the protein and are e.g. used for developing Markov models that describe a protein trajectory. 
Thus, one goal is to separate metastable states of a protein and the transitions between those states. 





§ MARKOV STATE MODELING
 

Markov state models (MSMs) can be used to describe the dynamics of a MD trajectory by using a n × n transition matrix. To generate an MSM from an MD trajectory X={x_1,...,x_t}, 
discrete states of the trajectory need to be identified. 
State-of-the-art methods use a two-step approach in which they first reduce the dimensionality of the data by using PCA or TICA and then clustering it with k-Means into n discrete states S={S_1,...S_n}<cit.>. These states are used to calculate the transition probability matrix for a given lag time τ by counting the transitions from time step t to t+τ: 


    p_ij(τ)=ℙ(x_t+τ∈ S_j|x_t∈ S_i)


The resulting MSM can be used for further analysis of the trajectory. In this paper, we analyze the clustering performance of our method in comparison to state-of-the-art methods for clustering MD trajectories based on their suitability as Markov states.


The variational approach for Markov models (VAMP)<cit.> is a generalization of the Variational approach to conformational dynamics (VAC) <cit.>, <cit.> and provides a tool to evaluate MSMs. The VAC allows the approximation of eigenvalues and eigenfunctions with statistical observables <cit.>. It also defines a family of scoring functions, called the VAMP-r, which can be calculated from data. This score can be used to compare the choices of hyper-parameters, like the featurization of the data or the used clustering, used for constructing the MSM. The score is calculated from the singular value decomposition of the Koopman operator. The Koopman operator is explained below. It is important to note that the VAMP score can only be compared for MSMs constructed with the same lag time τ and cannot be used to find the optimal lag time τ itself.

The Koopman operator gives the conditional expected value of an arbitrary observable g at time t+τ for a given x_t. The Koopman operator K_τ of a Markov process is defined as follows <cit.>:


    K_τ g(x)=𝔼[g(x_t+τ|x_t=x]


When choosing the Dirac delta function δ_y centered at y, applying the Koopman operator evaluates the transition density of the dynamics <cit.>.


    K(τ)=C_00^-1/2C_01C_11^-1/2≈ U_mΣ_mV_m^T,

where Σ_m=diag(σ_1,...,σ_m) are the first m singular values. 



  
*VAMP-r Score
The VAMP-score <cit.> defines a function that can be used to score the quality of MSMs for a set lag-time τ.  This score can be used to compare the methods used to create the discrete trajectories from the same featured trajectory <cit.>.

With Σ_m as defined above (see <Ref>), the VAMP-r score can be written as:

    VAMP-r=∑_i=1^m σ_i^r


The VAMP-2-score is based on the sum of squared eigenvalues of the transition matrix and can be interpreted as kinetic content <cit.>. In this paper, we use the VAMP-2 score to score the performance of different discretization methods for MD trajectories since there usually is no given ground truth to compare to. The most commonly used scores are the VAMP-1, which is analogous to the Rayleigh trace, and the VAMP-2, which is analogous to the kinetic variance <cit.>. 




§ SUBSPACE CLUSTERING
 
In high-dimensional datasets where the curse of dimensionality renders traditional clustering infeasible, subspace clustering finds clusters in a union of lower dimensional subspaces <cit.>. For a dataset X=[x_1, x_2, ..., x_n]∈ℝ^d it is often assumed that vectors x_j are drawn from a union of k subspaces {S_i}_i=1^k of unknown dimensionality <cit.>. As MD data is typically very high dimensional, subspace clustering is a suitable approach.
For details on subspace clustering, see <cit.>.
In this paper, we focus on spectral-based subspace clustering.



The advantages of spectral subspace clustering methods include a relatively simple implementation and robustness regarding data corruption and initialization <cit.>. Spectral subspace clustering methods usually perform two steps. First, a subspace representation of the original data is learned and used to construct an affinity matrix. Then spectral clustering is used on the affinity matrix to obtain the final clustering. Many different spectral algorithms like sparse subspace clustering (SSC) <cit.> or Low-Rank Representation (LRR) <cit.> have been proposed, for an overview see <cit.>.

While spectral clustering has been successfully applied to numeric data, it lacks the ability to handle temporal data. We, thus, turn our attention to methods that have been developed with a different application goal in mind, but, as we argue, with similarities that make them attractive for the design of our MD subspace clustering method.

Temporal subspace clustering (TSC) <cit.> is a subspace clustering algorithm designed to segment video sequences into separate human motions. 
Video sequences are high-dimensional time series, with one frame of the video at each timestep. 
Significant changes in a video sequence usually are not immediate, but take place over many frames. 
Neighboring time frames are more correlated to each other than more distant frames. 
In that sense, molecular dynamics data is similar to video data. 
Unlike classical subspace clustering methods, which do not take any relations between the dimensions into account, TSC is designed to take advantage of the sequential relationship in time series data. 
TSC uses a dictionary learning approach together with a temporal Laplacian regularization function to encode those relationships in the dataset. Dictionary learning is a method that learns a set of basis vectors, a so-called dictionary <cit.>. The original data can then be represented as linear combinations of the basis vectors. TSC combines the dictionary learning approach with a temporal regularization function to encode the sequential relationships in the data. 
A non-negative dictionary and a corresponding coding are learned from the given data creating a robust affinity graph.



§ MOSCITO: TEMPORAL SUBSPACE CLUSTERING FOR MD DATA
  

We are now ready to describe our MOSCITO (MOlecular dynamics Subspace Clustering with Temporal Observance) approach, the first subspace clustering method for molecular dynamics data that captures temporal relations. 
Existing methods <cit.> often rely on a two- or three-step approach consisting of 1) dimensionality reduction, 2) clustering, and 3) finding macrostates by combining several clusters. As these steps are handled in isolation, their separate results may not be optimal overall. To avoid such undesirable side effects,  MOSCITO takes a holistic approach: 
Firstly, in subspace clustering, it inherently combines clustering with dimensionality reduction. 
Secondly, MOSCITO incorporates temporal relation by weighting sequential neighbors in a time window. In contrast, existing subspace clustering algorithms cannot capture e.g., correlation between consecutive time steps.

For our new method MOSCITO, we exploit the similarity between MD data and video data (see <Ref>) as well as the advances in the field of data mining on videos: TSC <cit.> clusters video data while incorporating the temporal relations between time steps as well as allowing clusters in subspaces of the data. Thus, we base MOSCITO on TSC. However, TSC is not able to handle MD data, because despite some similarities between the videos and MD data (e.g., smooth temporal transitions between time steps and potential clusters in subspaces), they also have strong differences: for every time step , MD data describes a chain of atoms with its 3d coordinates, while video data contains colors for every pixel, where pixels are in a 2d grid. 
Thus, we need to change several steps and include knowledge from the field of MD as shown in Figure <ref>, which provides an overview of MOSCITO. 
The blue parts are equivalent to those of TSC <cit.>, see <Ref>. 
The green parts are adapted or changed completely. 
At first, essential MD features 
are extracted from the MD trajectory. 
From the features and a temporal regularization function, we learn a dictionary and the corresponding representation coding matrix. 
We then construct an affinity graph, and generate temporal segments by clustering the affinity graph.






  
*Feature extraction 
From the input MD trajectors, we extract features that capture essential properties. 

Cartesian coordinates: 
The C^α atoms determine the shape and function of a protein, thus we use the common approach to focus solely on their 3d coordinates, rather than considering all atoms. 
For each frame,  coordinates are centered and aligned to the first frame of the trajectory.

Backbone torsions: 
The backbone torsions determine the main structure of a protein. The torsion angles ϕ and ψ describe the dihedral angles between atoms of the backbone. Backbone torsions better describe the structure of a protein than absolute coordinates <cit.> as internal angles are independent of the global rotation of the protein, removing the need to align all frames to a fixed rotation.  

Distance based: Heavy atoms are all atoms that are not hydrogen. We extract the minimal distance between heavy atoms <cit.> (we exclude the pairs at indices(i, i+1) and (i, i+2), as their distance is largely determined by their chemical bonds). Following <cit.>, distances d are transformed to the negative exponential function e^-d  

Flexible torsions: The flexible torsion feature consists of the χ_1 to χ_5 side chain torsions. The side chains of a protein are chains of atoms, connected to the backbone. The angles are transformed into sine and cosine angles.

Solvent accessible surface area: For each frame of the trajectory, the solvent accessible surface area (SASA) of each residue is calculated using the Shrake-Rupley algorithm <cit.>. The SASA of a residue is calculated by summing up the SASA of each of the residues atoms

3D shape histogram: Based on <cit.>, a combined 3D shape histogram for each frame of the trajectory is created. The state space is separated into five equally spaced shells around the origin of the coordinate system. The shells are split up further by combining them with 12 sectors into 60 cells. Histograms are created by counting the number of atoms in each cell. Unlike <cit.>, all atoms instead of only the surface atoms were used as we are not only interested in the shape and function of the protein at a certain time step, but also in its internal folded structure. Final histograms are normalized between 0 and 1.


  
*Temporal regularization
For trajectories, data points at neighboring time steps are usually more closely related than those at more distant time steps. Let the i-th column of the coding matrix Z a representation of vector x_i, then the encodings of its sequential neighbors, e.g., z_i-1, z_i+1 should be similar to z_i. To achieve this, we use a temporal Laplacian regularization function f(Z). 


    f(Z)=1/2∑_i=1^n ∑_j=1^n w_ij ||z_i - z_j||_2^2 = tr(ZL_T Z^T)

L_T is the temporal Laplacian matrix defined as L_T = D̃ - W. W is the weight matrix, capturing the sequential relationship in X, and D̃_ii = ∑_j=1^nw_ij is a diagonal matrix with the sum of the rows of W on its diagonal. 


The temporal regularization in MOSCITO weighs the sequential neighbors using a weight matrix W∈ℝ^n× n for a trajectory with n timesteps. 
Row i of W represents the weights of x_i's neighbors. 
In the base case, binary weights are used. 
This weighs all s sequential neighbors of x_i the same, independent of the actual temporal distance between the data points. 
In contrast to TSC, MOSCITO offers several 
Naturally, points in close temporal proximity are more strongly correlated than points further apart in time. To model the decrease in correlation, we introduce different decaying weightings: 

Binary:  weights of 1 for points in the neighborhood and 0 outside

Gaussian: Gaussian curve fitted to the neighborhood

Logarithmic: Logarithmic decay with a value of 1 for the direct neighbors dropping logarithmically to 0 for x_i+s and x_i-s

Exponential: Exponential decay w.r.t. the temporal distance 



  
*Dictionary learning Similar to TSC <cit.>, MOSCITO learns to represent dataset X by a dictionary D and a coding matrix Z, such that X≈ DZ <cit.>. 

Adopting a least square regression approach to dictionary learning, we use the following objective function <cit.>

    min_Z,D||X-DZ||_F^2+λ_1||Z||_F^2
    s.t.  Z ≥ 0, D ≥ 0, ||d_i||_2^2 ≤ 1,

where ||Z||_F^2 denotes the Frobenius norm, and we use λ_1 for balancing both terms. 

Importantly, MOSCITO integrates temporal regularization to capture relationships between nearby time steps. We achieve this by adding temporal regularization function f(Z) such that our new objective function is the same as in TSC <cit.>

    min_Z,D||X-DZ||_F^2+λ_1||Z||_F^2 + λ_2f(Z) 
    s.t.  Z ≥ 0, D ≥ 0, ||d_i||_2^2 ≤ 1,

with λ_2 tradeoff parameter <cit.>.

To solve the dictionary learning problem efficiently, we follow the alternating direction method of multipliers (ADMM) <cit.>. It decomposes the problem into subproblems that are more easy to solve. Using auxiliary variables U, V, Eq. <ref> is equivalent to

    min_Z,D,U,V||X-UV||_F^2+λ_1||V||_F^2 + λ_2f(V) 
    s.t.  U=D, V=Z, Z ≥ 0, D ≥ 0, ||d_i||_2^2 ≤ 1.

The augmented Lagrangian of Eq. <ref> is:

    ℒ    = 1/2||X-UV||_F^2 + λ_1||V||_F^2 + λ_2 tr(VL_TV^T) 
        + ⟨Λ, U_D⟩ + ⟨Π, V-Z⟩ + α/2||U-D||_F^2 + β/2||V-Z||_F^2,

with Λ and Π Lagrangian multipliers <cit.>, which can now be solved by alternatingly minimizing ℒ respective to V, U, Z and D. 

Updating V: To find the minimum of ℒ, its derivative with respect to V is set to zero, resulting in the following equation <cit.>:

    [I⊗ (U^TU+(λ_1+α)I) + λ_2L_T⊗ I] vec(V)=vec(U^TX-Π+α Z)


Updating U: As for V, set derivative of ℒ wrt. U to zero <cit.>:

    U=(XV^T-Λ+α D)(VV^T+α I)^-1


Updating Z and D: The update rules for Z and D are <cit.>: 

    Z=F_+(V+Π/β)
    ,     
        D=F_+(U+Λ/α)

where (F_+(A))_ij=max{A_ij, 0}, non-negativity constraint. Additionally, each column of D is normalized to unit length  ||d_i||_2^2≤1.



  
*Clustering
Using the learned coding matrix Z, an affinity graph is constructed. To capture sequential relationships, the following similarity measure is used <cit.>: 

    G(i, j)=z_i^T z_j/||z_i||_2||z_j||_2

Spectral clustering on graph G yields the final clusters.





§ EXPERIMENTS
 



  
Implementation 
We extract features of MD trajectories using libraries PyEMMA <cit.> and MDTraj[<https://www.mdtraj.org>].
Matrices are represented as NUMPY arrays and sparse matrices by scipy sparse matrices. 
We efficiently solve linear equations of the form Ax=b using the PyPardiso 0.4.2[<https://github.com/haasad/PyPardisoProject>] (version 2021.4) wrapper for the Intel oneMKL PARDISO library. 
For spectral clustering, we use the scikit-learn implementation. 
Our Python code for MOSCITO is https://github.com/Mhae98/MOSCITOpublicly available [<https://github.com/Mhae98/MOSCITO>]. 
For PCA, TICA and k-Means we use the implementations in PyEMMA.
The implementation of SSC is based on <cit.> and a Python translation of SSC [<https://github.com/abhinav4192/sparse-subspace-clustering-python>] 
All experiments run on a virtual machine with 64 CPUs and 512GB of RAM running Ubuntu 22.04. 





  
Datasets and parameter settings
We use MD trajectories of four proteins: 2F4K, Ace, 2WXC, and Savinase, which differ in the number of atoms | A|, number of Cα atoms | Cα|, and number of residues | R|, as summarized in Table <ref>, together with number of  trajectories and timesteps in each trajectory.

If not specified otherwise, parameter default values are 
 d= 60, s= 3, λ_1= 0.01, λ_2= 15, a= 0.1, b= 0.1.





 §.§ MOSCITO settings and features
 
In this section, we analyze the parameter sensitivity and impact of feature extraction for MOSCITO. 
For the following experiments, 10 trajectories of the 2WAV-protein are used. The Markov state models are constructed for k∈{10, 50, 100, 200} clusters with a lag time τ∈{1, 10, 100}. The average VAMP2-score of the 10 trajectories is calculated for each choice of clusters k and lag time τ. The VAMP2-score is calculated for five eigenvalues.
It is important to keep in mind that the VAMP2-score for different MSM lag times is not directly comparable, only for different numbers of clusters.




  §.§.§ Dictionary size
  d defines the dimensionality of the learned dictionary and thus the number of subspaces. As such, it is crucial for the clustering and runtime performance of MOSCITO, where larger dictionary size d provides more subspaces to represent the data.
We study d∈{10, 20, ..., 100}, and additionally s∈{2, 3, 4} number of sequential neighbors. Figure <ref> shows VAMP2-scores for different numbers of clusters (rows) and MSM lag times (columns): clustering performance improves for larger dictionary size d until it converges. For smaller MSM lag times, the VAMP2-score converges faster than for larger lag times, independent of the number of clusters. With an increasing number of sequential neighbors s, the overall shape of the curves is highly similar, with a slight preference for more neighbors that capture more of the temporal relationships.






  §.§.§ Sequential neighbors
 
Setting the number of sequential neighbors s in the temporal regularization to 0, the temporal aspect is not considered, with growing s longer temporal relationships are captured. The scores for s∈{0,...,12} are shown in Figure <ref> for different MSM lags (columns) and number of clusters (rows). 
The number of sequential neighbors s clearly affects clustering performance, with too few or too neighbors reducing performance, but with a broad stable range where temporal relationships are successfully captured.

Figure <ref> shows the clustering of a 2F4K protein trajectory with a long folded state into 5 clusters represented by different colors. With no (top) or few (center) sequential neighbors, time intervals comprised by a cluster can be very short.
Increasing the number of sequential neighbors (bottom) leads to continuous clusters with mostly uninterrupted blocks.  





  §.§.§ λ parameters
 
We study settings of the tradeoff parameters  λ_1 ∈{0.01, 0.05, …, 15} for sparsity  of the coding matrix Z, λ_2 ∈{0.01, 0.05, …, 35} for the temporal regularization function.
Figure <ref> shows that best performance is achieved for low values of λ_1  and higher values of λ_2 (light areas in the heatmap). This shows that temporal regularization is the more important factor for optimal clustering performance. 








  §.§.§ Learning rates
 
α, β∈{0.01, 0.05, …, 15} 
define how much Lagrangian multipliers Π and Λ change in each iteration of the algorithm.
Heat maps in Figure <ref> show that the results are stable across parameter settings, with VAMP2-scores in each heatmap relatively close, meaning our default values work well.





  §.§.§ Weighting mode
 
All weighting modes for the Laplacian regularization function: binary, Gaussian, logarithmic, and exponential, are tested for different numbers of sequential neighbors.
In Figure <ref> we see that for few sequential neighbors, the performance for all weighting modes is similar. For many sequential neighbors, performance starts to decrease. In most cases, the performance when using the binary weighting mode decreases earlier than for the other methods, but all reliably reach the same maximal score.






  §.§.§ Feature comparison
 
We analyze the above described features and their combinations via simple concatenation. 
Figure <ref> 
shows the average VAMP2-score for 10 trajectories of the 2WAX-protein, using the five largest eigenvalues for scoring. 






Best results for individual features are achieved with backbone torsions. Overall, combinations of backbone torsions, SASA, and shape histograms yield the best results. Figure <ref> shows the importance that MOSCITO gives to each of the features in its dictionary when clustering the full-dimensional feature space. While, e.g., the feature "flex" (flexible torsions) yields the worst VAMP2 score (left), it still has a high influence on the results (right), explaining the suboptimal VAMP2 score when using the full feature set.




 §.§ Clustering performance
 
In this section, the clustering performance of MOSCITO is compared to that of state-of-the-art approaches that are currently used in the field. We compare to PCA with k-Means and to TICA with k-Means because of their widespread use <cit.>. Additionally, we compare against the sparse spectral clustering algorithm <cit.>, a subspace clustering approach for molecular dynamics data.
For PCA and TICA, the default parameter settings of PyEmma <cit.> were used. SSC is a subspace clustering algorithm, thus it does not need any dimensionality reduction in the preprocessing. 

A general challenge in this kind of evaluation is the fact that no ground truth is given. As such, clustering performance is compared by constructing a MSM and scoring it using the VAMP2-score, in order to obtain quantitative insights. Still, visual inspection plays an important role in assessing the clustering performance, in particular, the temporal connectivity of clusters.



  
*Quantitative analysis
Figure <ref> shows the average VAMP2-score for 20 trajectories of the 2F4K protein in the first row. For MSM lag time 1, the performance of all methods is comparable with the exception of 10 clusters. There, MOSCITO and TICA + k-Means deliver the best performances. For longer MSM lag times, the scores separate with SSC performing better.




In the second row of Figure <ref>, we see the average VAMP2-score for 10 trajectories of the 2WXC protein. For the MSM lag times 1 and 10, the performance for all methods is relatively similar for large cluster sizes. For small cluster sizes, SSC performs poorly, while the other methods already reach close to their maximal score for only 10 clusters. For MSM lag time of 100, PCA + k-Means drastically drops in performance, while the other methods maintain comparable scores to each other.


The third row of Figure <ref> shows the average VAMP2-score for 28 trajectories of the Ace protein. Similar to the results of 2WXC, the performance for a MSM lag time of 1 and 10 is comparable between all methods for a large number of clusters. For few clusters, SSC again performs significantly worse. For MSM lag time of 100, the gap between the methods grows, with SSC performing best.

The last row of Figure <ref> shows the average VAMP2-score for 2 trajectories of Savinase. For all MSM lag times, SSC performs best while TICA + k-Means performs worst. Unlike for the other proteins, all methods require a large number of clusters to reach the maximal VAMP2-score. This might be due to the large size of Savinase and shorter trajectories compared to the other proteins.



  
*Qualitative analysis
The purpose of modeling temporal aspects is to obtain clusters with larger continuous segments, ideally retrieving macrostates directly.  Figure <ref> visually compares the segmentation of clustering methods, either by directly clustering the trajectory into five clusters or, alternatively by coarse-graining 50 clusters into five clusters using PCCA+ <cit.> (see <Ref>). We regard 
a 2F4K trajectory with a known long folded segment.


Directly clustering, i.e. without PCCA+, PCA + k-Means and SSC struggle to find the large folded state as one large block. MOSCITO and TICA+k-Means find similar clusterings and succeed to uncover the large folded state. When applying PCCA+, PCA + k-Means, TICA + k-Means, and MOSCITO generate similar clusterings, comparable to the ones of MOSCITO and TICA + k-Means without PCCA+. In this specific case, SSC fails to find the large segment, which is reflected in the respective VAMP2-score for this specific trajectory which is unusually low.



 §.§ Clustering runtime
 
In this section, the runtime of MOSCITO is studied and compared to the runtime of PCA + k-Means, TICA + k-Means, and SSC. The runtimes for MOSCITO are measured for 20 iterations. The following runtimes are the average of 10 2WXC-trajectories clustered into ten clusters, using all atom coordinates as features. 
The main factors influencing the runtime of MOSCITO, besides the size of the input data, are the dictionary size d and the number of sequential neighbors s. The effect of different dictionary sizes [10, ..., 100] on the runtime is shown left in Figure <ref>. The effect of different numbers of sequential neighbors [0, ..., 25] is on the right. 




The runtime of MOSCITO grows with increasing dictionary sizes. For optimal cluster performance, a large enough dictionary size has to be selected. Large values do not affect clustering performance negatively, but directly impact runtime. A dictionary size between 60 and 80 provides a good tradeoff.
The runtime generally grows with increasing number of regarded sequential neighbors. 
The overall run-to-run variance for the different trajectories is close to zero, indicating a stable runtime for a fixed number of iterations. 

For the comparison of the different clustering methods, backbone torsions were used. The comparison between the different clustering algorithms is shown in Figure <ref>.
PCA + k-Means and TICA + k-Means are both much faster than the subspace clustering methods.  Compared to SSC, MOSCITO is substantially faster. 



 §.§ Discussion
 
We conclude that MOSCITO provides at least comparable performance to state-of-the-art clustering methods for MD data. We evaluated the clustering performance by the clusters' suitability as Markov states using the VAMP2-score. The visual analysis of MOSCITOs clustering results shows a better clustering for higher numbers of sequential neighbors. Thus, MOSCITO is a very promising approach for creating MSMs in a one-step-approach.

MOSCITO's parameter robustness shows that there are good tradeoffs in dictionary size between runtime and clustering performance. The number of sequential neighbors matters, and our experiments indicate a good default value is between 3 and 5 to capture nearby temporal relations. 
In terms of feature extraction, MOSCITO on backbone torsions leads to best results, followed by Cα coordinates and SASA. Combining features does not improve the overall clustering performance. 
While simpler non-subspace methods are even faster, MOSCITO provides a faster runtime than SSC, and state-of-the-art quantitative results, with qualitative analysis showing the promise of the approach.




§ RELATED WORK
 
Distance-based clustering algorithms suffer from the curse of dimensionality when it comes to high-dimensional data like MD data.
Thus, the dimensionality is often reduced in a preprocessing step, e.g., with principal component analysis (PCA) or time-lagged independent component analysis (TICA) <cit.>. Both are widely used for analyzing MD data <cit.>. 
PCA transforms the data onto the axes with the highest variance, also known as principal components. 
PCA is a commonly used for MD data, e.g., in PyEMMA <cit.>, MSMBuilder [<http://www.msmbuilder.org>], or MDAnalysis [<https://www.mdanalysis.org/>].
TICA (Time-lagged independent component analysis) <cit.> linearly projects the data to independent, uncorrelated components (IC). 
The ICs are computed by maximizing the autocovariance for a fixed lagtime τ, and are frequently used for MD data (<cit.>, <cit.>).


State-of-the-art methods often follow a two-step approach, where the trajectories are discretized into a large number of microstates, and subsequently combined to the relevant macrostates.
Since the transitions between the metastable states are taking place over multiple timesteps
it is hard to assign time steps that are part of a transition to a metastable state clearly. 
Perron-Cluster Cluster Analysis (PCCA) (<cit.>,<cit.>) and a more robust version PCCA+ (<cit.>, <cit.>) fuzzily assign the microstates to their corresponding macrostates based on spectral methods for transition matrices:


The transition matrix P of a decomposable MSM with n_c macrostates can be reordered to a block-diagonal matrix P̃ with n_c blocks. 
For the matrix X containing the n_c dominant eigenvectors of P̃, the rows belonging to states in the same block are identical <cit.>. 
These rows represent the n_c corners of a (n_c-1) dimensional simplex.

For MSMs with transition states, i.e., nearly decomposable MSMs, the structure of the simplex is only slightly perturbed. To assign the microstates to the macrostates, they are fuzzily assigned to the n_c corners of the simplex based on their position in the simplex. The goal is to find a linear transformation A∈ℝ^n_c× n_c so that χ=XA where χ_i are the membership vectors. Those memberships all have to be positive and sum up to 1. 
The resulting macrostates generally cannot be considered to be a Markov chain <cit.>.


Time series data can be found in many fields, like traffic analysis, geology, computer vision, or molecular dynamics data. Many existing subspace clustering algorithms do not take the sequential properties of time series data into account <cit.>. One of the first subspace clustering methods taking advantage of sequential relationships in data was SpatSC introduced by <cit.>. A penalty term used to preserve sequential relationships in the learned affinity graph was added to the subspace clustering problem. Based on the same idea, OSC, a more robust subspace clustering method for sequential data was introduced by <cit.>. 
TSC <cit.> further improved on the idea by defining a Laplacian regularization to encode the temporal information. 
Our method MOSCITO builds upon TSC. 

To the best of our knowledge, only one subspace clustering method has been used on molecular dynamics data, namely SSC <cit.>, which was first introduced in the field of image processing <cit.>. We include SSC in our experiments as comparative method, thus we introduce it in detail in the following. 
Note that SSC does not take advantage of any properties that distinguish time series from other high-dimensional data, leading to suboptimal use of the available information in MD data.
The main idea behind SSC is to take advantage of the self-expressiveness of the data, i.e., assuming that every point y_i∈ Y can be rewritten as a linear combination of other points in the dataset. Each datapoint y∈∪_i=1^nS_i can be written as

    y_i=Yc_i,   c_ii=0,

where Y=[y_1,...,y_N] and c_i:=[c_i1  c_i2  ...  c_iN]^T with the constraint c_ii=0 eliminating the trivial solution of representing the point as itself. 
Since each subspace usually has more data points than dimensions the self-expressive dictionary Y is generally not unique. 
Among all those possible solutions for Equation <ref> there is a sparse solution so that the non-zero entries of c_i correspond to datapoints in the same subspace as y_i. This solution can be found by minimizing the l_1-norm of c_i, which can be solved efficiently and is known to prefer sparse solutions <cit.>.

These sparse representations are transferred to an undirected graph where points that are part of another point's sparse representation are connected. Based on this graph, spectral clustering detects the desired subspace clusters. 




§ CONCLUSION
 
We introduced MOSCITO, a subspace clustering method for high-dimensional molecular dynamics data. It uses temporal regularization to capture the relationship of sequential neighbors in time-series data.
While traditional and state-of-the-art methods rely on coarse-graining using PCCA+ in a two-step approach, MOSCITO achieves similar overall performance in a single clustering step.

Especially for small numbers of clusters, the temporal regularization in MOSCITO yields clusters consisting of connected time steps, fitting the real world more closely than alternative methods without relying on any postprocessing steps. 
Compared to current subspace clustering methods on MD data, MOSCITO provides a better runtime.
In future work, the integration of a multi-view approach into MOSCITO might take advantage of the additional information provided by using multiple features.


ACM-Reference-Format




 Paper Title: DistillGrasp: Integrating Features Correlation with Knowledge Distillation for Depth Completion of Transparent Objects.


	
	
	
	IEEE Robotics and Automation Letters. Preprint Version. Accepted Month, Year
	FirstAuthorSurname et al.: ShortTitle 
	
	
	
	
	
	
	
	DistillGrasp: Integrating Features Correlation with Knowledge Distillation for Depth Completion of Transparent Objects
    Yiheng Huang^0009-0007-9527-6569,
			Junhong Chen^0000-0002-4874-9550, 
			Nick Michiels^0000-0002-7047-5867, 
			Muhammad Asim^0000-0002-6423-9809Y. Huang, J. Chen, M. Asim and W. Liu are with the College of Computer Science and Technology, Guangdong University of Technology, Guangzhou, 510006, China. E-mails: (huangyiheng.gdut@gmail.com, CSChenjunhong@hotmail.com, asimpk@gdut.edu.cn, liuwy@gdut.edu.cn.)  (Yiheng Huang and Junhong Chen contributed equally to this work.)    (Corresponding author: Junhong Chen and Wenyin Liu.),  
			Luc Claesen^0000-0003-0405-6290, and Wenyin Liu^0000-0002-6237-6607
				J. Chen and N. Michiels are with Hasselt University - Flanders Make, Expertise Centre for Digital Media, Hasselt, Belgium.
				L. Claesen is with Hasselt University, Hasselt, Belgium.
			 W. Liu is also with Zhongguancun Laboratory, Beijing, China.
		
	
    
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

empty  
	empty 
	
	
	§ ABSTRACT 
		Due to the visual properties of reflection and refraction, RGB-D cameras cannot accurately capture the depth of transparent objects, leading to incomplete depth maps. To fill in the missing points, recent studies tend to explore new visual features and design complex networks to reconstruct the depth, however, these approaches tremendously increase computation, and the correlation of different visual features remains a problem. To this end, we propose an efficient depth completion network named DistillGrasp which distillates knowledge from the teacher branch to the student branch. Specifically, in the teacher branch, we design a position correlation block (PCB) that leverages RGB images as the query and key to search for the corresponding values, guiding the model to establish correct correspondence between two features and transfer it to the transparent areas. For the student branch, we propose a consistent feature correlation module (CFCM) that retains the reliable regions of RGB images and depth maps respectively according to the consistency and adopts a CNN to capture the pairwise relationship for depth completion. To avoid the student branch only learning regional features from the teacher branch, we devise a distillation loss that not only considers the distance loss but also the object structure and edge information. Extensive experiments conducted on the ClearGrasp dataset manifest that our teacher network outperforms state-of-the-art methods in terms of accuracy and generalization, and the student network achieves competitive results with a higher speed of 48 FPS. In addition, the significant improvement in a real-world robotic grasping system illustrates the effectiveness and robustness of our proposed system.
	
	
	
		Distillation learning, depth completion, transparent object grasping.
	
	
	
	
	

§ INTRODUCTION

	Transparent objects are widely used in various scenes, such as manufacturing, household services, etc. However, the perception of transparent objects has been recognized as a critical challenge <cit.> due to their reflective and refractive surfaces that cannot be perceived by the depth cameras, resulting in inaccurate and missing depth values. Without accurate depth maps, many downstream robotics applications cannot be applied, e.g., object grasping, autonomous driving, human-robot interaction, etc.
	
	The task of the depth completion of transparent objects is to recover the complete depth maps from the input of RGB images and raw depth maps, which contains two aspects: correcting the drifting depth values caused by refraction and generating the missing depth values caused by reflection. To prevent the drifting depth values from affecting the learning of the network, authors in <cit.> adopted a Deeplabv3+ network to remove the depth of transparent objects, and then extracted multiple visual maps as complementary information to complete the depth maps. However, the extraction of various visual maps not only increases the computational cost but also brings the problem of feature correlation. Instead of leveraging multiple visual maps, Chen et al. <cit.> introduced Transformer to the depth completion of transparent objects in order to acquire contextual information for completing depth maps. Hong et al. <cit.> integrated the DenseNet and Swin Transformer to extract both local and global features from RGB images and concatenated them with the depth maps to a U-Net architecture network for depth completion. Although these methods utilize advanced networks for fine-grained feature extraction, they still suffer from huge computational costs due to the complexity of the networks.
	
	In order to exploit the advantages of advanced networks while also maintaining model efficiency, in this paper, we propose a novel approach called DistillGrasp which preserves the advanced networks as the teacher networks to transfer the knowledge to the low-capacity student networks. Specifically, for the teacher network, we devise a position correlation block (PCB) which uses a transformer-based method to capture the structure information and establish the correspondence between RGB images and depth maps. For the student network, an efficient consistent feature correlation module (CFCM) is designed to capture the pairwise relationship based on reliable regions according to their consistency. To ensure that the student network can learn comprehensive features, a distillation loss incorporating distance, object structure, and edge information is introduced. Extensive experiments conducted on the ClearGrasp dataset verify the effectiveness and robustness of our methods. 
	
	In summary, our main contributions are as follows:
	
		
  * To the best of our knowledge, this is the first work on depth completion of transparent objects using knowledge distillation
		
		
  * For the teacher and student network, we separately devise two different correlation strategies named position correlation block (PCB) and consistent feature correlation module (CFCM) to capture the pairwise relationship, which guarantees accuracy and speed.
		
		
  * Considering the gap between the teacher and student network, we design a composite distillation loss consisting of distance loss, structural loss, and edge loss to ensure the student network can learn both local and global features.
		
		
  * Extensive experiments conducted on the ClearGrasp dataset illustrate that our teacher network outperforms state-of-the-art methods in accuracy and generalization, while the student network achieves competitive results with higher speed. The successful deployment of our system on a UR10e robot for grasping transparent objects verifies the effectiveness and robustness of our method.
	
	
	
	

§ RELATED WORK

	

 §.§ Depth Completion for Transparent Object Perception

	Depth completion for transparent objects aims to generate the missing depth values by referring to the existing RGB images and depth maps. Existing methods can be divided into two categories based on the number of viewpoints <cit.>, the multi-view approaches and single-view approaches. The multi-view approaches complete the depth maps of transparent objects by utilizing information collected from different perspectives. For example, Klank et al. <cit.> proposed to match two different views to locate the contradictory places, and then leveraged triangulation to correct the wrong point clouds. Authors in <cit.> introduced the Neural Radiance Field (NeRF) to depth completion for transparent objects, but they suffered from huge computation costs. Compared with multi-view approaches, single-view approaches not only avoid capturing multiple views, but also reduce the time of processing large amounts of data from different perspectives. The primary single-view approaches are designed based on particular patterns <cit.>, which cannot handle diverse scenarios and objects. To extend the application scenes, Sajjan et al. <cit.> proposed an end-to-end network to extract multiple visual maps from a large amount of data to complete the missing depth. Hong et al. <cit.> developed a U-Net network to complete the depth based on the correct depth filtered from the reflective and refractive regions. However, their methods focused on local features and ignored contextual information. Authors in <cit.> extended the Transformer to depth completion for transparent objects, offering an enhanced perception of global information. Although the utilization of Transformer can improve accuracy, it also significantly increases computational costs, hindering the deployment in real-world environments.
	
	

 §.§ Knowledge Distillation in Robotic Application

	Knowledge distillation <cit.> was first proposed to transfer refined knowledge from sophisticated teacher models to low-capacity student models, which has been widely deployed in various robotic applications, such as 3D object detection <cit.>, scene segmentation <cit.>, etc. Chen et al. <cit.> proposed a cross-modal knowledge distillation approach that transferred the spatial information from LiDAR to multi-camera BEV for 3D object detection. Hong et al. <cit.> applied knowledge distillation in monocular 3D object detection and extended it as a semi-supervised framework. Cen et al. <cit.> designed a bidirectional fusion network to learn the enhanced 3D representation and leveraged a knowledge distillation framework for improving performance. Although a number of applications made in robotics, there are still relatively few works on robotic grasping. In this work, we introduce distillation learning to transparent object grasping, aiming to utilize the knowledge from advanced teacher networks to distill an efficient student network with high accuracy.
	

§ METHOD

	

 §.§ Overview

	

	As shown in Fig. <ref>, our proposed model contains a teacher network and a student network. In the teacher network, we devise an encoder-decoder network where the encoder leverages position correlation blocks (PCB) to establish the correspondence between RGB images and depth maps, while the decoder adopts NeWCRFs <cit.> to restore the depth. In the student network, we use the same encoder-decoder network where the encoder utilizes Consistent feature correlation modules (CFCM) to capture pairwise relationships based on reliable positional information while the decoder uses the upsampling layers to recover the depth. Finally, a distillation loss with distance loss, structures loss, and edge loss is proposed to transfer the knowledge from the teacher network to the student network.
	
	

 §.§ Teacher Network

	The goal of our work is to improve the performance of synthetic data while keeping robustness to the real-world data. Thus, for the teacher network, we choose the Transformer network to extract spatial features since it performs better in capturing the overall structure of the objects. Specifically, we first adopt DeeplabV3+ to recognize transparent objects on RGB images and remove the corresponding areas on depth maps, which can avoid capturing wrong pairwise relationships between RGB images and depth maps. After that, patch embedding is employed on both channels respectively to split the images into patches and extract features. Because these features come from different channels and cannot provide useful mapping relationships for the network, we proposed our position correlation block to set up the correspondence between them.
	
	Position correlation block (PCB). Given a set of RGB image feature I and depth feature D, we first initialize the features by utilizing a layer normalization:
	
		
		
		
    x_i  = LN(I) , x_d = LN(D)

	where LN refers to layer normalization. In order to correlate the two different features, we introduce the attention layer. Since there are incomplete points and missing points in the depth map, we choose RGB images as query Q to search for the corresponding key K in RGB images and value V in the depth map. The attention layer can be denoted as follows:
	
		
		
		
    Q  = x_i , K = x_i , V=x_d  
    
    			F_qkv = SoftMax(Q· K^T+ B)· V  + I  
    
    			F_corr = MLP(LN(F_qkv)) + F_qkv

	where B is the relative position bias, MLP denotes a two-layer perceptron, F_qkv and F_corr represent the attention score and the correlated features. Although F_corr establishes the correspondence between RGB images and depth maps, it only considers the correlation of the regional patches and ignores the context information. Inspired by the Swin Transformer <cit.>, we adopt shifted window based attention to set up the connection between patches. Particularly, we apply the shifted window partitioning to the correlated features:
	
		
		
		
    I_shift = Shift(F_corr),D_shift = Shift(x_d)

	where Shift denotes shifted window partitioning, I_shift and D_shift represent the shifted RGB image features and the shifted depth features. Similar to the previous attention layer, the shifted image features are used to search for the corresponding key K in shift RGB image features and value V in the shifted depth features. Finally, a patch merging layer is adopted to reduce the height and width by half and expand the number of channels to twice. By stacking up four sets of PCBs, we can obtain the encoding features.
	
	In terms of the decoding stage, we follow <cit.> which develops a bottom-up structure that consists of four neural windows fully-connected Conditional Random Fields modules (NeWCRFs), where each module receives the features from the previous layer as well as the low-level features from the encoder through skip connections and outputs the upscale features utilizing a shuffle operation. Finally, the decoder outputs the completed depth map.
	

 §.§ Student Network

	The student network aims to keep pace with the teacher network while staying efficient at the same time. Therefore, for the student network, we choose CNN as our feature extractor because of its efficiency. Specifically, we first remove the depth of transparent objects by leveraging DeeplabV3+. Different from the teacher network that leverages RGB images to query corresponding positions on depth maps, our student network utilizes a more direct and efficient module called consistent feature correlation module which establishes correspondence based on the reliable positions of the RGB images and depth maps. The detail of the module is introduced as follows.
	
	Consistent feature correlation module (CFCM). Given a set of RGB image feature I and depth feature D, we set up two branches to calculate the consistency respectively, where each branch has the same structure except for the input. For clarity, we take the RGB image branch as an example. In particular, we first leverage a 1  × 1 convolution network to aggregate the different channel information and increase the number of channels. To avoid using too many convolutional layers that destroy the original spatial structure, the aggregated features are concatenated with the original depth features and sent into a 1  × 1 convolution network to compute the consistency score C_I. The consistency score C_I is formulated as follows:
	
		
		
		
    C_I = Conv_1 × 1( Conv_1×1(I) ⊕ D )

	where Conv_1×1 represents the  1  × 1 convolution network,⊕ indicates the channel concatenation, By multiplying the original RGB image feature I and the consistency score C_I, we can obtain the reliable RGB image and P_I.
	
		
		
		
    P_I = C_I⊙ I

	where ⊙ denotes element-wise multiplication. Following the same process, we can obtain a reliable depth map P_D. These two reliable visual maps are combined through element-wise addition to generate the integrated features F.
	
		
		
		
    F = P_I + P_D

	Finally, we devise an efficient CNN block to capture the pairwise relationships, where each block consists of a ReLU activation, a batch normalization, and a convolution layer to capture pairwise relationships. After stacking up four sets of CFCM, we can obtain the decoded features.
	
	In terms of the decoder, four upsampling blocks are deployed, where each block integrates a CNN network and an upsampling layer. Similarly, the encoded features are transferred to the decoder via skip connections, ensuring the multiscale features can be fully utilized. Finally, the decoder outputs the completed depth map.
	
	
	

 §.§ Distillation Loss

	Considering that the CNN-based student network can not acquire sufficient context features and leads to an unsatisfactory performance on the overall completion, we design our distillation loss into three components: distance loss, structural loss, and edge loss, where the distance loss is used to shrink the distance error, and the structural loss and edge loss ensure the overall structure of the completed depth.
	
	Distance Loss. Scale-Invariant Logarithmic (SILog) loss <cit.> has been proven to have the ability to capture scale-invariant depth information, thus we introduce it as our distance loss. Particularly, we first calculated the logarithm difference between the predicted depth map Δ E_p and the real depth Δ E_gt:
	
		
		
		
    Δ E_p =logD_s - logD_t , Δ E_gt =logD_s - logD_gt

	where D_s and D_t denote the predicted depth map completed by the student network and teacher network respectively. D_gt is the groundtruth depth value. For K valid values in the depth map, the SILog loss ℒ_d is computed as follows:
	
		
		
		
    ℒ_d    = α√(1/K∑_^Δ E_gt^2 -  λ/K^2 (∑_^Δ E_gt)^2)
        + β√(1/K∑_^Δ E_p^2 -  λ/K^2 (∑_^Δ E_p)^2)

	where α and β are scale constants and λ is the variance minimizing factor. In our experiments, α and β are set to 3 and 7 respectively, λ is set to 0.85.
	
	
	
	Structural Loss. We utilize the structural metrics from the structural similarity index measure (SSIM) and calculate their mean square error (MSE) as our structural loss. Specifically, the structural metric is defined as follows:
	
		
		
		
    C(x,y) = σ_x,y+θ/σ_xσ_y+ θ

	where x and y represent the depth maps, σ_x is the variance of the depth maps, σ_x,y is the covariance of the depth maps, and θ is the constant to prevent the dominator from being zero. In our work, we compute two sets of structural metrics, where the first metric C(D_gt,D_s) is computed between the groundtruth depth maps and the predicted depth maps from the student network, and the other metric C(D_t,D_s) is calculated between the predicted depth maps from the teacher network and the student network. Finally, the structural loss ℒ_s is computed as follows:
	
		
		
		
    ℒ_s = MSE(C(D_gt,D_t),C(D_t,D_s))

	where MSE denotes the mean square error.
	
	Edge Loss. Since the boundary of the transparent objects is always confused with the background, our edge loss is designed to capture the horizontal and vertical variations of the consecutive points. Particularly, the horizontal variations are defined as follows:
	
		
		
		
    V_x^r = |D_x,y^r - D_x+1,y^r| ⊙ M_x
where V_x^r denotes the horizontal variations of the depth maps r, D_x,y^r represents the depth maps r, M_x is the mask of the transparent objects in the horizontal direction. Similarly, the vertical variations are defined as follows:
	
		
		
		
    V_y^r = |D_x,y^r - D_x,y+1^r| ⊙ M_y

	where V_y^r denotes the vertical variations of the depth maps r and M_y is the mask of the transparent objects in the vertical direction. In our work, we compute two edge losses, where the first loss ℒ_e^gt,s is computed between the groundtruth depth maps and the predicted depth maps from the student network.
	
		
		
		
    ℒ_e^gt,s = MSE(V_x^gt,V_x^s) + MSE(V_y^gt,V_y^s)

	and the other ℒ_e^t,s is calculated between the predicted depth maps from the teacher network and the student network.
	
		
		
		
    ℒ_e^t,s = MSE(V_x^t,V_x^s) + MSE(V_y^t,V_y^s)

	
	In summary, the total loss of DistillGrasp ℒ is defined as follows:
	
		
		
		
    ℒ = ℒ_d + λ_1 *  ℒ_s + λ_2 *  ℒ_e^gt,s + λ_3 *   ℒ_e^t,s

	where λ_1, λ_2 and λ_3 are empirically set as 0.1, 0.3, and 0.7, respectively.
	
	

§ EXPERIMENTS

	

 §.§ Dataset

	We conducted our experiments on a large-scale transparent object dataset called ClearGrasp <cit.>, which contains 9 types of synthetic objects and 10 types of real-world objects with a total number of over 50k images. The objects are shown in Fig. <ref>. In this work, we follow the data divisions in Sajjan et al. <cit.> by using 5 known synthetic objects for training, and 5 overlapping real-world objects for testing. To verify the generalizability, 4 novel synthetic objects, and 5 novel real-world objects are used for testing.
	
	
	[table]
		labelsep=newline,
		singlelinecheck=true,
	
	[table]labelformat=simple, labelsep=newline, textfont=sc,justification=centering
	font=scriptsize
	
	
	

 §.§ Evaluation Metrics

	Following previous works  <cit.>, we adopt four common evaluation metrics including the root mean square error (RMSE), absolute relative difference (REL), mean absolute error (MAE), and accuracy with threshold δ_t. Particularly, the threshold t is set to 1.05, 1.10, and 1.25, where 1.05 is the strictest requirement for evaluating depth completion. Note that all these metrics are calculated based on the transparent object areas.
	
	

 §.§ Baselines

	We compare our teacher network (denoted as Our(T)) and student network (denoted as Our(S)) with recent deep learning methods as follows:
	
	ClearGrasp (CG) <cit.>, extracted multiple visual features from RGB-D images to infer the depths of transparent objects.
	
	DepthGrasp (DG) <cit.>, designed a self-attentive adversarial network with spectral residual blocks to capture the geometric information and generate the missing points.
	
	TransCG (DFNet) <cit.>, proposed an end-to-end depth completion network that uses U-Net architecture to extract fused features and outputs a refined depth map.
	
	Local Implicit Function (LIDF) <cit.>, leveraged a local implicit neural representation based on ray-voxel pairs to interface the depth maps.
	
	Fast Depth Completion (FDCT) <cit.>, introduced a fusion branch and cross-layer shortcuts to capture local information and designed a loss function to optimize the training process.
	

 §.§ Performance of the Approaches on Depth Completion

	Table <ref> summarizes the performance of different approaches on depth completion, from which we have some observations. 1) CG and DG achieve stable results in both synthetic and real-world environments, manifesting that the additional geometrics features can provide extra structural information in reducing the gap between synthetic data and real-world data. However, the direct concatenation of different visual features limits the performance of the methods. 2) DFNet and FDCT achieve excellent results on the synthetic known objects, but they perform poorly on real-world objects, potentially due to the architecture of CNNs that only consider the local information and are more likely to get overfitting. 3) LIDF performs excellent results in both synthetic and real-world environments, however, the offset ray-voxel pairs will affect the surrounding depth values, leading to regional bias. 4) Our teacher network achieves a competitive result on synthetic known objects and outperforms other methods in synthetic novel objects, real-known objects and real-novel objects, demonstrating the effectiveness and generalizability of our method. In particular, our network has achieved a maximum of 6% improvement on real known objects, which is beneficial to the architecture of the transformer network for capturing geometric structure and position correlation block for correlating positional information. 5) Although there is a gap between the student network and the teacher network, the student network still achieves promising results on the novel objects. We believe it is because the teacher network effectively distills the structure and the edge information from the RGB images and depth maps and successfully teaches them to the student network.
	
	In Fig. <ref>, we visualize some examples of completed depth maps and highlight the generated details with red circles. From the figures we can observe that both our teacher network and student network can complete the depth map with clear shape and contour, but the student network still needs to improve the completion of object boundaries.
	
	

 §.§ Speeds of the Approaches on Depth Completion

	

	To simulate the real-world deployment, we use real novel objects for testing the time cost. Table <ref> lists the time cost and parameters size of different approaches, from which we can observe that FDCT uses the least time to complete the depth map, however, its performance is not promising. Although our teacher network can achieve the best results in completing the depth map, it sacrifices twice as much time. Unlike the previous two networks, our student network achieves an ideal balance in time and accuracy, where it only sacrifices a little accuracy to acquire higher efficiency, reaching the speed of 48 FPS.

	

	

 §.§ Effect of Different Sets of Layers

	As one of the important hyperparameters, the number of layers plays an important role in the results. Thus, in Table <ref> we summarize the performance by choosing different sets of layers. Note that we use Teacher_X and Student_X to denote different types of networks, where X denotes the number of layers. From the table, we can notice that when the number of layers is set to 4, the network can achieve the best results. While using fewer or more layers, the performance will drop a little bit. We believe that it is because when using fewer sets of encoders, the network cannot further extract the details from the RGB images and depth maps, leading to lower accuracy. While using more encoders, the features will be compressed very small, where the spatial structure of the object will be violated. Moreover, the small size of features will increase the difficulties for the network to recover the depth maps.

	
	
	

 §.§ Ablation Study

	We design a series of ablation experiments to verify each component. For the teacher network, we adopt self-attention mechanisms based on RGB images and depth maps respectively to replace the position correlation block (PCB) and denote them as TCH_SA(RGB) and TCH_SA(Depth), respectively. For the student network, we denote the student network without consistent feature correlation module (CFCM) as STU_CFCM, the student network without distillation loss as STU_DL, the student network without CFCM and distillation loss as STU_CFCM_DL, and student network without teacher network as STU_Alone. Note that we use common loss RMSE to replace the distillation loss and adopt UNet as our baseline model. Additionally, we further verify the effect of each distillation loss.
	
	Table <ref> shows the experimental results, from which we can find that our PCB in the teacher network achieves a maximum 18.2% improvement compared with the single input of self-attention, illustrating that the correlation of features can facilitate depth completion. In the student network, the employment of CFCM and distillation loss obtain a certain improvement in accuracy, where the CFCM has a greater impact on the depth completion because it fuses the two different types of features and decides the entire quality of the depth completion, while distillation loss only plays a part in optimizing the generation of edge and object structure. 
	
	Intuitively, we visualize some examples of the completed depth maps generated by the ablation experiments and highlight the generated details with red circles. The results are shown in Fig. <ref>. From Fig. <ref>(a) we can observe that our position correlation block can better deal with the complex shapes objects owing to the correspondence established between RGB images and depth maps. From Fig. <ref>(b) we can notice that the adoption of CFCM has a better result on the overall depth completion and the usage of distillation loss can improve the generation of the object structure and contour.
	
	
	Table <ref> illustrates the effect of each distillation loss, from which we can notice that the distance loss plays a critical role in the performance as it computes the point-wise errors and decides the overall completion. The structure loss and edge loss determine the object structure and boundary details respectively, which improve the results to a certain extent. The combination of loss functions achieves the best results since it guarantees depth completion in overall effect and details.

	

	

 §.§ Robot Grasping

	

	In order to demonstrate the applicability of our network in practical tasks, we deployed our system on a single-arm robot UR10e to grasp real-world transparent objects. Specifically, we choose the GR-CNN <cit.>, which was verified by the previous work <cit.>, as our grasping method. Fig. <ref>(a) presents 8 novel transparent objects that do not appear in the ClearGrasp dataset for grasping, including  “Corrugated collins glass”, “Goblet”, “Rocks glass”, “Corrugated round glass”, “Hexagon cup”, “Handler cup”, “Cylindrical cup” and “Seasoning pot”. For each object, the robot will attempt to grasp it 10 times, and only when the object is held for more than 10 seconds will be considered successful. 
	
	Table <ref> lists the success rate of grasping, from which we can observe that our proposed network can tremendously improve the success rate compared with using raw depth maps and outperform the state-of-the-art method DepthGrasp in real-world grasping. Fig. <ref>(b) shows a few snapshots of UR10e grasping transparent objects, from which we can see that the robot can successfully hold the transparent objects, demonstrating the effectiveness of our proposed method.
	
	
	
	

§ CONCLUSION

	In this paper, we propose a novel distillation network called DistillGrasp for completing the depth of transparent objects. Specifically, in the teacher network, we proposed a position correlation block to search for the positional correspondence between RGB images and depth maps. For the student network, we designed an efficient consistent feature correlation module to capture the pairwise relationship based on reliable positional information. In order to transfer the overall knowledge from the teacher branch to the student branch, we present a distillation loss that takes distance loss, object structure, and edge information into consideration. Extensive experiments demonstrated that our teacher network achieves state-of-the-art performance on accuracy and generalization while the student network achieves ideally balanced results in terms of efficiency and speed. Moreover, the deployment of our system on a robot effectively verifies the applicability of our system.
	
	
	

§ ACKNOWLEDGMENT

	This work is supported by the National Natural Science Foundation of China (No. 91748107), the Special Research Fund (BOF) of Hasselt University (No. BOF23DOCBL11), the foundation of State Key Laboratory of Public Big Data(No. PBD2023-11), the Guangdong Innovative Research Team Program (No. 2014ZT05G157). Chen Junhong was sponsored by the China Scholarship Council (No. 202208440309).
	
	
	ieeetr
	40
		ref1
		J. Weibel, P. Sebeto, S. Thalhammer, and M. Vincze, “Challenges of Depth Estimation for Transparent Objects,” International Symposium on Visual Computing, pp. 277-288, 2023.
		ref4
		S. Sajjan, M. Matthew, M. Pan, N. Ganesh, J. Lee, A. Zeng, and S. Song, “ClearGrasp:3D shape estimation of transparent objects for manipulation,” IEEE International Conference on Robotics and Automation (ICRA), pp. 1-13, 2020.
		ref5
		Y. Tang, J. Chen, Z. Yang, Z. Lin, Q. Li, and W. Liu, “Depthgrasp: Depth completion of transparent objects using selfattentive adversarial network with spectral residual for grasping,” IEEE International Conference on Intelligent Robots and Systems (IROS), pp. 5710-5716,2021.
		ref6
		K. Chen, S. Wang, B. Xia, D. Li, Z. Kan, and B. Li, “TODETrans: Transparent Object Depth Estimation with Transformer,” IEEE International Conference on Robotics and Automation (ICRA), pp. 4880-4886, 2023.
		ref7
		Y. Hong, J. Chen, Y. Cheng, Y. Han, F. Van Reeth, L. Claesen, and W. Liu, “Cluedepth grasp: Leveraging positional clues of depth for completing depth of transparent objects,” Frontiers in Neurorobotics, vol. 16, 2022.
		ref8
		J. Jiang, G. Cao, J. Deng, T. -T. Do and S. Luo, "Robotic Perception of Transparent Objects: A Review," in IEEE Transactions on Artificial Intelligence, vol. 5, no. 6, pp. 2547-2567, 2023.
		ref9
		U. Klank, D. Carton, and M. Beetz, “Transparent object detection and reconstruction on a mobile platform,” IEEE International Conference on Robotics and Automation (ICRA), 2011, pp. 5971–5978.
		ref11
		J. Ichnowski, Y. Avigal, J. Kerr, and K. Goldberg, “Dex-nerf: Using a neural radiance field to grasp transparent objects,” Conference on Robot Learning (CoRL), 2021.
		ref12
		J. Kerr, L. Fu, H. Huang, J. Ichnowski, M. Tancik, Y. Avigal, A. Kanazawa, and K. Goldberg, “Evo-nerf: Evolving nerf for sequential robot grasping,” Conference on Robot Learning (CoRL), 2022.
		ref13
		Y. Qian, M. Gong, and Y. H. Yang, “3d reconstruction of transparent objects with position-normal consistency,” in Proc. IEEE/CVFInternational Conference on Pattern Recognition(ICPR), pp. 4369–4377, 2016.
		ref19
		Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
		ref32
		Z. Chen, Z. Li, S. Zhang, L. Fang, Q. Jiang, and F. Zhao, "Bevdistill: Cross-modal bev distillation for multi-view 3d object detection," International Conference on Learning Representations (ICLR), 2023.
		ref33
		Y. Hong, H. Dai, and Y. Ding, "Cross-modality knowledge distillation network for monocular 3d object detection," in European Conference on Computer Vision. Springer, 2022, pp. 87–104.
		ref34
		J. Cen, S. Zhang, Y. Pei, K. Li, H. Zheng, M. Luo, Y. Zhang, and Q. Chen, "Cmdfusion: Bidirectional fusion network with cross-modality knowledge distillation for lidar semantic segmentation," IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 771–778, 2023.
		ref35
		F. Jiang, H. Gao, S. Qiu, H. Zhang, R. Wan, and J. Pu., "Knowledge Distillation from 3D to Bird’s-Eye-View for LiDAR Semantic Segmentation," in 2023 IEEE International Conference on Multimedia and Expo (ICME), Brisbane, Australia, 2023 pp. 402-407.
		ref24
		W. Yuan, X. Gu, Z. Dai, S. Zhu, and P. Tan, “Neural Window Fully-connected CRFs for Monocular Depth Estimation,”  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3906-3915, 2022.
		ref26
		Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” IEEE/CVF International Conference on Computer Vision (ICCV),  pp. 10012-10022, 2021.
		ref27
		D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” Neural Information Processing Systems(NIPS), pp. 2366-2374, 2014.
		ref28
		H. Fang, H. -S. Fang, S. Xu and C. Lu, “TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 7383-7390, 2022.	
		ref31
		L. Zhu, A. Mousavian, Y. Xiang, H. Mazhar, J. Eenbergen, S. Debnath, and D. Fox, “RGB-D Local Implicit Function for Depth Completion of Transparent Objects,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),  pp. 4647-4656, 2021.
		ref29
		T. Li, Z. Chen, H. Liu, and C. Wang, “FDCT: Fast Depth Completion for Transparent Objects,”  IEEE Robotics and Automation Letters, vol. 8, no. 9, pp. 5823-5830, 2023.
		ref30
		S. Kumra, S. Joshi, and F. Sahin, “Antipodal robotic grasping using generative residual convolutional neural network,” IEEE International Conference on Intelligent Robots and Systems (IROS), pp. 9626-9633, 2020.
	
	


 Paper Title: Simple but Efficient: A Multi-Scenario Nearline Retrieval Framework for Recommendation on Taobao.









yingcai.myc@taobao.com

  Alibaba Group
  China



shanyi.wzy@taobao.com

  Alibaba Group
  China



yuliang.yyl@alibaba-inc.com

  Alibaba Group
  China



joshuawu.wujian@taobao.com

  Alibaba Group
  China



mengzhu.jyn@taobao.com

  Alibaba Group
  China





















§ ABSTRACT 
    In recommendation systems, the matching stage is becoming increasingly critical, serving as the upper limit for the entire recommendation process. Recently, some studies have started to explore the use of multi-scenario information for recommendations, such as model-based and data-based approaches. However, the matching stage faces significant challenges due to the need for ultra-large-scale retrieval and meeting low latency requirements. As a result, the methods applied at this stage (collaborative filtering and two-tower models) are often designed to be lightweight, hindering the full utilization of extensive information. On the other hand, the ranking stage features the most sophisticated models with the strongest scoring capabilities, but due to the limited screen size of mobile devices, most of the ranked results may not gain exposure or be displayed.
    
    In this paper, we introduce an innovative multi-scenario nearline retrieval framework. It operates by harnessing ranking logs from various scenarios through Flink, allowing us to incorporate finely ranked results from other scenarios into our matching stage in near real-time. Besides, we propose a streaming scoring module, which selects a crucial subset from the candidate pool. Implemented on the "Guess You Like" (homepage of the Taobao APP), China's premier e-commerce platform, our method has shown substantial improvements—most notably, a 5% uptick in product transactions. Furthermore, the proposed approach is not only model-free but also highly efficient, suggesting it can be quickly implemented in diverse scenarios and demonstrate promising performance.








<ccs2012>
   <concept>
       <concept_id>10002951.10003317.10003347.10003350</concept_id>
       <concept_desc>Information systems Recommender systems</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Information systems Recommender systems












Simple but Efficient: A Multi-Scenario Nearline Retrieval Framework for Recommendation on Taobao
    Yuning Jiang
    
================================================================================================





§ INTRODUCTION & RELATED WORK


With the rapid advancement of e-commerce platforms, recommendation systems <cit.> have become vital for personalized content recommendation, improving user experiences, and boosting business revenues.
In most industrial scenarios <cit.>, recommendation systems are challenged with vast candidates and strict latency constraints, which hence consist of several stages (matching <cit.>, pre-ranking <cit.> and ranking <cit.>) to balance efficiency and accuracy.
The matching stage is designed to quickly narrow down relevant items from a vast pool of candidates, prioritizing low computational complexity for scalability. This sets the stage for the subsequent pre-ranking and ranking phases, where more granular scoring takes place building on the initial filtering.



Positioned at the very outset of the recommendation systems, the matching phase inherently determines the maximum potential efficacy of recommendation quality.
Existing matching algorithms in recommendation systems predominantly fall under two broad categories: collaborative filtering-based methods <cit.> and deep learning-based methods <cit.>. Collaborative filtering-based methods leverage historical interactions between users and items or latent factors to understand user preferences, yet they often grapple with issues related to data sparsity. On the other hand, deep learning-based approaches <cit.> typically make use of comprehensive neural network models in an end-to-end fashion to deduce how users might feel about each item. Although effective, these methods rely on simpler model structures <cit.>, which might not fully grasp the complexities of user preferences. To address the limitations of model complexity in the matching phase, Netflix proposed a nearline retrieval framework <cit.> that utilizes indirect calculations for complex model inference. However, this approach suffers from delayed real-time responses and significantly increases resource expenditure.

Recently the concept of multi-scenario modeling <cit.> has been introduced to recommendation systems. Using Taobao as a case in point, Figure <ref> illustrates various personalized recommendation scenarios including the homepage, search results page, and post-purchase page. User behavior observed in one scenario can potentially inform and enhance recommendations in another. Nevertheless, most current multi-scenario modeling approaches <cit.> employ complex frameworks, leading to a significant uptick in computational cost, rendering them impractical for the matching phase. In practice, within each scenario, the ranking stage evaluates thousands of candidates; however, due to page limitations, typically only tens of items are ultimately displayed to the user. As a result, a substantial number of high-scoring items from the ranking stage remain unseen by users despite their latent potential.







To address the above challenges, our paper presents a novel Multi-scenario Nearline Retrieval (MNR) approach for the matching stage. We leverage the power of complex ranking models from various scenarios by aggregating the ranking results for each user into a consolidated candidate pool. We then introduce a real-time scoring module that operates online, assessing the pool by considering both the initial ranking and the access time of each item to identify the optimal set for matching.
Implemented on Taobao, China's leading e-commerce platform, our approach has yielded significant performance gains in live environments, signaling a major advancement in the field of recommendation systems.

The major contributions are summarized as follows,



  * We exploit a multi-scenario nearline retrieval framework in the matching stage, which is both model-free and efficient. To the best of our knowledge, we are the first to harness the capabilities of complex ranking models across different scenarios for the matching stage.


  * The proposed MNR performs a streaming scoring module, which evaluates the candidate pool by considering both the initial order and access time of each item.


  * We carried out online experiments on Taobao, one of the globe's most expansive recommendation platforms, and observed a 5% uptick in transaction volume. These findings underscore the efficiency and superiority of the framework we've introduced.







§ METHOD






 §.§ Problem Formalization

The goal of the matching stage in industrial recommendation systems is to sift through a vast item pool E = {e⃗_1, e⃗_2, ⋯, e⃗_|E|}, numbering in the billions, for each user u⃗. This process aims to narrow down the items to a tailored subset ℰ_u (|ℰ_u| << |E|), ensuring that each item within this subset aligns with the user's interests,

    ℰ_u = ℱ_Matching (E),




 §.§ Multi-Scenario Nearline Retrieval

In this work, we emphasize the significance of utilizing the ranking results from other scenarios to enhance the matching stage of the current scenario. Specifically, a user may be involved in multiple scenarios (s⃗_⃗1⃗, s⃗_⃗2⃗, ⋯, s⃗_⃗l⃗) in a platform. In each scenario s⃗, the ranking stage will provide a set of scored items based on the ranking model. 
Taking into account that a user may visit a scenario multiple times, the ranking results for a user within scenario s⃗ can be represented as follows:

    R^t1_s   = ( e⃗_1^t1, e⃗_2^t1, ⋯, e⃗_n^t1), 
    R^t2_s   = ( e⃗_1^t2, e⃗_2^t2, ⋯, e⃗_n^t2),

where t1 and t2 denote different access times of the user, n denotes the number of scoring candidate sets for ranking stage. For each scenario s⃗, the goal of MNR can be formalized as follow, 

    ℰ_s = ℱ_NMR (R^t1_s, R^t2_s, ⋯, R^tN_s),

where ℱ_NMR denotes the function that identifies the top m items critical to the current scenario from various refined ranking candidate sets. ℰ_s = { e_1, e_2, ⋯, e_m } comprises the ultimate selection of items retrieved based on the ranking results within scenario s⃗.

Scenario-aware Ranking Results Collecting.
In each scenario, items filtered by the pre-ranking stage are then scored in the ranking stage. Despite obtaining scores for all items during the ranking stage, not every item in the candidate set aligns with user preferences. Consequently, we initially truncate the ranking results obtained from each visit,

    ℛ^t1_s = Truncate_s(R^t1_s).

Then it is non-trivial to aggregate several ranking results at different access times (ℛ^t1_s, ℛ^t2_s, ⋯, ℛ^tl_s) into a unified set. With online storage being a limiting factor, we utilize a priority queue for each scenario s⃗ to maintain a record of each user's ranking history,

    𝒞_s = Queue(ℛ^t1_s, ℛ^t2_s, ⋯, ℛ^tl_s),

here we adopt a first-in-first-out (FIFO) strategy to retain the user's latest preferences. In this way, we can retain user interests from other scenarios without incurring extra computational overhead.

Streaming Candidate Scoring.
Upon compiling the aggregated ranking results 𝒞s that reflect user preferences in scenario s⃗, it's crucial to refine and limit 𝒞s to manage the size of the retrieval set. We prioritize two aspects: the initial order of ranking results and the timing of user interaction. The ranking phase typically employs sophisticated models with robust feature interactions, suggesting that items with higher scores are more attuned to user interests. On the other hand, as user tastes can evolve, rankings closer to the present moment often correlate better with current preferences. To reconcile ranking with access timing, we can define the scoring function as follows,

    finalScore = (α/α + rank_index) × (β/β + time_index),

where rank_index indicates an item's position in the initial ranking results, arranged in descending order. The term time_index refers to the sequence of a user's visits, with the index of the most recent visit being 0. α and β act as balancing hyper-parameters, influencing the significance of item ranking and timing of access, respectively. By assigning a higher value to α, the scoring formula places greater importance on the timing of the user's visit. Alternatively, elevating β steers the focus towards the original ranking score of the items. Based on Equation <ref>, the final retrieval results ℰ_s is obtained by by choosing the top-k items from 𝒞s.

Implementation Details.
The proposed framework employs the Flink platform for subscribing to rank logs from different scenarios (records of candidate items and their scoring during ranking processes). Quasi-real-time operations such as queue management, aggregation, and top-k calculations are performed. To promote category diversity, we disperse items across categories when generating TopK results, limiting the number of items per category to a predefined threshold. 





§ EXPERIMENT





 §.§ Experiment Setting.

Experiment Scenarios.
In the work, we evaluate the effectiveness of MNR on Taobao Homepage and MNR utilizes the ranking results from several other scenarios on Taobao. The involved scenarios are as follows:

    
  * Taobao Homepage  is the front page of one of the world's largest e-commerce platforms.
    
  * Main Search is the largest e-commerce search system in China.
    
  * Photo Search is an image search feature integrated within Taobao, enabling users to search for products using images.
    
  * Post-purchase activates with recommendations after a user makes a purchase, often spotlighting items that complement the recent buy.
    
  * In-shop takes into account the user's interactions within the shop and the shop's inventory to suggest items.
    
  * Mini-Detail is a high-traffic scenario on Taobao, presenting users with additional product recommendations alongside the currently viewed item after a click-through.


Baselines.
To further evaluate the effectiveness of our proposed framework, we compare our method with three representative methods in industrial recommendation systems as follows,


    
  * SWING <cit.> emerges as a state-of-the-art approach in collaborative filtering. It generates recommendations by directly learning a sparse similarity matrix over items and operates by optimizing a regularized objective function.
    
  * MIND <cit.> is a deep learning-based algorithm specifically tailored for the recommendation system domain. It uses a dynamic routing mechanism to further enhance its capability to adaptively aggregate these different interests based on the target item.
    
  * Glare <cit.> operates as a nearline retrieval framework, focusing solely on the model's capabilities within its own scenario, with an implementation that unfolds in a non-real-time manner.



Evaluation Metric.
We adopt two indicators that the industrial recommendation systems pay the most attention to,

    
  * CTCVR signifies the likelihood of an item transitioning from exposure to a transaction.
    
  * Hitrate indicates the percentage of a user's actual behaviors at all scenarios accurately hit by the items retrieved, which have been normalized based on the number of retrievals.

We also present the PVR in the experiment, which represents the percentage of the finally exposed items that come from this particular matching approach. A higher PVR indicates that the following stages have a greater acceptance of the items retrieved.



 §.§ Overall Performance




Table <ref> illustrates the performance outcomes associated with different methods, where Online_ALL represents the overall performance at the matching stage.
Noteworthy in the table is the robust performance uplift observed in MNR_MiniDetail and MNR_MainSearch, marking improvements of 35.0% and 57.0% in CTCVR, respectively.  
These metrics underscore the efficacy of MNR's strategic utilization of cross-scenario insights, elevating its precision in predicting and matching user preference.
In terms of Hitrate, MNR_MiniDetail and MNR_MainSearch report substantial improvements, with Hitrate of 1.94% and 2.02%, outperforming the foundational 0.62% of Online_ALL.
Besides, the results from Table <ref> further indicate that MNR also surpasses other competing methods such as SWING, MIND, and Glare. It can be observed that MNR represents significant improvements over Swing's 0.43%, Mind's 1.21%, and Glare's 0.66%, respectively, translating to more accurately targeted recommendations.
Moreover, scenarios like MNR_In-Shop and MNR_PhotoSearch are challenged by a lack of ample finely ranked scores, leading to the implementation of a lower truncation threshold. Consequently, this curtails the quantity of items funneled into the matching pool, resulting in a diminished PVR.
The underwhelming performance of MNR_In-Shop can be linked to the nature of in-shop scenarios, where users are limited to the items of a specific shop once they enter it, typically ranging from a few dozen to about a hundred items. The majority of products are likely to be exposed, and so those captured by the MNR framework for recall are predominantly tail-end items.



 §.§ Ablation Study




Impact of Online Stream Computing.
To evaluate the impact of online stream computing, we compare the MNR with online (real-time stream computing) and offline (daily updated) modes. From Table <ref>,  MNR with  offline mode shows a significant reduction in performance with a 36.0% decline, registering a PVR of just 3.02%. This stark contrast highlights the critical advantage of real-time data processing offered by the online mode, cementing the importance of stream computing in enhancing the accuracy and efficiency of recommendation models.



Impact of Streaming Candidate Scoring.
MNR utilizes Equation <ref> to score the ranking results collected in the queue from each scenario. To validate the effectiveness of our proposed scoring function, we evaluate against both a First-In-First-Out (FIFO) strategy as well as a random strategy, to evaluate their impacts.
From Table <ref>, we can observe that MNR shows a significant improvement in effectiveness when compared to alternative strategies. The base PVR of MNR stands at 6.99%. In contrast, the MNR with FIFO strategy, focusing solely on the temporal aspect and ignoring the ranking scores, shows a substantial decrease in performance with only a 1.93% effectiveness rate and an 8.9% reduction compared to the base. Similarly, the MNR with random strategy, which employs a completely random selection method, results in an even more pronounced drop to 0.35% effectiveness, with a 23.6% reduction. This comparison highlights the superior efficacy of the MNR's targeted scoring approach in optimizing recommendation outcomes over simpler FIFO or random strategies.



Impact of hyper-parameter α.
α plays a crucial role as a trade-off hyper-parameter, mediating the balance between ranking influence and access time. 
The results of the impact of hyper-parameter α are shown in Table <ref>. Notably, as α increases, we observe a trend where PVR remains relatively stable but CTCVR experiences a significant decline. At α values of 200 and 500, the CTCVR drops by -12.0% and -15.0%, respectively, indicating that setting α too high might lead to an overemphasis on visit time at the expense of relevant ranking factors. This trend highlights the importance of carefully calibrating α to strike an optimal balance between recency and relevance in our MNR framework.



 §.§ Online performance.

We conducted online experiments by deploying MNR with multiple selected scenarios data to handle the real traffic of Taobao's homepage for a month. 
The results from real traffic on the Taobao homepage demonstrate that our proposed method, compared to the baseline, achieves a consistent and stable increase of 5% in transaction volume. This highlights the commercial value and practicality of our approach. Moreover, the proposed algorithm exhibits efficient computational performance and low resource consumption, rendering it a promising solution for real-time personalized recommendation scenarios in recommendation systems.





§ CONCLUSION


In conclusion, the proposed multi-scenario nearline retrieval framework enhances the matching stage of recommendation systems, addressing the constraints of scale and latency by leveraging ranking logs from various scenarios in near real-time. The integration of this framework within the Taobao APP has shown a commendable increase in product transactions by 5%, indicating its effectiveness and potential for broader application. This advancement underscores the benefits of cross-scenario information utilization and in future work, we intend to optimize our pre-ranking model by integrating recalls from multiple scenarios.





ACM-Reference-Format




 Paper Title: DiscipLink: Unfolding Interdisciplinary Information Seeking Process via Human-AI Co-Exploration.



DiscipLink]DiscipLink: Unfolding Interdisciplinary Information Seeking Process via Human-AI Co-Exploration



cb.zheng@connect.ust.hk
0000-0003-0226-9399

  Hong Kong University of Science and Technology
  Hong Kong
  China



yzhangiy@connect.ust.hk
0000-0001-8263-1823

  Hong Kong University of Science and Technology
  Hong Kong
  China



zhuangbi@connect.ust.hk
0000-0001-8199-071X

  Hong Kong University of Science and Technology
  Hong Kong
  China



chuhanshi@seu.edu.cn
0000-0002-3370-1626

  Southeast University
  Nanjing
  China



mxubh@connect.ust.hk
0009-0001-0591-7526

  Hong Kong University of Science and Technology
  Hong Kong
  China



mxj@cse.ust.hk
0000-0002-9847-7784

  Hong Kong University of Science and Technology
  Hong Kong
  China



§ ABSTRACT 
  Interdisciplinary studies often require researchers to explore literature in diverse branches of knowledge. 
  Yet, navigating through the highly scattered knowledge from unfamiliar disciplines poses a significant challenge. 
  In this paper, we introduce , a novel interactive system that facilitates collaboration between researchers and large language models (LLMs) in interdisciplinary information seeking ().
  Based on users' topic of interest, initiates exploratory questions from the perspectives of possible relevant fields of study, and users can further tailor these questions.
  then supports users in searching and screening papers under selected questions by automatically expanding queries with disciplinary-specific terminologies, extracting themes from retrieved papers, and highlighting the connections between papers and questions.
  Our evaluation, comprising a within-subject comparative experiment and an open-ended exploratory study, reveals that can effectively support researchers in breaking down disciplinary boundaries and integrating scattered knowledge in diverse fields. 
  The findings underscore the potential of LLM-powered tools in fostering information-seeking practices and bolstering interdisciplinary research.







<ccs2012>
   <concept>
       <concept_id>10002951.10003317.10003331</concept_id>
       <concept_desc>Information systems Users and interactive retrieval</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10003120.10003121.10003129</concept_id>
       <concept_desc>Human-centered computing Interactive systems and tools</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178</concept_id>
       <concept_desc>Computing methodologies Artificial intelligence</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
 </ccs2012>


[500]Information systems Users and interactive retrieval
[500]Human-centered computing Interactive systems and tools
[300]Computing methodologies Artificial intelligence
























[
    Xiaojuan Ma
    August 5, 2024
==================





§ INTRODUCTION





The knowledge of humankind is classified into disciplines, and many researchers are trained and specialize in a single discipline <cit.>. 
However, there is a growing trend towards adopting interdisciplinary approaches to research, especially for complex, real-world problems <cit.>. 
Although forming research teams composed of experts from various fields is one preferred way of carrying out interdisciplinary studies, resource constraints and budget limitations sometimes make it impractical to establish such teams <cit.>. 
Researchers trained in one discipline often have to independently seek out information from the literature from various, maybe unfamiliar, research fields to develop their project ideas and approaches, a process we refer to as interdisciplinary information seeking () in this paper.
However, this process is challenging for several reasons.
First, relevant literature to an interdisciplinary project is often highly scattered in various weak-connected research areas <cit.>.
It often requires multiple iterations for researchers to identify what to search <cit.>.
Second, different disciplines use different terminologies, which becomes an obstacle for researchers to form accurate queries for searching literature in their unfamiliar fields <cit.>.
Also, interdisciplinary researchers face the challenge of interpreting vast amounts of information from various unfamiliar fields and evaluating the relevance to their interests during the search <cit.>.



While prior human-computer interaction (HCI) work has proposed various approaches for seeking scholarly information, few of them are targeted at the needs of .
For example, when a computer science researcher is looking to find existing work on computational approaches to combat misinformation, existing literature search tools can facilitate the search by expanding on citations and references <cit.>, iterating search keywords <cit.>, or tracking key actor activities <cit.>.
However, when the researcher is curious about what knowledge in sociology and education, in particular, can be blended into research on this topic, these tools provide limited guidance within the vast search space.

















In this paper, we aim to leverage the recent breakthroughs in Large Language Models (LLMs) to tackle the challenges of . 
LLMs, such as GPT4 [<https://openai.com/gpt-4>] and Gemini [<https://deepmind.google/technologies/gemini>], are pre-trained on vast amounts of internet-wide text data, enabling them to embed the knowledge from various academic fields and can potentially tackle challenges mentioned above <cit.>.
Various LLM-powered literature search tools have emerged, such as Consensus [<https://consensus.app/search/>], Perplexity [<https://www.perplexity.ai/>], and SciSpace [<https://scispace.com/>].
These tools generally retrieve relevant papers based on user questions and provide answers grounded on the retrieved papers, which lowers the risks of hallucination <cit.>.
However, current LLM tools might still not be a perfect solution for for several reasons.
First, interdisciplinary projects often target “ill-structured problem space” <cit.>. 
As a result, the search process is by nature exploratory, and the intents of researchers will change along with the search process <cit.>.
Automating the search with LLMs does not account for the shifts and leaves users with the burden of iterating exploration.
Second, existing workflows allocate the paper screening task entirely to LLMs.
However, the paper screening process is critical as interdisciplinary researchers need to make sense of outside domains <cit.> continuously, have serendipitous discovery <cit.>, and build cognitive contexts <cit.> from the process. 
Moreover, it is found that the information seeking by LLMs might be biased, creating a “generative echo chamber” <cit.>.
Thus, it is critical to engage researchers in the exploration process instead of solely relying on the responses provided by LLMs.




To better assist researchers in with LLM, we design and develop , a human-centered AI system to co-explore literature in various disciplines with users.
integrates a mixed-initiative <cit.> and human-AI collaboration <cit.> workflow.
With users' topic of interest as input, first generates exploratory questions (EQs) around the topic from the perspectives of various relevant disciplines.
In this process, LLMs simulate domain experts from specific fields to provide EQs.
Users can tailor the EQs or elicit to generate more EQs by either making direct requests or sharing papers they find interesting, thus guiding the system to uncover new directions for exploration.

Besides, when users want to delve into any specific EQs, surfaces the existing knowledge about the EQs for users.
Specifically, we design an LLM-based query formulation strategy by including discipline-specific terminologies in the queries to extensively search papers around the EQs while maintaining relevance to the intended field(s).
After retrieving papers from queries, extracts key themes that are related to users' topics of interest as well as selected EQs to facilitate users in finding answers for the EQs.
Additionally, annotates returned papers by highlighting their connection with users' exploration focus to facilitate users in efficiently screening papers.







We evaluated through two complementary studies.
First, we tested the usability of with 12 graduate students by comparing our system with a baseline condition on searching papers for specific interdisciplinary research topics.
The quantitative findings indicate that enabled participants to complete their tasks more efficiently and to uncover a more comprehensive range of knowledge.
In the second study, we invited seven experienced interdisciplinary researchers to use for literature exploration in their own projects.
The qualitative feedback from participants highlighted their appreciation for 's co-exploration workflow. 
Participants also identified potential limitations of , particularly in meeting the unique needs of individual researchers.

In summary, the contribution of this paper is threefold:









    
  * , an interactive system featuring a human-AI co-exploration workflow, empowering researchers in interdisciplinary information seeking.
    
  * A comparative experiment and an open-ended exploratory study to evaluate the effectiveness of .
    
  * Design implications derived from our design process and the user studies on how to support interdisciplinary researchers with LLMs.








§ RELATED WORK







In this section, we first introduce the key processes and challenges of interdisciplinary information seeking.
Then, we discuss the existing systems that support the literature review.
Lastly, we discuss existing human-AI collaboration research on information-seeking and sensemaking.



 §.§ Interdisciplinary Information Seeking ()



Interdisciplinary research involves integrating knowledge from multiple disciplines to address research problems <cit.>.
A critical component of this type of research is interdisciplinary information seeking (), the process of gathering relevant information across various fields.
Compared with normal scholarly information seeking, is typically more complex and time-consuming due to its highly dynamic process <cit.>.



<cit.> describe through a nonlinear model comprising three main stages: orientation, opening, and consolidation, which are often intertwined during the process (see <ref>). 
Orientation involves researchers determining their search direction.
The key challenge in orientation is the highly-scattered nature of the necessary knowledge for an interdisciplinary project <cit.>.
Researchers aim to construct an interdisciplinary knowledge picture for their projects, yet they cannot be experts in every field. 
Consequently, they often seek out resources, including experts, to discern relevant knowledge <cit.>. 
In the absence of such guidance, they have to rely on their own general knowledge and intuition to determine the disciplines from which to gather information <cit.>. 
This approach, however, introduces considerable uncertainty into their search process.
Furthermore, researchers often need to revisit their search goals based on insights from the opening and consolidation stages, adding to the time required for  <cit.>.

Opening is the stage where researchers explore literature in their chosen search directions.
Interdisciplinary researchers tend to deliberately increase the breadth of their search and maintain an open-minded attitude towards all encountered information, considering it potentially valuable <cit.>. 
This approach is driven by the necessity to gather widely dispersed knowledge inherent to .
A significant challenge in this stage is the terminology gap: researchers often struggle to craft queries that use appropriate terminologies to effectively bridge their research interests with literature from unfamiliar disciplines <cit.>.
Previous work suggests that terminology may “have a different meaning” <cit.>, and “not always appropriate or transferable” <cit.> across disciplines.
Researchers might not be able to immediately find the papers for the search direction they identified in orientation due to the terminology gap.
Thus, they typically need to gradually accumulate their knowledge in unfamiliar fields and iterative refining search terms to uncover relevant literature <cit.>.

Consolidation is the stage where researchers decide whether and how to include found papers in their projects, typically following the Opening and Orientation stages. 

Two primary challenges emerge during consolidation. First, the Opening stage results in the accumulation of a large volume of information, much of which may be irrelevant noise, thereby imposing a significant information load on researchers <cit.>. 
Second, the found papers have their own contexts in their belonging disciplines, and to sift, verify, and incorporate the found papers, the interdisciplinary researcher must make sense of the connection between the found papers and their own project
This task is challenging for researchers who are not accustomed to reading literature from unfamiliar fields <cit.>.




In summary, is challenging because of the highly scattered relevant knowledge, terminology gap, information load, and the difficulty in identifying relevant papers.
These challenges identified by prior work motivate our system design (see <ref>).



 §.§ Interactive Systems Supporting Literature Review


Getting a comprehensive understanding of prior works is generally challenging for researchers <cit.>.
Although survey papers synthesizing previous works into themes are commonly used by researchers to grasp fields <cit.>, it is uncertain whether existing surveys can effectively cover all fields and capture rapidly emerging research topics. 
To overcome the obstacles associated with literature reviews, some studies have aimed to simplify the process of digesting research papers <cit.>, while others have focused on aiding researchers in compiling relevant literature through mechanisms like citation networks <cit.> and identification of key authors <cit.>.

Specifically, <cit.> outline two primary strategies employed by prior interactive tools to assist researchers in synthesizing literature reviews from extensive bodies of work: the bottom-up approach, which supports researchers in developing key themes through improved navigation at the paper level, as seen in tools like Threddy <cit.> and Apolo <cit.>, and the top-down approach, which offers an overarching view of the literature, exemplified by Metro Maps <cit.>. 
Building on these concepts, <cit.> propose for a mixed-initiative approach that combines clustering papers via citation networks to provide an overview for researchers while also allowing them to customize this overview from the bottom up. 
Inspired by <cit.>, our system incorporates a mixed-initiative workflow once a collection of relevant papers has been assembled.

But differently, our design focuses on a more expansive search process across various disciplines to meet the unique demands of . Therefore, the system we propose takes users’ descriptions of their research interests as input, instead of using seed papers as in <cit.>’s work.
Moreover, we specifically tailor the discovery and synthesis workflow to preserve the link between discovered papers and the user's contexts, considering the potential diverge between them in .












Previous studies have also endeavored to enable researchers to uncover relevant knowledge from other disciplines, such as developing analogical literature search engines that facilitate the discovery of papers from distant fields sharing similar research aims <cit.>, directions or challenges <cit.>.

Nevertheless, the analogical search has a clear goal in performing literature-based discovery <cit.> to uncover solutions that could be transferred for solving the target research problem <cit.>.
Differently, our work targets a more expansive search process (i.e., ), and focuses on supporting researchers oriented to relevant research in various disciplines.
Additionally, <cit.> introduce a visual analytics system designed to automatically collate and synthesize research data from several disciplines, aiding in the selection of research directions in experimental medicinal chemistry. While their visual analytics method is tailored to a specific interdisciplinary domain, our study explores the broader potential of LLMs to facilitate the general processes involved in .







 §.§ AI-powered Information Seeking and Sensemaking
















Numerous initiatives have been undertaken to develop AI systems aimed at enhancing information-seeking and sensemaking activities. 
For instance, <cit.> introduced DocFlow, a visual analytics system that facilitates document retrieval and categorization through natural language questions within the biomedical field. 
<cit.> design in-context positioning of query suggestions and found it improves the exploratory search process.

The advent of large language models (LLMs) has positioned them as potential knowledge bases for information retrieval <cit.>. There is a growing body of research focused on refining the user experience in sourcing information from 
LLMs. For example, <cit.> argue that the typical linear conversational workflow found in many LLM-based tools falls short in aiding users to synthesize and comprehend information generated by LLMs. 
To address this, they advocate for the adoption of interactive hierarchical views to enrich information interaction practices. 
<cit.> present a novel approach by organizing LLM responses into interactive graph-based diagrams, thereby streamlining the process for users to pinpoint the information they need.



While LLMs in the above works mainly serve by directly generating answers for information seekers, hallucination remains a significant issue. 
To address this, some research focuses on using LLMs to extract and summarize information from search results. 
Selenite, an LLM-powered system recently proposed by <cit.>, employs LLMs to generate comprehensive overviews of options and criteria grounded in search results, guiding users through complex decisions.
Recent emerging works also see opportunities to leverage potential hallucinated LLM generations.
For example, <cit.> use LLMs to generate “hypothetical documents” for input queries to improve dense retrieval.
Similarly, <cit.> utilize LLM generation for visual search, but to help users better express their intents and refine their searches. 
<cit.> connect retrieved papers with users’ existing collections using LLMs, which may include hallucinations, to aid in sensemaking. 
They found that in their design, hallucinations had minimal negative impacts.
In our work, we also explore how potentially hallucinated generations can support information seeking.
We use LLMs to produce high-level exploratory questions, instead of facts, from various disciplinary perspectives.
These questions enable users to co-explore with LLMs.
Additionally, we leverage LLMs to aggregate discipline-specific terminology by generating pseudo-answers, thereby expanding search queries with relevant terminologies to help users bridge terminology gaps in .


























§ DESIGN GOALS

















Drawing on insights from prior research on interdisciplinary information seeking (refer to <ref>) and reflections from our design iterations (detailed in the supplementary material), we have identified four design goals (DG) essential for designing a human-AI collaborative system to aid in .


    
  DG1 Support orienting the exploration in various disciplines.
    Engaging in interdisciplinary research often necessitates acquiring knowledge from various disciplines, which challenges researchers in determining relevant search areas in unfamiliar fields <cit.>. 
    Although LLMs can assist researchers in delving into topics across various disciplines, their tendency to generate hallucinations may lead to confusion. 
    Additionally, prior studies <cit.> and our preliminary design experiments underscore the importance of maintaining high self-agency for researchers who prefer to be actively involved in the exploration process. 
    Accordingly, the system should not solely provide answers for with LLMs but should utilize LLMs as a knowledge resource, guiding users in their exploration and enabling a collaborative exploration experience.

    
  DG2 Support formulating search queries with domain-specific terminologies.
    Formulating effective search queries, especially with terminology spanning multiple fields, poses a significant challenge for researchers in  <cit.>. 
    This often involves iterative refinement of search keywords, informed by ongoing discoveries from the literature <cit.>. Given that LLMs excel in linking concepts and generating diverse query sets <cit.>, the system should leverage LLMs to assist users in crafting queries, thereby saving time and enhancing search efficiency.

    
  DG3 Organize retrieved papers around exploration focus.
    The sheer volume of papers encountered in can overwhelm researchers <cit.>. 
    Moreover, search results often include irrelevant noises that detract from the exploration focus, as revealed in our design iterations. 
    Previous HCI research suggests the aggregation of similar information and providing overviews is helpful in such cases to alleviate user burden <cit.>. 
    Therefore, the system should categorize retrieved papers into overviews, enabling users to swiftly pinpoint relevant groups of literature.


  DG4 Provide Information Scent for Efficient Paper Sifting.
In the process, researchers often need to meticulously review papers to verify their relevance to specific aspects of their research <cit.>. 
This labor-intensive task necessitates a mechanism for quickly identifying relevant information within papers. 
To facilitate this process, the system should highlight critical paper details and offer “information scents” that guide researchers in their review process <cit.>.


    
    
    
    
    
    
    
    
    
    
    
    
    
















§ THE SYSTEM

We introduce , a human-AI collaborative system designed to enhance interdisciplinary information seeking (). 
Unlike previous efforts in scholarly information seeking, focuses on enabling researchers to effectively collaborate with LLMs for broader exploration across disciplines, thereby linking their own projects with diverse knowledge.

Generally, inspired by prior research <cit.>, facilitates both top-down and bottom-up mixed-initiative literature discovery workflows (<ref>).
In the top-down workflow, users decide the exploration direction they are interested in first, avoiding being lost in the details of a number of papers.
To aid this process, generates diverse EQs from various disciplines based on the user's research topic (<ref>-1), crafts comprehensive queries for each EQ (<ref>-2), and summarizes key themes from retrieved papers to streamline screening (<ref>-3).
On the other hand, the bottom-up approach empowers users to spontaneously navigate through papers, citations, and references.
Users can organize the papers of interest and formulate new EQs based on these papers, thereby starting new research directions.
enhances this exploration by organizing citations and references based on user exploration history and suggesting EQs informed by user-selected papers.

As a research prototype, uses GPT-4 as the underlying LLM, API service for searching papers, and “” <cit.> for text embedding.
The subsequent subsections detail a usage scenario and describe 's key features in addressing our design goals.

























 §.§ Usage Scenario

Zoey, a computer science researcher, is tackling a new project aimed at raising awareness about social media misinformation among older adults. 
While she possesses expertise in misinformation detection through natural language processing, she finds herself stumped by the challenges older adults face in navigating social media and the human factors influencing misinformation awareness.
To bridge these gaps, Zoey turns her attention to unfamiliar disciplines like sociology and psychology for insights. 
Beginning with search terms such as “older adults misinformation,” through Google Scholar, she finds a mountain of papers that seem relevant across disciplines, including education, sociology, and psychology, each offering unique perspectives on the subject. 
Determined not to miss out on anything potentially useful, Zoey digs into each paper, one by one. 
However, the scattered insights across different fields make her research feel like assembling a vast puzzle without a guiding image.
Moreover, she needs to understand different perspectives during her search, each requiring a shift in mindset. 
Struggling to manage the information overload and the constant shifts in disciplinary perspectives, Zoey considers seeking help from .

Upon describing her research project to , Zoey receives a list of exploratory questions (EQs) organized by specific disciplines.
Zoey is particularly interested in the perspectives of psychology, education, and sociology.
These EQs, framed with easy-to-understand language, introduce some key factors, such as “What can older adults be motivated to learn digital literacy?” (education), “What cognitive strategies reduce belief in false information?” (psychology), that has never come to her mind before, possibly due to her unfamiliarity with these disciplines.
Among these, the EQ “How effective are peer-led education programs for seniors?” catches her interest. 
However, aiming to align it more closely with her focus on misinformation, she refines it to “How can peer-led education programs combat misinformation among older adults.”

After tailoring several EQs across psychology, education, and sociology to her preferences, Zoey prompts to fetch relevant papers. 
After searching, organizes the findings into themes for each selected EQ in the Exploration view (<ref>-middle).
Delving into the search results on cognitive strategies to counter misinformation, Zoey discovers themes such as “Critical Thinking and Cognitive Improvement” and “Inoculation and Overconfidence Strategies to Combat Misinformation”.
This overview allows her to navigate the themes without getting bogged down in the details of individual papers.

Attracted by the concept of “psychological inoculation,”, a term new to her, she explores a linked wiki page, uncovering its relevance in preemptively addressing misinformation. 
Through annotated key phrases in paper titles and abstracts, such as “misperception” and “credibility,” Zoey confirms the relevance of these papers, collecting them by dragging the theme to the Collection view (<ref>-right).



Later, Zoey's exploration leads her to a highly cited paper, and habitually, she checks its citations. 
categorizes these citations by discipline, with “Medicine” surprisingly ranking in the top three. 
Out of curiosity, she examines the citations under Medicine, and based on the highlighted terms, she finds a paper that discusses the impacts of health misinformation on older adults.
She realizes the strong connection between health misinformation and the older adults, but she hasn't explored this direction.
To extend her exploration, Zoey drags the paper to the orientation view (<ref>-left).
Based on the paper, recommends three new EQs around health misinformation's impact on older adults in the context of social media, thus broadening her exploration horizon.


















 §.§ System Design




  §.§.§ [DG1] Elicit Exploration across Disciplines


To support users in expanding their exploration to relevant information scattered in various academic fields (DG1), encapsulates potential literature search objectives into interactive elements named exploratory questions (EQs).
In other words, proposes EQs from different disciplines' perspectives to elicit users' breath exploration.
For example, when exploring a research topic of facilitating older adults to learn about AI tools, possible EQs could be “what cognitive challenges older adults may face in understanding complex concepts?”, which concerns psychology, and “how to design interactive tools to support learning for older adults”, which is more related to HCI literature.
Below, we introduce the enabled interactions around EQs in , the design rationale, and the EQ-generation process.


Supported Interactions around Exploratory Questions.
Upon users entering a research topic (<ref>-A1), generates a list of EQs tailored to the research topic of interest. 
In this process, we leverage the inherited knowledge of derived from its large-scale training data to ensure the coverage of diverse disciplines. 

Users have the flexibility to modify their topic description anytime, prompting to produce new EQs based on the updated description or to initiate a new exploration session by resetting the current ones (<ref>-A2). 
The EQs are categorized by their relevant disciplines (<ref>-D1), with a navigation bar allowing users to easily filter EQs by research field (<ref>-B). 
By selecting EQs, users can specify their exploration intentions for to identify literature closely aligned with their interests. 
Further interactions include editing EQs (<ref>-D2), adding or removing selected EQs (<ref>-C1), and generating additional EQs by specifying disciplines and particular needs or by manually entering questions (<ref>-C2).





Note that if users encounter an interesting paper on a yet-to-be-explored topic, they can obtain new EQs from based on the paper by dragging it from the exploration or collection view to the orientation view. 
Users have the option to pinpoint specific aspects of a paper, such as particular keywords, that pique their interest. 
Upon specifying these details, they can direct to suggest new EQs tailored to these focal points.
We set to provide three new EQs each time.
Users can choose one or several EQs to refine or opt to craft new EQs from scratch. 
Once these EQs are finalized and submitted, initiates a new round of search for papers concerning these EQs immediately.



Design Rational.
Our design of EQs draws inspiration from human-centered design principles, including the creation of a shared representation to balance automation with user agency <cit.>, and the reification of actions into objects for direct manipulation <cit.>.

EQs act as a bridge between users and the system, offering an intuitive means for users to convey their interests and for to interpret and further translate these interests into complex queries, thereby fetching a broad spectrum of relevant papers. 
We have explored other forms of LLM assistance in the design process for guiding and expanding user exploration in .
Initially, our prototype leveraged to generate research topic statements for users. 
However, this approach led to two main issues: the topics often contained inaccuracies due to LLM hallucinations, and they served more as end products of exploration instead of inspirations for users to explore.  


Another alternative we considered was utilizing to generate simple queries, similar to those users might input into search engines like Google Scholar. This method, while straightforward, can be inefficient in .
Effective queries for necessitate the use of discipline-specific terminology, which could be challenging for users to understand. 
To address the problems in the above two methods, EQs in are framed as questions rather than facts, aiming to expand the exploration space, and users can flexibly customize EQs to co-explore with LLMs. Also, EQs are crafted in user-friendly language as a high-level abstraction of research directions over low-level typical search queries. 

















Prompts Design.
takes two main types of inputs from users to generate EQs: the research topic of interest and/or the paper of interest, as illustrated in <ref>. To generate EQs from diverse disciplinary perspectives based on these seeds of exploration, we employ prompt-chaining and persona-prompting techniques <cit.>.


The process involves chaining prompts to guide in generating EQs. 

More specifically, given a research topic, we provide the list of disciplines used by for to determine which of them might be relevant.
The list includes primary disciplines, such as “computer science” and “psychology”.
then determines specific subfields within these disciplines (e.g., “developmental psychology” within “psychology”) that closely align with the topic of interest, enabling it to propose more targeted questions.
Then, for each identified field and subfield, our prompt to begins with, “You are an expert in field. A researcher is developing a research topic: topic. What knowledge from your field could assist in developing this idea? Present your advice in the form of questions.” After collecting EQs across different disciplines, we instruct to refine this list by removing any duplicates, resulting in the final EQs shared with users.

Preliminary experiments indicate that adopting specific expert personas enables to produce questions that closely match the disciplinary focus.
For instance, when exploring how AI tools can support older adults  learning in science without employing a persona, language models tend to produce questions centered on “AI for learning.” 
These questions focus predominantly on “AI,” steering the subsequent discovery process toward computer science literature rather than education. 
Conversely, utilizing persona prompting can generate EQs that lean more heavily towards aspects of older adults learning from an education perspective, such as “What motivates older adult learners in science education?”.


For generating EQs based on a paper of interest, perhaps discovered through navigating the references and citations of papers returned in the previous rounds of exploration, adopts a similar persona-prompting strategy. 
Here, the meta-information of the paper, including its title, abstract, designated fields, and keywords, is integrated into the prompt context, allowing users to specify keywords that highlight their areas of interest. Following this, we prompt to identify relevant fields within the primary disciplines and simulate experts in these fields to recommend EQs related to the provided context. 




  §.§.§ [DG2] Query Expansion to Retrieve Relevant Papers




When users express interest in a particular EQ, furthers the exploration by expansively generating numerous queries related to the EQ.
Next, it conducts searches for papers in using these queries concurrently.
In EQ generation, we guide to employ straightforward, easy-to-understand language so that users can quickly decide whether to dive into an EQ or not. 
However, when it comes to searching for papers, we instruct to use precise terminology and field-specific jargon relevant to the EQ to craft the search queries. 
This aims to fill the terminology gap of introduced in <ref>.

Moreover, we intend to retrieve a diverse set of papers using the output queries to allow to map out the knowledge landscape around the EQ.
We design a prompt to craft queries according to these requirements, as shown in <ref>.

To be more specific, we begin by requesting to provide (pseudo-)answers from various perspectives on the current EQ in the form of bullet points. 
Although there is a risk of hallucination, these responses, even when flawed, are found to contain keywords and concepts beneficial for formulating effective search queries, as noted by <cit.>.
Subsequently, compiles a list of relevant terms associated with each bullet point. 
These terms lay the groundwork for creating a set of targeted search queries,  which are later fed into paper search engines.
This process, inspired by <cit.>, helps to enrich the conceptual breadth and depth of inquiries into varied branches of knowledge.












  §.§.§ [DG3] Organize the Retrieved Papers of around EQs


It is challenging for users to go through the extensive papers retrieved from the queries, especially considering that many papers are likely from unfamiliar fields of study.
To alleviate this information overload, distills key themes from the papers, focusing on the connection to the user's current exploration context (i.e., the research topic and the selected exploratory question). 
 

Themes Extraction. 
While the papers retrieved from the expansive queries could contain many themes, users would be more interested in those more related to their current exploration context, i.e., the research topic and the selected EQ.
A LLM-based clustering pipeline (<ref>) is deployed to sift through papers and extract themes pertinent to the user’s interests.


Several LLM-based clustering techniques have been proposed by researchers, while trade-offs exist.
For example, LLooM <cit.> and ClusterLLM <cit.> produce high-quality clusters but need to operate at the data instance level – LLooM involves distilling info from each input data instance and ClusterLLM uses triplet comparisons – leading to increased time costs. 
To balance quality and time costs and enable interactive theme exploration in , our pipeline begins by generating contextualized embeddings for papers, proceeds to cluster these embeddings, and concludes with refining the clustering outcomes.


In the first step, we are inspired by the keyphrase expansion technique proposed by <cit.> to construct the contextualized paper embedding.
This technique concatenates original text embedding with keyphrase embedding related to user intents. 
Similarly, in , the contextualized embedding comprises text embedding of the paper's metadata (title and abstract) and embeddings of the paper's discipline, if matching the EQ's discipline, as well as key phrases from the paper's metadata resonating with the research topic and EQ. 

By considering whether the paper's discipline matches the EQ's discipline in the paper embedding, we add a soft constraint on clustering papers that fall into the same field of study. 
To identify relevant phrases from academic papers, we first convert key concepts C = {c_1, c_2, ..., c_m} from the research topic and EQs, along with phrases P = {p_1, p_2, ..., p_n} from the paper's metadata, into embeddings. 
A phrase p_i is considered relevant if its cosine similarity with any concept c_j, defined as sim(v_c_j, v_p_i) > 0.6, surpasses a predetermined threshold of 0.6 [This threshold is determined through empirical testing during development.]. 


In the clustering phase, we employ DBSCAN <cit.> to group the contextualized embeddings.
While, through constructing the contextualized embeddings, the algorithm is more likely to put papers relevant to the exploration context together, it can produce other clusters less related to the contexts.
Besides, the resulting clusters might still have many papers that making it challenging for users to quickly screen them.
To this end, we harness to curate the clustering results.
Specifically, we prompt with two questions for each cluster: “Is this cluster related to the research topic and the selected question?” and “Is there a clear division between papers that a subclustering would be helpful?”
Based on the results, we remove irrelevant clusters and decide whether to divide any existing clusters further.
We still keep the papers under the removed clusters in a “possibly relevant paper” list in for participants to browse if they want.
This design decision is driven by the observation that researchers would not only search on purpose but also be interested in serendipitous information discovery <cit.>.

Supported Interactions. 
Upon selecting an EQ for which papers have already been queried, presents the extracted research themes. 
It displays the disciplinary distribution and key phrases for papers within each theme (<ref>-E) and links the key phrases to their respective wiki pages via DBPedia Spotlight <cit.> when possible. This feature aids users in demystifying unfamiliar terms. Users can delve into themes to review the papers classified under them.






  §.§.§ [DG4] Support Sifting Papers

presents retrieved papers with their disciplines, titles, and meta information such as authors, published year, venue, abstract, and references as well as citations (<ref>-F1).
Users can also click a link button to navigate to the page of the papers for further investigation.

Besides, highlights the relevant phrases in the title and abstract that are aligned with the current exploration contexts, i.e.,  research topic as well as the selected EQ, as shown in <ref>-F1.
The method of extracting relevant keyphrases is the same as the one in the clustering process described in <ref>. 
Additionally, to prevent information overload in the user interface, only the sentence from the abstract deemed most relevant to the exploration context is displayed. 
Formally, given an abstract represented as a document consisting of a set of sentences D = {s_1, …, s_n} and the set of concepts in the exploration contexts C = {c_1, …, c_m}, the key sentence s^* is identified by:
s^* = max_s_i ∈ D|{c_j ∈ C |  exists a phrase relevant to  c_j  in  s_i}|.
Users can also expand the key sentence to see the full abstract at will.




  §.§.§ Other Features

Besides the aforementioned four features that directly target the four design goals, also includes some other features to facilitate the process.

Support Organization of Useful Papers.
allows users to drag and drop themes or papers from the exploration view to the collection view.
When dropping themes, a collection is created in with the same title as the themes and containing the same group of papers.
When papers are dropped, they would be added to collections based on their sources: from a theme or from citations and references of other papers.
Users can edit the title of the collections, move papers from one collection to another also through dragging and dropping, and delete collections.
This organization feature, together with the themes extraction feature, supports the mixed-initiative workflow proposed by <cit.>.

Ranking Citations and Reference.
displays the citations and references of a paper upon users' requests, and these citations and references are organized based on their disciplines (<ref>-F2).
In particular, we ranked the disciplines to prioritize the ones that contain relevant information to the exploration contexts but are less explored according to users' interaction history with the system.
This ranking aims to balance exploration and exploitation <cit.> and support DG1.
We define users' engagement level in each discipline as U_d = p_d + q_d, where p_d is the number of papers collected and q_d is the number of EQs queried in discipline d. 
The exploration score, E_d, is set to 1 / (U_d + 1) to encourage exploring lesser-engaged disciplines.
The relevance of each discipline, V_d, is computed as the average cosine similarity between the paper embeddings in d and the user-specified research topic embedding. 
The combined score C_d for ranking disciplines is simply computed as C_d = β V_d + E_d, where β is set to 1 in our prototype.
For papers under each discipline, we rank them by their cosine similarity score with the users' topic of interest.













































§ STUDY 1: USABILITY EVALUATION



To evaluate , we first conducted a usability study on with a focus on how useful and easy it is to use in helping researchers search papers regarding an interdisciplinary research topic.
Specifically, we explore the following research questions:

    
  RQ1 Can help researchers gather more comprehensive literature? 
    
  RQ2 Does improve user efficiency in the process?
    
  RQ3 How do users perceive the usefulness and experience of interacting with ??    









 §.§ Participants

We recruited 12 participants (six females, five males, and one prefer not to say) from social media. 
They are all postgraduate research students.
Eight participants have 1-3 years of research experience, three have 3-5 years, and one have 5-10 years.
Their research directions include HCI (6), visualization (2), machine learning (2), programming language (1) and cognitive science (1). 
All but one participant confirmed that they have experience in interdisciplinary research.
Additionally, five participants suggested they often use AI tools, including Consensus [<https://consensus.app/>], Elicit [<https://elicit.com/>], Perplexity [<https://www.perplexity.ai/>] and Paper Digest [<https://www.paperdigest.org/>], for searching literature. 




 §.§ Protocol


We adopted a within-subject experiment design.
Participants were asked to complete tasks under two conditions in a counterbalanced order:  the system condition (using ) and the baseline condition (using tools they usually use).
We prepare two interdisciplinary research topics for participants to perform :


  * Develop digital teachers to mentor high school students in rural areas, focusing on training their problem-solving capabilities. We refer to this task as AI for education in later texts.

  * Help designers use AR to creatively document intangible cultural heritage, ensuring its preservation for future generations. We refer to this task as AR for ICH in later texts.












Participants were instructed to widely survey papers from various academic fields for each topic.
In both conditions, participants were required to complete an outline in Google Docs as the task outcomes. 
In the outline, participants needed to list the research papers they found relevant and group them into topics.
Participants were also asked to write a short explanation of how the topics related to the given project.

Before the task starts in the system condition, we first provide a 5-minute tutorial on and give 5 minutes for participants to try .
Then, they are given 30 minutes to search papers and complete the outline for each task.
After each task, they are asked to complete a corresponding questionnaire.
Upon completing both tasks, we ask participants to share their experience in using .



 §.§ Measurement


To answer RQ1, we compare participants' outlines in the two conditions from three aspects:
statistics of the outlines, participants' subjective ratings, and ratings from external experts.

For the statistics of the outlines, we computed the number of topics and papers in each outline.


For the participant's ratings, in the questionnaire after each task, participants self-rated the outline's helpfulness, comprehensiveness, organization, and relevance using the 7-point Likert Scale.
Moreover, before each task started, we asked participants to tell us what they planned to search for during the task.

Subsequently, upon completion of the assigned task, we directed the participants to provide ratings indicating the extent to which they identified relevant literature they had planned to search, as well as the degree to which they discovered relevant literature that had not been part of their initial search strategy.
These ratings help to assess whether achieves the goal of expanding researchers' exploration scope.
For the expert ratings, we recruited two experts to score the outlines.
One is a final-year PhD student, and another is an assistant professor.
Both are HCI researchers and have experience in multiple interdisciplinary research projects widely covering AI, XR, education, and design.


They evaluated the participants' outlines while blind to the condition using a 10-point Likert scale by rating the following statements:

    
  * Overall outline helpfulness: “I found the outline is overall helpful for developing the research project.”
    
  * Outline comprehensiveness: “I believe this outline covers a comprehensive set of relevant topics.”
    
  * Topic relevance for each topic: “I found this topic relevant to the project.”
    
  * Topic support for each topic: “I found the topic to be well-supported by papers under it.”

We refer to <cit.> in designing these statements except for the outline comprehensiveness.
We add the outline comprehensiveness as a measurement as an important design goal of is to assist researchers in gathering highly-scatter knowledge.
Before their evaluation, experts first engaged in a session to discuss how to give specific points.  
Experts' given ratings were averaged as the final score.

To answer RQ2, we measure the task workload using the NASA task load index <cit.>.
We also measure participants' knowledge gain during the search process, as researchers typically need to learn knowledge from other disciplines on the fly in .
Participants rated their knowledge on the given research topic using a seven-point Likert scale before and after the task, and we take the change in self-rated knowledge as the knowledge gain <cit.>.
They also rate their self-efficiency during the task.

To answer RQ3, we adopt the technology acceptance model  <cit.> to assess the perceived usefulness, ease to use, ease to learn, workflow compatibility of , and participants' intention to use it.
Participants also rate their perceived helpfulness of to key stages of , including orienting to various academic fields, breadth searching papers, filtering irrelevant papers, and making sense of the connection between the papers and the focused research project.











 §.§ Findings

For statistical results presented in this section, we used the paired samples t-test for comparison between conditions, unless the Shapiro-Wilk test of normality indicated a deviation from normality. 
In cases of non-normal data, we utilized the Wilcoxon Signed Rank test.





  §.§.§ [RQ1] Quality of the outline

Participants in the system condition, on average, collected 5.08 topics (SD=2.28), more than the baseline condition (M=3.92, SD=1.51), but there are no significant differences (p=0.057).
There are also no significant differences regarding the number of papers (the system condition: M=20.58, SD=14.39; the baseline condition: M=17.83, SD=10.31; p=0.58).

The experts ratings indicated significantly higher outline helpfulness of the system condition (M=6.67, SD=1.09) than the baseline condition (M=6.04, SD=0.96, p=0.003).
Similarly, experts' ratings of outline comprehensiveness are significantly higher in the system condition (M=6.63, SD=1.23) than in the baseline condition (M=5.83, SD=0.69, p=0.02).
Nevertheless, there are no significant differences regarding averaged topic relevance and averaged topic support for the two conditions.

For participants' subjective ratings (shown in <ref>), there are no significant differences between the self-report outline helpfulness (p=0.16), organization (p=0.31), and relevance (p=0.18).
However, participants reported higher comprehensiveness of the outline in the system condition (M=5.00, SD=1.60) compared to the baseline condition (M=3.08, SD=1.31, p=0.002).
Besides, there is no significant difference between how well participants consider they found literature they plan to search (p=0.07).
But regarding how well they found relevant literature they originally did not plan to search, the system condition (M=6.08, SD=1.44) is significantly better than the baseline condition (M=4.67, SD=1.44, p=0.05).

These findings suggest that is beneficial in helping researchers to gather comprehensive literature from various disciplines as intended.



  §.§.§ [RQ2] Search process


Participants in the system condition reported significantly larger learning gains (M=2.50, SD=2.468) compared to the baseline condition (M=1.08, SD=1.56, p=0.02), suggesting helps users learn more during the exploration.
Also, they self-reported higher efficiency in the system condition (M=6.18, SD=0.94) compared to the baseline condition (M=4.33, SD=1.50, p=0.01).
Participants exhibited a significantly higher task workload in the baseline conditions based on the averaging score of the NASA TLX items (M=4.01, SD=0.74) compared to the system condition (M=2.93, SD=0.97, p<0.01).
For the individual items (illustrated in <ref>), the system condition is significantly better than the baseline condition in regard to the physical demands (p=0.01), performance (p=0.03), effort (p=0.02), and frustration (p=0.04).
These results suggest that improves participants' search efficiency, lowers their workload, and helps them effectively learn more in the process.





  §.§.§ [RQ3] User experience

Participants generally held positive attitudes towards the usefulness, ease of use, ease of learning, and workflow compatibility, as shown in <ref>.
Most participants appreciated the generated EQs.
P12 noted that “When I came to this new field, I didn't know where to go and how to search on Google. There might be some terms in this field, and then it [] could help me ask some questions. It is difficult for me to ask professional questions in this way.”

Nevertheless, some participants complained that provides too many EQs and thus adds load to users: “It has many [EQ] recommendations, and I may still need some efforts to screen them. I need more time to see if this is relevant, and it is unlikely to be completed very well in a relatively short time.” (P06).
P07 expects the EQs to be better connected, “There may be a connection or overlap between them, but it didn't tell me.”.

Besides, as illustrated in <ref>, there were mixed opinions among participants regarding 's effectiveness in filtering papers. 
Those who viewed positively highlighted its ability to deliver relevant themes and papers aligned with the selected EQs. (“I think the results of the query are very accurate. I can’t get such accurate results using Google Scholar because I feel that it only matches titles. You can search more accurately with this [system].”, P11) and appraise the keyphrase highlighting (“The papers provided by other search engines may not be relevant, and I don't know exactly what relevant keyword it has. But you can clearly see the fields, the key phrases [in ]”, P09).
However, some participants felt challenged by the volume of papers recommended, pointing out the difficulty of effectively screening them within the experiment time, “Because it recommends a lot, I still need manual effort to screen them. I can't complete it very well in a relatively short period of time.” (P6)

We also examined what tools participants used in the baseline condition.
We found that all participants used Google Scholar, while P6 and P10 used tools including LitMaps and Connected Papers to check citation networks of papers, and P01, P08, P11, P12 attempted AI-powered literature search tools including Consensus, Elicit, Paper Digest, and Perplexity.
Compared to , participants suggested the tools displaying citation networks offer better visual salience on key papers but provide limited assistance in guiding search directions.
P10 noted that “Some of them [papers] were clustered together [in the citation network], and then I took a quick look at those. [...] But I just picked the standout ones and need to read carefully to proceed, while your tool is much better for getting a comprehensive exploration quickly.”
Regarding AI-powered search tools, P01, P11, and P12 initially tried them but quickly reverted to Google Scholar due to a lack of helpful answers. Conversely, P08 found Paper Digest useful for creating outlines but felt that “Your system has several exploration levels, allowing me to understand the paper I gave better. In the summary generated by Paper Digest, I am more about reading and accepting papers, and my understanding of these papers may not be as clear.”
This feedback illuminates the unique strengths of in unfolding the process.



§ STUDY 2: CASE STUDY

Encouraged by the results of study 1, we further conducted an open-ended exploratory study with experienced interdisciplinary researchers to get qualitative feedback on the potential benefits and limitations of in .





 §.§ Methodology



  §.§.§ Participants

We recruited seven researchers (two females and five males) who are focused on interdisciplinary research, including one assistant professor, two postdoctoral researchers, and four senior PhD students.
Participants are from various backgrounds, including AI for healthcare, data storytelling, social psychology, AI for science, music technology, and AI for education.



  §.§.§ Protocal


We recommended that participants choose projects for which they haven't fully devoted themselves to searching for relevant literature yet, in order to explore those projects in this study.
If they do not have such projects or they feel uncomfortable doing that, participants could also explore papers under their other interdisciplinary projects.
Participants were informed that none of the information on their research projects would be reported.

After getting the consent of participants, at the beginning of the study, we gave a tutorial on , as the one in Study 1, and suggested participants get familiar with the system through a 5-minute practice with a given research topic.
We then asked participants to explore their interdisciplinary projects using .
We encouraged participants to think aloud and explore for at least 20 minutes.
Participants are allowed to explore for a longer time if they are willing to do so.
After the open-ended exploration, we conducted semi-structured interviews with participants.
The interview questions cover participants' general experience with , whether helps in uncovering relevant knowledge in multiple disciplines, whether found helpful papers, whether helps in screening papers, and how well can fit into their daily process.
We also asked whether participants felt the level of automation was appropriate.




  §.§.§ Data Collection and Analysis

All study sessions were audio-recorded with the informants' consent. 
The audio recordings were transcribed using automated tools and subsequently manually verified.
Two members of our research team analyzed the transcripts using thematic analysis <cit.>. The coding results were thoroughly discussed and integrated through iterative discussions.



 §.§ Findings

We report our findings based on the participants' qualitative feedback, focused on those not mentioned by participants in Study 1.
Our reports do not include information about participants' research projects to protect participants' privacy.



  §.§.§ Helpfulness of EQs

All participants found some EQs generated by relevant and provided insights to their research projects, enhancing their literature exploration process.
P7 considered EQs “introduces a creative and playful way to pick your sense of curiosity.”
P3 specifically noted that the EQs aligned closely with the fields most pertinent to his research and matched the disciplines associated with the references cited in his paper.
He noted how exploring these EQs can be helpful to his paper: “There are some [EQs] that are relevant and can be helpful for developing the introduction part of the paper. There are also three or four questions that can be explored to develop follow-up future studies or some other discussions.”

Besides, P4 and P7 both mentioned that the generated EQs would particularly be beneficial to researchers in the onboarding stage of interdisciplinary research.
P4 found the EQs are related to “the core knowledge in each field”; thus “it is very helpful for understanding knowledge for first-year Ph.D. students.”
Similarly, P7 commented that “most of the researchers, including myself, have problems with the onboarding process. So, these types of applications or surveys could help us with the onboarding process and also learning the new research area step by step.”

Moreover, most participants found that streamlines the workflow that aligns with and augments their original workflow.
P1 noted that:


    In the past, I would think of directions in different disciplines and search for them. This process was in my mind, but it would definitely not be so comprehensive. It () would ask some very related questions that I had not thought of before.


An exception is P6, who usually did not search literature by himself.
He collaborated with experts from other disciplines and read the literature they suggested.
However, he also found helpful in his workflow:

    Sometimes, you don't want to follow the advice of experts exactly. What experts say is not always correct. [...] This tool is very useful when I want to develop something new [beyond their strengths].




  §.§.§ Various expectation of EQs


Although participants generally found EQs helpful, they also found some EQs did not meet their expectations.
P4, whose background is chemistry and physics, found that all EQs start with “How” or “What”, but not “Why”.
He considered “asking why” is more important as “
Understanding the underlying causes is important to conducting my research”.

Another concern is about the connection between the generated EQs and their projects.
Interestingly, there are diverging opinions from participants on this.
P1, P3, and P6 consider some EQs as not good because the connection between these EQs and their projects is loose and only covers a portion of the key concepts in the projects.
P1 noted that “some questions are simply digging into one field, and it seems that they are less relevant to my interdisciplinary project.”

However, P5 and P7 desired more EQs to delve deeply into other disciplines and not strongly connected to their own projects.
One reason is the lack of existing research on their project; thus, EQs strongly connected to the project “won't be able to help me find something very useful.” 
Another reason is the need to delve deeply into other disciplines of research, as explained by P5 with her previous experience:


    When I was doing [a certain concept] [related interdisciplinary research], I had to read a lot of psychological literature to see how psychologists used [a certain concept]. [...] If I don’t read real psychological papers and do it directly, then I will always read second-hand literature. I'll never know why people chose that model, and it is problematic.





Moreover, some participants believed the provided EQs should be different in different stages of the interdisciplinary research.

P2 elaborated on the development of the exploration needs and suggested adding more customization over the EQs generation:


    When you first explore this [interdisciplinary] field, you can have more possibilities [...] But I'm actually pretty sure I only want papers with [a concept] and [a concept]. I think before [providing EQs], I can have a choice. For example, am I a person who is exploring this [interdisciplinary] topic for the first time, or am I actually an expert in this topic?












  §.§.§ supports human-AI co-exploration


Participants generally considered the level of automation to be appropriate, and they were able to engage and direct the exploration.
We observed how participants actively tailor the EQs for their own goals.
For example, P1 edited EQs by adding concepts more related to his project to strengthen the connection between the found papers and the project; for P7, the original question was about how an intervention affects an element, and P7 edited the question by adding more elements could be affected to enlarge the search scope.
Participants also initiated the creation of EQs based on their insights during the exploration process.
P1 found “gamification,” and P2 found “music” as elements that they hadn't considered in their own projects but were relevant through browsing retrieved papers, and they created EQs based on these findings.

Additionally, P7 highlighted how the language framing of the EQs encourages him to tailor the EQs and join the co-exploration:


    If your system was giving me very sophisticated questions. Then, it might restrict my own creativity. But because the questions are broad, easy-to-understand. I think it helps me also think by myself. And these [EQs] can act like hints.


We also found most of the participants not only rely on the provided themes and the corresponding papers in to seek information, but also deliberately attempt more ways to discover papers, including digging into the “possibly relevant paper” list, the citation and references of papers, and even browse the recommended paper list in the paper page after they clicked the link of papers in .
This behavior aligns with the nonlinear model that researchers would deliberately increase the breadth of the search in  <cit.>.

When asked about their sense of control during the exploration process with , all participants affirmed they were in control. 
They attributed this to having a clear objective for their exploration to advance their own projects. 
Thus, their engagement with the system's recommendations was driven not by the system's guidance alone but by the perceived value of these suggestions to their projects. 
P6 encapsulated this sentiment by stating, “I have been deeply thinking about this research topic for a while. It's always on my mind. AI cannot diminish my goal or change my focus.”



















§ DISCUSSION



This paper explores the potential of LLMs in supporting interdisciplinary information seeking and contributes to the growing body of research on human-LLM interaction across various domains <cit.>. It is worth noting that there are differing opinions in academia regarding the role of LLMs in enhancing search systems.
<cit.> envision that future search systems can build on LLMs to offer answers to user queries instead of references that require users to further explore.
Conversely, <cit.> argue against “dropping a LLM-based agent as a one-size-fits-all solution” and emphasize the importance of user interaction with information in search processes, which enables activities such as learning and serendipity.
Our work aligns with <cit.>'s perspective by offering a human-AI co-exploration solution that integrates LLM support while enabling rich user interaction during the search process.
In this section, we discuss the lessons learned in designing and evaluating , as well as its limitations and potential directions for future work.



 §.§ Exploratory Questions as Share Representation in AI-powered 

In , we conceptualize exploratory questions as a shared representation between users and AI to facilitate co-exploration. 
The concept of shared representation, as introduced by <cit.>, refers to the use of a representation, be it textual or visual, in human-AI interaction that is mutually intelligible to both humans and AI. 
This shared foundation allows for collaborative contributions to the task at hand, thereby striking a balance between automation and user agency. 
Participants in Study 1 infrequently modified the EQs, possibly due to time constraints. 
However, during open-ended exploratory studies, we noticed participants customizing the generated EQs and creating their own, showing the potential of EQs to act as a shared representation that encourages engagement from both humans and AI.

Feedback from participants in our first two studies was generally positive regarding the utility of EQs. Yet, some noted the lack of visible connections between EQs, particularly when they span across disciplines and overlap in themes after querying. 

currently does not highlight these links by presenting EQs in a list.
To address this, future efforts could consider better supporting the curation and exploration of EQs.
One possible direction is to consider the existing research on curating examples for creativity support, and visually contextualizing the EQs when they are generated <cit.>.



 §.§ Support Learning with Human-AI Co-Exploration

As noted in the literature <cit.> and commented by our participants in study 2 (<ref>), learning in helps researchers conduct interdisciplinary research over a long period of time.
This insight underpins our decision to create a human-AI co-exploration workflow rather than merely employing AI agents to autonomously compile research papers for scholars <cit.>.
Our approach is in line with the idea of AI aiding human cognitive processes <cit.>. 
Feedback from study 2 indicates that not only piques user curiosity but also facilitates their initiation into interdisciplinary research, thereby supporting learning within .

However, much of the existing research on human-AI collaboration has prioritized enhancing collaboration to boost immediate performance outcomes <cit.>, often focusing on short-term metrics such as the accuracy of decision-making with AI support. While these immediate results are valuable, we advocate for a broader focus within the field, emphasizing support for enduring outcomes of human-AI collaborations. This includes promoting long-term project success (as in interdisciplinary research), enhancing worker well-being, and contributing to societal benefits.




 §.§ Contextualization in 


Interdisciplinary research often involves sifting through a vast array of potentially relevant papers. However, the specific contexts of researchers guide them in concentrating on particular studies. 
acknowledges this by using the user-provided research topic and selected EQs as a framework for offering contextualized support: recommending EQs tailored to the research topic and synthesizing themes from the papers based on these EQs.
Feedback from user studies confirms the relevance of EQs to their research topics and the extracted themes to the EQs.

Yet, findings from our Study 2 also indicate that interdisciplinary researchers expect to grasp their research contexts in a more nuanced way, beyond just the topics at hand. 
This includes understanding their preferences regarding the framing of questions (e.g., P4 shows a preference for “why” questions over “what” and “how”), their desired exploration scope (e.g., some participants favor an in-depth exploration into other disciplines while others prefer a stronger linkage with their own research project), and their current phase in the research process (e.g., P2 suggested questions could initially be broader and become more targeted as the research progresses). 
Although allows for customization of EQs, offering more nuanced EQ recommendations could significantly enhance the user experience by saving time and boosting efficiency.
Therefore, future endeavors should delve deeper into the specific factors of the needs of interdisciplinary researchers to offer more finely tuned exploration and research support.












 §.§ Limitation and Future Work








  §.§.§ Deepening the Support to the Nonlinear Exploration Process

The design of is motivated by the nonlinear model of behavior <cit.>.
supports the nonlinear exploration process primarily through generating EQs for orientation, query expansion for opening, and theme extraction for consolidation.
However, support for researchers could be further extended, especially in the consolidation stage and the transitions between different stages.
For example, does not provide direct supports for activities in consolidation such as incorporation and verifying, which is challenging due to the diverging tastes and paradigms across disciplines <cit.>.
Future work can explore methods to help researchers understand how to use the found papers effectively.
One inspiration could come from research on analogical search <cit.>, which supports researchers in transferring solutions from other disciplines. 
Also, improving the interdisciplinary reading experience through features such as guiding questions or in-situ summarization as suggested by  <cit.> can be helpful.

Moreover, participants indicated that is limited in assisting researchers in exploring the citation networks of papers of interest, which could facilitate transitioning from consolidation to opening or orientation. Existing work, such as Synergi <cit.> and Relatedly <cit.>, can enhance this support. For example, Synergi can aid in theme extraction from citation networks based on text snippets of seed papers <cit.>.








  §.§.§ Scale of the user study

In our usability experiment, the task scenarios are mainly related to HCI research.

We acknowledge that this design may limit the generalizability of our findings to other research contexts. 
Testing in a broader range of scenarios quantitatively would enable us to better evaluate its efficacy in supporting .
Additionally, both the usability experiment and the open-ended exploratory study are relatively short in comparison to the substantial amount of time typically required for researchers in .
Long-term field studies are needed to fully assess 's support in , and benefits as well as limitations of human-AI co-exploration.
For example, some participants in study 2 exhaustively explore all presented papers by the system, which can be explained by their open-minded search habits <cit.>.
It would be interesting to see whether long-term usage of would change such habits. 







§ CONCLUSION

In this paper, we present , a novel interactive system designed to enhance the process of interdisciplinary information seeking through human-AI co-exploration. leverages the capabilities of large language models (LLMs) to assist users in their exploration efforts by generating exploratory questions (EQs) and facilitating automatic query expansion. It enables users to actively participate in the exploration by customizing EQs. Additionally, employs contextualized clustering to extract themes from the gathered papers and annotates key phrases within those papers, aiding users in quickly identifying how these papers relate to their research objectives.
Our evaluation of includes two user studies. The first study assesses usability, gathering both quantitative and qualitative feedback from 12 researchers, which indicates that significantly enhances researchers' ability to conduct thorough paper searches. The second study examines the potential integration of into the daily routines of interdisciplinary researchers, with qualitative findings highlighting the value researchers find in co-exploring with . However, these studies also highlight certain limitations of , pointing towards areas for future investigation and improvement.





















This work is supported by the seed fund of the Big Data for Bio-Intelligence Laboratory (Z0428) from The Hong Kong University of Science and Technology. We thank the anonymous reviewers for their constructive comments and our user study participants for their efforts and insights. We also thank Yujie Zheng for helping with the literature review and Chengzhong Liu for providing suggestions during the early system design stage.





ACM-Reference-Format








67


















#1     #1#1#1     #1     #1     #1     #1#1          #1#1           






[Abbott(2010)]
        abbott2010chaos
authorpersonAndrew Abbott.
  year2010.
booktitleChaos of disciplines.
publisherUniversity of Chicago Press.



[AlKhamissi et al(2022)]
        alkhamissi2022review
authorpersonBadr AlKhamissi, personMillicent
  Li, personAsli Celikyilmaz, personMona Diab, and
  personMarjan Ghazvininejad. year2022.
A review on language models as knowledge bases.
journalarXiv preprint arXiv:2204.06031
  (year2022).



[Athukorala et al(2016)]
        athukorala2016beyond
authorpersonKumaripaba Athukorala,
  personAlan Medlar, personAntti Oulasvirta,
  personGiulio Jacucci, and personDorota Glowacka.
  year2016.
Beyond relevance: Adapting exploration/exploitation
  in information retrieval. In booktitleProceedings of the
  21st international conference on intelligent user interfaces.
  pages359–369.



[August et al(2023)]
        august2023paper
authorpersonTal August, personLucy Lu Wang,
  personJonathan Bragg, personMarti A Hearst,
  personAndrew Head, and personKyle Lo.
  year2023.
Paper plain: Making medical research papers
  approachable to healthcare consumers with natural language processing.
journalACM Transactions on Computer-Human
  Interaction volume30, number5
  (year2023), pages1–38.



[Bansal et al(2021)]
        bansal2021does
authorpersonGagan Bansal, personTongshuang
  Wu, personJoyce Zhou, personRaymond Fok,
  personBesmira Nushi, personEce Kamar,
  personMarco Tulio Ribeiro, and personDaniel Weld.
  year2021.
Does the whole exceed its parts? the effect of ai
  explanations on complementary team performance. In
  booktitleProceedings of the 2021 CHI conference on human
  factors in computing systems. pages1–16.



[Bates(1996)]
        bates1996learning
authorpersonMarcia J Bates.
  year1996.
Learning about the information seeking of
  interdisciplinary scholars and students.
(year1996).



[Beaudouin-Lafon and Mackay(2000)]
        beaudouin2000reification
authorpersonMichel Beaudouin-Lafon and
  personWendy E Mackay. year2000.
Reification, polymorphism and reuse: three
  principles for designing visual interfaces. In
  booktitleProceedings of the working conference on Advanced
  visual interfaces. pages102–109.



[Braun and Clarke(2012)]
        braun2012thematic
authorpersonVirginia Braun and
  personVictoria Clarke. year2012.
booktitleThematic analysis.
publisherAmerican Psychological Association.



[Brown et al(2015)]
        brown2015interdisciplinarity
authorpersonRebekah R Brown, personAna
  Deletic, and personTony HF Wong.
  year2015.
Interdisciplinarity: How to catalyse
  collaboration.
journalNature volume525,
  number7569 (year2015), pages315–317.



[Brown et al(2020)]
        brown2020language
authorpersonTom Brown, personBenjamin Mann,
  personNick Ryder, personMelanie Subbiah,
  personJared D Kaplan, personPrafulla Dhariwal,
  personArvind Neelakantan, personPranav Shyam,
  personGirish Sastry, personAmanda Askell,
  et al year2020.
Language models are few-shot learners.
journalAdvances in neural information processing
  systems  volume33 (year2020),
  pages1877–1901.



[Bruce(1994)]
        bruce1994research
authorpersonChristine Susan Bruce.
  year1994.
Research students' early experiences of the
  dissertation literature review.
journalStudies in Higher Education
  volume19, number2 (year1994),
  pages217–229.



[Cai et al(2019)]
        cai2019human
authorpersonCarrie J Cai, personEmily Reif,
  personNarayan Hegde, personJason Hipp,
  personBeen Kim, personDaniel Smilkov,
  personMartin Wattenberg, personFernanda Viegas,
  personGreg S Corrado, personMartin C Stumpe,
  et al year2019.
Human-centered tools for coping with imperfect
  algorithms during medical decision-making. In
  booktitleProceedings of the 2019 chi conference on human
  factors in computing systems. pages1–14.



[Chan et al(2018)]
        chan2018solvent
authorpersonJoel Chan, personJoseph Chee
  Chang, personTom Hope, personDafna Shahaf, and
  personAniket Kittur. year2018.
Solvent: A mixed initiative system for finding
  analogies between research papers.
journalProceedings of the ACM on Human-Computer
  Interaction volume2, numberCSCW
  (year2018), pages1–21.



[Chan et al(2024)]
        chan2024formulating
authorpersonJoel Chan, personZijian Ding, personEesh Kamrah, and personMark Fuge.
  year2024.
Formulating or Fixating: Effects of Examples on Problem Solving Vary as a Function of Example Presentation Interface Design. In
  booktitleProceedings of the CHI Conference on Human Factors in Computing Systems. pages1–16.



[Chang et al(2023)]
        chang2023citesee
authorpersonJoseph Chee Chang, personAmy X
  Zhang, personJonathan Bragg, personAndrew Head,
  personKyle Lo, personDoug Downey, and
  personDaniel S Weld. year2023.
CiteSee: Augmenting Citations in Scientific Papers
  with Persistent and Personalized Historical Context.
journalarXiv preprint arXiv:2302.07302
  (year2023).



[Chau et al(2011)]
        chau2011apolo
authorpersonDuen Horng Chau, personAniket
  Kittur, personJason I Hong, and personChristos
  Faloutsos. year2011.
Apolo: making sense of large network data by
  combining rich user interaction and machine learning. In
  booktitleProceedings of the SIGCHI conference on human
  factors in computing systems. pages167–176.



[Choe et al(2021)]
        choe2021papers101
authorpersonKiroong Choe, personSeokweon
  Jung, personSeokhyeon Park, personHwajung Hong,
  and personJinwook Seo. year2021.
Papers101: Supporting the discovery process in the
  literature review workflow for novice researchers. In
  booktitle2021 IEEE 14th Pacific Visualization Symposium
  (PacificVis). IEEE, pages176–180.



[Choi and Pak(2006)]
        choi2006multidisciplinarity
authorpersonBernard CK Choi and
  personAnita WP Pak. year2006.
Multidisciplinarity, interdisciplinarity and
  transdisciplinarity in health research, services, education and policy: 1.
  Definitions, objectives, and evidence of effectiveness.
journalClinical and investigative medicine
  volume29, number6 (year2006),
  pages351.


[Coltheart(1981)]
        coltheart1981mrc
authorpersonMax Coltheart.
  year1981.
The MRC psycholinguistic database.
journalThe Quarterly Journal of Experimental Psychology Section A volume33, number4 (year1981),
  pages497–505.



[Foster(2004)]
        foster2004nonlinear
authorpersonAllen Foster.
  year2004.
A nonlinear model of information-seeking behavior.
journalJournal of the American society for
  information science and technology volume55,
  number3 (year2004), pages228–237.



[Gajos and Mamykina(2022)]
        gajos2022people
authorpersonKrzysztof Z Gajos and
  personLena Mamykina. year2022.
Do people engage cognitively with AI? Impact of AI
  assistance on incidental learning. In booktitle27th
  international conference on intelligent user interfaces.
  pages794–806.



[Gao et al(2022)]
        gao2022precise
authorpersonLuyu Gao, personXueguang Ma, personJimmy Lin, and personJamie Callan.
  year2022.
Precise zero-shot dense retrieval without relevance labels.
journalarXiv preprint arXiv:2212.10496 (year2022).



[Hart and Staveland(1988)]
        hart1988development
authorpersonSandra G Hart and
  personLowell E Staveland. year1988.
Development of NASA-TLX (Task Load Index): Results
  of empirical and theoretical research.
In booktitleAdvances in psychology.
  Vol. volume52. publisherElsevier,
  pages139–183.



[Head et al(2021)]
        head2021augmenting
authorpersonAndrew Head, personKyle Lo,
  personDongyeop Kang, personRaymond Fok,
  personSam Skjonsberg, personDaniel S Weld, and
  personMarti A Hearst. year2021.
Augmenting scientific papers with just-in-time,
  position-sensitive definitions of terms and symbols. In
  booktitleProceedings of the 2021 CHI Conference on Human
  Factors in Computing Systems. pages1–18.



[Heer(2019)]
        heer2019agency
authorpersonJeffrey Heer.
  year2019.
Agency plus automation: Designing artificial
  intelligence into interactive systems.
journalProceedings of the National Academy of
  Sciences volume116, number6 (year2019),
  pages1844–1850.



[Horvitz(1999)]
        horvitz1999principles
authorpersonEric Horvitz.
  year1999.
Principles of mixed-initiative user interfaces. In
  booktitleProceedings of the SIGCHI conference on Human
  Factors in Computing Systems. pages159–166.



[Jesson et al(2011)]
        jesson2011doing
authorpersonJill Jesson, personLydia
  Matheson, and personFiona M Lacey.
  year2011.
Doing your literature review: Traditional and
  systematic techniques.
(year2011).



[Jiang et al(2023)]
        jiang2023graphologue
authorpersonPeiling Jiang, personJude
  Rayan, personSteven P Dow, and personHaijun Xia.
  year2023.
Graphologue: Exploring large language model
  responses with interactive diagrams. In
  booktitleProceedings of the 36th Annual ACM Symposium on
  User Interface Software and Technology. pages1–20.



[Kang et al(2022a)]
        kang2022threddy
authorpersonHyeonsu Kang, personJoseph Chee
  Chang, personYongsung Kim, and personAniket
  Kittur. year2022a.
Threddy: An Interactive System for Personalized
  Thread-based Exploration and Organization of Scientific Literature. In
  booktitleProceedings of the 35th Annual ACM Symposium on
  User Interface Software and Technology. pages1–15.



[Kang et al(2022b)]
        kang2022you
authorpersonHyeonsu B Kang, personRafal
  Kocielnik, personAndrew Head, personJiangjiang Yang,
  personMatt Latzke, personAniket Kittur,
  personDaniel S Weld, personDoug Downey, and
  personJonathan Bragg. year2022b.
From who you know to what you read: augmenting
  scientific recommendations with implicit social networks. In
  booktitleProceedings of the 2022 CHI Conference on Human
  Factors in Computing Systems. pages1–23.



[Kang et al(2022c)]
        kang2022augmenting
authorpersonHyeonsu B Kang, personXin Qian,
  personTom Hope, personDafna Shahaf,
  personJoel Chan, and personAniket Kittur.
  year2022c.
Augmenting scientific creativity with an analogical
  search engine.
journalACM Transactions on Computer-Human
  Interaction volume29, number6
  (year2022), pages1–36.



[Kang et al(2023a)]
        kang2023comlittee
authorpersonHyeonsu B Kang, personNouran
  Soliman, personMatt Latzke, personJoseph Chee Chang,
  and personJonathan Bragg. year2023a.
ComLittee: Literature Discovery with Personal
  Elected Author Committees.
journalarXiv preprint arXiv:2302.06780
  (year2023).



[Kang et al(2023b)]
        kang2023synergi
authorpersonHyeonsu B Kang, personTongshuang
  Wu, personJoseph Chee Chang, and personAniket
  Kittur. year2023b.
Synergi: A Mixed-Initiative System for Scholarly
  Synthesis and Sensemaking. In booktitleProceedings of the
  36th Annual ACM Symposium on User Interface Software and Technology.
  pages1–19.



[Knapp(2012)]
        knapp2012plugging
authorpersonJeffrey A Knapp.
  year2012.
Plugging the “whole”: Librarians as
  interdisciplinary facilitators.
journalLibrary Review volume61,
  number3 (year2012), pages199–214.



[Lahav et al(2022)]
        lahav2022search
authorpersonDan Lahav, personJon Saad
  Falcon, personBailey Kuehl, personSophie Johnson,
  personSravanthi Parasa, personNoam Shomron,
  personDuen Horng Chau, personDiyi Yang,
  personEric Horvitz, personDaniel S Weld,
  et al year2022.
A search engine for discovery of scientific
  challenges and directions. In booktitleProceedings of the
  AAAI Conference on Artificial Intelligence, Vol. volume36.
  pages11982–11990.



[Lam et al(2024)]
        lam2024concept
authorpersonMichelle S Lam, personJanice
  Teoh, personJames A Landay, personJeffrey Heer,
  and personMichael S Bernstein.
  year2024.
Concept Induction: Analyzing Unstructured Text with
  High-Level Concepts Using LLooM. In booktitleProceedings of
  the CHI Conference on Human Factors in Computing Systems.
  pages1–28.



[Lee et al(2024)]
        lee2024paperweaver
authorpersonYoonjoo Lee, personHyeonsu B Kang, personMatt Latzke, personJuho Kim, personJonathan Bragg, personJoseph Chee Chang, and personPao Siangliulue.
  year2024.
PaperWeaver: Enriching Topical Paper Alerts by Contextualizing Recommended Papers with User-collected Papers. In
  booktitleProceedings of the CHI Conference on Human Factors in Computing Systems. pages1–19.



[Lewis et al(2020)]
        lewis2020retrieval
authorpersonPatrick Lewis, personEthan
  Perez, personAleksandra Piktus, personFabio Petroni,
  personVladimir Karpukhin, personNaman Goyal,
  personHeinrich Küttler, personMike Lewis,
  personWen-tau Yih, personTim Rocktäschel,
  et al year2020.
Retrieval-augmented generation for
  knowledge-intensive nlp tasks.
journalAdvances in Neural Information Processing
  Systems  volume33 (year2020),
  pages9459–9474.



[Liang et al(2022)]
        liang2022holistic
authorpersonPercy Liang, personRishi
  Bommasani, personTony Lee, personDimitris Tsipras,
  personDilara Soylu, personMichihiro Yasunaga,
  personYian Zhang, personDeepak Narayanan,
  personYuhuai Wu, personAnanya Kumar, et al
  year2022.
Holistic evaluation of language models.
journalarXiv preprint arXiv:2211.09110
  (year2022).



[Liu et al(2023)]
        liu2023selenite
authorpersonMichael Xieyang Liu,
  personTongshuang Wu, personTianying Chen,
  personFranklin Mingzhe Li, personAniket Kittur, and
  personBrad A Myers. year2023.
Selenite: Scaffolding Online Sensemaking with
  Comprehensive Overviews Elicited from Large Language Models.
journalarXiv preprint arXiv:2310.02161
  (year2023).


[Lupfer et al(2016)]
        lupfer2016patterns
authorpersonNic Lupfer, personAndruid Kerne, personAndrew M Webb, and personRhema Linder.
  year2016.
Patterns of free-form curation: Visual thinking with web content. In
  booktitleProceedings of the 24th ACM international conference on Multimedia. pages12–21.



[Ma et al(2024)]
        ma2024towards
authorpersonShuai Ma, personQiaoyi Chen, personXinru Wang, personChengbo Zheng, personZhenhui Peng, personMing Yin, and personXiaojuan Ma.
  year2024.
Towards human-ai deliberation: Design and evaluation of llm-empowered deliberative ai for ai-assisted decision-making.
journalarXiv preprint arXiv:2403.16812 (year2024).



[Ma et al(2023)]
        ma2023should
authorpersonShuai Ma, personYing Lei,
  personXinru Wang, personChengbo Zheng,
  personChuhan Shi, personMing Yin, and
  personXiaojuan Ma. year2023.
Who should i trust: Ai or myself? leveraging human
  and ai correctness likelihood to promote appropriate trust in ai-assisted
  decision-making. In booktitleProceedings of the 2023 CHI
  Conference on Human Factors in Computing Systems. pages1–19.



[Mendes et al(2011)]
        mendes2011dbpedia
authorpersonPablo N Mendes, personMax
  Jakob, personAndrés García-Silva, and
  personChristian Bizer. year2011.
DBpedia spotlight: shedding light on the web of
  documents. In booktitleProceedings of the 7th international
  conference on semantic systems. pages1–8.



[Metzler et al(2021)]
        metzler2021rethinking
authorpersonDonald Metzler, personYi Tay,
  personDara Bahri, and personMarc Najork.
  year2021.
Rethinking search: making domain experts out of
  dilettantes. In booktitleAcm sigir forum,
  Vol. volume55. ACM New York, NY, USA, pages1–27.



[Miller(2023)]
        miller2023explainable
authorpersonTim Miller.
  year2023.
Explainable ai is dead, long live explainable ai!
  hypothesis-driven decision support using evaluative ai. In
  booktitleProceedings of the 2023 ACM Conference on Fairness,
  Accountability, and Transparency. pages333–342.



[Mote(1962)]
        mote1962reasons
authorpersonLionel JB Mote.
  year1962.
Reasons for the variations in the information needs
  of scientists.
journalJournal of Documentation
  volume18, number4 (year1962),
  pages169–175.



[Newby(2011)]
        newby2011entering
authorpersonJill Newby.
  year2011.
Entering unfamiliar territory: Building an
  information literacy course for graduate students in interdisciplinary
  areas.
journalReference & User Services Quarterly
  volume50, number3 (year2011),
  pages224–229.



[OpenAI(2024)]
        openai2024embedding
authorpersonOpenAI.
  year2024.
New Embedding Models and API Updates.
howpublished<https://openai.com/index/new-embedding-models-and-api-updates/>, noteAccessed on 7.22.2024.



[Palani et al(2023)]
        palani2023relatedly
authorpersonSrishti Palani, personAakanksha Naik, personDoug Downey, personAmy X Zhang, personJonathan Bragg, and personJoseph Chee Chang.
  year2023.
Relatedly: Scaffolding literature reviews with existing related work sections. In
  booktitleProceedings of the 2023 CHI Conference on Human Factors in Computing Systems. pages1–20.



[Palani et al(2022)]
        palani2022interweave
authorpersonSrishti Palani, personYingyi
  Zhou, personSheldon Zhu, and personSteven P Dow.
  year2022.
InterWeave: Presenting Search Suggestions in
  Context Scaffolds Information Search and Synthesis. In
  booktitleProceedings of the 35th Annual ACM Symposium on
  User Interface Software and Technology. pages1–16.



[Palmer and Fenlon(2010)]
        palmer2010information
authorpersonCarole L Palmer and
  personKatrina Fenlon. year2010.
Information research on interdisciplinarity.
journalThe Oxford handbook of interdisciplinarity
  (year2010), pages174–188.



[Pirolli and Card(1999)]
        pirolli1999information
authorpersonPeter Pirolli and personStuart
  Card. year1999.
Information foraging.
journalPsychological review volume106,
  number4 (year1999), pages643.



[Pirolli et al(1996)]
        pirolli1996scatter
authorpersonPeter Pirolli, personPatricia
  Schank, personMarti Hearst, and personChristine
  Diehl. year1996.
Scatter/gather browsing communicates the topic
  structure of a very large text collection. In
  booktitleProceedings of the SIGCHI conference on Human
  factors in computing systems. pages213–220.



[Qiu et al(2022)]
        qiu2022docflow
authorpersonRui Qiu, personYamei Tu,
  personYu-Shuen Wang, personPo-Yin Yen, and
  personHan-Wei Shen. year2022.
DocFlow: A Visual Analytics System for
  Question-based Document Retrieval and Categorization.
journalIEEE Transactions on Visualization and
  Computer Graphics (year2022).



[Rowland(2002)]
        rowland2002overcoming
authorpersonStephen Rowland.
  year2002.
Overcoming fragmentation in professional life: The
  challenge for academic development.
journalHigher education quarterly
  volume56, number1 (year2002),
  pages52–64.



[Schubert et al(2017)]
        schubert2017dbscan
authorpersonErich Schubert, personJörg
  Sander, personMartin Ester, personHans Peter
  Kriegel, and personXiaowei Xu.
  year2017.
DBSCAN revisited, revisited: why and how you should
  (still) use DBSCAN.
journalACM Transactions on Database Systems (TODS)
  volume42, number3 (year2017),
  pages1–21.



[Shah and Bender(2022)]
        shah2022situating
authorpersonChirag Shah and personEmily M
  Bender. year2022.
Situating search. In
  booktitleProceedings of the 2022 Conference on Human
  Information Interaction and Retrieval. pages221–232.



[Shahaf et al(2012)]
        shahaf2012metro
authorpersonDafna Shahaf, personCarlos
  Guestrin, and personEric Horvitz.
  year2012.
Metro maps of science. In
  booktitleProceedings of the 18th ACM SIGKDD international
  conference on Knowledge discovery and data mining.
  pages1122–1130.



[Sharma et al(2024)]
        sharma2024generative
authorpersonNikhil Sharma, personQ Vera
  Liao, and personZiang Xiao. year2024.
Generative Echo Chamber? Effects of LLM-Powered
  Search Systems on Diverse Information Seeking.
journalarXiv preprint arXiv:2402.05880
  (year2024).



[Shi et al(2023)]
        shi2023nl2color
authorpersonChuhan Shi, personWeiwei Cui, personChengzhong Liu, personChengbo Zheng, personHaidong Zhang, personQiong Luo, and personXiaojuan Ma.
  year2023.
NL2Color: Refining color palettes for charts with natural language.
journalIEEE Transactions on Visualization and Computer Graphics (year2023).



[Shi et al(2022)]
        shi2022medchemlens
authorpersonChuhan Shi, personFei Nie,
  personYicheng Hu, personYige Xu, personLei
  Chen, personXiaojuan Ma, and personQiong Luo.
  year2022.
MedChemLens: An Interactive Visual Tool to Support
  Direction Selection in Interdisciplinary Experimental Research of Medicinal
  Chemistry.
journalIEEE Transactions on Visualization and
  Computer Graphics volume29, number1
  (year2022), pages63–73.



[Shi et al(2023)]
        shi2023retrolens
authorpersonChuhan Shi, personYicheng Hu, personShenan Wang, personShuai Ma, personChengbo Zheng, personXiaojuan Ma, and personQiong Luo.
  year2023.
RetroLens: A Human-AI Collaborative System for Multi-step Retrosynthetic Route Planning. In
  booktitleProceedings of the 2023 CHI Conference on Human Factors in Computing Systems. pages1–20.


[Smalheiser(2017)]
        smalheiser2017rediscovering
authorpersonNeil R Smalheiser.
  year2017.
Rediscovering Don Swanson: the past, present and future of literature-based discovery.
journalJournal of Data and Information Science volume2, number4 (year2017),
  pages43–64.



[Son et al(2024)]
        son2024genquery
authorpersonKihoon Son, personDaEun Choi, personTae Soo Kim, personYoung-Ho Kim, and personJuho Kim.
  year2024.
Genquery: Supporting expressive visual search with generative models. In
  booktitleProceedings of the CHI Conference on Human Factors in Computing Systems. pages1–19.



[Spanner(2001)]
        spanner2001border
authorpersonDon Spanner.
  year2001.
Border crossings: Understanding the cultural and
  informational dilemmas of interdisciplinary scholars.
journalThe Journal of Academic Librarianship
  volume27, number5 (year2001),
  pages352–360.



[Suh et al(2023)]
        suh2023sensecape
authorpersonSangho Suh, personBryan Min,
  personSrishti Palani, and personHaijun Xia.
  year2023.
Sensecape: Enabling Multilevel Exploration and
  Sensemaking with Large Language Models.
journalarXiv preprint arXiv:2305.11483
  (year2023).



[Swanson(1986)]
        swanson1986undiscovered
authorpersonDon R Swanson.
  year1986.
Undiscovered public knowledge.
journalThe Library Quarterly volume56, number2 (year1986),
  pages103–118.



[Swanson and Smalheiser(1996)]
        swanson1996undiscovered
authorpersonDon R Swanson and personNeil R Smalheiser.
  year1996.
Undiscovered Public Knowledge: A Ten-Year Update. In
  booktitleProceedings of the KDD. volume96,
  pages295–298.



[Tang et al(2008)]
        tang2008arnetminer
authorpersonJie Tang, personJing Zhang,
  personLimin Yao, personJuanzi Li, personLi
  Zhang, and personZhong Su. year2008.
Arnetminer: extraction and mining of academic
  social networks. In booktitleProceedings of the 14th ACM
  SIGKDD international conference on Knowledge discovery and data mining.
  pages990–998.



[Teevan et al(2004)]
        teevan2004perfect
authorpersonJaime Teevan, personChristine
  Alvarado, personMark S Ackerman, and personDavid R
  Karger. year2004.
The perfect search engine is not enough: a study of
  orienteering behavior in directed search. In
  booktitleProceedings of the SIGCHI conference on Human
  factors in computing systems. pages415–422.



[Van Noorden et al(2015)]
        van2015interdisciplinary
authorpersonRichard Van Noorden et al
  year2015.
Interdisciplinary research by the numbers.
journalNature volume525,
  number7569 (year2015), pages306–307.



[Venkatesh and Bala(2008)]
        venkatesh2008technology
authorpersonViswanath Venkatesh and
  personHillol Bala. year2008.
Technology acceptance model 3 and a research agenda
  on interventions.
journalDecision sciences volume39,
  number2 (year2008), pages273–315.



[Viswanathan et al(2023)]
        viswanathan2023large
authorpersonVijay Viswanathan, personKiril
  Gashteovski, personCarolin Lawrence, personTongshuang
  Wu, and personGraham Neubig. year2023.
Large language models enable few-shot clustering.
journalarXiv preprint arXiv:2307.00524
  (year2023).



[Wang et al(2023b)]
        wang2023query2doc
authorpersonLiang Wang, personNan Yang,
  and personFuru Wei. year2023b.
Query2doc: Query expansion with large language
  models.
journalarXiv preprint arXiv:2303.07678
  (year2023).



[Wang et al(2023a)]
        wang2023can
authorpersonShuai Wang, personHarrisen
  Scells, personBevan Koopman, and personGuido
  Zuccon. year2023a.
Can ChatGPT write a good boolean query for
  systematic review literature search?. In
  booktitleProceedings of the 46th International ACM SIGIR
  Conference on Research and Development in Information Retrieval.
  pages1426–1436.


[Webb et al(2016)]
        webb2016free
authorpersonAndrew M Webb, personAndruid Kerne, personRhema Linder, personNic Lupfer, personYin Qu, personKade Keith, personMatthew Carrasco, and personYvonne Chen.
  year2016.
A free-form medium for curating the digital.
journalCurating the Digital: Space for Art and Interaction pages73–87.



[White et al(2023)]
        white2023prompt
authorpersonJules White, personQuchen Fu,
  personSam Hays, personMichael Sandborn,
  personCarlos Olea, personHenry Gilbert,
  personAshraf Elnashar, personJesse Spencer-Smith,
  and personDouglas C Schmidt. year2023.
A prompt pattern catalog to enhance prompt
  engineering with chatgpt.
journalarXiv preprint arXiv:2302.11382
  (year2023).



[Wu et al(2022)]
        wu2022ai
authorpersonTongshuang Wu, personMichael
  Terry, and personCarrie Jun Cai.
  year2022.
Ai chains: Transparent and controllable human-ai
  interaction by chaining large language model prompts. In
  booktitleProceedings of the 2022 CHI Conference on Human
  Factors in Computing Systems. pages1–22.



[Xiao(2023)]
        xiao2023autosurveygpt
authorpersonChang Xiao.
  year2023.
AutoSurveyGPT: GPT-Enhanced Automated Literature
  Discovery. In booktitleAdjunct Proceedings of the 36th
  Annual ACM Symposium on User Interface Software and Technology.
  pages1–3.



[Yang et al(2022)]
        yang2022empirical
authorpersonZhengyuan Yang, personZhe Gan,
  personJianfeng Wang, personXiaowei Hu,
  personYumao Lu, personZicheng Liu, and
  personLijuan Wang. year2022.
An empirical study of gpt-3 for few-shot
  knowledge-based vqa. In booktitleProceedings of the AAAI
  Conference on Artificial Intelligence, Vol. volume36.
  pages3081–3089.



[Zhang et al(2023)]
        zhang2023clusterllm
authorpersonYuwei Zhang, personZihan Wang,
  and personJingbo Shang. year2023.
Clusterllm: Large language models as a guide for
  text clustering.
journalarXiv preprint arXiv:2305.14871
  (year2023).



[Zheng et al(2022)]
        zheng2022telling
authorpersonChengbo Zheng, personDakuo Wang, personApril Yi Wang, and personXiaojuan Ma.
  year2022.
Telling stories from computational notebooks: AI-assisted presentation slides creation for presenting data science work. In
  booktitleProceedings of the 2022 CHI Conference on Human Factors in Computing Systems. pages1–20.



[Zheng et al(2024)]
        zheng2024charting
authorpersonChengbo Zheng, personKangyu Yuan, personBingcan Guo, personReza Hadi Mogavi, personZhenhui Peng, personShuai Ma, and personXiaojuan Ma.
  year2024.
Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with Students. In
  booktitleProceedings of the CHI Conference on Human Factors in Computing Systems. pages1–19.
























§ EXAMPLE OUTPUTS OF PROMPTS FOR EXPLORATORY QUESTION (EQ) GENERATION AND SEARCH QUERY EXPANSION











We design to address the terminology gap by generating simple EQs to ease sensemaking and search queries with specific terminology to ensure search proficiency.
In this appendix, we provide example outputs of the generated EQs and queries.


 §.§ EQ Generation


uses EQs to help users explore unfamiliar fields by asking LLM to take expert personas and adding a condition in the prompt that the generated EQs need to be easy to understand.
We tested prompts for EQ generation, including the one used in , and two ablation prompts: 1) a prompt without persona; 2) a prompt without the simplification condition.
The prompts are presented in <ref>.

We used these prompts to generate three EQs under three different relevant disciplines for eight research topics (in total 72 EQs). Part of the results are presented in <ref>.
We include all the results in the supplementary material.
These results present hints that using persona makes EQs more concrete to domains and adding a simplification requirement enhances the comprehensibility.






 












 §.§ Search Query Expansion






first generates pseudo-answers (PA) and then uses PA to prompt for queries (<ref>).
We tested query expansion with PA generation first and without PA.
For each EQ generated in <ref>, nine search queries were generated.
Part of the results are presented in <ref> and all are included in the supplementary material.
We also compared the concreteness of the generated queries of the two query expansion approaches (with and without PA) using the MRC Psycholinguistic Database <cit.>.
The average concreteness ratings of the queries generated with PA is 319.56 (SD=52.37), and for the queries generated without PA is 292.42 (SD=39.69).
A comparison between the two conditions regarding the generated queries of each input EQ using paired t-test shows the queries generated with PA is significantly more concrete than the ones generated without PA (p<0.01).
These results provide evidence on the addition of PA helps inject domain-specific terms into the queries.



















 Paper Title: ProSpec RL: Plan Ahead, then Execute.


	
	
	
Enhancement of the molecular electron chirality by electronic excitation

    Masato Senami
    August 5, 2024
==========================================================================

§ ABSTRACT 
		Imagining potential outcomes of actions before execution helps agents make more informed decisions, a prospective thinking ability fundamental to human cognition. However, mainstream model-free Reinforcement Learning (RL) methods lack the ability to proactively envision future scenarios, plan, and guide strategies. These methods typically rely on trial and error to adjust policy functions, aiming to maximize cumulative rewards or long-term value, even if such high-reward decisions place the environment in extremely dangerous states. To address this, we propose the Prospective (ProSpec) RL method, which makes higher-value, lower-risk optimal decisions by imagining future n-stream trajectories. Specifically, ProSpec employs a dynamic model to predict future states (termed “imagined states”) based on the current state and a series of sampled actions. Furthermore, we integrate the concept of Model Predictive Control and introduce a cycle consistency constraint that allows the agent to evaluate and select the optimal actions from these trajectories. Moreover, ProSpec employs cycle consistency to mitigate two fundamental issues in RL: augmenting state reversibility to avoid irreversible events (low risk) and augmenting actions to generate numerous virtual trajectories, thereby improving data efficiency. We validated the effectiveness of our method on the DMControl benchmarks, where our approach achieved significant performance improvements. Code will be open-sourced upon acceptance.
	
	
	
	

§ INTRODUCTION

	Reinforcement Learning (RL) has proven indispensable for solving continuous decision problems <cit.>. Current RL research falls into two main categories: model-based and model-free methods. Model-based methods involve constructing a world model to simulate an interactive environment, which helps the agent predict future trajectories for optimal decision-making. However, these methods can be expensive and prone to failure due to discrepancies between the world model and reality <cit.>. On the other hand, model-free methods refine the policy function through trial-and-error learning, often outperforming model-based techniques in continuous control tasks <cit.>. However, these methods require frequent interaction with the environment, which is risky or costly in real-world applications <cit.>, and suffer from low data efficiency when learning from complex, high-dimensional observations (such as image pixels) based on limited experience.
	
	Training model-free methods using low data efficiency and sparse signals to fit high-performance feature encoders is challenging and prone to suboptimal convergence <cit.>. In contrast, humans can learn to play simple arcade games in minutes <cit.>, while even state-of-the-art RL algorithms require billions of frames to reach human-level performance <cit.>. Moreover, collecting such large amounts of interaction data is impractical, especially in domains such as autonomous driving and robot control. Currently, researchers are investigating data efficiency challenges from various angles. Including leveraging model-based world models to aid in policy learning <cit.>, employing data augmentation techniques to augment diversity in image appearances <cit.>, and introducing auxiliary tasks to assist in state representation learning <cit.>, among other approaches. While many of these methods involve creating additional training processes to accelerate agent training, the challenge of limited experience continues to impede the model's exceptional performance.
	
	Learning planning is often deemed more critical than learning policies <cit.>. In cognitive neuroscience and psychology, planning is considered a facet of prospective thinking <cit.>, which aids in predicting and adapting to future changes, envisioning potential problems or dangers, devising strategies to enhance problem-solving abilities, and setting future goals to drive action <cit.>. Ideally, prospective thinking is highly relevant to continuous decision-making tasks, where current actions can have irreversible consequences on future scenarios. However, current model-free methods lack planning capabilities, thus relying solely on trial and error may lead to unacceptable risks.
	
	In this work, we introduce ProSpec, a model-free RL method that harnesses prospective thinking to envision potential future scenarios and make optimal decisions with higher value and lower risk through planning. More importantly, ProSpec could address data efficiency concerns by generating numerous virtual trajectories through imagination. Specifically, we model prospective thinking using a Flow-based Reversible Dynamics Model (FDM) and a Model Predictive Control (MPC)-based block. The FDM predicts prospective states in latent space in the forward process. In the backward process, we enforce cycle consistency constraints on the reversibility of prospective states to prevent the failure of prospective trajectories in irreparable states. Furthermore, we augment actions in a self-supervised learning fashion to generate numerous virtual trajectories and improve the efficiency of RL data. The MPC block employs the concept of MPC to facilitate exploratory interaction between the agent and the FDM, with the objective of exploring future scenarios from diverse perspectives. We summarize the contributions of this work as follows:
	
		
		
		
		
  * We introduce a model-free method that integrates prospective thinking by incorporating a Flow-based Reversible Dynamics Model (FDM) for predicting prospective states, combined with cycle consistency to maintain state traceability. This enables the agent to explore future scenarios from various perspectives and identify optimal decisions with higher value and lower risk.
		
  * Within our reversible framework, we introduce action augmentation in a self-supervised manner, generating multiple state-action trajectories to improve data efficiency.
		
  * We validate the effectiveness of ProSpec through experiments on DeepMind benchmarks, showing significant performance improvements.
	
	
	

§ RELATED WORK

	

 §.§ Prospective thinking

	The capacity to anticipate the future represents as a crucial facet of human cognition, recently garnering substantial attention from researchers <cit.>. In psychology, prospective thinking is defined as the ability to mentally simulate the future, involving the extraction of information from episodic and semantic memory (such as detailed information about past states) as well as more abstract, schematic, and conceptual knowledge (such as envisioning general goals or events) <cit.>. Broadly, prospective thinking involves two main aspects: 1) predicting future scenarios and 2) guiding decisions for those scenarios. In deep learning, methods such as Model Predictive Control (MPC) and model-based Reinforcement Learning (RL) appear to encapsulate certain aspects of prospective thinking.
	
	Model-based RL methods aim to construct an internal model, often referred to as a "World Model," to simulate the environment. This model helps predict future states and rewards, guiding the agent's decision-making and planning. Examples include PILCO <cit.>, Dyna-Q <cit.>, and World models <cit.>. On the other hand, MPC-based methods utilize the world model to optimize trajectories within a shorter time horizon. Examples of such methods include MuZero, Eff.Zero <cit.>, and TD-MPC <cit.>. However, these approaches heavily rely on accurate environmental modeling and are limited by the complexity of state spaces.
	Consequently, model-based approaches often struggle to outperform simpler model-free methods in continuous control tasks <cit.>. Model-free methods, in contrast, learn policies through trial and error, directly interacting with the environment without the need for a World Model. This method is easy to implement and adapts well to complex environments. However, due to the frequent interaction with the environment and the lack of prediction of future states or consequences, it can lead to risky exploration of the environment <cit.>.
	
	Building upon this, we merge the planning capabilities of model-based methods to enhance model-free methods. To achieve this, we introduce a dynamics model into model-free methods, enabling the prediction of future scenarios. Additionally, we integrate the concept of MPC with value consistency to aid decision-making and guidance for future scenarios. Our objective is to leverage prospective thinking in selecting the optimal path while maintaining the trial-and-error learning approach of model-free methods. This integration aims to enhance the effectiveness of model-free methods in learning.
	

 §.§ Data-Efﬁcient Reinforcement Learning

	When confronted with a limited amount of interaction data, the performance of both model-based and model-free methods experiences a significant decline. Researchers have extensively investigated this issue from various angles <cit.>. For example, SiMPLe utilizes collected data to learn a world model and generate imagined trajectories. They achieved outstanding results in multiple games under the setting of 100k frames, albeit at the cost of several weeks of training time. Subsequent studies, such as DataEfficient Rainbow <cit.> and OTRainbow <cit.>, have demonstrated that merely increasing the number of steps in multi-step returns and conducting more frequent parameter updates can lead to improved performance under conditions of low data efficiency.
	
	Consequently, there is growing interest in using computer vision techniques to enhance representation learning in RL <cit.>. For example, DrQ <cit.> and RAD <cit.> showed that moderate image augmentation can significantly improve the efficiency of RL data, outperforming previous methods using world models in model-based approaches. Yarats et al. <cit.> introduced image reconstruction as an auxiliary loss function in RL to improve data efficiency. Inspired by contrastive learning in representation learning, CURL <cit.> incorporated contrastive loss into RL, allowing the algorithm to distinguish representations of different states through contrastive loss and promote the generation of similar embeddings for similar states. In addition, some studies aim to incorporate dynamical models to help agents predict future events, thereby generating multiple virtual trajectories to improve feature representation learning. For example, SPR <cit.>, PBL <cit.>, and SLAC <cit.>, among others. Subsequently, EfficientZero <cit.> integrated SPR's prediction loss into MuZero, improving the quality of internal state representations through self-generated predictions within the model.
	
	Recently, researchers have incorporated cycle consistency into dynamics models <cit.> to ensure that generated virtual trajectories maintain consistency and coherence with real-world behavior. The basic concept of cycle consistency involves performing a forward transformation from A to B using two transformation models, followed by a reverse transformation from A' back to B, and imposing consistency constraints between the original input A and the reconstructed input A'. For example, PlayVirtual <cit.> introduced backward dynamics models based on SPR, and established a loop through constraints between forward and backward dynamics models to improve feature representation learning via cycle consistency. VCR <cit.> introduced the concept of value consistency, where their dynamics model is constrained by Q-values computed with the real environment, enabling the model to learn representations directly related to decision-making, thereby improving model performance.
	
	These successful cases demonstrate the effectiveness of cycle consistency in improving data efficiency. However, this approach often requires additional training to construct a backward dynamics model (as seen in PlayVirtual <cit.>), or directly imposing consistency constraints on learning from real environment policies. The latter approach may be susceptible to deviations in value functions caused by high-bias environments, as observed in VCR <cit.>. Therefore, in this study, we propose a novel approach that utilizes a simple flow-based dynamics model for both forward and backward predictions. In addition, cycle consistency is employed to improve data and state, ensuring the generation of numerous imagined trajectories under low-risk constraints to mitigate data efficiency issues.
	
	

§ METHODS

	In this section, we will systematically introduce the proposed ProSpec method. First, we will review some preliminary work. Then, we will present the overall framework of ProSpec.
	

 §.§ Preliminaries: Reinforcement Learning

	Reinforcement learning (RL) typically uses Markov decision processes (MDPs) to solve continuous decision problems, denoted as <𝒮,𝒜,𝒯,ℛ,γ>. Here 𝒮 represents a finite set of states; 𝒜 is the action space; 𝒯(s_t,a_t,s_t+1)=P(s_t+1|s_t,a_t) is the dynamic function, which defines the probability of transition from state s_t to s_t+1 after taking action a_t; ℛ(s_t,a_t) is the reward function, and γ∈(0,1] is the discount factor. The goal of RL is to enable the agent to learn how to maximize the expected discounted cumulative payoff G_t=∑^∞_τ=0γ^τℛ(s_t+τ, a_t+τ) by choosing actions at each time t. The agent's decision process is guided by the policy π(a_t|s_t), which maps the current state to the action selection. The action-value function Q_π(s_t,a_t)=𝔼_π[G_t|s_t,a_t] evaluates the expected payoff of taking action a_t in state s_t and following policy π. This paper focuses on value-based RL methods, specifically Q-learning, which approximates the optimal policy π^* using the Bellman equation <cit.>. Therefore, Soft Actor-Critic (SAC) is employed as the policy algorithm, given that it is a widely utilized gradient-based algorithm for continuous control that incorporates policy entropy as an additional reward to encourage exploration. The overall training objective of SAC is as follows:
	
    𝒥_θ=	ℒ_critic+ℒ_actor+ℒ_α

	where, 	ℒ_critic, ℒ_actor and ℒ_α represent the critic loss, actor loss and temperature loss, respectively (see the supplementary material for details).
	

 §.§ Overall Framework

	
	In this paper, we introduce the ProSpec method, which aims to incorporate prospective thinking into model-free methods. ProSpec consists of two main components: first, the Flow-based Dynamics Model for predicting future scenarios, and second, the prospective unit that integrates Model Predictive Control and value consistency to guide decision-making. Figure<ref> shows the process of ProSpec RL. In the following sections, we provide detailed explanations of each component.	
	
	Flow-based Dynamics Model (FDM). The FDM predicts the future and retroactively infers previous latent states, crucial in ProSpec. Unlike PlayVirtual <cit.>, which employs separate models for these tasks, our approach employs a single model for both, avoiding training instability. We utilize Real-valued non-volume preserving (RealNVP) <cit.>, a flow-based neural network that performs bijective coupling transformations. RealNVP operates via bijective coupling layers, with each layer handling parts of the input a_t ∈ℝ^1:d and z_t ∈ℝ^d+1:D separately. It applies transformations using affine functions for scaling exp(sc_i(·)) and translation t_i(·), along with element-wise multiplication `⊙' and addition `+' operations. This enables the computation of the forward process as:
	
    [  ẑ_t+1^1:d = a_t ⊙exp(sc_2(z_t))+t_2(z_t); ẑ_t+1^d+1:D = z_t⊙exp(sc_1(a_t))+t_1(a_t) ]

	and the backward processes can be calculated as:
	
    [   z_t=(ẑ_t+1^d+1: D-t_1(ẑ_t+1^1: d)) ⊙exp(-sc_1(ẑ_t+1^1: d)); a_t=(ẑ_t+1^1: d -t_2(ẑ_t+1^d+1: D))⊙exp(-sc_2(ẑ_t+1^d+1: D)) ]

	Note that there exists a dimensional inconsistency between ẑ_t+1∈ℝ^D and z_t∈ℝ^d+1:D. Typically, a feed-forward neural network, represented as f(z_t; θ) with parameters θ, endeavors to fit training data and predict the subsequent latent state ẑ_t+1. The network's transformation can be expressed as ẑ_t+1 = W · z_t + B, where W ∈ℝ^n × n and B ∈ℝ^n × n denote the weights and biases, collectively denoted as θ = {W, B}. However, owing to the typical non-invertibility of W, conducting backward inference (i.e., directly computing z_t = W^-1(ẑ_t+1 - B)) poses challenges. To ensure reversibility, Orthogonal Weight Normalization (OWN) <cit.> is employed, supporting straightforward backward inference by enforcing orthogonality in the weights, unlike the structural design in invertible networks. Consequently, the backward process can be reformulated as:
	
    z_t=W^T(ẑ_t+1-B), W∈𝒪^n× n
	
	where the matrix family 𝒪^n × n = {W ∈ℝ^n × n| WW^T = W^-1W = I} (see <cit.> for more details). By using the combination of RealNVP and OWN, we can effectively achieve bidirectional transformation between z_t and ẑ_t+1.
	
	
	Forward States Prediction with FDM. In the MDP framework, a one-step transition (s_t, a_t, s_t+1) describes the process of transitioning from the current state s_t ∈𝒮 to the next state s_t+1 by taking action a_t ∈𝒜. Considering the crucial aspect of prospective thinking, i.e., the ability to predict future scenarios, is very important for RL tasks. Many studies focus on training dynamic models to predict future states to learn good feature representations <cit.>. In this study, the FDM h(·, ·) is used to simulate the dynamics function 𝒯 in the latent embedding space, allowing us to explore the future from multiple perspectives. As shown in Figure <ref>, the encoder f(·, ·) first converts the image-based observation/state s_0 into a latent representation z_0 = f(s_0). Then, based on the sampled action set Â(z_0) = {â_0, ⋯, â_k} from k perspectives, the FDM predicts the latent state set Ẑ_0:t for t future steps. Formally:
	
    [                                 ẑ_0=z_0=f(s_0);                             ẑ_t=h(â_t-1,ẑ_t-1); Ẑ_0:t = {h(Â_0(z_0), z_0),⋯, h(Â_t(ẑ_t),ẑ_t )} ]

	The optimization objective of the FDM h(·, ·) is to minimize the difference (error) between the predicted latent state sequence ẑ_t and the latent feature sequence z_t directly extracted from the original observation/state, where z_t = f_T(s_t). Following SPR <cit.>, we use “projection” and cosine similarity on DMControl to predict the error, aiming to achieve the following objective:
	
    ℒ_pred=
    		-∑_k=1^K(ẑ_t+k/ẑ_t+k_2)^⊤(z_t+k/z_t+k_2)

	Where z_t is referred to as the target embedding, generated by the stop-gradient version of the online encoder f(·) termed as the target encoder f_T(·).
	
	Backward Return to Previous States with FDM. The backward process of the FDM aims to derive the previous latent states z̆_t based on the given subsequent latent state z̆_t+1 and the causal action â_t. We enforce constraints on the trajectories generated by FDM, ensuring cycle consistency to prevent forward trajectories from encountering irreparable states. Given â_t and z̆_t+1, the reversibility of FDM allows us to easily trace back to the previous latent dynamics z̆_t. Formally, given the hidden state z̆_t+K and a sequence of actions â_t+K-1:t={â_t+K-1,⋯,â_t}, we iteratively compute the previous K state sequence z̆_t+k-1:t={z̆_t+k-1,⋯,z̆_t} using the inverse function h^-1(· ,· ):
	
    z̆_t+k-1={[  h^-1(â_t+k-1, ẑ_t+k),                 if k=K; h^-1(â_t+k-1, z̆_t+k),          if k=K-1,⋯,1. ].

	Using FDM, we can easily generate both a forward and a backward trajectory given the current state and a sequence of actions, forming a cycle forward-inverse trajectory. Ideally, the initial latent state z_t of this cycle trajectory should coincide with the final latent state z̆_t. Therefore, as shown in the right side of Figure <ref>, we introduce cycle consistency constraints between the initial state z_0 and the final state z̆_0 to prevent prospective trajectories from entering irreparable states. In addition, we improve actions in an unsupervised manner by generating or sampling virtual actions to create diverse virtual trajectories, which promotes data efficiency without relying solely on real trajectories for supervised learning. The inclusion of cycle consistency constraints serves a dual purpose, enhancing both the learning of feature representations and the reversibility of states to preclude irreversible events, while also elevating the training efficiency of the weight matrix W in OWN. 
	
	The optimization objective of cycle consistency is to minimize the spatial distance d(z_t, z̆_t) between two latent states z_t and z̆_t. Given the latent state representation z_t at time t, we randomly sample M actions from the action space 𝒜. Therefore, the loss function for cycle consistency is defined as:
	
    ℒ_c=1/M∑^M_m=1d(z_t,z̆_t)

	Guiding Future Scenario Decisions with Model Predictive Control. 
	In RL, policies π are formed by neural networks to approximate the optimal policy π_θ(·|s_t). In classical control tasks, however, π takes the form of trajectory optimization, often seeking a local best solution at each time step <cit.>. This involves predicting the best actions a_t:t+H over a short horizon H and executing the first action a_t, a method known as MPC:
	
    π_θ^ℳ𝒫𝒞=a_t:t+Harg max𝔼[∑^H_j=tγ^jℛ(s_j,a_j)]

	In the SAC framework, the actor selects samples from the distribution of state-action pairs (s_t, a_t) according to the current policy π: a_t ∼π_θ(·, s_t).
	To incorporate prospective thinking into model-free methods, we introduce the concept of MPC. Specifically, a_t is chosen as the first action that maximizes the long-term reward within a finite range H from j ∈ [1, n]:
	
    a_t=a_t:t+Harg max∑^H_j=tγ^jℛ(s_j,a_j)

	Due to its deep understanding of the environment model, MPC autonomously computes rewards from simulated trajectories (i.e., state-action pairs), which is distinct from model-free methods where the reward function ℛ(·,·) is provided by the environment, not computed by itself. Furthermore, it's essential to distinguish this approach from the iterative Q-value estimation in reinforcement learning, which prioritizes short-term gains over long-term cumulative rewards. 
	
	Therefore, we introduce the value consistency to guide the prospective action selection process. The evaluation of actions through prospective thinking should ideally be consistent with the Q-values determined by the RL policy π, ensuring the agent selects actions that provide the maximum long-term benefit in each state. As shown in Figure <ref>, we construct an action set Â(z_0)={â_0,..,â_t} from the current latent state z_0, which includes t potential actions. For each pair of trajectories (ẑ_i,a_i∈Â(s_t)), we use Q(ẑ_i,â_i) as a proxy for the reward function ℛ(·,·), where i ∈ [1,t]. Subsequently, after computing the cumulative discounted rewards CQ_i=∑^t_j=0γ^jQ(ẑ_j,â_j) for each trajectory set, we determine the locally optimal action a^*_0=arg max_a_0{CQ_1,⋯,CQ_k}. Ensuring the consistency of Q-values guarantees that the selected action a_t^* at each time step is the optimal choice according to the current policy, thus fulfilling the function of decision-making and guiding future scenarios.
	
		
		
	
	
	Overall Objective. As shown in Figure <ref>, the overall training objective of our approach can be summarized as follows:
	
    [ ℒ_total(θ)=𝒥_θ_RL  
    			+λ_predℒ_pred(θ)_pred
    			+λ_cℒ_c(θ)_Cycle Consistency ]

	where θ represents all model parameters used to compute the above loss functions, 𝒥_θ is the loss function for RL (see section <ref>), and λ_pred and λc are hyperparameters that weight different losses (more detailed can be seen in the supplementary material).
	
	

§ EXPERIMENTS

	In this section, we will provide a detailed introduction to the evaluation environment used in this paper, including experimental settings, experimental details, and evaluation metrics. Then, we perform a series of ablation experiments to analyze the key components of our method.
	

 §.§ Setup for Evaluation

	Environments. We conducted benchmark tests on ProSpec in environments with limited interaction, using the DeepMind Control Suite (DMControl) for continuous control testing <cit.>. Following the practices of most researchers <cit.>, we evaluated the performance after training for 100k and 500k interaction steps in the DMControl environment, selecting a total of 6 games for performance evaluation.
	
	Baseline. In this work, we consider SPR <cit.> as a strong baseline. We also selected Dreamer <cit.>, SAC+AE <cit.>, SLAC <cit.>, CURL <cit.>, DrQ <cit.>, SPR <cit.>, PlayVirtual <cit.>, and VCR <cit.> as baselines. It is worth noting that since SPR was originally designed for offline tasks, we adapted SPR for continuous tasks using a SAC-based approach (SPR^†).
	
	Implementation Datails. We used the encoder and policy network architecture of CURL <cit.> as the basic network. The contrastive loss component was removed from CURL, and we introduced BYOL <cit.>, creating a baseline solution similar to SPR. To generate imaginary trajectories in this environment, the prediction horizon was set to 6, and action generation was achieved by randomly sampling from a uniform distribution over the continuous action space. Additional implementation details are provided in the Appendix.
	
	
	Evalation Metrics. In DMControl, the maximum achievable score for each environment is capped at 1000 <cit.>. Consistent with previous studies <cit.>, we evaluated model performance across six common environments and used the median score to represent overall performance. To reduce potential variance in benchmarking results, we performed ProSpec evaluations on DMControl using 10 random seeds for each game. For the 100k step, we reran the official code for SPR and PlayVirtual with 10 random seeds and recorded the final results. We did not rerun the code for VCR as we were unable to obtain the official implementation. Due to limited computational resources, we report the 500k results directly from the respective papers. The entire implementation was using PyTorch <cit.>, with network training performed on an NVIDIA RTX 3090 GPU.
	
	

 §.§ Performance Comparison with State-of-the-Arts

	As shown in Table <ref>, our method surpassed the performance of most other methods by achieving five of the top scores in six environments, regardless of the limited 100k or 500k interactions. Furthermore, when considering the median scores, our method achieved the highest rankings in the following two aspects:
	(i) In the DMControl-100k environment with limited interaction data, our method achieves the highest median score of 807.5. This outperforms PlayVirtual by 8.32%, VCR by 3.66%, SPR by 9.64%, DrQ by 17.80%, and CURL by 44.20%.
	(ii) Our method achieves a median score of 959.5 and comes close to the maximum score of 1000 in four environments.
	These results demonstrate the exceptional performance of our method even with limited data. Furthermore, our method shows a significant improvement in performance across different environments.
	
	
	
	

 §.§ Analysis

	In this section, we performed an ablation study in the DMControl-100k to analyze the key components of ProSpec. Due to limited computing resources, we experimented using 5 random seeds.
	
	r4cm
		

		Effectiveness of FDM and Prospective Thinking.
		
		
			1l     Median Score 
 SPR^†              736.5                   

			PlayVirtual              745.5                  

			ProSpec-NP                  752.0           
 ProSpec           807.5          
 
		

	
	Effectiveness of FDM and Prospective Thinking. To explore the effect of prospective thinking on RL tasks, we evaluated ProSpec without prospective thinking (ProSpec-NP) and compared it to SPR^† and PlayVirtual. The results are detailed in Table <ref>. As can be seen: (i) ProSpec improved by 55.5 scores over ProSpec-NP; (ii) ProSpec-NP outperformed PlayVirtual by 6.5 scores; (iii) ProSpec increased PlayVirtual's median score from 752.0 to 807.5. Several factors may explain these results: First, using human-like prospective thinking, ProSpec explores the future from multiple perspectives and selects higher-value, lower-risk decisions, improving its performance by 55.5 scores over ProSpec-NP. Typcally, Prospective thinking resembles residual structures in neural networks, allowing the model to approach local optima with each decision, leading to consistent performance improvements. Second, the flow-based dynamics model provides more accurate backward predictions, while cycle consistency augments state traceability and guides low-risk decisions, improving ProSpec's performance over PlayVirtual. Finally, self-supervised methods generate multiple state-action trajectories, increasing data efficiency and allowing ProSpec to outperform SPR^†.
	
	r4cm
		

		The impact of cycle consistency on ProSpec.
		
		
			1l     Median Score 
 SPR^†              736.5                  

			ProSpec-NC                 752.8                
ProSpec           807.5          
 
		

	
	The Impact of Cycle Consistency on ProSpec. As described in Section 3.2, cycle consistency is used to address two fundamental issues in RL: augmenting state traceability to avoid irretrievable events (low risk), and generating a large number of virtual trajectories to improve data efficiency with augmented actions. We evaluated the performance of ProSpec without cycle consistency constraints (referred to as ProSpec-NC) and compared it to SPR methods. As shown in Table <ref>, regularizing the encoder with ℒ_c in ProSpec-NC resulted in a gain of 2.21%, while regularizing both the encoder and the FDM resulted in a gain of 7.27%. It is clear that cycle consistency can benefit the training of the FDM, but the greater benefit comes from the learning of the feature representations in the encoder <cit.>.
	
	r7cm
		

		Impact of perspective count k.
		
		
			                       k=1     k=3        k=6        k=9  
 Finger, spin           880.0              867.9               871.2               811.7         

			Cartpole, swingup      816.0              831.5          820.0                 800.0    

			Reacher, easy          675.8            762.6               770.3               763.6           

			Cheetah, run           349.8            388.2               384.5             419.7               

			Walker, walk           424.6            561.4               517.6               516.5               

			Ball in cup, catch     845.1            905.5               905.8               934.4             
Median Score           745.9            797.0     795.1    781.8                 
 
		
	
	Impact of Perspective Count k. We explored how the number of perspectives k affects ProSpec in lookahead tasks. Experiments were performed with different perspectives k ∈ [1, 9], and we recorded the median results under different prediction steps t ∈ [1, 15] (additional results are available in the supplementary material). Table <ref> presents these results: (i) at k = 3, ProSpec achieved peak performance, marking an improvement of 6.85% over k = 1 and 1.9% over k = 9; (ii) contrary to expectations, the performance improvement does not scale linearly with the number of perspectives. While k ≥ 3, the performance improvement reaches saturation and shows less differences. This may be because the actions sampled by the agent yield similar long-term rewards as the number of perspectives increases, resulting in insignificant differences in model performance.
	
	Impact of Prediction Steps t on ProSpec. We examined the effect of prediction steps t on ProSpec's performance, as shown in table <ref>. Similar to the previous experiment, we evaluated the median performance across different perspectives k. Our results can be summarized as follows: (i) Except for the finger-spin environments, ProSpec shows significant performance improvements in most environments. This suggests that ProSpec's prospective thinking effectively uses future information to improve the effectiveness of current decisions, and improves model performance across different scenarios. (ii) ProSpec achieves optimal performance at t = 6, showing improvements of 6.25% and 6.18% over t = 1 and t = 15, respectively. (iii) Notably, ProSpec's performance does not exhibit a linear increase with prediction steps t. Instead, beyond a certain threshold of t, performance growth begins to slow or even decline. This phenomenon may be due to the increased complexity of future states that the model attempts to capture with an excessive number of prediction steps, leading to additional prediction errors and compromising prediction accuracy.
	
	
	

§ CONCLUSION AND LIMITATION

	To imbue deep RL with cognitive abilities akin to those of humans—namely, prospective thinking—we propose a novel method named ProSpec.  ProSpec employs a reversible flow-based dynamics model to predict latent future outcomes, drawing on the concept of Model Predictive Control to guide decision-making in anticipation of future scenarios. This approach enables RL to simulate future n-stream trajectories, facilitating high-value, low-risk optimal decision-making. Additionally, ProSpec augments state traceability through cycle consistency, reducing environmental risks and leveraging this to augment actions, thus generating a wealth of virtual trajectories to improve data efficiency. We conducted a large number of experiments at DMControl and the results confirmed the effectiveness of ProSpec.  However, due to the necessity of exploring future scenarios from multiple perspectives, our method incurs additional overhead in training the agent. In addition, ProSpec is currently tailored for value-based RL methods; future research endeavors will aim to discover a universally applicable RL approach that incorporates prospective thinking.
	
	unsrtnat
	
	10
		
		mnih2015human
		Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness,
		Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg
		Ostrovski, et al.
		Human-level control through deep reinforcement learning.
		nature, 518(7540):529–533, 2015.
		
		silver2017mastering
		David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja
		Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton,
		et al.
		Mastering the game of go without human knowledge.
		nature, 550(7676):354–359, 2017.
		
		schrittwieser2020mastering
		Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan,
		Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis,
		Thore Graepel, et al.
		Mastering atari, go, chess and shogi by planning with a learned
		model.
		Nature, 588(7839):604–609, 2020.
		
		zhihe2023dmbp
		YANG Zhihe and Yunjian Xu.
		Dmbp: Diffusion model based predictor for robust offline
		reinforcement learning against state observation perturbations.
		In The Twelfth International Conference on Learning
			Representations, 2023.
		
		hansen2022temporal
		Nicklas Hansen, Xiaolong Wang, and Hao Su.
		Temporal difference learning for model predictive control.
		arXiv preprint arXiv:2203.04955, 2022.
		
		levine2020offline
		Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.
		Offline reinforcement learning: Tutorial, review, and perspectives on
		open problems.
		arXiv preprint arXiv:2005.01643, 2020.
		
		yarats2021improving
		Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob
		Fergus.
		Improving sample efficiency in model-free reinforcement learning from
		images.
		In Proceedings of the AAAI Conference on Artificial
			Intelligence, volume 35, pages 10674–10681, 2021.
		
		bellemare2013arcade
		Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.
		The arcade learning environment: An evaluation platform for general
		agents.
		Journal of Artificial Intelligence Research, 47:253–279, 2013.
		
		badia2020agent57
		Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo
		Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell.
		Agent57: Outperforming the atari human benchmark.
		In International conference on machine learning, pages
		507–517. PMLR, 2020.
		
		yuan2024task
		Huining Yuan, Hongkun Dou, Xingyu Jiang, and Yue Deng.
		Task-aware world model learning with meta weighting via bi-level
		optimization.
		Advances in Neural Information Processing Systems, 36, 2024.
		
		zhu2024repo
		Chuning Zhu, Max Simchowitz, Siri Gadipudi, and Abhishek Gupta.
		Repo: Resilient model-based reinforcement learning by regularizing
		posterior predictability.
		Advances in Neural Information Processing Systems, 36, 2024.
		
		yarats2020image
		Denis Yarats, Ilya Kostrikov, and Rob Fergus.
		Image augmentation is all you need: Regularizing deep reinforcement
		learning from pixels.
		In International conference on learning representations, 2020.
		
		laskin2020reinforcement
		Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind
		Srinivas.
		Reinforcement learning with augmented data.
		Advances in neural information processing systems,
		33:19884–19895, 2020.
		
		yu2021playvirtual
		Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo
		Chen.
		Playvirtual: Augmenting cycle-consistent virtual trajectories for
		reinforcement learning.
		Advances in Neural Information Processing Systems,
		34:5276–5289, 2021.
		
		jaderberg2016reinforcement
		Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z
		Leibo, David Silver, and Koray Kavukcuoglu.
		Reinforcement learning with unsupervised auxiliary tasks.
		arXiv preprint arXiv:1611.05397, 2016.
		
		schwarzer2020data
		Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and
		Philip Bachman.
		Data-efficient reinforcement learning with self-predictive
		representations.
		arXiv preprint arXiv:2007.05929, 2020.
		
		yue2023value
		Yang Yue, Bingyi Kang, Zhongwen Xu, Gao Huang, and Shuicheng Yan.
		Value-consistent representation learning for data-efficient
		reinforcement learning.
		In Proceedings of the AAAI Conference on Artificial
			Intelligence, volume 37, pages 11069–11077, 2023.
		
		janner2019trust
		Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine.
		When to trust your model: Model-based policy optimization.
		Advances in neural information processing systems, 32, 2019.
		
		lowrey2018plan
		Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor
		Mordatch.
		Plan online, learn offline: Efficient learning and exploration via
		model-based control.
		arXiv preprint arXiv:1811.01848, 2018.
		
		schacter2012future
		Daniel L Schacter, Donna Rose Addis, Demis Hassabis, Victoria C Martin,
		R Nathan Spreng, and Karl K Szpunar.
		The future of memory: remembering, imagining, and the brain.
		Neuron, 76(4):677–694, 2012.
		
		aspinwall2005psychology
		Lisa G Aspinwall.
		The psychology of future-oriented thinking: From achievement to
		proactive coping, adaptation, and aging.
		Motivation and emotion, 29:203–235, 2005.
		
		mckiernan2017prospective
		Peter McKiernan.
		Prospective thinking; scenario planning meets neuroscience.
		Technological Forecasting and Social Change, 124:66–76, 2017.
		
		kvavilashvili2020nature
		Lia Kvavilashvili and Jan Rummel.
		On the nature of everyday prospection: A review and theoretical
		integration of research on mind-wandering, future thinking, and prospective
		memory.
		Review of General Psychology, 24(3):210–237, 2020.
		
		schacter2007cognitive
		Daniel L Schacter and Donna Rose Addis.
		The cognitive neuroscience of constructive memory: remembering the
		past and imagining the future.
		Philosophical Transactions of the Royal Society B: Biological
			Sciences, 362(1481):773–786, 2007.
		
		hassabis2007deconstructing
		Demis Hassabis and Eleanor A Maguire.
		Deconstructing episodic memory with construction.
		Trends in cognitive sciences, 11(7):299–306, 2007.
		
		suddendorf2007evolution
		Thomas Suddendorf and Michael C Corballis.
		The evolution of foresight: What is mental time travel, and is it
		unique to humans?
		Behavioral and brain sciences, 30(3):299–313, 2007.
		
		deisenroth2011pilco
		Marc Deisenroth and Carl E Rasmussen.
		Pilco: A model-based and data-efficient approach to policy search.
		In Proceedings of the 28th International Conference on machine
			learning (ICML-11), pages 465–472, 2011.
		
		peng2018deep
		Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong, and Shang-Yu
		Su.
		Deep dyna-q: Integrating planning for task-completion dialogue policy
		learning.
		arXiv preprint arXiv:1801.06176, 2018.
		
		ha2018world
		David Ha and Jürgen Schmidhuber.
		World models.
		arXiv preprint arXiv:1803.10122, 2018.
		
		yu2020mopo
		Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey
		Levine, Chelsea Finn, and Tengyu Ma.
		Mopo: Model-based offline policy optimization.
		Advances in Neural Information Processing Systems,
		33:14129–14142, 2020.
		
		van2019use
		Hado P Van Hasselt, Matteo Hessel, and John Aslanides.
		When to use parametric models in reinforcement learning?
		Advances in Neural Information Processing Systems, 32, 2019.
		
		kielak2019recent
		Kacper Piotr Kielak.
		Do recent advancements in model-based deep reinforcement learning
		really improve data efficiency?
		2019.
		
		laskin2020curl
		Michael Laskin, Aravind Srinivas, and Pieter Abbeel.
		Curl: Contrastive unsupervised representations for reinforcement
		learning.
		In International conference on machine learning, pages
		5639–5650. PMLR, 2020.
		
		guo2020bootstrap
		Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-Bastien Grill,
		Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar.
		Bootstrap latent-predictive representations for multitask
		reinforcement learning.
		In International Conference on Machine Learning, pages
		3875–3886. PMLR, 2020.
		
		lee2020stochastic
		Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine.
		Stochastic latent actor-critic: Deep reinforcement learning with a
		latent variable model.
		Advances in Neural Information Processing Systems, 33:741–752,
		2020.
		
		hafner2020mastering
		Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba.
		Mastering atari with discrete world models.
		arXiv preprint arXiv:2010.02193, 2020.
		
		sutton2018reinforcement
		Richard S Sutton and Andrew G Barto.
		Reinforcement learning: An introduction.
		MIT press, 2018.
		
		dinh2016density
		Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.
		Density estimation using real nvp.
		arXiv preprint arXiv:1605.08803, 2016.
		
		huang2018orthogonal
		Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li.
		Orthogonal weight normalization: Solution to optimization over
		multiple dependent stiefel manifolds in deep neural networks.
		In Proceedings of the AAAI Conference on Artificial
			Intelligence, volume 32, 2018.
		
		tassa2018deepmind
		Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las
		Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al.
		Deepmind control suite.
		arXiv preprint arXiv:1801.00690, 2018.
		
		edwards2018forward
		Ashley D Edwards, Laura Downs, and James C Davidson.
		Forward-backward reinforcement learning.
		arXiv preprint arXiv:1803.10227, 2018.
		
		vinyals2019grandmaster
		Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu,
		Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds,
		Petko Georgiev, et al.
		Grandmaster level in starcraft ii using multi-agent reinforcement
		learning.
		Nature, 575(7782):350–354, 2019.
		
		wang2016dueling
		Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando
		Freitas.
		Dueling network architectures for deep reinforcement learning.
		In International conference on machine learning, pages
		1995–2003. PMLR, 2016.
		
		hafner2019dream
		Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.
		Dream to control: Learning behaviors by latent imagination.
		arXiv preprint arXiv:1912.01603, 2019.
		
		grill2020bootstrap
		Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre
		Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan
		Guo, Mohammad Gheshlaghi Azar, et al.
		Bootstrap your own latent-a new approach to self-supervised learning.
		Advances in neural information processing systems,
		33:21271–21284, 2020.
		
		hafner2019learning
		Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha,
		Honglak Lee, and James Davidson.
		Learning latent dynamics for planning from pixels.
		In International conference on machine learning, pages
		2555–2565. PMLR, 2019.
		
		paszke2019pytorch
		Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
		Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.
		Pytorch: An imperative style, high-performance deep learning library.
		Advances in neural information processing systems, 32, 2019.
		
	
	



