Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To the best of our knowledge, it is the first method that has the ability to model lifelong behaviors in the field of universal user representation learning.

The review comment to be evaluated is:

Strengths:

1. This paper is easy to understand.
2. The problem studied in this paper is important.
3. The proposed multi-scale aggregation method is interesting.

Weaknesses:

1. The contribution of this paper can be somewhat incremental. First, the multi-anchor attention network is also called "multi-aspect attention", which has been widely used in many scenarios [1,2,3]. A very similar contrastive model pertaining approach is also studied in [4]. Thus, the contributions of this paper are somewhat incremental.
2. Discussions and comparisons on universal user modeling methods are insufficient. The authors only include PTUM as the baseline but do not discuss it in Section 2.1. There are also many other methods like [4,5]. They are also not cited nor compared.
3. The design of the item embedding method is problematic. Different from texts that there are rich relations between adjacent words, even adjacent behaviors can be rather random. Thus, it may not be suitable to learn item embeddings by regarding items as words and user behavior sequences as sentences. There are many methods for item embedding learning such as PinSage [6].
4. Compared baselines are also insufficient. As mentioned in the second point, there are many user model pretraining methods that are not compared. In addition, the compared basic user modeling techniques are also rather weak (e.g., CNN, HAN, Doc2Vec and TF-IDF).
5. Some details are unclear. PTUM and PeterRec require fine-tuning the user model in downstream tasks. However, in this work it is not clear whether the pretrained user model is finetuned.
6. The selection of some hyperparameters seems to be arbitrary, e.g., the number of anchors and the window size $\beta$. The authors do not provide experiments to show the influence of several key hyperparameters.
7. The generality of the proposed method is not verified. It is not clear whether the proposed method can be generalized to other types of user models.

Minor:

There are some typos and grammatical issues. For example, "committed to learn" and "Comparisonn".

[1] Event detection using hierarchical multi-aspect attention. WWW 2019.

[2] ARP: Aspect-aware neural review rating prediction. CIKM 2019.

[3] Hybrid graph convolutional networks with multi-head attention for location recommendation. World Wide Web 23.6 (2020): 3125-3151.

[4] Contrastive Pre-training for Sequential Recommendation. arXiv e-prints (2020): arXiv-2010.

[5] Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. SIGIR. 2020.

[6] Graph convolutional neural networks for web-scale recommender systems. KDD 2018.

The evidence to be evaluated is:Ren _et al._ proposed a hierarchical periodic memory network for lifelong sequential modeling. They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi _et al._ decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency. Specifically, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally. Pi _et al._ also designed a search-based interest model (SIM) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item.

In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length. For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN).

In order to model extremely long lifelong user behavior sequence, we propose to aggregate the content of items under user purchases, clicks, or other behaviors at a certain granularity.

As mentioned above, a BoI feature \(\mathbf{b_{t}}\) for each user is obtained, through the BoI model given behavior data in any time period \(\mathbf{t}\). Most directly, a user representation with super-high dimension \(\mathbf{b_{T}}\) can be obtained by applying the BoI model to the whole lifelong time \(\mathbf{T}\).

Therefore, we propose to get a BoI feature sequence by applying the BoI model at each time period of the whole lifelong time. Then, a Self-supervised Multi-anchor Encoder Network (SMEN) is designed to learn compressed user representations from the sequence of BoI features. The lifelong time \(\mathbf{T}\) is divided into \(N\) parts at a fixed time interval, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}\) and \(\mathbf{t_{i}}\) denotes the \(i\)-th time period.

In this way, a sequence of BoI features \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) can be obtained, where \(\mathbf{b_{t_{i}}}\) denotes the BoI feature corresponding to the \(i\)-th time period \(\mathbf{t_{i}}\). After that, SMEN is used to map \(\mathbf{B}\) to low-dimensional user representations. As shown in Fig. 1 (b), SMEN consists of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are \(M\) anchors, and each of them indicates a certain preference of users. Let \(\mathbf{b}\) be a BoI feature, the module converts \(\mathbf{b}\) to \(M\) low-dimensional representations as shown in Fig. 2.

Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module. Through the multi-anchor module, a sequence of representation groups \(\mathbf{\mathcal{R}}=\{\mathbf{R_{t_{1}}},\mathbf{R_{t_{2}}},...,\mathbf{R_{t_{N}}}\}\) for each user is obtained, where \(\mathbf{R_{t_{i}}}=\{\mathbf{r_{t_{i}}^{1}},\mathbf{r_{t_{i}}^{2}},...,\mathbf{r_{t_{i}}^{M}}\}\) is a representation group generated by multi-anchor module corresponding to the BoI feature \(\mathbf{b_{t_{i}}}\) in time period \(\mathbf{t_{i}}\). In the time aggregation module, each sequence \(\mathbf{R^{i}}=\{\mathbf{r_{t_{i}}^{i}},\mathbf{r_{t_{i}}^{i}},...,\mathbf{r_{t_{N}}^{i}}\}\) is aggregated separately to yield a new representation \(\mathbf{\tilde{r}^{i}}\in\mathbb{R}^{H}\). Thus the variable-size representation sequence \(\mathbf{\mathcal{R}}\) can be transformed into \(M\) fixed-size representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\).

Compared to average and max pooling, RNNs are more appropriate to capture variations over time.

To address this trade-off, multi-scale aggregation module is designed. The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities. Generally, user behavior in this work is aggregated monthly and yearly, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}=\big{\{}\mathbf{t_{1}^{\prime}},\bm {t_{2}^{\prime}},...,\mathbf{t_{N^{\prime}}^{\prime}}\big{\}}\) (e.g., if the length of time period is 5 years, \(\mathbf{N}\) is 60 for monthly granularity, and \(\mathbf{N^{\prime}}\) is 5 for yearly granularity). Thus two sequences of BoI feature \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) and \(\mathbf{B^{\prime}}=\big{\{}\mathbf{b_{t_{1}^{\prime}}},\mathbf{b_{t_{2}^{\prime}}},...,\bm {b_{t_{N^{\prime}}^{\prime}}}\big{\}}\) at different scales correspondingly are obtained. Then, two groups of representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) and \(\mathbf{\tilde{R^{\prime}}}=\big{\{}\mathbf{\tilde{r^{\prime}}^{1}},\mathbf{\tilde{r^{ \prime}}^{2}},...,\mathbf{\tilde{r^{\prime}}^{M}}\big{\}}\) are obtained after the multi-anchor module and time aggregation module with parameter sharing.

From Table 2 and Table 3, we have observed that LURM has achieved better performance compared to other self-supervised and supervised methods. Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.

It can be seen that the richer the behavior, the better the performance on user profiling prediction tasks. Compared to using only behaviors of two months, an average improvement of 9.75%/16.03% can be achieved when using behaviors of 5 years on these tasks.

This is because these tasks are more focused on the short-term interests of users, and it can be observed that the best performance can be obtained when input with a length of one year is used on these tasks.

In this work, a novel framework named LURM is proposed to model lifelong user behaviors with any length. With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets. \begin{table}
\begin{tabular}{c c c c c} \hline Method & Books of Literature & Games of Sports & Outdoor-hunting & Musical Instrument \\ \hline TextCNN & 73.847/2.83 & 61.00/69.92 & 66.06/72.76 & 68.28/71.81 \\ HAN & 79.12/77.49 & 66.62/70.01 & 71.19/73.42 & 70.01/72.12 \\ \hline TF-IDF & 78.62/77.21 & 65.69/68.69 & 71.90/72.19 & 69.16/68.62 \\ Doc2Vec & 71.21/68.26 & 66.75/66.85 & 71.58/66.36 & 70.29/67.37 \\ \hline PTUM & 78.66/77.91 & 66.06/68.97 & 70.34/73.01 & 69.50/72.92 \\ SUMN & 79.57/77.73 & 66.83/70.63 & 72.27/73.44 & 70.61/73.43 \\ \hline BoI & 76.29/75.75 & 59.34/67.58 & 67.80/71.28 & 66.36/69.88 \\ LURM & **82.55/79.13** & **68.02/72.43** & **77.59/75.19** & **74.04/74.43** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

 \begin{table}
\begin{tabular}{c c c c c c} \hline \hline Method & Age & Baby Age & Clothing & Shoe & Coffee \\ \hline TextCNN & 86.40/61.02 & 72.19/66.01 & 76.64/76.13 & 84.33/80.24 & 80.49/79.46 \\ TF-IDF & 87.73/61.75 & 73.08/67.26 & 77.61/78.01 & 85.04/81.15 & 81.75/80.78 \\ SUMN & 85.35/60.61 & 74.20/67.63 & 73.86/74.16 & 83.47/79.63 & 79.89/79.42 \\ \hline BoI(short) & 89.57/63.99 & 82.23/70.22 & 78.93/78.85 & 85.53/81.68 & 82.61/81.41 \\ LURM(short) & 88.48/61.99 & 82.71/69.29 & 78.37/78.57 & 85.15/81.18 & 81.94/80.78 \\ BoI & **96.48**/**80.19** & 93.29/83.57 & **80.83**/**79.33** & **86.69**/**82.16** & **83.78**/**81.74** \\ LURM & 96.09/78.43 & **94.59**/**84.91** & 80.37/79.10 & 86.30/81.65 & 83.11/81.18 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input: Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

Input: Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "To the best of our knowledge, it is the first method that has the ability to model lifelong behaviors in the field of universal user representation learning."

The claim appears to be overstated based on the evidence and reviewer comments. The reviewer points out several existing works that tackle similar problems, including other universal user modeling methods mentioned in points 1 and 2 of the weaknesses. Specifically, they note that "multi-anchor attention" has been widely used in many scenarios and similar contrastive model approaches exist, citing multiple papers. While the method may have novel elements in combining different techniques for lifelong behavior modeling, claiming it as the "first" method appears to overstate its novelty given the existing literature cited by the reviewer.

<score>4</score>

This score reflects that while the method may be innovative, the claim of being "first" is not well-supported given the evidence of prior related work and similar approaches. The wording should be modified to better reflect its incremental contributions and acknowledge existing work in the field.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To the best of our knowledge, it is the first method that has the ability to model lifelong behaviors in the field of universal user representation learning.

The review comment to be evaluated is:

[Advantages]
- Modeling users and user behaviors are essential, especially for web-based applications. The problem dealt with in this paper will be a promising application of representation learning based on neural networks.
- The structure of the proposed method is reasonable, especially for e-commerce applications.

[Drawbacks]
- I could not understand the novelty of the proposed method. The bag-of-features representation seems to be standard in several research fields such as computer vision, natural language processing, and data mining. The "multi-anchor module" seems to be almost the same as the multi-head attention module that is famous in the transformer. The time aggregation module does not contain a special trick. According to the introduction of this paper, the novelty of the proposed method is the ability for modeling life-long behaviors. However, I could not find any special techniques for handling life-long sequences. Please clearly state the technical novelty of the proposed method.
- The proposed method for modeling user behaviors dismisses both microscopic and macroscopic time-series characteristics, which would be significant for tracing user interest. More specifically, bag-of-interest representations remove microscopic characteristics, and the time aggregation module erases macroscopic features. If the authors believe that I have some misunderstanding, please justify it.
- When obtaining stable user representations, the knowledge of life-long learning will be required, since (1) users might have several completely different interests in their minds, (2) each interest lasts only a short term, especially in e-commerce applications, and (3) interests containing old behavior sequences might come back suddenly. This implies that almost all the information containing behavior sequences should be kept in user representations and this problem is exactly in the context of life-long learning. Please perform bibliographic surveys for life-long learning, describe the critical problem for the previous work, and present the novelty and advantage(s)  of the proposed method against the previous work.
- The downstream tasks in the experimental evaluations seem to be too easy and not suitable for evaluating the ability to handle long behavior sequences.

The evidence to be evaluated is:Ren _et al._ proposed a hierarchical periodic memory network for lifelong sequential modeling. They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi _et al._ decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency. Specifically, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally. Pi _et al._ also designed a search-based interest model (SIM) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item.

In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length. For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN).

In order to model extremely long lifelong user behavior sequence, we propose to aggregate the content of items under user purchases, clicks, or other behaviors at a certain granularity.

As mentioned above, a BoI feature \(\mathbf{b_{t}}\) for each user is obtained, through the BoI model given behavior data in any time period \(\mathbf{t}\). Most directly, a user representation with super-high dimension \(\mathbf{b_{T}}\) can be obtained by applying the BoI model to the whole lifelong time \(\mathbf{T}\).

Therefore, we propose to get a BoI feature sequence by applying the BoI model at each time period of the whole lifelong time. Then, a Self-supervised Multi-anchor Encoder Network (SMEN) is designed to learn compressed user representations from the sequence of BoI features. The lifelong time \(\mathbf{T}\) is divided into \(N\) parts at a fixed time interval, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}\) and \(\mathbf{t_{i}}\) denotes the \(i\)-th time period.

In this way, a sequence of BoI features \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) can be obtained, where \(\mathbf{b_{t_{i}}}\) denotes the BoI feature corresponding to the \(i\)-th time period \(\mathbf{t_{i}}\). After that, SMEN is used to map \(\mathbf{B}\) to low-dimensional user representations. As shown in Fig. 1 (b), SMEN consists of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are \(M\) anchors, and each of them indicates a certain preference of users. Let \(\mathbf{b}\) be a BoI feature, the module converts \(\mathbf{b}\) to \(M\) low-dimensional representations as shown in Fig. 2.

Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module. Through the multi-anchor module, a sequence of representation groups \(\mathbf{\mathcal{R}}=\{\mathbf{R_{t_{1}}},\mathbf{R_{t_{2}}},...,\mathbf{R_{t_{N}}}\}\) for each user is obtained, where \(\mathbf{R_{t_{i}}}=\{\mathbf{r_{t_{i}}^{1}},\mathbf{r_{t_{i}}^{2}},...,\mathbf{r_{t_{i}}^{M}}\}\) is a representation group generated by multi-anchor module corresponding to the BoI feature \(\mathbf{b_{t_{i}}}\) in time period \(\mathbf{t_{i}}\). In the time aggregation module, each sequence \(\mathbf{R^{i}}=\{\mathbf{r_{t_{i}}^{i}},\mathbf{r_{t_{i}}^{i}},...,\mathbf{r_{t_{N}}^{i}}\}\) is aggregated separately to yield a new representation \(\mathbf{\tilde{r}^{i}}\in\mathbb{R}^{H}\). Thus the variable-size representation sequence \(\mathbf{\mathcal{R}}\) can be transformed into \(M\) fixed-size representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\).

Compared to average and max pooling, RNNs are more appropriate to capture variations over time.

To address this trade-off, multi-scale aggregation module is designed. The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities. Generally, user behavior in this work is aggregated monthly and yearly, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}=\big{\{}\mathbf{t_{1}^{\prime}},\bm {t_{2}^{\prime}},...,\mathbf{t_{N^{\prime}}^{\prime}}\big{\}}\) (e.g., if the length of time period is 5 years, \(\mathbf{N}\) is 60 for monthly granularity, and \(\mathbf{N^{\prime}}\) is 5 for yearly granularity). Thus two sequences of BoI feature \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) and \(\mathbf{B^{\prime}}=\big{\{}\mathbf{b_{t_{1}^{\prime}}},\mathbf{b_{t_{2}^{\prime}}},...,\bm {b_{t_{N^{\prime}}^{\prime}}}\big{\}}\) at different scales correspondingly are obtained. Then, two groups of representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) and \(\mathbf{\tilde{R^{\prime}}}=\big{\{}\mathbf{\tilde{r^{\prime}}^{1}},\mathbf{\tilde{r^{ \prime}}^{2}},...,\mathbf{\tilde{r^{\prime}}^{M}}\big{\}}\) are obtained after the multi-anchor module and time aggregation module with parameter sharing.

From Table 2 and Table 3, we have observed that LURM has achieved better performance compared to other self-supervised and supervised methods. Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.

It can be seen that the richer the behavior, the better the performance on user profiling prediction tasks. Compared to using only behaviors of two months, an average improvement of 9.75%/16.03% can be achieved when using behaviors of 5 years on these tasks.

This is because these tasks are more focused on the short-term interests of users, and it can be observed that the best performance can be obtained when input with a length of one year is used on these tasks.

In this work, a novel framework named LURM is proposed to model lifelong user behaviors with any length. With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets. \begin{table}
\begin{tabular}{c c c c c} \hline Method & Books of Literature & Games of Sports & Outdoor-hunting & Musical Instrument \\ \hline TextCNN & 73.847/2.83 & 61.00/69.92 & 66.06/72.76 & 68.28/71.81 \\ HAN & 79.12/77.49 & 66.62/70.01 & 71.19/73.42 & 70.01/72.12 \\ \hline TF-IDF & 78.62/77.21 & 65.69/68.69 & 71.90/72.19 & 69.16/68.62 \\ Doc2Vec & 71.21/68.26 & 66.75/66.85 & 71.58/66.36 & 70.29/67.37 \\ \hline PTUM & 78.66/77.91 & 66.06/68.97 & 70.34/73.01 & 69.50/72.92 \\ SUMN & 79.57/77.73 & 66.83/70.63 & 72.27/73.44 & 70.61/73.43 \\ \hline BoI & 76.29/75.75 & 59.34/67.58 & 67.80/71.28 & 66.36/69.88 \\ LURM & **82.55/79.13** & **68.02/72.43** & **77.59/75.19** & **74.04/74.43** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

 \begin{table}
\begin{tabular}{c c c c c c} \hline \hline Method & Age & Baby Age & Clothing & Shoe & Coffee \\ \hline TextCNN & 86.40/61.02 & 72.19/66.01 & 76.64/76.13 & 84.33/80.24 & 80.49/79.46 \\ TF-IDF & 87.73/61.75 & 73.08/67.26 & 77.61/78.01 & 85.04/81.15 & 81.75/80.78 \\ SUMN & 85.35/60.61 & 74.20/67.63 & 73.86/74.16 & 83.47/79.63 & 79.89/79.42 \\ \hline BoI(short) & 89.57/63.99 & 82.23/70.22 & 78.93/78.85 & 85.53/81.68 & 82.61/81.41 \\ LURM(short) & 88.48/61.99 & 82.71/69.29 & 78.37/78.57 & 85.15/81.18 & 81.94/80.78 \\ BoI & **96.48**/**80.19** & 93.29/83.57 & **80.83**/**79.33** & **86.69**/**82.16** & **83.78**/**81.74** \\ LURM & 96.09/78.43 & **94.59**/**84.91** & 80.37/79.10 & 86.30/81.65 & 83.11/81.18 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input: Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

Input: Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim: "To the best of our knowledge, it is the first method that has the ability to model lifelong behaviors in the field of universal user representation learning."

Based on the review comments and evidence:
1. The reviewer points out that no special techniques for handling life-long sequences were found and questions the technical novelty.
2. The evidence shows prior work by Ren et al. and Pi et al. that already addressed lifelong sequential modeling and long-term interest capturing.
3. The method uses fairly standard techniques (bag-of-features, multi-head attention-like mechanisms) without clearly demonstrating revolutionary advances in lifelong behavior modeling.
4. While the method shows good performance, the experimental results don't specifically validate the claim of being "first" or demonstrate unique lifelong modeling capabilities.

The claim appears significantly overstated given the existence of prior work and lack of clearly novel lifelong learning techniques. A more measured claim about the method's effectiveness would have been more appropriate.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We introduce a sub-model named BoI which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors.

The evidence to be evaluated is:Previous works have shown that considering long-term historical behavior sequences for user modeling can significantly improve the performance of different tasks

Ren _et al._ proposed a hierarchical periodic memory network for lifelong sequential modeling. They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi _et al._ decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency. Specifically, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally. Pi _et al._ also designed a search-based interest model (SIM) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item.

In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length. For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN).

In order to model extremely long lifelong user behavior sequence, we propose to aggregate the content of items under user purchases, clicks, or other behaviors at a certain granularity.

Therefore, we propose a model called Bag of Interests (BoI) to aggregate user behavior data at 'interest' granularity.

As shown in Fig. 1 (a), BoI consists of an item embedding module and a large-scale clustering module.

Like the BoVW model, an 'interest' vocabulary are supposed to be built in our BoI model. Therefore, the embedding of each item is required, so that similar items with close distance in the embedding space can be clustered together.

Inspired by these works, we design a contrastive learning task based on the relation between items drawn from a user to learn item embedding similar to item2vec Barkan & Koenigstein (2016).

Considering the large-scale item set, a subset \(\mathbf{E^{\prime}}\subset\mathbf{E}\) at the million level is sampled, and an efficient clustering algorithm named HDSC Yi et al. (2014) on \(\mathbf{E^{\prime}}\) is employed to cluster similar items into the same 'interest'. After clustering, the cluster centers \(\mathbf{C}\) make up an 'interest' vocabulary. Therefore each item can be attributed to one/multiple 'interest(s)' by hard/soft cluster assignment. Take hard cluster assignment as an example, each user can obtain his/her sparsely high-dimensional BoI feature \(\mathbf{b_{t}}\in\mathbb{R}^{D}\) in time period \(\mathbf{t}\) according to his/her behavior sequence \(\mathbf{S_{t}}=\big{\{}\mathbf{x_{1}},\mathbf{x_{2}},...,\mathbf{x_{|\mathbf{S_{t}}|}}\big{\}}\) : \[\mathbf{b_{t}}=[log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{1}}),log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{2}}), \tag{4}\] \[...,log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c _{D}})]\] where \(\mathbf{x_{i}}\in c_{j}\) means item \(i\) is assigned to the cluster \(c_{j}\in\mathbf{C}\), \(\mathbbm{1}\) is an indicator function. As mentioned above, a BoI feature \(\mathbf{b_{t}}\) for each user is obtained, through the BoI model given behavior data in any time period \(\mathbf{t}\).

Therefore, we propose to get a BoI feature sequence by applying the BoI model at each time period of the whole lifelong time.

In this way, a sequence of BoI features \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) can be obtained, where \(\mathbf{b_{t_{i}}}\) denotes the BoI feature corresponding to the \(i\)-th time period \(\mathbf{t_{i}}\).

To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are \(M\) anchors, and each of them indicates a certain preference of users. Let \(\mathbf{b}\) be a BoI feature, the module converts \(\mathbf{b}\) to \(M\) low-dimensional representations as shown in Fig. 2.

where \(f(b_{j})=b_{j}\mathbf{W_{j}^{e}}\) is the 'interest' embedding function, \(b_{j}\) is the \(j\)-th element of \(\mathbf{b}\), \(\mathbf{W^{e}}=\left(\mathbf{W_{1}^{e}},\mathbf{W_{2}^{e}},...,\mathbf{W_{D}^{e}}\right)^{T}\) is the embedding matrix. And \(\alpha_{ij}\) is the attention weight between the \(i\)-th anchor and the \(j\)-th 'interest', which measures the portion assigned to the \(i\)-th preference from the \(j\)-th behavior 'interest'. The weight \(\alpha_{ij}\) is defined as \[\alpha_{ij}=\frac{\exp(\mathbf{W_{i}^{a}}\mathbf{k_{j}})}{\sum_{l}\exp(\mathbf{W_{l}^{a}} \mathbf{k_{j}})}, \tag{6}\]

In this way, we can learn different aspects of user preferences. Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module. Through the multi-anchor module, a sequence of representation groups \(\mathbf{\mathcal{R}}=\{\mathbf{R_{t_{1}}},\mathbf{R_{t_{2}}},...,\mathbf{R_{t_{N}}}\}\) for each user is obtained, where \(\mathbf{R_{t_{i}}}=\{\mathbf{r_{t_{i}}^{1}},\mathbf{r_{t_{i}}^{2}},...,\mathbf{r_{t_{i}}^{M}}\}\) is a representation group generated by multi-anchor module corresponding to the BoI feature \(\mathbf{b_{t_{i}}}\) in time period \(\mathbf{t_{i}}\). In the time aggregation module, each sequence \(\mathbf{R^{i}}=\{\mathbf{r_{t_{i}}^{i}},\mathbf{r_{t_{i}}^{i}},...,\mathbf{r_{t_{N}}^{i}}\}\) is aggregated separately to yield a new representation \(\mathbf{\tilde{r}^{i}}\in\mathbb{R}^{H}\). Thus the variable-size representation sequence \(\mathbf{\mathcal{R}}\) can be transformed into \(M\) fixed-size representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\).

Under a time division \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}\), we have obtained \(M\) representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) through the BoI, multi-anchor module and time aggregation module as introduced above.

The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities. Generally, user behavior in this work is aggregated monthly and yearly, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}=\big{\{}\mathbf{t_{1}^{\prime}},\bm {t_{2}^{\prime}},...,\mathbf{t_{N^{\prime}}^{\prime}}\big{\}}\) (e.g., if the length of time period is 5 years, \(\mathbf{N}\) is 60 for monthly granularity, and \(\mathbf{N^{\prime}}\) is 5 for yearly granularity). Thus two sequences of BoI feature \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) and \(\mathbf{B^{\prime}}=\big{\{}\mathbf{b_{t_{1}^{\prime}}},\mathbf{b_{t_{2}^{\prime}}},...,\bm {b_{t_{N^{\prime}}^{\prime}}}\big{\}}\) at different scales correspondingly are obtained. Then, two groups of representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) and \(\mathbf{\tilde{R^{\prime}}}=\big{\{}\mathbf{\tilde{r^{\prime}}^{1}},\mathbf{\tilde{r^{ \prime}}^{2}},...,\mathbf{\tilde{r^{\prime}}^{M}}\big{\}}\) are obtained after the multi-anchor module and time aggregation module with parameter sharing.

\[\mathbf{\hat{r}^{i}}=s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}])\odot \mathbf{\tilde{r}^{i}}+(1-s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}]) )\odot\mathbf{\tilde{r^{\prime}}^{i}}, \tag{7}\] where \(\odot\) is the Hadamard product, \([\cdot]\) is a operation concatenating vectors along the last dimension, and \(s^{i}(\cdot)\) is the switch function which is implemented as a fully-connected layer followed by a sigmoid function. The switch function controls the fusion between BoI feature sequences of different scales. The output of this module \(\mathbf{\hat{R}}=[\mathbf{\hat{r}^{1}},\mathbf{\hat{r}^{2}},...,\mathbf{\hat{r}^{M}}]\) is served as the final user representation.

We constructed a training dataset \(D_{pt}^{I}\) for self-supervised user modeling by randomly sampling hundred million of users who have behaviors between 2016-06 and 2021-05, while taking behaviors occurred between 2021-07 and 2021-08 to form datasets \(\{D_{ds}^{I}(t)\}_{t}\).

The first one uses the high-dimensional output of BoI module as input in the downstream tasks, while the second uses the output of LURM.

In particular, it can be observed that LURM has a significant improvement over BoI.

Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.

Compared to using only behaviors of two months, an average improvement of 9.75%/16.03% can be achieved when using behaviors of 5 years on these tasks.

Taking BoI as a benchmark, it can be seen that LURM achieves almost lossless dimensionality reduction compared to LURM(1anchor-128) and LURM(1anchor-1280), which verifies the effectiveness of the multi-anchor module.

And It can be seen that the representation was improved through the further learning of SMEN with self-supervised tasks and the ability to encode temporal variation. In this work, a novel framework named LURM is proposed to model lifelong user behaviors with any length. With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets.

Input: Figure 1: Illustration of our Lifelong User Representation Model (LURM) for user understanding. LURM consists of two sub-model: (a) Bag of Interests (BoI) is used to aggregate user behavior data at ‘interest’ granularity, and is composed of an item embedding module and a large scale clustering module. We apply the BoI model on lifelong behavior data to generate multi-scale BoI feature sequences, (b) Self-supervised Multi-anchor Encoder Network (SMEN) is used to learn compressed user representations from BoI features sequences, and is composed of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the claim appears to be well-stated (score of 1). The evidence thoroughly demonstrates that BoI can indeed encode behaviors in any time period through sparse vectors. The evidence shows:

1. Clear technical details of how BoI works, including item embedding, clustering, and feature calculation
2. Mathematical formulas demonstrating how behavior sequences are converted to sparse vectors
3. Implementation details showing it works across different time periods (monthly/yearly granularity)
4. Experimental validation using 5 years of behavior data
5. Architecture diagrams illustrating the model components

The claim avoids exaggeration and accurately reflects what the model accomplishes according to the technical evidence provided.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We introduce a sub-model named BoI which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors.

The review comment to be evaluated is:

Strengths:

--Previous work cannot model universal user representation based on lifelong sequences of user behavior since registration and challenging to make full use of highly long user behavior sequences accumulated in Internet service platforms.

-- This work proposes a universal Lifelong User Representation Model to encode user behaviors in any period with a sparse vector with super-high dimension and map these user features to user representations with almost lossless dimensionality reduction. In this way, the model could easily learn different aspects of user preferences in extremely long sequences and generalize to other downstream tasks.

Weaknesses:

-- The structure of related works is not clear enough. At the end of universal user modeling, this work focuses on the conclusion that previous work dedicated to learning universal user representations cannot leverage information from lifelong user behaviors instead of their generalization ability. The same problem also appeared at the end of lifelong user modeling.

-- In a multi-scale aggregation module, this work captures diverse patterns from user behavior by aggregating several representations generated at different time granularities. However, this method could only use information intra-corresponding representation of every time granularity and cannot explore the information inter different time granularities.

-- The explanation of the considerable gap between BoI and LURM on the two datasets is unconvincing. On the industrial dataset, the performance of BOI is better than LURM, which proved that the design of SMEN cannot bring benefits for BOI.

-- In different downstream tasks, the benefits on category preference identification tasks are tiny, even using behaviors of 5 years on this task. There is no explanation about this phenomenon, and it proves BOI cannot learn much preference information about user behaviors on different categories.

-- This work did not design experiments about multi-scale aggregation modules to prove the effectiveness of process representations of different time granularities.

-- There are some format problems in the paper, such as no period at the end of the first paragraph of the introduction.

The evidence to be evaluated is:Previous works have shown that considering long-term historical behavior sequences for user modeling can significantly improve the performance of different tasks

Ren _et al._ proposed a hierarchical periodic memory network for lifelong sequential modeling. They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi _et al._ decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency. Specifically, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally. Pi _et al._ also designed a search-based interest model (SIM) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item.

In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length. For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN).

In order to model extremely long lifelong user behavior sequence, we propose to aggregate the content of items under user purchases, clicks, or other behaviors at a certain granularity.

Therefore, we propose a model called Bag of Interests (BoI) to aggregate user behavior data at 'interest' granularity.

As shown in Fig. 1 (a), BoI consists of an item embedding module and a large-scale clustering module.

Like the BoVW model, an 'interest' vocabulary are supposed to be built in our BoI model. Therefore, the embedding of each item is required, so that similar items with close distance in the embedding space can be clustered together.

Inspired by these works, we design a contrastive learning task based on the relation between items drawn from a user to learn item embedding similar to item2vec Barkan & Koenigstein (2016).

Considering the large-scale item set, a subset \(\mathbf{E^{\prime}}\subset\mathbf{E}\) at the million level is sampled, and an efficient clustering algorithm named HDSC Yi et al. (2014) on \(\mathbf{E^{\prime}}\) is employed to cluster similar items into the same 'interest'. After clustering, the cluster centers \(\mathbf{C}\) make up an 'interest' vocabulary. Therefore each item can be attributed to one/multiple 'interest(s)' by hard/soft cluster assignment. Take hard cluster assignment as an example, each user can obtain his/her sparsely high-dimensional BoI feature \(\mathbf{b_{t}}\in\mathbb{R}^{D}\) in time period \(\mathbf{t}\) according to his/her behavior sequence \(\mathbf{S_{t}}=\big{\{}\mathbf{x_{1}},\mathbf{x_{2}},...,\mathbf{x_{|\mathbf{S_{t}}|}}\big{\}}\) : \[\mathbf{b_{t}}=[log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{1}}),log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{2}}), \tag{4}\] \[...,log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c _{D}})]\] where \(\mathbf{x_{i}}\in c_{j}\) means item \(i\) is assigned to the cluster \(c_{j}\in\mathbf{C}\), \(\mathbbm{1}\) is an indicator function. As mentioned above, a BoI feature \(\mathbf{b_{t}}\) for each user is obtained, through the BoI model given behavior data in any time period \(\mathbf{t}\).

Therefore, we propose to get a BoI feature sequence by applying the BoI model at each time period of the whole lifelong time.

In this way, a sequence of BoI features \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) can be obtained, where \(\mathbf{b_{t_{i}}}\) denotes the BoI feature corresponding to the \(i\)-th time period \(\mathbf{t_{i}}\).

To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are \(M\) anchors, and each of them indicates a certain preference of users. Let \(\mathbf{b}\) be a BoI feature, the module converts \(\mathbf{b}\) to \(M\) low-dimensional representations as shown in Fig. 2.

where \(f(b_{j})=b_{j}\mathbf{W_{j}^{e}}\) is the 'interest' embedding function, \(b_{j}\) is the \(j\)-th element of \(\mathbf{b}\), \(\mathbf{W^{e}}=\left(\mathbf{W_{1}^{e}},\mathbf{W_{2}^{e}},...,\mathbf{W_{D}^{e}}\right)^{T}\) is the embedding matrix. And \(\alpha_{ij}\) is the attention weight between the \(i\)-th anchor and the \(j\)-th 'interest', which measures the portion assigned to the \(i\)-th preference from the \(j\)-th behavior 'interest'. The weight \(\alpha_{ij}\) is defined as \[\alpha_{ij}=\frac{\exp(\mathbf{W_{i}^{a}}\mathbf{k_{j}})}{\sum_{l}\exp(\mathbf{W_{l}^{a}} \mathbf{k_{j}})}, \tag{6}\]

In this way, we can learn different aspects of user preferences. Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module. Through the multi-anchor module, a sequence of representation groups \(\mathbf{\mathcal{R}}=\{\mathbf{R_{t_{1}}},\mathbf{R_{t_{2}}},...,\mathbf{R_{t_{N}}}\}\) for each user is obtained, where \(\mathbf{R_{t_{i}}}=\{\mathbf{r_{t_{i}}^{1}},\mathbf{r_{t_{i}}^{2}},...,\mathbf{r_{t_{i}}^{M}}\}\) is a representation group generated by multi-anchor module corresponding to the BoI feature \(\mathbf{b_{t_{i}}}\) in time period \(\mathbf{t_{i}}\). In the time aggregation module, each sequence \(\mathbf{R^{i}}=\{\mathbf{r_{t_{i}}^{i}},\mathbf{r_{t_{i}}^{i}},...,\mathbf{r_{t_{N}}^{i}}\}\) is aggregated separately to yield a new representation \(\mathbf{\tilde{r}^{i}}\in\mathbb{R}^{H}\). Thus the variable-size representation sequence \(\mathbf{\mathcal{R}}\) can be transformed into \(M\) fixed-size representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\).

Under a time division \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}\), we have obtained \(M\) representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) through the BoI, multi-anchor module and time aggregation module as introduced above.

The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities. Generally, user behavior in this work is aggregated monthly and yearly, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}=\big{\{}\mathbf{t_{1}^{\prime}},\bm {t_{2}^{\prime}},...,\mathbf{t_{N^{\prime}}^{\prime}}\big{\}}\) (e.g., if the length of time period is 5 years, \(\mathbf{N}\) is 60 for monthly granularity, and \(\mathbf{N^{\prime}}\) is 5 for yearly granularity). Thus two sequences of BoI feature \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) and \(\mathbf{B^{\prime}}=\big{\{}\mathbf{b_{t_{1}^{\prime}}},\mathbf{b_{t_{2}^{\prime}}},...,\bm {b_{t_{N^{\prime}}^{\prime}}}\big{\}}\) at different scales correspondingly are obtained. Then, two groups of representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) and \(\mathbf{\tilde{R^{\prime}}}=\big{\{}\mathbf{\tilde{r^{\prime}}^{1}},\mathbf{\tilde{r^{ \prime}}^{2}},...,\mathbf{\tilde{r^{\prime}}^{M}}\big{\}}\) are obtained after the multi-anchor module and time aggregation module with parameter sharing.

\[\mathbf{\hat{r}^{i}}=s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}])\odot \mathbf{\tilde{r}^{i}}+(1-s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}]) )\odot\mathbf{\tilde{r^{\prime}}^{i}}, \tag{7}\] where \(\odot\) is the Hadamard product, \([\cdot]\) is a operation concatenating vectors along the last dimension, and \(s^{i}(\cdot)\) is the switch function which is implemented as a fully-connected layer followed by a sigmoid function. The switch function controls the fusion between BoI feature sequences of different scales. The output of this module \(\mathbf{\hat{R}}=[\mathbf{\hat{r}^{1}},\mathbf{\hat{r}^{2}},...,\mathbf{\hat{r}^{M}}]\) is served as the final user representation.

We constructed a training dataset \(D_{pt}^{I}\) for self-supervised user modeling by randomly sampling hundred million of users who have behaviors between 2016-06 and 2021-05, while taking behaviors occurred between 2021-07 and 2021-08 to form datasets \(\{D_{ds}^{I}(t)\}_{t}\).

The first one uses the high-dimensional output of BoI module as input in the downstream tasks, while the second uses the output of LURM.

In particular, it can be observed that LURM has a significant improvement over BoI.

Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.

Compared to using only behaviors of two months, an average improvement of 9.75%/16.03% can be achieved when using behaviors of 5 years on these tasks.

Taking BoI as a benchmark, it can be seen that LURM achieves almost lossless dimensionality reduction compared to LURM(1anchor-128) and LURM(1anchor-1280), which verifies the effectiveness of the multi-anchor module.

And It can be seen that the representation was improved through the further learning of SMEN with self-supervised tasks and the ability to encode temporal variation. In this work, a novel framework named LURM is proposed to model lifelong user behaviors with any length. With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets.

Input: Figure 1: Illustration of our Lifelong User Representation Model (LURM) for user understanding. LURM consists of two sub-model: (a) Bag of Interests (BoI) is used to aggregate user behavior data at ‘interest’ granularity, and is composed of an item embedding module and a large scale clustering module. We apply the BoI model on lifelong behavior data to generate multi-scale BoI feature sequences, (b) Self-supervised Multi-anchor Encoder Network (SMEN) is used to learn compressed user representations from BoI features sequences, and is composed of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We introduce a sub-model named BoI which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors" appears to be well-stated and accurately reflects the evidence presented. The paper provides detailed technical explanations of how BoI works, including its item embedding module, clustering approach, and mathematical formulations for generating sparse vectors (equation 4). Figure 1 clearly illustrates BoI's architecture and its role in encoding behaviors. The evidence demonstrates that BoI can indeed encode behaviors from different time periods into sparse vector sequences through interest-based aggregation.

<score>1</score>

Justification:
1. The claim is precise and matches the technical implementation described in the evidence
2. The mathematical formulations (equation 4) directly support how behaviors are encoded into sparse vectors
3. The temporal flexibility is demonstrated through descriptions of monthly and yearly granularities
4. Figure 1 provides clear visual evidence of the model architecture
5. No exaggeration or overstatement is present - the claim strictly describes BoI's demonstrated capabilities


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We introduce a sub-model named BoI which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors.

The review comment to be evaluated is:

Strengths:

1. This paper is easy to understand.
2. The problem studied in this paper is important.
3. The proposed multi-scale aggregation method is interesting.

Weaknesses:

1. The contribution of this paper can be somewhat incremental. First, the multi-anchor attention network is also called "multi-aspect attention", which has been widely used in many scenarios [1,2,3]. A very similar contrastive model pertaining approach is also studied in [4]. Thus, the contributions of this paper are somewhat incremental.
2. Discussions and comparisons on universal user modeling methods are insufficient. The authors only include PTUM as the baseline but do not discuss it in Section 2.1. There are also many other methods like [4,5]. They are also not cited nor compared.
3. The design of the item embedding method is problematic. Different from texts that there are rich relations between adjacent words, even adjacent behaviors can be rather random. Thus, it may not be suitable to learn item embeddings by regarding items as words and user behavior sequences as sentences. There are many methods for item embedding learning such as PinSage [6].
4. Compared baselines are also insufficient. As mentioned in the second point, there are many user model pretraining methods that are not compared. In addition, the compared basic user modeling techniques are also rather weak (e.g., CNN, HAN, Doc2Vec and TF-IDF).
5. Some details are unclear. PTUM and PeterRec require fine-tuning the user model in downstream tasks. However, in this work it is not clear whether the pretrained user model is finetuned.
6. The selection of some hyperparameters seems to be arbitrary, e.g., the number of anchors and the window size $\beta$. The authors do not provide experiments to show the influence of several key hyperparameters.
7. The generality of the proposed method is not verified. It is not clear whether the proposed method can be generalized to other types of user models.

Minor:

There are some typos and grammatical issues. For example, "committed to learn" and "Comparisonn".

[1] Event detection using hierarchical multi-aspect attention. WWW 2019.

[2] ARP: Aspect-aware neural review rating prediction. CIKM 2019.

[3] Hybrid graph convolutional networks with multi-head attention for location recommendation. World Wide Web 23.6 (2020): 3125-3151.

[4] Contrastive Pre-training for Sequential Recommendation. arXiv e-prints (2020): arXiv-2010.

[5] Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. SIGIR. 2020.

[6] Graph convolutional neural networks for web-scale recommender systems. KDD 2018.

The evidence to be evaluated is:Previous works have shown that considering long-term historical behavior sequences for user modeling can significantly improve the performance of different tasks

Ren _et al._ proposed a hierarchical periodic memory network for lifelong sequential modeling. They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi _et al._ decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency. Specifically, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally. Pi _et al._ also designed a search-based interest model (SIM) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item.

In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length. For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN).

In order to model extremely long lifelong user behavior sequence, we propose to aggregate the content of items under user purchases, clicks, or other behaviors at a certain granularity.

Therefore, we propose a model called Bag of Interests (BoI) to aggregate user behavior data at 'interest' granularity.

As shown in Fig. 1 (a), BoI consists of an item embedding module and a large-scale clustering module.

Like the BoVW model, an 'interest' vocabulary are supposed to be built in our BoI model. Therefore, the embedding of each item is required, so that similar items with close distance in the embedding space can be clustered together.

Inspired by these works, we design a contrastive learning task based on the relation between items drawn from a user to learn item embedding similar to item2vec Barkan & Koenigstein (2016).

Considering the large-scale item set, a subset \(\mathbf{E^{\prime}}\subset\mathbf{E}\) at the million level is sampled, and an efficient clustering algorithm named HDSC Yi et al. (2014) on \(\mathbf{E^{\prime}}\) is employed to cluster similar items into the same 'interest'. After clustering, the cluster centers \(\mathbf{C}\) make up an 'interest' vocabulary. Therefore each item can be attributed to one/multiple 'interest(s)' by hard/soft cluster assignment. Take hard cluster assignment as an example, each user can obtain his/her sparsely high-dimensional BoI feature \(\mathbf{b_{t}}\in\mathbb{R}^{D}\) in time period \(\mathbf{t}\) according to his/her behavior sequence \(\mathbf{S_{t}}=\big{\{}\mathbf{x_{1}},\mathbf{x_{2}},...,\mathbf{x_{|\mathbf{S_{t}}|}}\big{\}}\) : \[\mathbf{b_{t}}=[log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{1}}),log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{2}}), \tag{4}\] \[...,log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c _{D}})]\] where \(\mathbf{x_{i}}\in c_{j}\) means item \(i\) is assigned to the cluster \(c_{j}\in\mathbf{C}\), \(\mathbbm{1}\) is an indicator function. As mentioned above, a BoI feature \(\mathbf{b_{t}}\) for each user is obtained, through the BoI model given behavior data in any time period \(\mathbf{t}\).

Therefore, we propose to get a BoI feature sequence by applying the BoI model at each time period of the whole lifelong time.

In this way, a sequence of BoI features \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) can be obtained, where \(\mathbf{b_{t_{i}}}\) denotes the BoI feature corresponding to the \(i\)-th time period \(\mathbf{t_{i}}\).

To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are \(M\) anchors, and each of them indicates a certain preference of users. Let \(\mathbf{b}\) be a BoI feature, the module converts \(\mathbf{b}\) to \(M\) low-dimensional representations as shown in Fig. 2.

where \(f(b_{j})=b_{j}\mathbf{W_{j}^{e}}\) is the 'interest' embedding function, \(b_{j}\) is the \(j\)-th element of \(\mathbf{b}\), \(\mathbf{W^{e}}=\left(\mathbf{W_{1}^{e}},\mathbf{W_{2}^{e}},...,\mathbf{W_{D}^{e}}\right)^{T}\) is the embedding matrix. And \(\alpha_{ij}\) is the attention weight between the \(i\)-th anchor and the \(j\)-th 'interest', which measures the portion assigned to the \(i\)-th preference from the \(j\)-th behavior 'interest'. The weight \(\alpha_{ij}\) is defined as \[\alpha_{ij}=\frac{\exp(\mathbf{W_{i}^{a}}\mathbf{k_{j}})}{\sum_{l}\exp(\mathbf{W_{l}^{a}} \mathbf{k_{j}})}, \tag{6}\]

In this way, we can learn different aspects of user preferences. Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module. Through the multi-anchor module, a sequence of representation groups \(\mathbf{\mathcal{R}}=\{\mathbf{R_{t_{1}}},\mathbf{R_{t_{2}}},...,\mathbf{R_{t_{N}}}\}\) for each user is obtained, where \(\mathbf{R_{t_{i}}}=\{\mathbf{r_{t_{i}}^{1}},\mathbf{r_{t_{i}}^{2}},...,\mathbf{r_{t_{i}}^{M}}\}\) is a representation group generated by multi-anchor module corresponding to the BoI feature \(\mathbf{b_{t_{i}}}\) in time period \(\mathbf{t_{i}}\). In the time aggregation module, each sequence \(\mathbf{R^{i}}=\{\mathbf{r_{t_{i}}^{i}},\mathbf{r_{t_{i}}^{i}},...,\mathbf{r_{t_{N}}^{i}}\}\) is aggregated separately to yield a new representation \(\mathbf{\tilde{r}^{i}}\in\mathbb{R}^{H}\). Thus the variable-size representation sequence \(\mathbf{\mathcal{R}}\) can be transformed into \(M\) fixed-size representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\).

Under a time division \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}\), we have obtained \(M\) representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) through the BoI, multi-anchor module and time aggregation module as introduced above.

The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities. Generally, user behavior in this work is aggregated monthly and yearly, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}=\big{\{}\mathbf{t_{1}^{\prime}},\bm {t_{2}^{\prime}},...,\mathbf{t_{N^{\prime}}^{\prime}}\big{\}}\) (e.g., if the length of time period is 5 years, \(\mathbf{N}\) is 60 for monthly granularity, and \(\mathbf{N^{\prime}}\) is 5 for yearly granularity). Thus two sequences of BoI feature \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) and \(\mathbf{B^{\prime}}=\big{\{}\mathbf{b_{t_{1}^{\prime}}},\mathbf{b_{t_{2}^{\prime}}},...,\bm {b_{t_{N^{\prime}}^{\prime}}}\big{\}}\) at different scales correspondingly are obtained. Then, two groups of representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) and \(\mathbf{\tilde{R^{\prime}}}=\big{\{}\mathbf{\tilde{r^{\prime}}^{1}},\mathbf{\tilde{r^{ \prime}}^{2}},...,\mathbf{\tilde{r^{\prime}}^{M}}\big{\}}\) are obtained after the multi-anchor module and time aggregation module with parameter sharing.

\[\mathbf{\hat{r}^{i}}=s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}])\odot \mathbf{\tilde{r}^{i}}+(1-s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}]) )\odot\mathbf{\tilde{r^{\prime}}^{i}}, \tag{7}\] where \(\odot\) is the Hadamard product, \([\cdot]\) is a operation concatenating vectors along the last dimension, and \(s^{i}(\cdot)\) is the switch function which is implemented as a fully-connected layer followed by a sigmoid function. The switch function controls the fusion between BoI feature sequences of different scales. The output of this module \(\mathbf{\hat{R}}=[\mathbf{\hat{r}^{1}},\mathbf{\hat{r}^{2}},...,\mathbf{\hat{r}^{M}}]\) is served as the final user representation.

We constructed a training dataset \(D_{pt}^{I}\) for self-supervised user modeling by randomly sampling hundred million of users who have behaviors between 2016-06 and 2021-05, while taking behaviors occurred between 2021-07 and 2021-08 to form datasets \(\{D_{ds}^{I}(t)\}_{t}\).

The first one uses the high-dimensional output of BoI module as input in the downstream tasks, while the second uses the output of LURM.

In particular, it can be observed that LURM has a significant improvement over BoI.

Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.

Compared to using only behaviors of two months, an average improvement of 9.75%/16.03% can be achieved when using behaviors of 5 years on these tasks.

Taking BoI as a benchmark, it can be seen that LURM achieves almost lossless dimensionality reduction compared to LURM(1anchor-128) and LURM(1anchor-1280), which verifies the effectiveness of the multi-anchor module.

And It can be seen that the representation was improved through the further learning of SMEN with self-supervised tasks and the ability to encode temporal variation. In this work, a novel framework named LURM is proposed to model lifelong user behaviors with any length. With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets.

Input: Figure 1: Illustration of our Lifelong User Representation Model (LURM) for user understanding. LURM consists of two sub-model: (a) Bag of Interests (BoI) is used to aggregate user behavior data at ‘interest’ granularity, and is composed of an item embedding module and a large scale clustering module. We apply the BoI model on lifelong behavior data to generate multi-scale BoI feature sequences, (b) Self-supervised Multi-anchor Encoder Network (SMEN) is used to learn compressed user representations from BoI features sequences, and is composed of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We introduce a sub-model named BoI which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors" appears to be well-stated and accurately reflects what is demonstrated in the evidence.

The evidence thoroughly details how the BoI model:
- Aggregates user behavior data at an 'interest' granularity
- Uses item embedding and clustering to create an 'interest' vocabulary
- Converts behavior sequences into sparse high-dimensional BoI features for any time period
- Can be applied across multiple time periods to create a sequence of BoI features

The claim makes no exaggerated assertions and is directly supported by the technical details and methodology presented in both the text and Figure 1.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We introduce a sub-model named BoI which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors.

The review comment to be evaluated is:

[Advantages]
- Modeling users and user behaviors are essential, especially for web-based applications. The problem dealt with in this paper will be a promising application of representation learning based on neural networks.
- The structure of the proposed method is reasonable, especially for e-commerce applications.

[Drawbacks]
- I could not understand the novelty of the proposed method. The bag-of-features representation seems to be standard in several research fields such as computer vision, natural language processing, and data mining. The "multi-anchor module" seems to be almost the same as the multi-head attention module that is famous in the transformer. The time aggregation module does not contain a special trick. According to the introduction of this paper, the novelty of the proposed method is the ability for modeling life-long behaviors. However, I could not find any special techniques for handling life-long sequences. Please clearly state the technical novelty of the proposed method.
- The proposed method for modeling user behaviors dismisses both microscopic and macroscopic time-series characteristics, which would be significant for tracing user interest. More specifically, bag-of-interest representations remove microscopic characteristics, and the time aggregation module erases macroscopic features. If the authors believe that I have some misunderstanding, please justify it.
- When obtaining stable user representations, the knowledge of life-long learning will be required, since (1) users might have several completely different interests in their minds, (2) each interest lasts only a short term, especially in e-commerce applications, and (3) interests containing old behavior sequences might come back suddenly. This implies that almost all the information containing behavior sequences should be kept in user representations and this problem is exactly in the context of life-long learning. Please perform bibliographic surveys for life-long learning, describe the critical problem for the previous work, and present the novelty and advantage(s)  of the proposed method against the previous work.
- The downstream tasks in the experimental evaluations seem to be too easy and not suitable for evaluating the ability to handle long behavior sequences.

The evidence to be evaluated is:Previous works have shown that considering long-term historical behavior sequences for user modeling can significantly improve the performance of different tasks

Ren _et al._ proposed a hierarchical periodic memory network for lifelong sequential modeling. They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi _et al._ decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency. Specifically, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally. Pi _et al._ also designed a search-based interest model (SIM) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item.

In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length. For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN).

In order to model extremely long lifelong user behavior sequence, we propose to aggregate the content of items under user purchases, clicks, or other behaviors at a certain granularity.

Therefore, we propose a model called Bag of Interests (BoI) to aggregate user behavior data at 'interest' granularity.

As shown in Fig. 1 (a), BoI consists of an item embedding module and a large-scale clustering module.

Like the BoVW model, an 'interest' vocabulary are supposed to be built in our BoI model. Therefore, the embedding of each item is required, so that similar items with close distance in the embedding space can be clustered together.

Inspired by these works, we design a contrastive learning task based on the relation between items drawn from a user to learn item embedding similar to item2vec Barkan & Koenigstein (2016).

Considering the large-scale item set, a subset \(\mathbf{E^{\prime}}\subset\mathbf{E}\) at the million level is sampled, and an efficient clustering algorithm named HDSC Yi et al. (2014) on \(\mathbf{E^{\prime}}\) is employed to cluster similar items into the same 'interest'. After clustering, the cluster centers \(\mathbf{C}\) make up an 'interest' vocabulary. Therefore each item can be attributed to one/multiple 'interest(s)' by hard/soft cluster assignment. Take hard cluster assignment as an example, each user can obtain his/her sparsely high-dimensional BoI feature \(\mathbf{b_{t}}\in\mathbb{R}^{D}\) in time period \(\mathbf{t}\) according to his/her behavior sequence \(\mathbf{S_{t}}=\big{\{}\mathbf{x_{1}},\mathbf{x_{2}},...,\mathbf{x_{|\mathbf{S_{t}}|}}\big{\}}\) : \[\mathbf{b_{t}}=[log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{1}}),log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{2}}), \tag{4}\] \[...,log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c _{D}})]\] where \(\mathbf{x_{i}}\in c_{j}\) means item \(i\) is assigned to the cluster \(c_{j}\in\mathbf{C}\), \(\mathbbm{1}\) is an indicator function. As mentioned above, a BoI feature \(\mathbf{b_{t}}\) for each user is obtained, through the BoI model given behavior data in any time period \(\mathbf{t}\).

Therefore, we propose to get a BoI feature sequence by applying the BoI model at each time period of the whole lifelong time.

In this way, a sequence of BoI features \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) can be obtained, where \(\mathbf{b_{t_{i}}}\) denotes the BoI feature corresponding to the \(i\)-th time period \(\mathbf{t_{i}}\).

To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are \(M\) anchors, and each of them indicates a certain preference of users. Let \(\mathbf{b}\) be a BoI feature, the module converts \(\mathbf{b}\) to \(M\) low-dimensional representations as shown in Fig. 2.

where \(f(b_{j})=b_{j}\mathbf{W_{j}^{e}}\) is the 'interest' embedding function, \(b_{j}\) is the \(j\)-th element of \(\mathbf{b}\), \(\mathbf{W^{e}}=\left(\mathbf{W_{1}^{e}},\mathbf{W_{2}^{e}},...,\mathbf{W_{D}^{e}}\right)^{T}\) is the embedding matrix. And \(\alpha_{ij}\) is the attention weight between the \(i\)-th anchor and the \(j\)-th 'interest', which measures the portion assigned to the \(i\)-th preference from the \(j\)-th behavior 'interest'. The weight \(\alpha_{ij}\) is defined as \[\alpha_{ij}=\frac{\exp(\mathbf{W_{i}^{a}}\mathbf{k_{j}})}{\sum_{l}\exp(\mathbf{W_{l}^{a}} \mathbf{k_{j}})}, \tag{6}\]

In this way, we can learn different aspects of user preferences. Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module. Through the multi-anchor module, a sequence of representation groups \(\mathbf{\mathcal{R}}=\{\mathbf{R_{t_{1}}},\mathbf{R_{t_{2}}},...,\mathbf{R_{t_{N}}}\}\) for each user is obtained, where \(\mathbf{R_{t_{i}}}=\{\mathbf{r_{t_{i}}^{1}},\mathbf{r_{t_{i}}^{2}},...,\mathbf{r_{t_{i}}^{M}}\}\) is a representation group generated by multi-anchor module corresponding to the BoI feature \(\mathbf{b_{t_{i}}}\) in time period \(\mathbf{t_{i}}\). In the time aggregation module, each sequence \(\mathbf{R^{i}}=\{\mathbf{r_{t_{i}}^{i}},\mathbf{r_{t_{i}}^{i}},...,\mathbf{r_{t_{N}}^{i}}\}\) is aggregated separately to yield a new representation \(\mathbf{\tilde{r}^{i}}\in\mathbb{R}^{H}\). Thus the variable-size representation sequence \(\mathbf{\mathcal{R}}\) can be transformed into \(M\) fixed-size representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\).

Under a time division \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}\), we have obtained \(M\) representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) through the BoI, multi-anchor module and time aggregation module as introduced above.

The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities. Generally, user behavior in this work is aggregated monthly and yearly, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}=\big{\{}\mathbf{t_{1}^{\prime}},\bm {t_{2}^{\prime}},...,\mathbf{t_{N^{\prime}}^{\prime}}\big{\}}\) (e.g., if the length of time period is 5 years, \(\mathbf{N}\) is 60 for monthly granularity, and \(\mathbf{N^{\prime}}\) is 5 for yearly granularity). Thus two sequences of BoI feature \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) and \(\mathbf{B^{\prime}}=\big{\{}\mathbf{b_{t_{1}^{\prime}}},\mathbf{b_{t_{2}^{\prime}}},...,\bm {b_{t_{N^{\prime}}^{\prime}}}\big{\}}\) at different scales correspondingly are obtained. Then, two groups of representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) and \(\mathbf{\tilde{R^{\prime}}}=\big{\{}\mathbf{\tilde{r^{\prime}}^{1}},\mathbf{\tilde{r^{ \prime}}^{2}},...,\mathbf{\tilde{r^{\prime}}^{M}}\big{\}}\) are obtained after the multi-anchor module and time aggregation module with parameter sharing.

\[\mathbf{\hat{r}^{i}}=s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}])\odot \mathbf{\tilde{r}^{i}}+(1-s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}]) )\odot\mathbf{\tilde{r^{\prime}}^{i}}, \tag{7}\] where \(\odot\) is the Hadamard product, \([\cdot]\) is a operation concatenating vectors along the last dimension, and \(s^{i}(\cdot)\) is the switch function which is implemented as a fully-connected layer followed by a sigmoid function. The switch function controls the fusion between BoI feature sequences of different scales. The output of this module \(\mathbf{\hat{R}}=[\mathbf{\hat{r}^{1}},\mathbf{\hat{r}^{2}},...,\mathbf{\hat{r}^{M}}]\) is served as the final user representation.

We constructed a training dataset \(D_{pt}^{I}\) for self-supervised user modeling by randomly sampling hundred million of users who have behaviors between 2016-06 and 2021-05, while taking behaviors occurred between 2021-07 and 2021-08 to form datasets \(\{D_{ds}^{I}(t)\}_{t}\).

The first one uses the high-dimensional output of BoI module as input in the downstream tasks, while the second uses the output of LURM.

In particular, it can be observed that LURM has a significant improvement over BoI.

Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.

Compared to using only behaviors of two months, an average improvement of 9.75%/16.03% can be achieved when using behaviors of 5 years on these tasks.

Taking BoI as a benchmark, it can be seen that LURM achieves almost lossless dimensionality reduction compared to LURM(1anchor-128) and LURM(1anchor-1280), which verifies the effectiveness of the multi-anchor module.

And It can be seen that the representation was improved through the further learning of SMEN with self-supervised tasks and the ability to encode temporal variation. In this work, a novel framework named LURM is proposed to model lifelong user behaviors with any length. With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets.

Input: Figure 1: Illustration of our Lifelong User Representation Model (LURM) for user understanding. LURM consists of two sub-model: (a) Bag of Interests (BoI) is used to aggregate user behavior data at ‘interest’ granularity, and is composed of an item embedding module and a large scale clustering module. We apply the BoI model on lifelong behavior data to generate multi-scale BoI feature sequences, (b) Self-supervised Multi-anchor Encoder Network (SMEN) is used to learn compressed user representations from BoI features sequences, and is composed of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the available evidence, I analyzed the claim "We introduce a sub-model named BoI which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors."

This claim appears well-stated and accurately reflects what is demonstrated in the evidence. The paper provides detailed technical explanations of how BoI works, including:
- Clear mathematical formulations showing how behaviors are encoded into sparse vectors (Equation 4)
- Specific implementation details about item embedding and clustering
- Visual representation in Figure 1(a) showing the BoI architecture
- Experimental validation showing the model's effectiveness with 5 years of behavior data

The reviewer comments, while critical of other aspects, do not directly challenge this specific claim about BoI's encoding capabilities.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We can obtain compressed user representations with little information loss with the help of a designed sub-model named SMEN.

The evidence to be evaluated is:They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi _et al._ decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency. Specifically, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally.

In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length. For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN).

Recently, in natural language and image processing, dis criminative approaches based on contrastive learning in the latent space have shown great success in the field of representation learning, achieving state-of-the-art results. Inspired by these works, we design a contrastive learning task based on the relation between items drawn from a user to learn item embedding similar to item2vec Barkan & Koenigstein (2016). Given a set of users \(U=\{u_{1},u_{2},...,u_{|U|}\}\), each user \(u\in U\) corresponds to a behavior sequence \(\mathbf{S}=\big{\{}\mathbf{x_{1}},\mathbf{x_{2}},...,\mathbf{x_{|\mathcal{S}|}}\big{\}}\), where \(\mathbf{x_{i}}\in\mathbf{S}\) denotes the \(i\)-th item. \(|\mathbf{U}|\) and \(|\mathbf{S}|\) denote the number of users and the length of \(u\)'s behaviors respectively. Generally, the content of an item \(\mathbf{x}\) can be expressed as \(\{w_{1},w_{2},...,w_{|\mathbf{x}|}\}\), where \(w_{i}\) denotes a word from a vocabulary \(V\), and \(|\mathbf{x}|\) denotes the number of words in the content of \(\mathbf{x}\). Firstly, an encoder with average operation is used to generate item embedding \(c\): \[\mathbf{e_{\mathbf{x}}}=encoder(w_{1},w_{2},...,w_{|\mathbf{x}|})=proj(\frac{1}{|\mathbf{x}|} \sum_{i=1}^{|\mathbf{x}|}\mathbf{W_{i}}), \tag{1}\] where \(\mathbf{W_{i}}\in\mathbb{R}^{d}\) is the embedding of word \(w_{i}\) and will be learned during training, \(proj(\cdot)\) includes two residual blocks, and a \(L_{2}\) normalization layer.

After the encoder above has been trained, an item embedding set \(\mathbf{E}=\{\mathbf{e_{i}}\}_{i\in I}\) is obtained, where \(I\) is the complete collection of items at the billion level.

Considering the large-scale item set, a subset \(\mathbf{E^{\prime}}\subset\mathbf{E}\) at the million level is sampled, and an efficient clustering algorithm named HDSC Yi et al. (2014) on \(\mathbf{E^{\prime}}\) is employed to cluster similar items into the same 'interest'. After clustering, the cluster centers \(\mathbf{C}\) make up an 'interest' vocabulary. Therefore each item can be attributed to one/multiple 'interest(s)' by hard/soft cluster assignment. Take hard cluster assignment as an example, each user can obtain his/her sparsely high-dimensional BoI feature \(\mathbf{b_{t}}\in\mathbb{R}^{D}\) in time period \(\mathbf{t}\) according to his/her behavior sequence \(\mathbf{S_{t}}=\big{\{}\mathbf{x_{1}},\mathbf{x_{2}},...,\mathbf{x_{|\mathbf{S_{t}}|}}\big{\}}\) : \[\mathbf{b_{t}}=[log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{1}}),log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c_{2}}), \tag{4}\] \[...,log(1+\Sigma_{i=1}^{|\mathbf{S_{t}}|}\mathbbm{1}_{\mathbf{x_{i}}\in c _{D}})]\]

However, there are two main disadvantages: 1) it is not friendly to the downstream tasks since the dimension of representation is too high, and 2) it is too crude to aggregate the information in the whole lifelong time without considering variations over time.

Then, a Self-supervised Multi-anchor Encoder Network (SMEN) is designed to learn compressed user representations from the sequence of BoI features.

After that, SMEN is used to map \(\mathbf{B}\) to low-dimensional user representations. As shown in Fig. 1 (b), SMEN consists of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are \(M\) anchors, and each of them indicates a certain preference of users. Let \(\mathbf{b}\) be a BoI feature, the module converts \(\mathbf{b}\) to \(M\) low-dimensional representations as shown in Fig. 2. Each representation is computed as \[\mathbf{r^{i}}=ReLU(\mathbf{\alpha_{i}^{T}}f(\mathbf{b}))=ReLU(\sum_{j}^{D}\alpha_{ij}f(b_ {j}))=ReLU(\sum_{j}^{D}\alpha_{ij}b_{j}\mathbf{W_{j}^{e}}), \tag{5}\] where \(f(b_{j})=b_{j}\mathbf{W_{j}^{e}}\) is the 'interest' embedding function, \(b_{j}\) is the \(j\)-th element of \(\mathbf{b}\), \(\mathbf{W^{e}}=\left(\mathbf{W_{1}^{e}},\mathbf{W_{2}^{e}},...,\mathbf{W_{D}^{e}}\right)^{T}\) is the embedding matrix. And \(\alpha_{ij}\) is the attention weight between the \(i\)-th anchor and the \(j\)-th 'interest', which measures the portion assigned to the \(i\)-th preference from the \(j\)-th behavior 'interest'. The weight \(\alpha_{ij}\) is defined as \[\alpha_{ij}=\frac{\exp(\mathbf{W_{i}^{a}}\mathbf{k_{j}})}{\sum_{l}\exp(\mathbf{W_{l}^{a}} \mathbf{k_{j}})}, \tag{6}\]

In this way, we can learn different aspects of user preferences. Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module. Through the multi-anchor module, a sequence of representation groups \(\mathbf{\mathcal{R}}=\{\mathbf{R_{t_{1}}},\mathbf{R_{t_{2}}},...,\mathbf{R_{t_{N}}}\}\) for each user is obtained, where \(\mathbf{R_{t_{i}}}=\{\mathbf{r_{t_{i}}^{1}},\mathbf{r_{t_{i}}^{2}},...,\mathbf{r_{t_{i}}^{M}}\}\) is a representation group generated by multi-anchor module corresponding to the BoI feature \(\mathbf{b_{t_{i}}}\) in time period \(\mathbf{t_{i}}\). In the time aggregation module, each sequence \(\mathbf{R^{i}}=\{\mathbf{r_{t_{i}}^{i}},\mathbf{r_{t_{i}}^{i}},...,\mathbf{r_{t_{N}}^{i}}\}\) is aggregated separately to yield a new representation \(\mathbf{\tilde{r}^{i}}\in\mathbb{R}^{H}\). Thus the variable-size representation sequence \(\mathbf{\mathcal{R}}\) can be transformed into \(M\) fixed-size representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\).

To address this trade-off, multi-scale aggregation module is designed. The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities.

\[\mathbf{\hat{r}^{i}}=s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}])\odot \mathbf{\tilde{r}^{i}}+(1-s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}]) )\odot\mathbf{\tilde{r^{\prime}}^{i}}, \tag{7}\] where \(\odot\) is the Hadamard product, \([\cdot]\) is a operation concatenating vectors along the last dimension, and \(s^{i}(\cdot)\) is the switch function which is implemented as a fully-connected layer followed by a sigmoid function. The switch function controls the fusion between BoI feature sequences of different scales. The output of this module \(\mathbf{\hat{R}}=[\mathbf{\hat{r}^{1}},\mathbf{\hat{r}^{2}},...,\mathbf{\hat{r}^{M}}]\) is served as the final user representation.

The entire behavior sequences were input into SMEN because the length of input sequence only depends on the time interval.

In order to retain details as many as possible, the size of the 'interest' vocabulary was set to \(10^{5}\), and the number of anchors \(M\) was set to 10 in our experiments. So the final user embedding dimension was 1280.

Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.

Taking BoI as a benchmark, it can be seen that LURM achieves almost lossless dimensionality reduction compared to LURM(1anchor-128) and LURM(1anchor-1280), which verifies the effectiveness of the multi-anchor module.

It may be that the item embedding we learned is better on the industrial dataset and then LURM achieved similar results compared with BoI, indicating the ability of SMEN to compress representation losslessly.

And It can be seen that the representation was improved through the further learning of SMEN with self-supervised tasks and the ability to encode temporal variation.

Input: Figure 1: Illustration of our Lifelong User Representation Model (LURM) for user understanding. LURM consists of two sub-model: (a) Bag of Interests (BoI) is used to aggregate user behavior data at ‘interest’ granularity, and is composed of an item embedding module and a large scale clustering module. We apply the BoI model on lifelong behavior data to generate multi-scale BoI feature sequences, (b) Self-supervised Multi-anchor Encoder Network (SMEN) is used to learn compressed user representations from BoI features sequences, and is composed of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

Input: Figure 2: Illustration of the multi-anchor module in SMEN. The module outputs a group of diverse representations by assigning different portions of each ‘interest’ to different anchors. The • denotes the dot product operation, ⊗ denotes the scalar multiply operation, and ⊕ denotes vector sum operation.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find the claim is moderately overstated. While the evidence shows SMEN can achieve dimensionality reduction through its multi-anchor module architecture, the claim of "little information loss" is not fully substantiated. The evidence mentions "almost lossless dimensionality reduction" in experiments, but this is primarily comparative to other versions of their own model (LURM variants) rather than comprehensive benchmarks. The paper states "it may be that the item embedding we learned is better," suggesting some uncertainty about the true extent of information preservation.

The figures demonstrate the architectural components but don't quantitatively prove minimal information loss. The evidence shows promising results but uses cautious language about the conclusions, indicating the claim could be more precisely stated with appropriate caveats about the comparative nature of the performance measures.

<score>3</score>

Justification:
1. The evidence uses more qualified language ("almost lossless" and "may be") compared to the definitive claim
2. Performance comparisons are mainly internal rather than against broad benchmarks
3. While the technical architecture is well-documented, quantitative evidence specifically demonstrating "little information loss" is limited
4. The results appear promising but the claim would be better stated with appropriate scope limitations and comparative context