Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Experiments on several benchmark datasets show that our approach outperforms state-of-the-art unsupervised representation methods in downstream tasks.

The review comment to be evaluated is:

Strengths:

--Previous work cannot model universal user representation based on lifelong sequences of user behavior since registration and challenging to make full use of highly long user behavior sequences accumulated in Internet service platforms.

-- This work proposes a universal Lifelong User Representation Model to encode user behaviors in any period with a sparse vector with super-high dimension and map these user features to user representations with almost lossless dimensionality reduction. In this way, the model could easily learn different aspects of user preferences in extremely long sequences and generalize to other downstream tasks.

Weaknesses:

-- The structure of related works is not clear enough. At the end of universal user modeling, this work focuses on the conclusion that previous work dedicated to learning universal user representations cannot leverage information from lifelong user behaviors instead of their generalization ability. The same problem also appeared at the end of lifelong user modeling.

-- In a multi-scale aggregation module, this work captures diverse patterns from user behavior by aggregating several representations generated at different time granularities. However, this method could only use information intra-corresponding representation of every time granularity and cannot explore the information inter different time granularities.

-- The explanation of the considerable gap between BoI and LURM on the two datasets is unconvincing. On the industrial dataset, the performance of BOI is better than LURM, which proved that the design of SMEN cannot bring benefits for BOI.

-- In different downstream tasks, the benefits on category preference identification tasks are tiny, even using behaviors of 5 years on this task. There is no explanation about this phenomenon, and it proves BOI cannot learn much preference information about user behaviors on different categories.

-- This work did not design experiments about multi-scale aggregation modules to prove the effectiveness of process representations of different time granularities.

-- There are some format problems in the paper, such as no period at the end of the first paragraph of the introduction.

The evidence to be evaluated is:We compared the proposed LURM model against a rich set of user representation learning methods, including TF-IDF, Doc2Vec, TextCNN, HAN, PTUM and SUMN.

We also compared LURM with two methods named TextCNN and HAN which learn task-specific user representations and classifiers in a supervised manner. Finally, we compared LURM with two newly self-supervised methods for user modeling mentioned before, PTUM and SUMN.

Table 2 shows the comparison of category preference identification on the public Amazon dataset.

It can be seen that LURM consistently outperforms other unsupervised methods, e.g., about 3.23%/1.49% average improvements than SUMN and 4.41%/2.09% average improvements than PUTM in terms of AUC and ACC respectively. Even compared with supervised methods, LURM outperforms TextCNN and HAN about 8.26%/3.47% and 3.82%/2.04% respectively.

Table 3 shows the comparison of two user profiling prediction tasks and three category preference identification tasks on the industrial dataset.

When using the same input, LURM(short) achieved better results than other methods on both user profiling prediction tasks and category preference identification tasks. When using longer input, we can obtain more improvements, e.g., about 15.55%/17.55% average improvement on user profiling prediction tasks, and 4.19%/2.91% average improvement on category preference identification tasks compared with SUMN. From Table 2 and Table 3, we have observed that LURM has achieved better performance compared to other self-supervised and supervised methods.

With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets. \begin{table}
\begin{tabular}{c c c c c} \hline Method & Books of Literature & Games of Sports & Outdoor-hunting & Musical Instrument \\ \hline TextCNN & 73.847/2.83 & 61.00/69.92 & 66.06/72.76 & 68.28/71.81 \\ HAN & 79.12/77.49 & 66.62/70.01 & 71.19/73.42 & 70.01/72.12 \\ \hline TF-IDF & 78.62/77.21 & 65.69/68.69 & 71.90/72.19 & 69.16/68.62 \\ Doc2Vec & 71.21/68.26 & 66.75/66.85 & 71.58/66.36 & 70.29/67.37 \\ \hline PTUM & 78.66/77.91 & 66.06/68.97 & 70.34/73.01 & 69.50/72.92 \\ SUMN & 79.57/77.73 & 66.83/70.63 & 72.27/73.44 & 70.61/73.43 \\ \hline BoI & 76.29/75.75 & 59.34/67.58 & 67.80/71.28 & 66.36/69.88 \\ LURM & **82.55/79.13** & **68.02/72.43** & **77.59/75.19** & **74.04/74.43** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

 \begin{table}
\begin{tabular}{c c c c c c} \hline \hline Method & Age & Baby Age & Clothing & Shoe & Coffee \\ \hline TextCNN & 86.40/61.02 & 72.19/66.01 & 76.64/76.13 & 84.33/80.24 & 80.49/79.46 \\ TF-IDF & 87.73/61.75 & 73.08/67.26 & 77.61/78.01 & 85.04/81.15 & 81.75/80.78 \\ SUMN & 85.35/60.61 & 74.20/67.63 & 73.86/74.16 & 83.47/79.63 & 79.89/79.42 \\ \hline BoI(short) & 89.57/63.99 & 82.23/70.22 & 78.93/78.85 & 85.53/81.68 & 82.61/81.41 \\ LURM(short) & 88.48/61.99 & 82.71/69.29 & 78.37/78.57 & 85.15/81.18 & 81.94/80.78 \\ BoI & **96.48**/**80.19** & 93.29/83.57 & **80.83**/**79.33** & **86.69**/**82.16** & **83.78**/**81.74** \\ LURM & 96.09/78.43 & **94.59**/**84.91** & 80.37/79.10 & 86.30/81.65 & 83.11/81.18 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input: Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

Input: Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the evidence and review comments:

The claim "Experiments on several benchmark datasets show that our approach outperforms state-of-the-art unsupervised representation methods in downstream tasks" requires careful assessment.

Looking at the evidence:
1. Tables 2 and 3 show that LURM does outperform several baseline methods on multiple tasks
2. However, important limitations exist:
- The review points out that benefits on category preference tasks are minimal
- BoI actually performs better than LURM on the industrial dataset for several metrics
- The review notes that explanations for performance gaps are unconvincing
- There's a lack of ablation studies for key components like multi-scale aggregation
- The evaluation only includes a limited set of benchmarks and tasks

While LURM shows improvements in some cases, the claim overstates the conclusiveness and breadth of the results by not acknowledging these important limitations and mixed performance compared to baselines.

<score>4</score>

This claim rates a 4 (significantly overstated) because it makes a broad, unqualified statement about outperforming state-of-the-art methods when the evidence shows mixed results and several methodological limitations noted in the review that weaken the strength of the conclusions.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Experiments on several benchmark datasets show that our approach outperforms state-of-the-art unsupervised representation methods in downstream tasks.

The review comment to be evaluated is:

Strengths:

1. This paper is easy to understand.
2. The problem studied in this paper is important.
3. The proposed multi-scale aggregation method is interesting.

Weaknesses:

1. The contribution of this paper can be somewhat incremental. First, the multi-anchor attention network is also called "multi-aspect attention", which has been widely used in many scenarios [1,2,3]. A very similar contrastive model pertaining approach is also studied in [4]. Thus, the contributions of this paper are somewhat incremental.
2. Discussions and comparisons on universal user modeling methods are insufficient. The authors only include PTUM as the baseline but do not discuss it in Section 2.1. There are also many other methods like [4,5]. They are also not cited nor compared.
3. The design of the item embedding method is problematic. Different from texts that there are rich relations between adjacent words, even adjacent behaviors can be rather random. Thus, it may not be suitable to learn item embeddings by regarding items as words and user behavior sequences as sentences. There are many methods for item embedding learning such as PinSage [6].
4. Compared baselines are also insufficient. As mentioned in the second point, there are many user model pretraining methods that are not compared. In addition, the compared basic user modeling techniques are also rather weak (e.g., CNN, HAN, Doc2Vec and TF-IDF).
5. Some details are unclear. PTUM and PeterRec require fine-tuning the user model in downstream tasks. However, in this work it is not clear whether the pretrained user model is finetuned.
6. The selection of some hyperparameters seems to be arbitrary, e.g., the number of anchors and the window size $\beta$. The authors do not provide experiments to show the influence of several key hyperparameters.
7. The generality of the proposed method is not verified. It is not clear whether the proposed method can be generalized to other types of user models.

Minor:

There are some typos and grammatical issues. For example, "committed to learn" and "Comparisonn".

[1] Event detection using hierarchical multi-aspect attention. WWW 2019.

[2] ARP: Aspect-aware neural review rating prediction. CIKM 2019.

[3] Hybrid graph convolutional networks with multi-head attention for location recommendation. World Wide Web 23.6 (2020): 3125-3151.

[4] Contrastive Pre-training for Sequential Recommendation. arXiv e-prints (2020): arXiv-2010.

[5] Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. SIGIR. 2020.

[6] Graph convolutional neural networks for web-scale recommender systems. KDD 2018.

The evidence to be evaluated is:We compared the proposed LURM model against a rich set of user representation learning methods, including TF-IDF, Doc2Vec, TextCNN, HAN, PTUM and SUMN.

We also compared LURM with two methods named TextCNN and HAN which learn task-specific user representations and classifiers in a supervised manner. Finally, we compared LURM with two newly self-supervised methods for user modeling mentioned before, PTUM and SUMN.

Table 2 shows the comparison of category preference identification on the public Amazon dataset.

It can be seen that LURM consistently outperforms other unsupervised methods, e.g., about 3.23%/1.49% average improvements than SUMN and 4.41%/2.09% average improvements than PUTM in terms of AUC and ACC respectively. Even compared with supervised methods, LURM outperforms TextCNN and HAN about 8.26%/3.47% and 3.82%/2.04% respectively.

Table 3 shows the comparison of two user profiling prediction tasks and three category preference identification tasks on the industrial dataset.

When using the same input, LURM(short) achieved better results than other methods on both user profiling prediction tasks and category preference identification tasks. When using longer input, we can obtain more improvements, e.g., about 15.55%/17.55% average improvement on user profiling prediction tasks, and 4.19%/2.91% average improvement on category preference identification tasks compared with SUMN. From Table 2 and Table 3, we have observed that LURM has achieved better performance compared to other self-supervised and supervised methods.

With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets. \begin{table}
\begin{tabular}{c c c c c} \hline Method & Books of Literature & Games of Sports & Outdoor-hunting & Musical Instrument \\ \hline TextCNN & 73.847/2.83 & 61.00/69.92 & 66.06/72.76 & 68.28/71.81 \\ HAN & 79.12/77.49 & 66.62/70.01 & 71.19/73.42 & 70.01/72.12 \\ \hline TF-IDF & 78.62/77.21 & 65.69/68.69 & 71.90/72.19 & 69.16/68.62 \\ Doc2Vec & 71.21/68.26 & 66.75/66.85 & 71.58/66.36 & 70.29/67.37 \\ \hline PTUM & 78.66/77.91 & 66.06/68.97 & 70.34/73.01 & 69.50/72.92 \\ SUMN & 79.57/77.73 & 66.83/70.63 & 72.27/73.44 & 70.61/73.43 \\ \hline BoI & 76.29/75.75 & 59.34/67.58 & 67.80/71.28 & 66.36/69.88 \\ LURM & **82.55/79.13** & **68.02/72.43** & **77.59/75.19** & **74.04/74.43** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

 \begin{table}
\begin{tabular}{c c c c c c} \hline \hline Method & Age & Baby Age & Clothing & Shoe & Coffee \\ \hline TextCNN & 86.40/61.02 & 72.19/66.01 & 76.64/76.13 & 84.33/80.24 & 80.49/79.46 \\ TF-IDF & 87.73/61.75 & 73.08/67.26 & 77.61/78.01 & 85.04/81.15 & 81.75/80.78 \\ SUMN & 85.35/60.61 & 74.20/67.63 & 73.86/74.16 & 83.47/79.63 & 79.89/79.42 \\ \hline BoI(short) & 89.57/63.99 & 82.23/70.22 & 78.93/78.85 & 85.53/81.68 & 82.61/81.41 \\ LURM(short) & 88.48/61.99 & 82.71/69.29 & 78.37/78.57 & 85.15/81.18 & 81.94/80.78 \\ BoI & **96.48**/**80.19** & 93.29/83.57 & **80.83**/**79.33** & **86.69**/**82.16** & **83.78**/**81.74** \\ LURM & 96.09/78.43 & **94.59**/**84.91** & 80.37/79.10 & 86.30/81.65 & 83.11/81.18 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input: Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

Input: Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim to be somewhat overstated. While the tables do show LURM outperforming other methods on several metrics across different tasks, there are several important considerations:

1. Based on reviewer comments, some key baseline methods are missing from the comparison, particularly recent user modeling methods.

2. From Table 3, BoI actually achieves better performance than LURM on several metrics (Age, Clothing, Shoe, Coffee tasks).

3. The claim states "state-of-the-art" but the baselines appear limited, as noted by the reviewer (e.g., CNN, HAN, Doc2Vec and TF-IDF are considered "rather weak" baselines).

4. The generalizability and robustness of the method (e.g., sensitivity to hyperparameters) is not well demonstrated.

While the method does show improvements over the included baselines, claiming "state-of-the-art" performance without comprehensive comparisons to recent methods is an overstatement.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Experiments on several benchmark datasets show that our approach outperforms state-of-the-art unsupervised representation methods in downstream tasks.

The review comment to be evaluated is:

[Advantages]
- Modeling users and user behaviors are essential, especially for web-based applications. The problem dealt with in this paper will be a promising application of representation learning based on neural networks.
- The structure of the proposed method is reasonable, especially for e-commerce applications.

[Drawbacks]
- I could not understand the novelty of the proposed method. The bag-of-features representation seems to be standard in several research fields such as computer vision, natural language processing, and data mining. The "multi-anchor module" seems to be almost the same as the multi-head attention module that is famous in the transformer. The time aggregation module does not contain a special trick. According to the introduction of this paper, the novelty of the proposed method is the ability for modeling life-long behaviors. However, I could not find any special techniques for handling life-long sequences. Please clearly state the technical novelty of the proposed method.
- The proposed method for modeling user behaviors dismisses both microscopic and macroscopic time-series characteristics, which would be significant for tracing user interest. More specifically, bag-of-interest representations remove microscopic characteristics, and the time aggregation module erases macroscopic features. If the authors believe that I have some misunderstanding, please justify it.
- When obtaining stable user representations, the knowledge of life-long learning will be required, since (1) users might have several completely different interests in their minds, (2) each interest lasts only a short term, especially in e-commerce applications, and (3) interests containing old behavior sequences might come back suddenly. This implies that almost all the information containing behavior sequences should be kept in user representations and this problem is exactly in the context of life-long learning. Please perform bibliographic surveys for life-long learning, describe the critical problem for the previous work, and present the novelty and advantage(s)  of the proposed method against the previous work.
- The downstream tasks in the experimental evaluations seem to be too easy and not suitable for evaluating the ability to handle long behavior sequences.

The evidence to be evaluated is:We compared the proposed LURM model against a rich set of user representation learning methods, including TF-IDF, Doc2Vec, TextCNN, HAN, PTUM and SUMN.

We also compared LURM with two methods named TextCNN and HAN which learn task-specific user representations and classifiers in a supervised manner. Finally, we compared LURM with two newly self-supervised methods for user modeling mentioned before, PTUM and SUMN.

Table 2 shows the comparison of category preference identification on the public Amazon dataset.

It can be seen that LURM consistently outperforms other unsupervised methods, e.g., about 3.23%/1.49% average improvements than SUMN and 4.41%/2.09% average improvements than PUTM in terms of AUC and ACC respectively. Even compared with supervised methods, LURM outperforms TextCNN and HAN about 8.26%/3.47% and 3.82%/2.04% respectively.

Table 3 shows the comparison of two user profiling prediction tasks and three category preference identification tasks on the industrial dataset.

When using the same input, LURM(short) achieved better results than other methods on both user profiling prediction tasks and category preference identification tasks. When using longer input, we can obtain more improvements, e.g., about 15.55%/17.55% average improvement on user profiling prediction tasks, and 4.19%/2.91% average improvement on category preference identification tasks compared with SUMN. From Table 2 and Table 3, we have observed that LURM has achieved better performance compared to other self-supervised and supervised methods.

With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets. \begin{table}
\begin{tabular}{c c c c c} \hline Method & Books of Literature & Games of Sports & Outdoor-hunting & Musical Instrument \\ \hline TextCNN & 73.847/2.83 & 61.00/69.92 & 66.06/72.76 & 68.28/71.81 \\ HAN & 79.12/77.49 & 66.62/70.01 & 71.19/73.42 & 70.01/72.12 \\ \hline TF-IDF & 78.62/77.21 & 65.69/68.69 & 71.90/72.19 & 69.16/68.62 \\ Doc2Vec & 71.21/68.26 & 66.75/66.85 & 71.58/66.36 & 70.29/67.37 \\ \hline PTUM & 78.66/77.91 & 66.06/68.97 & 70.34/73.01 & 69.50/72.92 \\ SUMN & 79.57/77.73 & 66.83/70.63 & 72.27/73.44 & 70.61/73.43 \\ \hline BoI & 76.29/75.75 & 59.34/67.58 & 67.80/71.28 & 66.36/69.88 \\ LURM & **82.55/79.13** & **68.02/72.43** & **77.59/75.19** & **74.04/74.43** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

 \begin{table}
\begin{tabular}{c c c c c c} \hline \hline Method & Age & Baby Age & Clothing & Shoe & Coffee \\ \hline TextCNN & 86.40/61.02 & 72.19/66.01 & 76.64/76.13 & 84.33/80.24 & 80.49/79.46 \\ TF-IDF & 87.73/61.75 & 73.08/67.26 & 77.61/78.01 & 85.04/81.15 & 81.75/80.78 \\ SUMN & 85.35/60.61 & 74.20/67.63 & 73.86/74.16 & 83.47/79.63 & 79.89/79.42 \\ \hline BoI(short) & 89.57/63.99 & 82.23/70.22 & 78.93/78.85 & 85.53/81.68 & 82.61/81.41 \\ LURM(short) & 88.48/61.99 & 82.71/69.29 & 78.37/78.57 & 85.15/81.18 & 81.94/80.78 \\ BoI & **96.48**/**80.19** & 93.29/83.57 & **80.83**/**79.33** & **86.69**/**82.16** & **83.78**/**81.74** \\ LURM & 96.09/78.43 & **94.59**/**84.91** & 80.37/79.10 & 86.30/81.65 & 83.11/81.18 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input: Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

Input: Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim based on the evidence and review comments:

The claim states that the approach outperforms state-of-the-art unsupervised representation methods in downstream tasks. Let's analyze this:

1. The evidence shows LURM does outperform other unsupervised methods (PTUM, SUMN) on both Amazon and Industrial datasets, with improvements of 3-4% on Amazon and 4-15% on Industrial dataset.

2. However, several critical limitations noted in the review raise concerns:
- The downstream tasks may be too simple to properly evaluate long behavior sequences
- The method lacks clear technical novelty (similar to existing approaches)
- The approach dismisses important temporal characteristics
- There's insufficient handling of life-long learning aspects

While the numerical results support basic performance improvements, the reviewer's technical concerns and experimental limitations suggest the claim is moderately overstated. The claim would be better stated with qualifications about the specific tasks and limitations.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose INFERNO, a method to infer object-centric representations of visual scenes without relying on annotations.

The evidence to be evaluated is:We propose INFERNO (Infer NeRF Objects). The goal of our method is to infer object-centric 3D scene representations from single 2D views. Given an image \(x\in\mathbb{R}^{H\times W\times 3}\), we learn an inference function \(f_{\theta}\) that maps images to scene representations \(s=f_{\theta}(x)=(o_{1},o_{2},...,o_{K},o_{bg},c)\).

We define a rendering function \(g_{\gamma}\) that takes as input a scene representation and generates a 2D view of that scene \(\hat{x}=g_{\gamma}(f_{\theta}(x))\).

assume a isotropic Gaussian likelihood model with unit covariance and optimize the probability of the data under our model, which is equivalent to minimizing the mean squared error of inputs and reconstructions: \[\gamma^{*},\theta^{*}=\operatorname*{arg\,min}_{\gamma,\theta}p(X|\gamma,\theta) =\operatorname*{arg\,min}_{\gamma,\theta}\frac{1}{N}\sum_{i\leq N}(x_{i}-g_{ \gamma}(f_{\theta}(x_{i})))^{2} \tag{1}\] In the following sections we describe in more detail our inference mechanism, our rendering pipeline and their implementation. We represent 3D objects as Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) with a similar setup as that of GIRAFFE (Niemeyer and Geiger, 2021). A NeRF is a function \(g_{\tau}\) that defines a 3D shape implicitly. It takes as input a 3D location \(\mathbf{l}=(x,y,z)\) and a 2D viewing direction \(\mathbf{d}=(\psi,\phi)\) and outputs an occupancy value \(\sigma\) and a color value \(\mathbf{a}=(r,g,b)\). NeRFs are usually implemented using fully connected neural networks. Additionally, the inputs are usually embedded into a higher-dimensional space using positional encodings \(\gamma\) that embed locations and viewing directions into higher dimensional spaces \(\mathbb{R}^{P_{l}}\) and \(\mathbb{R}^{P_{d}}\), respectively.

In INFERNO, we share a single parametrization of a NeRF function across all objects. Each object specific shape and appearance is defined by the shape and appearance latent variables, which correspond to the object attributes \(o^{shape},o^{app}\). The background is defined as another NeRF with separate parameters. The background NeRF also has shape and appearance latent variables to model different backgrounds.

Given the rendering pipeline, the goal of our method is to infer representations that reconstruct a given scene. Our inference mechanism computes image features through a neural network encoder and then extracts \(K\) object and a background slots. These slots are then mapped to our structured scene representation through learned neural networks. To extract image features for each object and background slots, we use Slot Attention (Locatello et al., 2020). Slot Attention is a mechanism that maps a set of \(K\) entities, called slots, to image features without annotations.

Slots attend spatial chunks of the input features \(I\) through soft-attention \(u=TQ^{T}\), where \(T=k(I)\) and \(Q=q(\pi)\) are the embeddings of the inputs and slots respectively. The attention weights are normalized through a softmax operating on the slots axis, which makes slots compete among themselves and discourages multiple slots from attending the same input region \(w=softmax(u)\). The weighted average of \(I\) according to the attention weights is then computed and fed to a GRU network, to update each slot value: \(\pi_{j}=GRU(w_{j}*I,\pi_{j})\ \forall j\in{1,K}\). Multiple rounds of soft attention are performed to iteratively refine the slots.

Object slots are unstructured tensors that result from aggregating image features. We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects. More concretely, we map object slots to their 3D pose in the scene through a 2-layer MLP. To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin et al., 2018).

NeRFs have obtained superior reconstructions compared to other implicit methods, and our model uses NeRFs to represent multiple objects and the background of a scene.

GIRAFFE (Niemeyer and Geiger, 2021) is a GAN-based method that represents multiple objects in a scene using NeRFs as part of their generator.

However, we focus on recovering scene representations from existing images, while GIRAFFE does not have an inference mechanism. ObjSURF (Stelzner et al., 2021) and UORF (Yu et al., 2021) infer scene representations composed of multiple objects, each object represented with a differently instantiated NeRF.

Recently, there is a line of work on object-centric generative models of scenes (Kosiorek et al., 2021; Burgess et al., 2019; Locatello et al., 2020; Lin et al., 2020; Greff et al., 2019).

When equipped with an inference mechanism, these models learn to segment objects in a scene without annotations, driven by their compositional generative process.

Our model uses a variant of Slot Attention to decide on which part of a 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object.

Contrary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space. In this section we showcase the capabilities of INFERNO with three main experiments. First, we demonstrate the interpretability of the scene representations through manipulating scenes and verifying the corresponding effects in the rendered outputs. Then we show that it learns to identify and segment the objects in a scene without supervision.

Our model infers 3D object-centric scene representations from single 2D views. These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise.

Additionally, we verify that decomposing the scene into multiple objects leads to better reconstructions.

In INFERNO, we generate novel scenes by inferring representations for ground-truth images and then manipulating them.

We highlight that GIRAFFE is an unconditional model, while INFERNO generates novel scenes conditioned on existing ones.

We validate that our model is able to render out-of-distribution scenes not corresponding to training examples, such as scenes having 1 or 6 objects, and verify that the pose manipulations have semantically coherent effects.

Additionally, our model can perform inference and manipulations on existing scenes.

Given a scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales. While some of these manipulations can be performed with regular object-centric models, modifications to the scale and location of the objects are hard to implement without explicitly modeling 3D object pose. Unsupervised object discovery consists in learning to segment the objects in a scene without using annotations.

Note that this setup evaluates 2D segmentation masks, although our model naturally provides 3D segmentations.

However, our model learns to segment objects in 3D, while the other baselines extract 2D segmentation masks. Figure 3 shows some examples of the discovered object masks. We can see that the model learns to segment different objects and properly discards object slots when the number of objects in the scene is lower than needed.

It indicates that INFERNO pretraining, and not the encoder architecture, is key to learn useful representation for the task.

We next compare our approach with Aloe (Ding et al., 2020), an object-centric baseline, R3D and R3D NL, 3D convolutional models proposed by (Girdhar and Ramanan, 2019) and architectures using strong inductive biases toward object tracking such as OPNet (Shamsian et al., 2020) or Hopper (Zhou et al., 2021). Our model outperforms the R3D and R3D NL models that rely on unstructured representation. This result suggests that the object-centric representation learned by INFERNO is useful for the visual reasoning task. Our model also outperforms Aloe, an object-centric method using 2D object representation, when both methods do not use additional SSL loss.

We observe that our model significantly outperforms the slot-attention model which focus on the 2D geometry of the scene. This result supports the advantage of 3D-aware representation for solving the CATER task. We propose INFERNO, a model for inferring object-centric 3D scene representations. Our model is able to discover objects in a scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations.

Our model is composed of five main modules: encoder, slot attention, slot to object mapping, NeRF decoder, neural network upscaler.

We employ Slot Attention to map image features to object slots. Slots are sampled randomly from a Gaussian distribution with learned parameters. We use different distributions for the background slot and the object slots. During training we employ three iterations of slot attention to refine the image features to slot assignments.

The background and object slots are mapped to scene parameters using a series of MLP. For each object slot, we first map the object to its pose parameters. We use a 2-layer MLP with LayerNorm to map a slot to its pose.

Once we have inferred the object poses, we infer object shapes and appearances. These are inferred individually for each object using a common 2-layer MLP. The MLPs are conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of a condition. Specifically, we map the 7-dimensional pose tensor to LayerNorm parameters with a single linear layer with no activation or normalization. Shapes and appearances are defined by the output of the MLPs, which produce two 128-dimensional tensors.

Other concurrent approaches (Yu et al., 2021; Stelzner et al., 2021) use ground-truth camera locations to generate novel views, while we focus on recovering scene representations without the use of ground-truth annotations.

For generating manipulated scenes we consider the CLEVR2345 dataset (Niemeyer and Geiger, 2021). Images in the CLEVR2345 dataset contain from 2 to 5 objects.

We use 5 objects and rely on the model to not use additional slots if the scene shows less than 5 objects.

We first pretrain our INFERNO to reconstruct individual frames from the CATER dataset. We bootstrap a model trained on CLEVR6 for object discovery for 400k iterations and train it on the CATER dataset for an additional 100k iterations to speed-up training, as the iteration time of a model with 10 objects is larger.

When training on CLEVR6 we use 6 object slots, while when training on CATER we use 10 object slots. After pretraining, we finetune the INFERNO encoder to the supervised task of snitch localization. We discard the rendering pipeline of our model, and instead feed the inferred object slots representations to a transformer that aims at predicting the final position of the snitch.

The SSL loss randomly masks one object per-frame and tries to predict its representation at the corresponding output.

While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modifications and render novel views of a scene

While these generations have reduced details due to their resolution, they clearly show the different objects and their location in the scene.

The occupancy maps are obtained by integrating rays going from the image plane to the scene according to the NeRF density, but without taking into account the RGB output. We observe that each object is rendering a part of the scene corresponding to a single object instance.

To perform unsupervised object discovery we use the outputs of slot attention, since they do not require any upsampling and thus can provide more accurate segmentations.

Input: Figure 3: Object Discovery on CLEVR6: INFERNO identifies the different objects in a scene without supervision. For each input image, we show which regions of the input are attended by each object as well as the background. We include an example of a failed segmentation in the last row, where one object slot (4) is trying to represent multiple objects at the same time.

Input: Figure 2: Manipulations on CLEVR: we show some examples of the manipulations we perform to CLEVR2345 images, including object removal and addition, changing the scale of an object, and object translation. Our model can perform these transformations because it disentangles object pose and appearance.

Input: Figure 9: Object occupancy maps on CLEVR2345. For each input scene we show the reconstructed image as well as each individual object occupancy map. Occupancy maps are obtained by integrating the density outputs of the NeRF function along rays going from the image plane to the scene. We observe that each object is rendering a part of the scene corresponding to a single object instance. Outputs of the NeRF function have low resolution and are upscaled by a neural network. The last row shows an example of a bad grouping of two objects in the scene.

Input: Figure 1: Model Overview: We propose INFERNO, a model that infers and renders object-centric 3D scene representations. 1 Our model first decomposes an input observation into multiple object slots. 2 For each slot we infer a structured 3D representation. 3 The shape and appearance determine canonical objects rendered through NeRFs. 4 Objects are transformed and located in the overall scene according to their pose. 5 We combine objects and background and render a low-resolution scene given a camera location. 6 The input is reconstructed by upscaling the low resolution scene.

Input: Figure 6: Novel view synthesis on CLEVR2345. For each scene, we move the camera ±15◦ on the azimuth axis. Additionally, we zoom in the scene twice. While our model is trained with a fixed default camera pose and single scene views, it is able to generalize to small camera pose modifications and render novel views of a scene.

Input: Figure 7: NeRF outputs on CLEVR2345. For each input scene we show the reconstructed image as well as each the low resolution output of the NeRF function. This output is then upscaled with a neural network to obtain the reconstructed scene. While NeRF output lack full detail, they correctly depict each individual object and their position in the scene.

Input: Figure 5: Additional object additions and removals on CLEVR2345. For each scene, we show images with one randomly added object, and with 1-3 random objects removed. Some of these images show out-of-distribution samples with a number of objects not seen during training (2-5 objects).

Input: Figure 8: Shape and appearance disentanglement on CLEVR2345. For each input scene we alter the appearance vector of an object in the scene. First, we show the reconstructed scene. Then, for one of the objects in the scene, we replace its appearance vector by that of another object in the scene, while keeping its shape vector fixed. We perform this operation two times. We observe that, while the shading and color of the object changes as we change the appearance vector, its shape remains constant.

Input: Figure 4: Additional reconstructions on CLEVR2345.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We propose INFERNO, a method to infer object-centric representations of visual scenes without relying on annotations" appears well-stated. The evidence thoroughly demonstrates:

1. The model successfully infers and segments objects without supervision, as shown in Figure 3's attention maps
2. The method generates meaningful 3D object representations, validated through manipulations (Figure 2) and novel view synthesis (Figure 6)
3. The technical details provided clearly explain how INFERNO achieves this through slot attention and NeRF-based rendering
4. Limitations are acknowledged, like occasional grouping errors (Figure 9)

The claim accurately represents the method's capabilities without exaggeration and is supported by comprehensive empirical evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose INFERNO, a method to infer object-centric representations of visual scenes without relying on annotations.

The review comment to be evaluated is:

Strengths:

1. It is interesting to use NeRFs to represent objects in a scene, which encourages the understanding of 2D visual scene taking account into its 3D structure.
2. Explicitly modeling the 3D structure of 2D scene may help better deal with occlusion, ambiguity from similar appearance which can be a big challenge in 2D scene understanding.
3. It looks great to show how learned object features can be used for downstream tasks.

Weaknesses:
1. The technical novelty is somewhat limited. The paper mainly combines the Slot-attention and NeRF. Personally, it is hard to identify the exciting point(s).

2. The motivation of this work seems not well justified. It is claimed that the model is proposed to infer 3D representation for objects that are interpretable and amenable. But the experiments presented with such 3D representation cannot show its superiority and it is
hard to see the necessity of 3D object representation in this case considering the task presented can also be fulfilled with 2D object representation. In terms of the interpretability and manipulability of latent code, this paper presents objects removal/addition, object scaling and object translation. All of the above operations can actually be done just by explicitly modeling 2D object pose such as AIR (NIPS'16 paper). In terms of scene understanding in the form of object segmentation, this model cannot beat Slot Attention. In terms of downstream tasks, this model is not compared with other object-centric models. Thus, based on your current experiments, the motivation for learning such 3D representation for objects is not well justified.

3. Is it appropriate to use NeRF? If I understand correctly, NeRF learns the radiance field of a scene with 2D images rendered from different viewpoints. This model tries to recover the object radiance field from a single view of the objects. But the model training
objective can actually be fulfilled by just learning a single-view of the scene since there is only one image for each scene to generate reconstruction loss.

4. From my understanding, the 3D rendering model in this paper seems more similar to a decoder in VAE-wise architecture. It could be more convincing to provide some novel view synthesis (not novel scene) to claim it as a NeRF. For example, you may consider
object rotation.

5. All experiment datasets are a bit too simple. It is unclear how good/bad the proposed model will perform on realistic 3D scenes such as LLFF dataset or ScanNet. Otherwise, it is hard to see the real impact of this proposed method.

The evidence to be evaluated is:We propose INFERNO (Infer NeRF Objects). The goal of our method is to infer object-centric 3D scene representations from single 2D views. Given an image \(x\in\mathbb{R}^{H\times W\times 3}\), we learn an inference function \(f_{\theta}\) that maps images to scene representations \(s=f_{\theta}(x)=(o_{1},o_{2},...,o_{K},o_{bg},c)\).

We define a rendering function \(g_{\gamma}\) that takes as input a scene representation and generates a 2D view of that scene \(\hat{x}=g_{\gamma}(f_{\theta}(x))\).

assume a isotropic Gaussian likelihood model with unit covariance and optimize the probability of the data under our model, which is equivalent to minimizing the mean squared error of inputs and reconstructions: \[\gamma^{*},\theta^{*}=\operatorname*{arg\,min}_{\gamma,\theta}p(X|\gamma,\theta) =\operatorname*{arg\,min}_{\gamma,\theta}\frac{1}{N}\sum_{i\leq N}(x_{i}-g_{ \gamma}(f_{\theta}(x_{i})))^{2} \tag{1}\] In the following sections we describe in more detail our inference mechanism, our rendering pipeline and their implementation. We represent 3D objects as Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) with a similar setup as that of GIRAFFE (Niemeyer and Geiger, 2021). A NeRF is a function \(g_{\tau}\) that defines a 3D shape implicitly. It takes as input a 3D location \(\mathbf{l}=(x,y,z)\) and a 2D viewing direction \(\mathbf{d}=(\psi,\phi)\) and outputs an occupancy value \(\sigma\) and a color value \(\mathbf{a}=(r,g,b)\). NeRFs are usually implemented using fully connected neural networks. Additionally, the inputs are usually embedded into a higher-dimensional space using positional encodings \(\gamma\) that embed locations and viewing directions into higher dimensional spaces \(\mathbb{R}^{P_{l}}\) and \(\mathbb{R}^{P_{d}}\), respectively.

In INFERNO, we share a single parametrization of a NeRF function across all objects. Each object specific shape and appearance is defined by the shape and appearance latent variables, which correspond to the object attributes \(o^{shape},o^{app}\). The background is defined as another NeRF with separate parameters. The background NeRF also has shape and appearance latent variables to model different backgrounds.

Given the rendering pipeline, the goal of our method is to infer representations that reconstruct a given scene. Our inference mechanism computes image features through a neural network encoder and then extracts \(K\) object and a background slots. These slots are then mapped to our structured scene representation through learned neural networks. To extract image features for each object and background slots, we use Slot Attention (Locatello et al., 2020). Slot Attention is a mechanism that maps a set of \(K\) entities, called slots, to image features without annotations.

Slots attend spatial chunks of the input features \(I\) through soft-attention \(u=TQ^{T}\), where \(T=k(I)\) and \(Q=q(\pi)\) are the embeddings of the inputs and slots respectively. The attention weights are normalized through a softmax operating on the slots axis, which makes slots compete among themselves and discourages multiple slots from attending the same input region \(w=softmax(u)\). The weighted average of \(I\) according to the attention weights is then computed and fed to a GRU network, to update each slot value: \(\pi_{j}=GRU(w_{j}*I,\pi_{j})\ \forall j\in{1,K}\). Multiple rounds of soft attention are performed to iteratively refine the slots.

Object slots are unstructured tensors that result from aggregating image features. We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects. More concretely, we map object slots to their 3D pose in the scene through a 2-layer MLP. To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin et al., 2018).

NeRFs have obtained superior reconstructions compared to other implicit methods, and our model uses NeRFs to represent multiple objects and the background of a scene.

GIRAFFE (Niemeyer and Geiger, 2021) is a GAN-based method that represents multiple objects in a scene using NeRFs as part of their generator.

However, we focus on recovering scene representations from existing images, while GIRAFFE does not have an inference mechanism. ObjSURF (Stelzner et al., 2021) and UORF (Yu et al., 2021) infer scene representations composed of multiple objects, each object represented with a differently instantiated NeRF.

Recently, there is a line of work on object-centric generative models of scenes (Kosiorek et al., 2021; Burgess et al., 2019; Locatello et al., 2020; Lin et al., 2020; Greff et al., 2019).

When equipped with an inference mechanism, these models learn to segment objects in a scene without annotations, driven by their compositional generative process.

Our model uses a variant of Slot Attention to decide on which part of a 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object.

Contrary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space. In this section we showcase the capabilities of INFERNO with three main experiments. First, we demonstrate the interpretability of the scene representations through manipulating scenes and verifying the corresponding effects in the rendered outputs. Then we show that it learns to identify and segment the objects in a scene without supervision.

Our model infers 3D object-centric scene representations from single 2D views. These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise.

Additionally, we verify that decomposing the scene into multiple objects leads to better reconstructions.

In INFERNO, we generate novel scenes by inferring representations for ground-truth images and then manipulating them.

We highlight that GIRAFFE is an unconditional model, while INFERNO generates novel scenes conditioned on existing ones.

We validate that our model is able to render out-of-distribution scenes not corresponding to training examples, such as scenes having 1 or 6 objects, and verify that the pose manipulations have semantically coherent effects.

Additionally, our model can perform inference and manipulations on existing scenes.

Given a scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales. While some of these manipulations can be performed with regular object-centric models, modifications to the scale and location of the objects are hard to implement without explicitly modeling 3D object pose. Unsupervised object discovery consists in learning to segment the objects in a scene without using annotations.

Note that this setup evaluates 2D segmentation masks, although our model naturally provides 3D segmentations.

However, our model learns to segment objects in 3D, while the other baselines extract 2D segmentation masks. Figure 3 shows some examples of the discovered object masks. We can see that the model learns to segment different objects and properly discards object slots when the number of objects in the scene is lower than needed.

It indicates that INFERNO pretraining, and not the encoder architecture, is key to learn useful representation for the task.

We next compare our approach with Aloe (Ding et al., 2020), an object-centric baseline, R3D and R3D NL, 3D convolutional models proposed by (Girdhar and Ramanan, 2019) and architectures using strong inductive biases toward object tracking such as OPNet (Shamsian et al., 2020) or Hopper (Zhou et al., 2021). Our model outperforms the R3D and R3D NL models that rely on unstructured representation. This result suggests that the object-centric representation learned by INFERNO is useful for the visual reasoning task. Our model also outperforms Aloe, an object-centric method using 2D object representation, when both methods do not use additional SSL loss.

We observe that our model significantly outperforms the slot-attention model which focus on the 2D geometry of the scene. This result supports the advantage of 3D-aware representation for solving the CATER task. We propose INFERNO, a model for inferring object-centric 3D scene representations. Our model is able to discover objects in a scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations.

Our model is composed of five main modules: encoder, slot attention, slot to object mapping, NeRF decoder, neural network upscaler.

We employ Slot Attention to map image features to object slots. Slots are sampled randomly from a Gaussian distribution with learned parameters. We use different distributions for the background slot and the object slots. During training we employ three iterations of slot attention to refine the image features to slot assignments.

The background and object slots are mapped to scene parameters using a series of MLP. For each object slot, we first map the object to its pose parameters. We use a 2-layer MLP with LayerNorm to map a slot to its pose.

Once we have inferred the object poses, we infer object shapes and appearances. These are inferred individually for each object using a common 2-layer MLP. The MLPs are conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of a condition. Specifically, we map the 7-dimensional pose tensor to LayerNorm parameters with a single linear layer with no activation or normalization. Shapes and appearances are defined by the output of the MLPs, which produce two 128-dimensional tensors.

Other concurrent approaches (Yu et al., 2021; Stelzner et al., 2021) use ground-truth camera locations to generate novel views, while we focus on recovering scene representations without the use of ground-truth annotations.

For generating manipulated scenes we consider the CLEVR2345 dataset (Niemeyer and Geiger, 2021). Images in the CLEVR2345 dataset contain from 2 to 5 objects.

We use 5 objects and rely on the model to not use additional slots if the scene shows less than 5 objects.

We first pretrain our INFERNO to reconstruct individual frames from the CATER dataset. We bootstrap a model trained on CLEVR6 for object discovery for 400k iterations and train it on the CATER dataset for an additional 100k iterations to speed-up training, as the iteration time of a model with 10 objects is larger.

When training on CLEVR6 we use 6 object slots, while when training on CATER we use 10 object slots. After pretraining, we finetune the INFERNO encoder to the supervised task of snitch localization. We discard the rendering pipeline of our model, and instead feed the inferred object slots representations to a transformer that aims at predicting the final position of the snitch.

The SSL loss randomly masks one object per-frame and tries to predict its representation at the corresponding output.

While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modifications and render novel views of a scene

While these generations have reduced details due to their resolution, they clearly show the different objects and their location in the scene.

The occupancy maps are obtained by integrating rays going from the image plane to the scene according to the NeRF density, but without taking into account the RGB output. We observe that each object is rendering a part of the scene corresponding to a single object instance.

To perform unsupervised object discovery we use the outputs of slot attention, since they do not require any upsampling and thus can provide more accurate segmentations.

Input: Figure 3: Object Discovery on CLEVR6: INFERNO identifies the different objects in a scene without supervision. For each input image, we show which regions of the input are attended by each object as well as the background. We include an example of a failed segmentation in the last row, where one object slot (4) is trying to represent multiple objects at the same time.

Input: Figure 2: Manipulations on CLEVR: we show some examples of the manipulations we perform to CLEVR2345 images, including object removal and addition, changing the scale of an object, and object translation. Our model can perform these transformations because it disentangles object pose and appearance.

Input: Figure 9: Object occupancy maps on CLEVR2345. For each input scene we show the reconstructed image as well as each individual object occupancy map. Occupancy maps are obtained by integrating the density outputs of the NeRF function along rays going from the image plane to the scene. We observe that each object is rendering a part of the scene corresponding to a single object instance. Outputs of the NeRF function have low resolution and are upscaled by a neural network. The last row shows an example of a bad grouping of two objects in the scene.

Input: Figure 1: Model Overview: We propose INFERNO, a model that infers and renders object-centric 3D scene representations. 1 Our model first decomposes an input observation into multiple object slots. 2 For each slot we infer a structured 3D representation. 3 The shape and appearance determine canonical objects rendered through NeRFs. 4 Objects are transformed and located in the overall scene according to their pose. 5 We combine objects and background and render a low-resolution scene given a camera location. 6 The input is reconstructed by upscaling the low resolution scene.

Input: Figure 6: Novel view synthesis on CLEVR2345. For each scene, we move the camera ±15◦ on the azimuth axis. Additionally, we zoom in the scene twice. While our model is trained with a fixed default camera pose and single scene views, it is able to generalize to small camera pose modifications and render novel views of a scene.

Input: Figure 7: NeRF outputs on CLEVR2345. For each input scene we show the reconstructed image as well as each the low resolution output of the NeRF function. This output is then upscaled with a neural network to obtain the reconstructed scene. While NeRF output lack full detail, they correctly depict each individual object and their position in the scene.

Input: Figure 5: Additional object additions and removals on CLEVR2345. For each scene, we show images with one randomly added object, and with 1-3 random objects removed. Some of these images show out-of-distribution samples with a number of objects not seen during training (2-5 objects).

Input: Figure 8: Shape and appearance disentanglement on CLEVR2345. For each input scene we alter the appearance vector of an object in the scene. First, we show the reconstructed scene. Then, for one of the objects in the scene, we replace its appearance vector by that of another object in the scene, while keeping its shape vector fixed. We perform this operation two times. We observe that, while the shading and color of the object changes as we change the appearance vector, its shape remains constant.

Input: Figure 4: Additional reconstructions on CLEVR2345.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I find that the claim "We propose INFERNO, a method to infer object-centric representations of visual scenes without relying on annotations" is well-stated (score 1). The evidence strongly supports this through:

1. Detailed technical implementation showing how INFERNO infers object representations without supervision using Slot Attention
2. Multiple experimental results demonstrating successful unsupervised object segmentation
3. Visual evidence in figures showing accurate object decomposition and manipulation capabilities
4. Validation through downstream tasks and comparisons with baselines

While reviewers note some limitations regarding technical novelty and dataset simplicity, these don't invalidate the core claim about what INFERNO does. The claim is carefully worded to state exactly what the method achieves without overstating capabilities.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose INFERNO, a method to infer object-centric representations of visual scenes without relying on annotations.

The review comment to be evaluated is:

Strengths:

* an interesting combination of slot attention and NeRF mixture models.
* paper is nicely presented
* results are nice, though see below

Weaknesses:

While I like the paper in general I feel there are several points missing which are probably worth addressing to:

* The objects are transformed using the pose - but that means the appearance is transformed as well - this could be a problem if some of the appearance is pose dependent, for example shading will change, or reflections. Can the authors comment how this is handled here? my guess is that the post rendering network (the upscaling one) takes some of this burden but that's probably not ideal in terms of representation.

* How does the model perform when scenes are rendered from other viewpoints? if the resulting representation is indeed 3D it should be no problem to render the scene from other viewpoints - is that the case here? if not, why? (again, my guess is the upscaling network may be problematic here as well)

* I see that the masks used for estimating segmentation performance (Figure 3) are from the slot attention mechanism - what happens if you use the masks as decoded from the slots (i.e. render the densitiy of each slot separately). This would be more convincing to demonstrate these are "3D" segmentation.

* I feel the experimental validation is quite limited - using only a single dataset (even if it's two variants) is not very convincing as to the generality of the method. How does it work on other datasets? more/less complex.

* I would be happy to see more discussion of the post rendering / upscaling network - this seems like a cruicial part and is not discussed enough.

The evidence to be evaluated is:We propose INFERNO (Infer NeRF Objects). The goal of our method is to infer object-centric 3D scene representations from single 2D views. Given an image \(x\in\mathbb{R}^{H\times W\times 3}\), we learn an inference function \(f_{\theta}\) that maps images to scene representations \(s=f_{\theta}(x)=(o_{1},o_{2},...,o_{K},o_{bg},c)\).

We define a rendering function \(g_{\gamma}\) that takes as input a scene representation and generates a 2D view of that scene \(\hat{x}=g_{\gamma}(f_{\theta}(x))\).

assume a isotropic Gaussian likelihood model with unit covariance and optimize the probability of the data under our model, which is equivalent to minimizing the mean squared error of inputs and reconstructions: \[\gamma^{*},\theta^{*}=\operatorname*{arg\,min}_{\gamma,\theta}p(X|\gamma,\theta) =\operatorname*{arg\,min}_{\gamma,\theta}\frac{1}{N}\sum_{i\leq N}(x_{i}-g_{ \gamma}(f_{\theta}(x_{i})))^{2} \tag{1}\] In the following sections we describe in more detail our inference mechanism, our rendering pipeline and their implementation. We represent 3D objects as Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) with a similar setup as that of GIRAFFE (Niemeyer and Geiger, 2021). A NeRF is a function \(g_{\tau}\) that defines a 3D shape implicitly. It takes as input a 3D location \(\mathbf{l}=(x,y,z)\) and a 2D viewing direction \(\mathbf{d}=(\psi,\phi)\) and outputs an occupancy value \(\sigma\) and a color value \(\mathbf{a}=(r,g,b)\). NeRFs are usually implemented using fully connected neural networks. Additionally, the inputs are usually embedded into a higher-dimensional space using positional encodings \(\gamma\) that embed locations and viewing directions into higher dimensional spaces \(\mathbb{R}^{P_{l}}\) and \(\mathbb{R}^{P_{d}}\), respectively.

In INFERNO, we share a single parametrization of a NeRF function across all objects. Each object specific shape and appearance is defined by the shape and appearance latent variables, which correspond to the object attributes \(o^{shape},o^{app}\). The background is defined as another NeRF with separate parameters. The background NeRF also has shape and appearance latent variables to model different backgrounds.

Given the rendering pipeline, the goal of our method is to infer representations that reconstruct a given scene. Our inference mechanism computes image features through a neural network encoder and then extracts \(K\) object and a background slots. These slots are then mapped to our structured scene representation through learned neural networks. To extract image features for each object and background slots, we use Slot Attention (Locatello et al., 2020). Slot Attention is a mechanism that maps a set of \(K\) entities, called slots, to image features without annotations.

Slots attend spatial chunks of the input features \(I\) through soft-attention \(u=TQ^{T}\), where \(T=k(I)\) and \(Q=q(\pi)\) are the embeddings of the inputs and slots respectively. The attention weights are normalized through a softmax operating on the slots axis, which makes slots compete among themselves and discourages multiple slots from attending the same input region \(w=softmax(u)\). The weighted average of \(I\) according to the attention weights is then computed and fed to a GRU network, to update each slot value: \(\pi_{j}=GRU(w_{j}*I,\pi_{j})\ \forall j\in{1,K}\). Multiple rounds of soft attention are performed to iteratively refine the slots.

Object slots are unstructured tensors that result from aggregating image features. We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects. More concretely, we map object slots to their 3D pose in the scene through a 2-layer MLP. To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin et al., 2018).

NeRFs have obtained superior reconstructions compared to other implicit methods, and our model uses NeRFs to represent multiple objects and the background of a scene.

GIRAFFE (Niemeyer and Geiger, 2021) is a GAN-based method that represents multiple objects in a scene using NeRFs as part of their generator.

However, we focus on recovering scene representations from existing images, while GIRAFFE does not have an inference mechanism. ObjSURF (Stelzner et al., 2021) and UORF (Yu et al., 2021) infer scene representations composed of multiple objects, each object represented with a differently instantiated NeRF.

Recently, there is a line of work on object-centric generative models of scenes (Kosiorek et al., 2021; Burgess et al., 2019; Locatello et al., 2020; Lin et al., 2020; Greff et al., 2019).

When equipped with an inference mechanism, these models learn to segment objects in a scene without annotations, driven by their compositional generative process.

Our model uses a variant of Slot Attention to decide on which part of a 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object.

Contrary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space. In this section we showcase the capabilities of INFERNO with three main experiments. First, we demonstrate the interpretability of the scene representations through manipulating scenes and verifying the corresponding effects in the rendered outputs. Then we show that it learns to identify and segment the objects in a scene without supervision.

Our model infers 3D object-centric scene representations from single 2D views. These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise.

Additionally, we verify that decomposing the scene into multiple objects leads to better reconstructions.

In INFERNO, we generate novel scenes by inferring representations for ground-truth images and then manipulating them.

We highlight that GIRAFFE is an unconditional model, while INFERNO generates novel scenes conditioned on existing ones.

We validate that our model is able to render out-of-distribution scenes not corresponding to training examples, such as scenes having 1 or 6 objects, and verify that the pose manipulations have semantically coherent effects.

Additionally, our model can perform inference and manipulations on existing scenes.

Given a scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales. While some of these manipulations can be performed with regular object-centric models, modifications to the scale and location of the objects are hard to implement without explicitly modeling 3D object pose. Unsupervised object discovery consists in learning to segment the objects in a scene without using annotations.

Note that this setup evaluates 2D segmentation masks, although our model naturally provides 3D segmentations.

However, our model learns to segment objects in 3D, while the other baselines extract 2D segmentation masks. Figure 3 shows some examples of the discovered object masks. We can see that the model learns to segment different objects and properly discards object slots when the number of objects in the scene is lower than needed.

It indicates that INFERNO pretraining, and not the encoder architecture, is key to learn useful representation for the task.

We next compare our approach with Aloe (Ding et al., 2020), an object-centric baseline, R3D and R3D NL, 3D convolutional models proposed by (Girdhar and Ramanan, 2019) and architectures using strong inductive biases toward object tracking such as OPNet (Shamsian et al., 2020) or Hopper (Zhou et al., 2021). Our model outperforms the R3D and R3D NL models that rely on unstructured representation. This result suggests that the object-centric representation learned by INFERNO is useful for the visual reasoning task. Our model also outperforms Aloe, an object-centric method using 2D object representation, when both methods do not use additional SSL loss.

We observe that our model significantly outperforms the slot-attention model which focus on the 2D geometry of the scene. This result supports the advantage of 3D-aware representation for solving the CATER task. We propose INFERNO, a model for inferring object-centric 3D scene representations. Our model is able to discover objects in a scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations.

Our model is composed of five main modules: encoder, slot attention, slot to object mapping, NeRF decoder, neural network upscaler.

We employ Slot Attention to map image features to object slots. Slots are sampled randomly from a Gaussian distribution with learned parameters. We use different distributions for the background slot and the object slots. During training we employ three iterations of slot attention to refine the image features to slot assignments.

The background and object slots are mapped to scene parameters using a series of MLP. For each object slot, we first map the object to its pose parameters. We use a 2-layer MLP with LayerNorm to map a slot to its pose.

Once we have inferred the object poses, we infer object shapes and appearances. These are inferred individually for each object using a common 2-layer MLP. The MLPs are conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of a condition. Specifically, we map the 7-dimensional pose tensor to LayerNorm parameters with a single linear layer with no activation or normalization. Shapes and appearances are defined by the output of the MLPs, which produce two 128-dimensional tensors.

Other concurrent approaches (Yu et al., 2021; Stelzner et al., 2021) use ground-truth camera locations to generate novel views, while we focus on recovering scene representations without the use of ground-truth annotations.

For generating manipulated scenes we consider the CLEVR2345 dataset (Niemeyer and Geiger, 2021). Images in the CLEVR2345 dataset contain from 2 to 5 objects.

We use 5 objects and rely on the model to not use additional slots if the scene shows less than 5 objects.

We first pretrain our INFERNO to reconstruct individual frames from the CATER dataset. We bootstrap a model trained on CLEVR6 for object discovery for 400k iterations and train it on the CATER dataset for an additional 100k iterations to speed-up training, as the iteration time of a model with 10 objects is larger.

When training on CLEVR6 we use 6 object slots, while when training on CATER we use 10 object slots. After pretraining, we finetune the INFERNO encoder to the supervised task of snitch localization. We discard the rendering pipeline of our model, and instead feed the inferred object slots representations to a transformer that aims at predicting the final position of the snitch.

The SSL loss randomly masks one object per-frame and tries to predict its representation at the corresponding output.

While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modifications and render novel views of a scene

While these generations have reduced details due to their resolution, they clearly show the different objects and their location in the scene.

The occupancy maps are obtained by integrating rays going from the image plane to the scene according to the NeRF density, but without taking into account the RGB output. We observe that each object is rendering a part of the scene corresponding to a single object instance.

To perform unsupervised object discovery we use the outputs of slot attention, since they do not require any upsampling and thus can provide more accurate segmentations.

Input: Figure 3: Object Discovery on CLEVR6: INFERNO identifies the different objects in a scene without supervision. For each input image, we show which regions of the input are attended by each object as well as the background. We include an example of a failed segmentation in the last row, where one object slot (4) is trying to represent multiple objects at the same time.

Input: Figure 2: Manipulations on CLEVR: we show some examples of the manipulations we perform to CLEVR2345 images, including object removal and addition, changing the scale of an object, and object translation. Our model can perform these transformations because it disentangles object pose and appearance.

Input: Figure 9: Object occupancy maps on CLEVR2345. For each input scene we show the reconstructed image as well as each individual object occupancy map. Occupancy maps are obtained by integrating the density outputs of the NeRF function along rays going from the image plane to the scene. We observe that each object is rendering a part of the scene corresponding to a single object instance. Outputs of the NeRF function have low resolution and are upscaled by a neural network. The last row shows an example of a bad grouping of two objects in the scene.

Input: Figure 1: Model Overview: We propose INFERNO, a model that infers and renders object-centric 3D scene representations. 1 Our model first decomposes an input observation into multiple object slots. 2 For each slot we infer a structured 3D representation. 3 The shape and appearance determine canonical objects rendered through NeRFs. 4 Objects are transformed and located in the overall scene according to their pose. 5 We combine objects and background and render a low-resolution scene given a camera location. 6 The input is reconstructed by upscaling the low resolution scene.

Input: Figure 6: Novel view synthesis on CLEVR2345. For each scene, we move the camera ±15◦ on the azimuth axis. Additionally, we zoom in the scene twice. While our model is trained with a fixed default camera pose and single scene views, it is able to generalize to small camera pose modifications and render novel views of a scene.

Input: Figure 7: NeRF outputs on CLEVR2345. For each input scene we show the reconstructed image as well as each the low resolution output of the NeRF function. This output is then upscaled with a neural network to obtain the reconstructed scene. While NeRF output lack full detail, they correctly depict each individual object and their position in the scene.

Input: Figure 5: Additional object additions and removals on CLEVR2345. For each scene, we show images with one randomly added object, and with 1-3 random objects removed. Some of these images show out-of-distribution samples with a number of objects not seen during training (2-5 objects).

Input: Figure 8: Shape and appearance disentanglement on CLEVR2345. For each input scene we alter the appearance vector of an object in the scene. First, we show the reconstructed scene. Then, for one of the objects in the scene, we replace its appearance vector by that of another object in the scene, while keeping its shape vector fixed. We perform this operation two times. We observe that, while the shading and color of the object changes as we change the appearance vector, its shape remains constant.

Input: Figure 4: Additional reconstructions on CLEVR2345.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Summary: The claim "We propose INFERNO, a method to infer object-centric representations of visual scenes without relying on annotations" is well-supported by extensive evidence. The paper demonstrates:
- A clear technical approach combining slot attention and NeRF mixture models
- Successful unsupervised object segmentation (shown in Fig. 3)
- Ability to manipulate and reconstruct scenes (Fig. 2)
- Novel view synthesis capabilities (Fig. 6)
- Object decomposition and occupancy mapping (Fig. 9)

While there are some limitations noted in the review comments regarding viewpoint generalization and appearance transformation handling, these don't invalidate the core claim about inferring object-centric representations without annotations.

<score>1</score>