Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We design three pre-training tasks: Masked Language Modeling (MLM), Image-Text Matching (ITM), and Masked Region Modeling (MRM, with three variants).

The review comment to be evaluated is:

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).

I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.

Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.

# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.

The evidence to be evaluated is:In this section, we first introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).

We introduce three main tasks to pre-train our model: Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM). As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer.

Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores.

Masked Language Modeling (MLM) We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

\[\mathcal{L}_{\text{MLM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}\log P_{ \theta}(\mathbf{w_{m}}|\mathbf{w_{\setminus m}},\mathbf{v}). \tag{1}\] where \(\theta\) is the trainable parameters. Each pair \((\mathbf{w},\mathbf{v})\) is sampled from the whole training set \(D\). Image-Text Matching (ITM) In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1. We denote the output score as \(s_{\theta}(\mathbf{w},\mathbf{v})\). The ITM supervision is over the [CLS] token.7 During training, we sample a positive or negative pair \((\mathbf{w},\mathbf{v})\) from the dataset \(D\) at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples. We denote the label as \(y\in\{0,1\}\), indicating if the sampled pair is a match.

\[\mathcal{L}_{\text{ITM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}[y\log s_{ \theta}(\mathbf{v},\mathbf{w})+(1-y)\log(1-s_{\theta}(\mathbf{v},\mathbf{w})) ]). \tag{2}\] Masked Region Modeling (MRM) Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros. Unlike textual tokens that are represented as discrete labels, visual features are high-dimensional and continuous, thus cannot be supervised via class likelihood. Instead, we propose three variants for Masked Region Modeling, which share the same objective base: \[\mathcal{L}_{\text{MRM}}(\theta)=E_{(\mathbf{w},\mathbf{v})\sim D}f_{\theta}( \mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w}). \tag{3}\] 1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). Note that there is no ground-truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\). 3) Masked Region Classification with KL-Divergence (MRC-kl) MRC takes the most likely object class from the object detection model as the hard label (w.p. 0 or 1), which assumes the detected object class is the ground-truth label for the region. However, this may not be true, as no ground-truth label is provided for the detected region. Thus, in MRC-kl, we avoid this assumption by using soft label as supervision signal, which is the raw output from the detector (i.e., a distribution of object classes \(\tilde{c}(\mathbf{v}_{m}^{(i)})\)). MRC-kl aims to distill such knowledge into UNITER as Hinton et al. (2015), by minimizing the KL divergence between two distributions: \(f_{\theta}(\mathbf{v}_{\mathbf{m}}|\mathbf{v}_{\mathbf{m}},\mathbf{w})=\sum_{i= 1}^{M}D_{KL}(\tilde{c}(\mathbf{v}_{\mathbf{m}}^{(i)})||g_{\theta}(\mathbf{v}_{ \mathbf{m}}^{(i)}))\).

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study.

On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks. ITM is discarded because the text in VCR does not explicitly describe the image.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I've evaluated the claim "We design three pre-training tasks: Masked Language Modeling (MLM), Image-Text Matching (ITM), and Masked Region Modeling (MRM, with three variants)."

This claim is well-stated and accurately reflects the evidence without exaggeration. The paper clearly details:
- MLM conditioned on image regions
- ITM for instance-level alignment
- MRM with three specific variants (MRFR, MRC, MRC-kl)

The evidence thoroughly describes each task's implementation and provides extensive ablation studies demonstrating their effectiveness. The reviewer's comments also confirm these as the core pre-training tasks, though questioning novelty relative to BERT. The claim makes no overreaching statements about innovation or superiority.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We design three pre-training tasks: Masked Language Modeling (MLM), Image-Text Matching (ITM), and Masked Region Modeling (MRM, with three variants).

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:In this section, we first introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).

We introduce three main tasks to pre-train our model: Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM). As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer.

Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores.

Masked Language Modeling (MLM) We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

\[\mathcal{L}_{\text{MLM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}\log P_{ \theta}(\mathbf{w_{m}}|\mathbf{w_{\setminus m}},\mathbf{v}). \tag{1}\] where \(\theta\) is the trainable parameters. Each pair \((\mathbf{w},\mathbf{v})\) is sampled from the whole training set \(D\). Image-Text Matching (ITM) In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1. We denote the output score as \(s_{\theta}(\mathbf{w},\mathbf{v})\). The ITM supervision is over the [CLS] token.7 During training, we sample a positive or negative pair \((\mathbf{w},\mathbf{v})\) from the dataset \(D\) at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples. We denote the label as \(y\in\{0,1\}\), indicating if the sampled pair is a match.

\[\mathcal{L}_{\text{ITM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}[y\log s_{ \theta}(\mathbf{v},\mathbf{w})+(1-y)\log(1-s_{\theta}(\mathbf{v},\mathbf{w})) ]). \tag{2}\] Masked Region Modeling (MRM) Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros. Unlike textual tokens that are represented as discrete labels, visual features are high-dimensional and continuous, thus cannot be supervised via class likelihood. Instead, we propose three variants for Masked Region Modeling, which share the same objective base: \[\mathcal{L}_{\text{MRM}}(\theta)=E_{(\mathbf{w},\mathbf{v})\sim D}f_{\theta}( \mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w}). \tag{3}\] 1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). Note that there is no ground-truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\). 3) Masked Region Classification with KL-Divergence (MRC-kl) MRC takes the most likely object class from the object detection model as the hard label (w.p. 0 or 1), which assumes the detected object class is the ground-truth label for the region. However, this may not be true, as no ground-truth label is provided for the detected region. Thus, in MRC-kl, we avoid this assumption by using soft label as supervision signal, which is the raw output from the detector (i.e., a distribution of object classes \(\tilde{c}(\mathbf{v}_{m}^{(i)})\)). MRC-kl aims to distill such knowledge into UNITER as Hinton et al. (2015), by minimizing the KL divergence between two distributions: \(f_{\theta}(\mathbf{v}_{\mathbf{m}}|\mathbf{v}_{\mathbf{m}},\mathbf{w})=\sum_{i= 1}^{M}D_{KL}(\tilde{c}(\mathbf{v}_{\mathbf{m}}^{(i)})||g_{\theta}(\mathbf{v}_{ \mathbf{m}}^{(i)}))\).

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study.

On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks. ITM is discarded because the text in VCR does not explicitly describe the image.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "We design three pre-training tasks: MLM, ITM, and MRM (with three variants)" is well-stated and accurately reflects the evidence. The paper clearly demonstrates:

1. Detailed descriptions of all three tasks (MLM, ITM, MRM)
2. Mathematical formulations for each task
3. Thorough ablation studies showing the contribution of each task
4. Clear explanation of MRM's three variants (MRFR, MRC, MRC-kl)
5. Visual illustration in Figure 1 showing all three tasks

The claim is precise, factual, and fully supported by extensive technical details and experimental validation. It makes no exaggerated claims about the tasks' effectiveness beyond what is demonstrated.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Detailed ablation study also demonstrates that the combination of MLM+ITM+MRC-kl+MRFR yields the best pre-training performance.

The evidence to be evaluated is:In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks.

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study.

When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim carefully:

The evidence strongly supports the claim about the combination of MLM+ITM+MRC-kl+MRFR being the best performing setup. Specifically:

1. The evidence shows thorough ablation studies across multiple benchmarks
2. Line 10 (MLM+ITM+MRC-kl+MRFR combination) achieved the highest Meta-Sum score
3. This configuration was consistently chosen as the "optimal pre-training setting" for both base and large models
4. The results demonstrate clear comparative analysis between different combinations

The claim is well-stated (score of 1) as it accurately reflects the empirical evidence without exaggeration. The word "demonstrates" is appropriate given the comprehensive ablation studies presented and clear performance advantages shown.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Detailed ablation study also demonstrates that the combination of MLM+ITM+MRC-kl+MRFR yields the best pre-training performance.

The review comment to be evaluated is:

This paper presents a novel method for image-text representations called UNITER. The proposed method has been subsequently tested in many downstream tasks. A detailed ablation study helps to understand the role of each pretrained task in the proposed model.

Although the empirical results are nice, performing the intensive set of experiments on many different tasks is definitely time-consuming and needs a lot of engineering efforts, the technical contribution does not seem significant to me. The paper modifies an existing pre-training procedure by conditional masking (Section 2). I agree this is well-motivated but it has little novelty and a similar idea is there in VQA (See “Dynamic fusion with intra and inter-modality attention flow for visual question answering”). MLM and MRM are not new training procedure either, they are basically extending the BERT’s training procedure with the consideration of multiple modalities.

I have some questions for the authors:
(1) What are the advantages of using single-stream transformer over two-stream transformer (page 2). I guess it leads to fewer parameters but I don’t think this is a big problem.
(2) Some visualization of attention weights would be helpful.
Minor
•	In “m \e N^M” (equation 1), what is N and M?

The evidence to be evaluated is:In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks.

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study.

When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the provided evidence and review comment:

The claim states that a "detailed ablation study demonstrates that the combination of MLM+ITM+MRC-kl+MRFR yields the best pre-training performance." The evidence shows:
1. Multiple ablation studies were conducted across different benchmarks
2. When MLM, ITM and MRM were jointly trained, consistent performance gains were observed
3. MRC-kl achieved the best performance when combined with MLM + ITM
4. Adding MRFR led to the highest Meta-Sum score
5. This combination was identified as "optimal" and used in subsequent experiments

The evidence directly supports the claim with quantitative results showing this specific combination performed best. The wording is measured and accurately reflects the experimental findings without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Detailed ablation study also demonstrates that the combination of MLM+ITM+MRC-kl+MRFR yields the best pre-training performance.

The review comment to be evaluated is:

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).

I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.

Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.

# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.

The evidence to be evaluated is:In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks.

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study.

When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim "Detailed ablation study also demonstrates that the combination of MLM+ITM+MRC-kl+MRFR yields the best pre-training performance."

1. Summary:
The evidence provides strong support for this claim through ablation studies across multiple benchmarks. The data shows that when MLM, ITM and MRM are combined, there are consistent performance gains. Specifically, MRC-kl achieves the best performance (397.09) when combined with MLM + ITM, and adding MRFR leads to the highest Meta-Sum score. The combination is repeatedly referred to as the "optimal pre-training setting" and is used for both base and large models. The claim accurately reflects these findings without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Detailed ablation study also demonstrates that the combination of MLM+ITM+MRC-kl+MRFR yields the best pre-training performance.

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks.

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study.

When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess this claim based on the provided evidence:

The evidence shows multiple ablation studies were conducted across different benchmarks, comparing various combinations of pre-training tasks. Specifically, when MLM, ITM, and MRC-kl are combined, they achieve strong performance (397.09 Meta-Sum score). Adding MRFR to this combination (MLM+ITM+MRC-kl+MRFR) resulted in the highest Meta-Sum score, which directly supports the claim. The evidence demonstrably shows this combination yielded the best performance through systematic ablation studies.

<score>1</score>

The claim is well-stated because:
1. It accurately reflects the empirical evidence showing this specific combination performed best
2. The term "ablation study" is appropriate as the evidence shows systematic removal/addition of components
3. The evidence clearly demonstrates comparison across different combinations
4. While the reviewer notes limitations about understanding "why" it works, this doesn't affect the validity of the performance claim itself


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Experiments show that UNITER achieves new state of the art with significant performance boost across all six downstream tasks.

The evidence to be evaluated is:On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.

Results show that our UNITER-large model achieves new state of the art across all the benchmarks. UNITER-base model also outperforms the others by a large margin across all tasks except VQA. Specifically, our UNITER-base model outperforms SOTA by approximately \(+2.8\%\) for VCR on Q\(\rightarrow\)AR, \(+2.5\%\) for NLVR2, \(+7\%\) for SNLI-VE, \(+4\%\) on R@1 for Image-Text Retrieval (\(+15\%\) for zero-shot setting), and \(+2\%\) for RE Comprehension.

However, when evaluated on unseen tasks such as NLVR2, UNITER-base achieves 3% gain over LXMERT. In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

The finetuning is efficient - we train the model with a batch size of 64 examples and a learning rate of \(5e-5\) for only 5 epochs, and achieve state-of-the-art performance.

The \(Q\to AR\) accuracy of our ensemble model outperforms ViLBERT (Lu et al., 2019) ensemble by a large margin of 7.0%. Note even single UNITER-large already outperforms ViLBERT ensemble and VLBERT-large by 3.0%. Besides, we also compare our UNITER-large with LXMERT (Tan and Bansal, 2019) and VisualBERT (Li et al., 2019) on an additional testing split of NLVR2 in Table. 8. Our results consistently outperform the previous SOTA on all metrics19 by a large margin of \(\sim\)4.0%.

Table. 9 shows that UNITER still consistently outperforms the other models by a visible margin on VQA and RefCOCO+.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim to be well-stated based on the following:

1. Table 4 clearly shows UNITER (both BASE and LARGE variants) achieving superior performance across all six tasks (VQA, VCR, NLVR2, SNLI-VE, Image-Text Retrieval, and RefCOCO) compared to previous SOTA and other models.

2. The improvements are statistically significant:
- VCR Q→AR: +3% over SOTA with single model
- NLVR2: +4% margin over previous SOTA
- SNLI-VE: +7% improvement
- Image-Text Retrieval: +4% on R@1 (+15% zero-shot)
- RefCOCO tasks: Consistent improvements across variants

The claim is precise in stating "achieves new state of the art" and "significant performance boost" as these are supported by the quantitative evidence provided in both the text and table.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Experiments show that UNITER achieves new state of the art with significant performance boost across all six downstream tasks.

The review comment to be evaluated is:

This paper presents a novel method for image-text representations called UNITER. The proposed method has been subsequently tested in many downstream tasks. A detailed ablation study helps to understand the role of each pretrained task in the proposed model.

Although the empirical results are nice, performing the intensive set of experiments on many different tasks is definitely time-consuming and needs a lot of engineering efforts, the technical contribution does not seem significant to me. The paper modifies an existing pre-training procedure by conditional masking (Section 2). I agree this is well-motivated but it has little novelty and a similar idea is there in VQA (See “Dynamic fusion with intra and inter-modality attention flow for visual question answering”). MLM and MRM are not new training procedure either, they are basically extending the BERT’s training procedure with the consideration of multiple modalities.

I have some questions for the authors:
(1) What are the advantages of using single-stream transformer over two-stream transformer (page 2). I guess it leads to fewer parameters but I don’t think this is a big problem.
(2) Some visualization of attention weights would be helpful.
Minor
•	In “m \e N^M” (equation 1), what is N and M?

The evidence to be evaluated is:On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.

Results show that our UNITER-large model achieves new state of the art across all the benchmarks. UNITER-base model also outperforms the others by a large margin across all tasks except VQA. Specifically, our UNITER-base model outperforms SOTA by approximately \(+2.8\%\) for VCR on Q\(\rightarrow\)AR, \(+2.5\%\) for NLVR2, \(+7\%\) for SNLI-VE, \(+4\%\) on R@1 for Image-Text Retrieval (\(+15\%\) for zero-shot setting), and \(+2\%\) for RE Comprehension.

However, when evaluated on unseen tasks such as NLVR2, UNITER-base achieves 3% gain over LXMERT. In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

The finetuning is efficient - we train the model with a batch size of 64 examples and a learning rate of \(5e-5\) for only 5 epochs, and achieve state-of-the-art performance.

The \(Q\to AR\) accuracy of our ensemble model outperforms ViLBERT (Lu et al., 2019) ensemble by a large margin of 7.0%. Note even single UNITER-large already outperforms ViLBERT ensemble and VLBERT-large by 3.0%. Besides, we also compare our UNITER-large with LXMERT (Tan and Bansal, 2019) and VisualBERT (Li et al., 2019) on an additional testing split of NLVR2 in Table. 8. Our results consistently outperform the previous SOTA on all metrics19 by a large margin of \(\sim\)4.0%.

Table. 9 shows that UNITER still consistently outperforms the other models by a visible margin on VQA and RefCOCO+.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me analyze the claim against the evidence and review comment:

Summary:
The evidence from Table 4 and text results strongly supports UNITER achieving state-of-the-art performance across multiple vision-language tasks. The improvements are quantifiable and significant in most cases (e.g., +2.8% for VCR, +2.5% for NLVR2, +7% for SNLI-VE). However, the reviewer raises valid points about the technical novelty being incremental rather than revolutionary, as it builds upon existing methods. That said, the claim specifically focuses on performance improvements, which are well-documented in the evidence, rather than technical innovation.

Looking at specific elements:
1. "State of the art" - Supported by comprehensive benchmark results
2. "Significant performance boost" - Verified by percentage improvements
3. "Across all six tasks" - Confirmed by Table 4 showing superior performance in all tested scenarios

The claim appears to be well-stated as it accurately reflects the empirical evidence without exaggeration. The performance improvements are clearly demonstrated and quantified in the data.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Experiments show that UNITER achieves new state of the art with significant performance boost across all six downstream tasks.

The review comment to be evaluated is:

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).

I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.

Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.

# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.

The evidence to be evaluated is:On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.

Results show that our UNITER-large model achieves new state of the art across all the benchmarks. UNITER-base model also outperforms the others by a large margin across all tasks except VQA. Specifically, our UNITER-base model outperforms SOTA by approximately \(+2.8\%\) for VCR on Q\(\rightarrow\)AR, \(+2.5\%\) for NLVR2, \(+7\%\) for SNLI-VE, \(+4\%\) on R@1 for Image-Text Retrieval (\(+15\%\) for zero-shot setting), and \(+2\%\) for RE Comprehension.

However, when evaluated on unseen tasks such as NLVR2, UNITER-base achieves 3% gain over LXMERT. In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

The finetuning is efficient - we train the model with a batch size of 64 examples and a learning rate of \(5e-5\) for only 5 epochs, and achieve state-of-the-art performance.

The \(Q\to AR\) accuracy of our ensemble model outperforms ViLBERT (Lu et al., 2019) ensemble by a large margin of 7.0%. Note even single UNITER-large already outperforms ViLBERT ensemble and VLBERT-large by 3.0%. Besides, we also compare our UNITER-large with LXMERT (Tan and Bansal, 2019) and VisualBERT (Li et al., 2019) on an additional testing split of NLVR2 in Table. 8. Our results consistently outperform the previous SOTA on all metrics19 by a large margin of \(\sim\)4.0%.

Table. 9 shows that UNITER still consistently outperforms the other models by a visible margin on VQA and RefCOCO+.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "Experiments show that UNITER achieves new state of the art with significant performance boost across all six downstream tasks" appears to be generally well-supported by the data, but slightly overstated. While Table 4 shows UNITER (both base and large versions) achieving superior performance across most tasks compared to previous SOTA and other models, there are a few nuances:

1. For VQA task, UNITER-base (72.46) only marginally outperforms LXMERT (72.54) on test-std
2. The performance improvements vary in magnitude - some are significant (e.g., VCR Q→AR with +7% gain) while others are more modest
3. The reviewer notes limited novelty in the approach, though acknowledges strong experimental results

The claim would be more precise if it acknowledged the varying degrees of improvement rather than broadly stating "significant performance boost across all tasks."

<score>2</score>

Justification:
- The claim is largely supported by empirical evidence showing UNITER achieving better results across tasks
- However, the use of "significant" is somewhat overstated given the variable margins of improvement
- The comprehensiveness of the experimental validation (across multiple tasks and datasets) and quantitative results support the general claim
- More measured language acknowledging the varying degrees of improvement would make the claim more accurate


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Experiments show that UNITER achieves new state of the art with significant performance boost across all six downstream tasks.

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.

Results show that our UNITER-large model achieves new state of the art across all the benchmarks. UNITER-base model also outperforms the others by a large margin across all tasks except VQA. Specifically, our UNITER-base model outperforms SOTA by approximately \(+2.8\%\) for VCR on Q\(\rightarrow\)AR, \(+2.5\%\) for NLVR2, \(+7\%\) for SNLI-VE, \(+4\%\) on R@1 for Image-Text Retrieval (\(+15\%\) for zero-shot setting), and \(+2\%\) for RE Comprehension.

However, when evaluated on unseen tasks such as NLVR2, UNITER-base achieves 3% gain over LXMERT. In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

The finetuning is efficient - we train the model with a batch size of 64 examples and a learning rate of \(5e-5\) for only 5 epochs, and achieve state-of-the-art performance.

The \(Q\to AR\) accuracy of our ensemble model outperforms ViLBERT (Lu et al., 2019) ensemble by a large margin of 7.0%. Note even single UNITER-large already outperforms ViLBERT ensemble and VLBERT-large by 3.0%. Besides, we also compare our UNITER-large with LXMERT (Tan and Bansal, 2019) and VisualBERT (Li et al., 2019) on an additional testing split of NLVR2 in Table. 8. Our results consistently outperform the previous SOTA on all metrics19 by a large margin of \(\sim\)4.0%.

Table. 9 shows that UNITER still consistently outperforms the other models by a visible margin on VQA and RefCOCO+.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I'll assess the claim "Experiments show that UNITER achieves new state of the art with significant performance boost across all six downstream tasks."

1. Summary Assessment:
The evidence strongly supports that UNITER achieves SOTA across multiple tasks, with detailed performance improvements documented in Table 4. The performance boosts are indeed significant in many cases (e.g., +7% for SNLI-VE, +4% for Image-Text Retrieval). However, the claim slightly overstates by implying uniform significance across "all six" tasks - while improvements exist everywhere, some are more modest (like VQA where UNITER-base doesn't outperform all others). The reviewer's comment also notes the lack of explanation for these improvements.

<score>2</score>

Justification:
- The claim is largely accurate but has minor overstatement issues:
- Evidence shows consistent SOTA achievement across tasks
- "Significant" boost is supported for many but not all tasks
- Table 4 demonstrates improvements ranging from modest (~0.5%) to substantial (>7%)
- The paper notes that "UNITER-base outperforms the others by a large margin across all tasks except VQA"
- The claim would be better stated by acknowledging varying degrees of improvement rather than implying uniform significance


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Moreover, training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only.

The evidence to be evaluated is:We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011). Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.

We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training9.

In this study, we pre-train our model on Out-of-domain data (Conceptual Captions

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions.

In conclusion, for downstream tasks that contain new data which is very different from the pre-training datasets, second-stage pre-training helps further boost the performance.

To further demonstrate our idea, we conduct a direct comparison to ViLBERT (Lu et al., 2019) and VLBERT (Su et al., 2019), trained on Conceptual Captions (Sharma et al., 2018). We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be well-supported. The evidence shows that:

1. Model performance consistently improves when trained on additional CC and SBU data across multiple tasks, as shown in Table 4 where UNITER (trained on all 4 datasets) outperforms previous models.

2. The paper explicitly states "for downstream tasks that contain new data which is very different from the pre-training datasets, second-stage pre-training helps further boost the performance."

3. The evidence demonstrates these performance improvements quantitatively through comparison tables and explains the methodology behind incorporating the additional datasets.

The claim accurately reflects what the evidence demonstrates without exaggeration or overstatement.

<score>1</score>

Justification:
- The claim is carefully worded ("further boosts") without overstating the magnitude of improvement
- Table 4 provides extensive quantitative evidence showing better performance across multiple tasks
- The methodology of incorporating additional datasets is clearly explained
- The paper acknowledges the specific conditions under which the improvements occur (for downstream tasks with new/different data)
- No instances of exaggeration or unsupported claims are found


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Moreover, training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only.

The review comment to be evaluated is:

This paper presents a novel method for image-text representations called UNITER. The proposed method has been subsequently tested in many downstream tasks. A detailed ablation study helps to understand the role of each pretrained task in the proposed model.

Although the empirical results are nice, performing the intensive set of experiments on many different tasks is definitely time-consuming and needs a lot of engineering efforts, the technical contribution does not seem significant to me. The paper modifies an existing pre-training procedure by conditional masking (Section 2). I agree this is well-motivated but it has little novelty and a similar idea is there in VQA (See “Dynamic fusion with intra and inter-modality attention flow for visual question answering”). MLM and MRM are not new training procedure either, they are basically extending the BERT’s training procedure with the consideration of multiple modalities.

I have some questions for the authors:
(1) What are the advantages of using single-stream transformer over two-stream transformer (page 2). I guess it leads to fewer parameters but I don’t think this is a big problem.
(2) Some visualization of attention weights would be helpful.
Minor
•	In “m \e N^M” (equation 1), what is N and M?

The evidence to be evaluated is:We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011). Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.

We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training9.

In this study, we pre-train our model on Out-of-domain data (Conceptual Captions

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions.

In conclusion, for downstream tasks that contain new data which is very different from the pre-training datasets, second-stage pre-training helps further boost the performance.

To further demonstrate our idea, we conduct a direct comparison to ViLBERT (Lu et al., 2019) and VLBERT (Su et al., 2019), trained on Conceptual Captions (Sharma et al., 2018). We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "Moreover, training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only."

Summary:
The evidence strongly supports this claim. Table 4 shows UNITER's superior performance across multiple tasks compared to other models. The paper explicitly states they used four datasets (COCO, VG, CC, and SBU) and concludes that "second-stage pre-training helps further boost the performance" for tasks with new data. The claim is well-stated and accurately reflects the empirical results without exaggeration. The reviewer's comments, while critical of novelty, don't dispute the performance improvements.

<score>1</score>

Justification:
1. The claim is precise and measured, using appropriate terms like "boosts" which is supported by numerical evidence
2. Evidence directly demonstrates performance improvements across multiple tasks
3. The authors clearly state their dataset composition and methodology
4. The conclusion about performance boost is directly supported by empirical results shown in Table 4
5. Even though the reviewer questions technical novelty, they acknowledge the "empirical results are nice"
6. The statement acknowledges the specific conditions (unseen images/text) under which the improvement occurs


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Moreover, training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only.

The review comment to be evaluated is:

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).

I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.

Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.

# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.

The evidence to be evaluated is:We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011). Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.

We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training9.

In this study, we pre-train our model on Out-of-domain data (Conceptual Captions

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions.

In conclusion, for downstream tasks that contain new data which is very different from the pre-training datasets, second-stage pre-training helps further boost the performance.

To further demonstrate our idea, we conduct a direct comparison to ViLBERT (Lu et al., 2019) and VLBERT (Su et al., 2019), trained on Conceptual Captions (Sharma et al., 2018). We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim that "training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only."

Summary: The evidence supports this claim in a measured way. Table 4 shows UNITER's strong performance across multiple tasks, and the text indicates that the model was trained on all four datasets (COCO, VG, CC, and SBU). The authors demonstrate improved results with the full dataset compared to prior models. The claim is relatively well-stated as it directly reflects the experimental findings without excessive extrapolation or unsupported assertions.

<score>1</score>

Justification:
1. The evidence directly supports the claim through empirical results shown in Table 4
2. The authors provide clear details about the datasets used and their impact
3. The conclusion explicitly states that "second-stage pre-training helps further boost the performance"
4. The claim avoids overgeneralization and stays focused on the observed performance improvements
5. The findings are grounded in comprehensive experimental results across multiple tasks

The reviewer's comments don't contradict this aspect of the work, focusing instead on other methodological concerns. Therefore, this specific claim about dataset benefits appears well-stated and supported by the evidence.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Moreover, training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only.

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011). Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.

We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training9.

In this study, we pre-train our model on Out-of-domain data (Conceptual Captions

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions.

In conclusion, for downstream tasks that contain new data which is very different from the pre-training datasets, second-stage pre-training helps further boost the performance.

To further demonstrate our idea, we conduct a direct comparison to ViLBERT (Lu et al., 2019) and VLBERT (Su et al., 2019), trained on Conceptual Captions (Sharma et al., 2018). We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim: "Moreover, training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only."

Brief evaluation:
The evidence strongly supports this claim. Table 4 shows UNITER (trained on all 4 datasets including CC and SBU) consistently outperforming previous models trained on fewer datasets. The text evidence explicitly states "Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin" and "second-stage pre-training helps further boost the performance." The claim is measured and accurately reflects the empirical results without exaggeration.

<score>1</score>

Detailed justification:
1. The claim makes a straightforward statement about performance improvement that is directly supported by empirical evidence
2. The language used is appropriately measured ("boosts" rather than stronger terms like "dramatically improves")
3. The study provides comprehensive experimental validation across multiple tasks and datasets
4. While the reviewer notes limitations in understanding "why" the improvements occur, this doesn't affect the validity of the performance claim itself
5. The claim acknowledges the specific conditions (additional CC and SBU data) rather than making broader generalizations

The evidence thoroughly supports the specific performance claim being made, making this a well-stated assertion that aligns with the available data.