Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our contributions can be summarized as follows: (\(i\)) We introduce UNITER, a powerful UNiversal Image-TExt Representations for Vision-and-Language tasks. (\(ii\)) We achieve new state of the art (SOTA) on multiple V+L benchmarks, outperforming existing SOTA and concurrent multimodal pre-training methods by a large margin. (\(iii\)) We present extensive experiments and analysis to provide useful insights on the effectiveness of each pre-training task/dataset for multimodal encoder training.

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training on large-scale image/video and text pairs, then finetuning on downstream tasks. For example, VideoBERT (Sun et al., 2019) applied BERT to learn a bidirectional joint distribution over quantized video frame features and linguistic tokens from video-text pairs. ViLBERT (Lu et al., 2019) and LXMERT (Tan & Bansal, 2019) introduced the two-stream architecture, where two Transformers are applied to images and text independently, which will be fused by a third Transformer in a later stage. On the other hand, VisualBERT (Li et al., 2019), Unicoder-VL (Li et al., 2019), VL-BERT (Su et al., 2019) and B2T2 (Alberti et al., 2019) proposed the single-stream architecture, where a single Transformer is applied to both image and text.

Our UNITER model belongs to the second family. One key difference between UNITER and the other methods is the masking approach on pre-training tasks.

In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

Compared with Gao et al. (2019), UNITER learns a relatively more generic V+L representation via pre-training. In this section, we first introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3). The model architecture of UNITER is illustrated in Figure 1. Given a pair of image and sentence, UNITER takes the visual regions of the image and textual tokens of the sentence as the input. We design an Image Embedder and a Text Embedder to extract their respective embeddings. These embeddings are then feed it into a multi-layer self-attention Transformer to learn a cross-modality contextualized embedding between visual regions and textual tokens.

Specifically, in _Image Embedder_, we first use Faster R-CNN1 to extract the visual features (pooled ROI features) for each region. We also encode the location features for each region via a 7-dimensional vector2. Both visual and location features are then fed through a fully-connected (FC) layer, to be projected into the same embedding space. The final visual embedding for each region is obtained by summing up the two FC outputs and then passing through a layer normalization (LN) layer. For _Text Embedder_, we follow BERT (Devlin et al., 2019) and tokenize the input sentence into WordPieces (Wu et al., 2016). The final representation for each sub-word token3 is obtained via summing up its word embedding and position embedding, followed by another LN layer4.

We introduce three main tasks to pre-train our model: Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM). As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer. Specifically, word masking is realized by replacing the token with a special token [MASK], and region masking is implemented by replacing the visual feature vector with all zeros. Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2). Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores. To pre-train UNITER with the aforementioned different tasks, we randomly sample one pre-training task for each mini-batch and train on only one objective per SGD update. Masked Language Modeling (MLM) We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

\[\mathcal{L}_{\text{MLM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}\log P_{ \theta}(\mathbf{w_{m}}|\mathbf{w_{\setminus m}},\mathbf{v}). \tag{1}\] where \(\theta\) is the trainable parameters. Each pair \((\mathbf{w},\mathbf{v})\) is sampled from the whole training set \(D\). Image-Text Matching (ITM) In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1. We denote the output score as \(s_{\theta}(\mathbf{w},\mathbf{v})\). The ITM supervision is over the [CLS] token.7 During training, we sample a positive or negative pair \((\mathbf{w},\mathbf{v})\) from the dataset \(D\) at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples. We denote the label as \(y\in\{0,1\}\), indicating if the sampled pair is a match. Then we apply a binary cross-entropy loss for optimization:

The supervision over the [CLS] token in pretraining also alleviates the input mismatch problem between pretraining tasks and downstream finetuning tasks, since most of the downstream tasks take the representation of [CLS] token as the joint representation.

Masked Region Modeling (MRM) Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros. Unlike textual tokens that are represented as discrete labels, visual features are high-dimensional and continuous, thus cannot be supervised via class likelihood. Instead, we propose three variants for Masked Region Modeling, which share the same objective base: \[\mathcal{L}_{\text{MRM}}(\theta)=E_{(\mathbf{w},\mathbf{v})\sim D}f_{\theta}( \mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w}). \tag{3}\] 1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\).

Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011). Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training. To study the effects of different datasets on pre-training, we divide the four datasets into two categories. The first one consists of image captioning data from COCO and dense captioning data from VG.

In this way, we obtain 5.6M image-text pairs for training and 131K image-text pairs for our internal validation, which is half the size of the dataset used in LXMERT (Tan and Bansal, 2019), due to the filtering of overlapping images and the use of image-text pairs only. We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training9.

We evaluate UNITER on six V+L tasks (listed in Table 2), by transferring the pre-trained model to each target task and finetuning through end-to-end training. We report experimental results on two model sizes: UNITER-base with 12 layers and UNITER-large with 24 layers10. In VQA, VCR and NLVR\({}^{2}\) tasks, given an input image (or a pair of images) and a natural language question (or description), the model predicts an answer (or judges the correctness of the description) based on the visual content in the image. For Visual Entailment, we evaluate on the SNLI-VE dataset. The goal is to predict whether a given image semantically entails an input sentence. Classification accuracy over three classes ("Entailment", "Neutral" and "Contradiction") is used to measure model performance. For Image-Text Retrieval, we consider two datasets (COCO and Flickr30K) and evaluate the model in two settings: Image Retrieval (IR) and Text Retrieval (TR). Referring Expression (RE) Comprehension requires the model to select the target from a set of image region proposals given the query description. Models are evaluated on both ground-truth objects and detected proposals11 (MAttNet (Yu et al., 2018)).

For VQA, VCR, NLVR\({}^{2}\), Visual Entailment and Image-Text Retrieval, we extract the joint embedding of the input image-text pairs via a multi-layer perceptron (MLP) from the representation of the [CLS] token. For RE Comprehension, we use the MLP to compute the region-wise alignment scores. These MLP layers are learned during the finetuning stage. Specifically, we formulate VQA, VCR, NLVR\({}^{2}\), Visual Entailment and RE Comprehension as classification problems and minimize the cross-entropy loss over the ground-truth answers/responses. For Image-Text Retrieval, we formulate it as a ranking problem. During finetuning, we sample three pairs of image and text, one positive pair from the dataset and two negative pairs by randomly replacing its sentence/image with others. We compute the similarity scores (based on the joint embedding) for both positive and negative pairs, and maximize the margin between them through triplet loss. We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks.

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study. Comparing L2 and L3, MRFR (L3) achieves better results than MLM (L2) only on NLVR\({}^{2}\). On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Additionally, we validate the contributions of conditional masking through a comparison study. When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively. Lastly, we study the effects of pre-training datasets. Our experiments so far have been focused on In-domain data. In this study, we pre-train our model on Out-of-domain data (Conceptual Captions Table 4 presents the results of UNITER on all downstream tasks. Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

We compare with both task-specific models and concurrent pre-trained models on each downstream task. SOTA task-specific models include: MCAN (Yu et al., 2019) for VQA, MaxEnt (Suhr et al., 2019) for NLVR2, 2BT2 (Albert et al., 2019) for VCR, SCAN (Lee et al., 2018) for Image-Text Retrieval, EVE-Image (Xie et al., 2019) for SNLI-VE, and MAttNet for RE Comprehension (RefCOCO, ReFCOCO+ and ReFCOCOg)12. Concurrent pre-trained models include: ViLBERT, LXBERT, Unicoder-VL, VisualBERT and VLBERT.

Results show that our UNITER-large model achieves new state of the art across all the benchmarks. UNITER-base model also outperforms the others by a large margin across all tasks except VQA. Specifically, our UNITER-base model outperforms SOTA by approximately \(+2.8\%\) for VCR on Q\(\rightarrow\)AR, \(+2.5\%\) for NLVR2, \(+7\%\) for SNLI-VE, \(+4\%\) on R@1 for Image-Text Retrieval (\(+15\%\) for zero-shot setting), and \(+2\%\) for RE Comprehension.

However, when evaluated on unseen tasks such as NLVR2, UNITER-base achieves 3% gain over LXMERT. In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

For VCR, we propose a two-stage pre-training approach: (\(i\)) pre-train on standard pre-training datasets; and then (\(ii\)) pre-train on downstream VCR dataset. Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

The _Pair_ setting achieves better performance than the _Triplet_ setting even without cross-attention between the image pairs. We hypothesize that it is due to the fact that our UNITER is pre-trained with image-text pairs.

The bidirectional attention mechanism in the _Pair-bidattn_ setting, however, compensates the lack of cross-attention between images, hence yielding the best performance with a large margin. This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very different from pre-training tasks. In this paper, we present UNITER, a large-scale pre-trained model providing UNiversal Image-TExt Representations for Vision-and-Language tasks. Three main pre-training tasks are proposed and evaluated through extensive ablation studies. Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

Visual Entailment is a task derived from Flickr30K images and Stanford Natural Language Inference (SNLI) dataset, where the goal is to determine the logical relationship between a natural language statement and an image. Similar to BERT for Natural Language Inference (NLI), we treat SNLI-VE as a three-way classification problem and apply an MLP Transform on [CLS] output. The UNITER model is finetuned using cross-entropy loss.

The finetuning is efficient - we train the model with a batch size of 64 examples and a learning rate of \(5e-5\) for only 5 epochs, and achieve state-of-the-art performance.

Similar to Kovaleva et al. (2019), we observe several patterns in the attention maps of the UNITER model, as shown in Fig. 3. Note that different from Kovaleva et al. (2019), our attention mechanism Note that _Reversed Block_ (Fig. 3f) shows cross-modality alignment between tokens and regions. In Fig. 4, 5, and 6, we visualize several examples of text-to-image attention to demonstrate the local cross-modality alignment between regions and tokens. We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

The \(Q\to AR\) accuracy of our ensemble model outperforms ViLBERT (Lu et al., 2019) ensemble by a large margin of 7.0%. Note even single UNITER-large already outperforms ViLBERT ensemble and VLBERT-large by 3.0%. Besides, we also compare our UNITER-large with LXMERT (Tan and Bansal, 2019) and VisualBERT (Li et al., 2019) on an additional testing split of NLVR2 in Table. 8. Our results consistently outperform the previous SOTA on all metrics19 by a large margin of \(\sim\)4.0%.

Table. 9 shows that UNITER still consistently outperforms the other models by a visible margin on VQA and RefCOCO+. \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline \multicolumn{2}{c}{Task} & \multicolumn{2}{c}{Datasets} & \multicolumn{2}{c}{Image Src.} & \multicolumn{2}{c}{\#Images} & \multicolumn{2}{c}{\#Text} & \multicolumn{1}{l}{Metric} \\ \hline
1 & VQA & VQA & COCO & 204K & 1.1M & VQA-score \\ \hline
2 & VCR & VCR & Movie Clips & 110K & 290K & Accuracy \\ \hline
3 & NLVR2  & NUVR2  \\ \hline
4 & Visual Entailment & SNLI-VE & Flickr30K & 31K & 507K & Accuracy \\ \hline
5 & Image-Text Retrieval & COCO & COCO & 92K & 460K & Recall@1,5,10 \\  & & Flickr30K & Flickr30K & 32K & 160K & \\ \hline \multirow{3}{*}{6} & RefCOCO & \multirow{3}{*}{COCO} & 20K & 142K & \multirow{3}{*}{Accuracy} \\  & & RefCOCO+ & & 20K & 142K & \\ \cline{1-1}  & RefCOCOg & & 26K & 95K & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics on the datasets of downstream tasks.

 \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 2: Statistics on the datasets of downstream tasks.

Input: Figure 4: Text-to-image attention visualization example.

Input: Figure 6: Text-to-image attention visualization example.

Input: Figure 3: Attention heatmaps of UNITER

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I'll evaluate whether the claims are well-stated or overstated:

The evidence strongly supports most aspects of the claims with comprehensive experimental results and ablation studies. The paper demonstrates:
- UNITER achieves state-of-the-art results across multiple V+L benchmarks with both base and large models
- Thorough experiments analyzing pre-training tasks and datasets
- Detailed architectural insights and comparisons with existing approaches

However, while the model shows impressive performance gains, some claims could be more precisely qualified:
- "Powerful" is subjective and could be more specifically tied to benchmark improvements
- "Large margin" improvements vary across tasks (some more modest than others)
- The analysis insights, while valuable, represent incremental rather than revolutionary advances

<score>2</score>

The claims are slightly overstated but generally well-supported by the evidence. The core technical contributions and empirical results are strong, but some of the broader descriptive language could be more measured.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Different from concurrent work on multimodal pre-training that apply joint random masking to both modalities, we use conditioned masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text).

The evidence to be evaluated is:One key difference between UNITER and the other methods is the masking approach on pre-training tasks. Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM).

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). Note that there is no ground-truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, this claim appears to be well-stated (Score 1). The evidence strongly supports the distinction between UNITER's conditional masking approach and concurrent work's joint random masking:

1. The paper provides detailed mathematical formulations and clear explanations of how conditional masking works in both MLM and MRM tasks
2. Figure 8 shows empirical evidence that conditional masking achieves better validation accuracy than joint random masking across multiple datasets
3. The ablation study demonstrates conditional masking improves the Meta-Sum score (399.97) compared to joint random masking (396.51)
4. Visual examples in Figure 7 illustrate the key differences between conditional and joint random masking approaches

The claim accurately reflects the technical implementation and is supported by both qualitative and quantitative evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Different from concurrent work on multimodal pre-training that apply joint random masking to both modalities, we use conditioned masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text).

The review comment to be evaluated is:

This paper presents a novel method for image-text representations called UNITER. The proposed method has been subsequently tested in many downstream tasks. A detailed ablation study helps to understand the role of each pretrained task in the proposed model.

Although the empirical results are nice, performing the intensive set of experiments on many different tasks is definitely time-consuming and needs a lot of engineering efforts, the technical contribution does not seem significant to me. The paper modifies an existing pre-training procedure by conditional masking (Section 2). I agree this is well-motivated but it has little novelty and a similar idea is there in VQA (See “Dynamic fusion with intra and inter-modality attention flow for visual question answering”). MLM and MRM are not new training procedure either, they are basically extending the BERT’s training procedure with the consideration of multiple modalities.

I have some questions for the authors:
(1) What are the advantages of using single-stream transformer over two-stream transformer (page 2). I guess it leads to fewer parameters but I don’t think this is a big problem.
(2) Some visualization of attention weights would be helpful.
Minor
•	In “m \e N^M” (equation 1), what is N and M?

The evidence to be evaluated is:One key difference between UNITER and the other methods is the masking approach on pre-training tasks. Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM).

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). Note that there is no ground-truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim about UNITER's conditional masking approach being different from concurrent work.

The claim appears well-stated (score 1) for several reasons:

1. The evidence clearly shows that UNITER's conditional masking is distinct from other methods' joint random masking, with specific experimental results demonstrating its benefits:
- Quantitative validation shows higher accuracy and faster convergence (Figure 8)
- Meta-Sum score improvements (399.97 vs 396.51)
- Clear visualization of the difference in Figure 7

2. The review comment acknowledges this is a modification to existing pre-training, though considers it incremental. However, the technical distinction claimed is accurately described and supported by empirical evidence.

The claim stays strictly factual about the technical difference without overstating its significance or impact.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Different from concurrent work on multimodal pre-training that apply joint random masking to both modalities, we use conditioned masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text).

The review comment to be evaluated is:

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).

I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.

Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.

# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.

The evidence to be evaluated is:One key difference between UNITER and the other methods is the masking approach on pre-training tasks. Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM).

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). Note that there is no ground-truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find this claim to be well-stated. The evidence strongly supports that UNITER uses conditional masking (masking one modality while keeping the other intact) rather than joint random masking. The empirical results demonstrate clear benefits of this approach:

1. Validation accuracy plots show conditional masking converges faster and achieves higher accuracy
2. Meta-Sum score is higher with conditional masking (399.97) vs without (396.51)
3. Detailed examples and figures illustrate how conditional masking helps prevent misalignment issues
4. Multiple ablation studies validate the effectiveness

The claim accurately reflects what is implemented and demonstrated in the evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Different from concurrent work on multimodal pre-training that apply joint random masking to both modalities, we use conditioned masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text).

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:One key difference between UNITER and the other methods is the masking approach on pre-training tasks. Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM).

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). Note that there is no ground-truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim that UNITER uses conditional masking (masking one modality while keeping the other intact) versus joint random masking is well-stated and supported by multiple pieces of evidence:

1. Clear explanations of the conditional masking approach are provided with mathematical formulations
2. Empirical results in Figure 8 demonstrate better validation accuracy and faster convergence with conditional masking vs joint masking
3. Ablation studies quantitatively show improved Meta-Sum scores (399.97 vs 396.51) with conditional masking
4. Visual examples in Figure 7 illustrate the difference between the masking approaches

The reviewer's comment, while noting limitations in explaining why the approach works better, does not dispute the claim about the masking approach itself. The claim accurately represents what was implemented and demonstrated through results.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking.

The evidence to be evaluated is:Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate whether the claim "Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking" is well-stated or overstated.

Brief Assessment:
The evidence strongly supports this claim through multiple angles:
1. Direct performance comparison showing higher Meta-Sum scores (399.97 vs 396.51)
2. Validation curves in Fig. 8 demonstrating faster convergence and higher accuracy for both MLM and MRC-kl tasks across two datasets (COCO and VG)
3. Quantitative results in Table 3 showing improved performance on downstream tasks
4. Theoretical justification provided for why conditional masking works better

The claim appears well-stated as it's supported by comprehensive empirical evidence including ablation studies, validation curves, and downstream task performance. The word "comprehensive" is justified by the multiple evaluation methods used.

<score>1</score>

Detailed Justification:
The evidence provides thorough support through:
- Empirical validation on multiple metrics and datasets
- Clear visualization of performance differences in Fig. 8
- Quantitative comparisons in controlled settings
- Theoretical framework explaining the benefits
- Results that consistently favor conditional masking across different evaluation methods

The claim avoids overstating by:
- Making a straightforward comparative statement
- Being supported by clear numerical evidence
- Not making excessive claims about the magnitude of improvement
- Having multiple lines of supporting evidence that justify the use of "comprehensive"


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking.

The review comment to be evaluated is:

This paper presents a novel method for image-text representations called UNITER. The proposed method has been subsequently tested in many downstream tasks. A detailed ablation study helps to understand the role of each pretrained task in the proposed model.

Although the empirical results are nice, performing the intensive set of experiments on many different tasks is definitely time-consuming and needs a lot of engineering efforts, the technical contribution does not seem significant to me. The paper modifies an existing pre-training procedure by conditional masking (Section 2). I agree this is well-motivated but it has little novelty and a similar idea is there in VQA (See “Dynamic fusion with intra and inter-modality attention flow for visual question answering”). MLM and MRM are not new training procedure either, they are basically extending the BERT’s training procedure with the consideration of multiple modalities.

I have some questions for the authors:
(1) What are the advantages of using single-stream transformer over two-stream transformer (page 2). I guess it leads to fewer parameters but I don’t think this is a big problem.
(2) Some visualization of attention weights would be helpful.
Minor
•	In “m \e N^M” (equation 1), what is N and M?

The evidence to be evaluated is:Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking."

Summary:
The evidence strongly supports this claim through multiple angles:
1. Quantitative results show higher Meta-Sum scores (399.97 vs 396.51)
2. Figure 8 demonstrates better validation accuracy and faster convergence for both MLM and MRC-kl tasks across COCO and VG datasets
3. Table 3 shows improved performance on downstream tasks
4. The evidence provides comprehensive analysis through ablation studies, validation curves, and multiple downstream tasks

The claim is well-stated (score 1) as it accurately reflects the evidence without exaggeration. The word "comprehensive" is justified by the multiple evaluation methods and metrics presented, and the performance improvements are clearly demonstrated across different measures.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking.

The review comment to be evaluated is:

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).

I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.

Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.

# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.

The evidence to be evaluated is:Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find it well-supported but slightly overstated. The evidence shows clear benefits of conditional masking through:
- Higher Meta-Sum score (399.97 vs 396.51)
- Better convergence and final accuracy in MLM and MRC-kl validation curves
- Improved downstream task performance

However, the claim stating "better performance" is somewhat broad and could benefit from more specific qualification about where and how the improvements occur. The evidence shows improvements in specific metrics and tasks rather than comprehensive superiority in all scenarios.

<score>2</score>

Justification:
1. The evidence demonstrates concrete improvements through:
- Tables showing higher Meta-Sum scores with conditional masking
- Figure 8 validation curves showing faster convergence and higher accuracy
- Improved performance on downstream tasks (Table 3)

2. Limitations/qualifications:
- Results are specific to certain tasks and metrics
- The reviewer notes limited novelty as an extension of BERT
- The reviewer requests better explanation of why conditional masking should theoretically perform better

While the claim has strong empirical support, a more precise statement specifying the nature and scope of improvements would be more accurate than the current broad phrasing.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking.

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking."

1. Summary of Evidence Assessment:
The evidence strongly supports this claim through multiple measurements:
- Quantitative comparison showing higher Meta-Sum score (399.97 vs 396.51)
- Validation curves in Figure 8 demonstrating faster convergence and higher final accuracy for conditional masking on both MLM and MRC-kl tasks across COCO and VG datasets
- Better performance on downstream tasks shown in Table 3
- Multiple experimental validations across different metrics and tasks

However, the word "comprehensive" is slightly strong as the analysis, while thorough, focuses mainly on specific metrics and tasks. The reviewer's comment also points out that the "why" behind these improvements remains unclear.

<score>2</score>

Justification:
The claim is largely well-stated because:
1. The evidence clearly demonstrates superior performance through:
- Empirical results across multiple metrics
- Consistent improvements shown in validation curves
- Better downstream task performance
2. The term "comprehensive" is slightly overstated since:
- The analysis, while thorough, has some limitations in explaining mechanisms
- As noted in the review comment, there are unanswered questions about why the improvements occur
3. However, this minor overstatement doesn't significantly impact the claim's core validity, as the performance benefits are well-documented across multiple measures.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also conduct a thorough ablation study to find an optimal setting for the combination of pre-training tasks.

The evidence to be evaluated is:In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). Note that there is no ground-truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011).

To study the effects of different datasets on pre-training, we divide the four datasets into two categories. The first one consists of image captioning data from COCO and dense captioning data from VG.

We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks. In addition to standard metrics for each benchmark (listed in Table 2), we also use Meta-Sum (sum of all the scores across all the benchmarks) as a global metric. Firstly, we establish two baselines:

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study. Comparing L2 and L3, MRFR (L3) achieves better results than MLM (L2) only on NLVR\({}^{2}\). On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Additionally, we validate the contributions of conditional masking through a comparison study. When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

For VCR, we propose a two-stage pre-training approach: (\(i\)) pre-train on standard pre-training datasets; and then (\(ii\)) pre-train on downstream VCR dataset. Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

Thus, we experimented with three modified settings on NLVR2: (\(i\)) _Triplet_: joint embedding of images pairs and query captions; (\(ii\)) _Pair_: individual embedding of each image and each query caption; and (\(iii\)) _Pair-bidattn_: a bidirectional attention is added to the _Pair_ model to learn the interactions between the paired images.

The _Pair_ setting achieves better performance than the _Triplet_ setting even without cross-attention between the image pairs. We hypothesize that it is due to the fact that our UNITER is pre-trained with image-text pairs. Thus, it is difficult to finetune a pair-based pre-trained model on triplet input. The bidirectional attention mechanism in the _Pair-bidattn_ setting, however, compensates the lack of cross-attention between images, hence yielding the best performance with a large margin. This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very different from pre-training tasks.

Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks. ITM is discarded because the text in VCR does not explicitly describe the image.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR). \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline \multicolumn{2}{c}{Task} & \multicolumn{2}{c}{Datasets} & \multicolumn{2}{c}{Image Src.} & \multicolumn{2}{c}{\#Images} & \multicolumn{2}{c}{\#Text} & \multicolumn{1}{l}{Metric} \\ \hline
1 & VQA & VQA & COCO & 204K & 1.1M & VQA-score \\ \hline
2 & VCR & VCR & Movie Clips & 110K & 290K & Accuracy \\ \hline
3 & NLVR2  & NUVR2  \\ \hline
4 & Visual Entailment & SNLI-VE & Flickr30K & 31K & 507K & Accuracy \\ \hline
5 & Image-Text Retrieval & COCO & COCO & 92K & 460K & Recall@1,5,10 \\  & & Flickr30K & Flickr30K & 32K & 160K & \\ \hline \multirow{3}{*}{6} & RefCOCO & \multirow{3}{*}{COCO} & 20K & 142K & \multirow{3}{*}{Accuracy} \\  & & RefCOCO+ & & 20K & 142K & \\ \cline{1-1}  & RefCOCOg & & 26K & 95K & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics on the datasets of downstream tasks.

 \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 2: Statistics on the datasets of downstream tasks.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is well-stated. The evidence shows a truly thorough ablation study examining:
- Different combinations of pre-training tasks (MLM, ITM, MRC, MRFR)
- Detailed performance comparisons across multiple benchmarks
- Impact of conditional vs random masking
- Effects of different datasets (in-domain vs out-of-domain)
- Validation curves during pre-training
The results are presented systematically in Table 3 with comprehensive metrics, and the effects are analyzed quantitatively. The ablation analysis directly led to finding the optimal setting (MLM+ITM+MRC-kl+MRFR), supporting the claim's validity.

<score>1</score>