Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As sparsity increase, top-1 and top-5 accuracy further erode, suggesting that sparse models are more brittle to adversarial examples.

The review comment to be evaluated is:

This paper presents an empirical study on the effect of pruning to the model performance on each class and example, which leads to a novel finding that it has disparate effects to each sample. Specifically, the authors have found out that examples that are affected the most by pruning are more difficult to classify even for the non-pruned network, due to low image quality, mislabeling, or being atypical from the class prototype, and performed a further human study to analyze the source of difficulty. Moreover, the authors performed an additional experiment, which shows that the sparse models are brittle against natural adversarial examples.

Pros
- The paper provides a novel insight on the effect of pruning at the class and the example level, which could lead to a more effective pruning approach that exploit this findings.

Cons

- The paper only provides a novel finding but not the solution on how to tackle this problem, and thus the paper looks incomplete. After section 3.3, I was expecting to see some approaches to tackle this problem but the paper abruptly ended.

- The effect of pruning could largely differ from one method to another, but the authors do not experimentally compare the effects of different pruning methods. Also, it is highly likely that the findings discussed in the paper may be only true for input-independent pruning approaches, and may not generalize to input-dependent pruning method. The authors need to perform extensive study of both input-dependent and input-independent pruning approaches to validate their points.

In sum, the paper provides a novel insight on how pruning affects the performance at the example level, but does not provide a solution, and the current set of experiments is insufficient to validate that the empirical findings that the authors report generalize to other types of pruning approaches, such as input-dependent pruning. Thus I believe that the paper is not ready for publication yet, and vote for rejecting this paper in its current form.

The evidence to be evaluated is:We show that deep neural networks pruned to different levels of sparsity "forget" certain classes and examples more than others. While a subset of classes are systematically impacted, the direction of this impact is surprising and nuanced. Our results show certain classes are relatively impervious to the reduction in model capacity while others bear the brunt of degradation in performance. Pruning identified exemplars are a subset of exemplars where there is a high level of disagreement between sparse and non-sparse models. We show that this subset is universally challenging for models levels at all levels of sparsity to classify and mirrors the sensitivity of sparse models to open source robustness benchmarks ImageNet-C and ImageNet-A.

Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions.

Input: Figure 2: Visualization of pruning identified exemplars (PIE30) for the CIFAR-10 dataset. This subset of impacted images is identified by considering a set of 30 non-sparse wide ResNet models and 30 models trained to 30% sparsity.

Input: Figure 10: Excluding pruning identified exemplars (PIE) improves test-set top-1 accuracy for both ImageNet and CIFAR-10. The sensitivity to PIE images is amplified at higher levels of sparsity.

Input: Figure 7: Sparse models are less robust to natural adversarial examples. At high levels of sparsity, models are also more brittle to common image corruptions. We measure the relative Top-1 and Top-5 test set ResNet-50 accuracy normalized by average sparse model performance on an uncorrupted ImageNet test set. Left: Mean test-set accuracy on ImageNet-A (across 30 models). Right: Test-set performance on a subset of ImageNet-C corruptions. An extended list of all corruptions considered is included in the appendix.

Input: Table 1: Summary of class-level results for ImageNet. Only classes passing the significance test are included. The model accuracy difference column reports mean βMt − βM0 as the percentage point difference between the pruned and baseline model accuracies; a negative value means the pruned model’s average accuracy is lower than the baseline model’s.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "As sparsity increase, top-1 and top-5 accuracy further erode, suggesting that sparse models are more brittle to adversarial examples."

Summary of evaluation:
The evidence, particularly Figure 7, directly supports this claim. The graphs show a clear decline in both top-1 and top-5 accuracy on ImageNet-A (natural adversarial examples) as sparsity increases. Additionally, the ImageNet-C results demonstrate increased brittleness to various corruptions at higher sparsity levels. The claim is carefully worded using "suggesting" rather than making absolute statements, and the conclusion aligns well with the quantitative evidence presented. The experimental data provides strong support for the relationship between increasing sparsity and decreased robustness.

<score>1</score>

Justification:
1. Figure 7 shows clear empirical evidence of accuracy degradation with increased sparsity
2. The claim uses appropriately cautious language ("suggesting")
3. Multiple datasets (ImageNet-A and ImageNet-C) demonstrate consistent trends
4. The claim doesn't overextend beyond what the data shows
5. The review comment also confirms this finding by mentioning "sparse models are brittle against natural adversarial examples"


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.

The evidence to be evaluated is:More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training on large-scale image/video and text pairs, then finetuning on downstream tasks.

Our UNITER model belongs to the second family. One key difference between UNITER and the other methods is the masking approach on pre-training tasks.

In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

Compared with Gao et al. (2019), UNITER learns a relatively more generic V+L representation via pre-training. In this section, we first introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).

Given a pair of image and sentence, UNITER takes the visual regions of the image and textual tokens of the sentence as the input. We design an Image Embedder and a Text Embedder to extract their respective embeddings. These embeddings are then feed it into a multi-layer self-attention Transformer to learn a cross-modality contextualized embedding between visual regions and textual tokens.

Specifically, in _Image Embedder_, we first use Faster R-CNN1 to extract the visual features (pooled ROI features) for each region. We also encode the location features for each region via a 7-dimensional vector2. Both visual and location features are then fed through a fully-connected (FC) layer, to be projected into the same embedding space. The final visual embedding for each region is obtained by summing up the two FC outputs and then passing through a layer normalization (LN) layer. For _Text Embedder_, we follow BERT (Devlin et al., 2019) and tokenize the input sentence into WordPieces (Wu et al., 2016). The final representation for each sub-word token3 is obtained via summing up its word embedding and position embedding, followed by another LN layer4.

We also use a special modality embedding to help the model distinguish between textual and visual input, which is similar to the ‘segment embedding’ in BERT.

We introduce three main tasks to pre-train our model: Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM). As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer. Specifically, word masking is realized by replacing the token with a special token [MASK], and region masking is implemented by replacing the visual feature vector with all zeros. Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word.

Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores. To pre-train UNITER with the aforementioned different tasks, we randomly sample one pre-training task for each mini-batch and train on only one objective per SGD update. Masked Language Modeling (MLM) We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

\[\mathcal{L}_{\text{MLM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}\log P_{ \theta}(\mathbf{w_{m}}|\mathbf{w_{\setminus m}},\mathbf{v}). \tag{1}\] where \(\theta\) is the trainable parameters. Each pair \((\mathbf{w},\mathbf{v})\) is sampled from the whole training set \(D\).

In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1.

During training, we sample a positive or negative pair \((\mathbf{w},\mathbf{v})\) from the dataset \(D\) at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples. We denote the label as \(y\in\{0,1\}\), indicating if the sampled pair is a match.

We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011). Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.

In this way, we obtain 5.6M image-text pairs for training and 131K image-text pairs for our internal validation, which is half the size of the dataset used in LXMERT (Tan and Bansal, 2019), due to the filtering of overlapping images and the use of image-text pairs only. We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training9.

We evaluate UNITER on six V+L tasks (listed in Table 2), by transferring the pre-trained model to each target task and finetuning through end-to-end training. We report experimental results on two model sizes: UNITER-base with 12 layers and UNITER-large with 24 layers10. In VQA, VCR and NLVR\({}^{2}\) tasks, given an input image (or a pair of images) and a natural language question (or description), the model predicts an answer (or judges the correctness of the description) based on the visual content in the image.

For Image-Text Retrieval, we consider two datasets (COCO and Flickr30K) and evaluate the model in two settings: Image Retrieval (IR) and Text Retrieval (TR). Referring Expression (RE) Comprehension requires the model to select the target from a set of image region proposals given the query description.

For VQA, VCR, NLVR\({}^{2}\), Visual Entailment and Image-Text Retrieval, we extract the joint embedding of the input image-text pairs via a multi-layer perceptron (MLP) from the representation of the [CLS] token. For RE Comprehension, we use the MLP to compute the region-wise alignment scores. These MLP layers are learned during the finetuning stage. Specifically, we formulate VQA, VCR, NLVR\({}^{2}\), Visual Entailment and RE Comprehension as classification problems and minimize the cross-entropy loss over the ground-truth answers/responses. For Image-Text Retrieval, we formulate it as a ranking problem. During finetuning, we sample three pairs of image and text, one positive pair from the dataset and two negative pairs by randomly replacing its sentence/image with others. We compute the similarity scores (based on the joint embedding) for both positive and negative pairs, and maximize the margin between them through triplet loss. We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks.

In this study, we pre-train our model on Out-of-domain data (Conceptual Captions Table 4 presents the results of UNITER on all downstream tasks. Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

SOTA task-specific models include: MCAN (Yu et al., 2019) for VQA, MaxEnt (Suhr et al., 2019) for NLVR2, 2BT2 (Albert et al., 2019) for VCR, SCAN (Lee et al., 2018) for Image-Text Retrieval, EVE-Image (Xie et al., 2019) for SNLI-VE, and MAttNet for RE Comprehension (RefCOCO, ReFCOCO+ and ReFCOCOg)12. Concurrent pre-trained models include: ViLBERT, LXBERT, Unicoder-VL, VisualBERT and VLBERT.

Results show that our UNITER-large model achieves new state of the art across all the benchmarks. UNITER-base model also outperforms the others by a large margin across all tasks except VQA. Specifically, our UNITER-base model outperforms SOTA by approximately \(+2.8\%\) for VCR on Q\(\rightarrow\)AR, \(+2.5\%\) for NLVR2, \(+7\%\) for SNLI-VE, \(+4\%\) on R@1 for Image-Text Retrieval (\(+15\%\) for zero-shot setting), and \(+2\%\) for RE Comprehension.

In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

For VCR, we propose a two-stage pre-training approach: (\(i\)) pre-train on standard pre-training datasets; and then (\(ii\)) pre-train on downstream VCR dataset. Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

We hypothesize that it is due to the fact that our UNITER is pre-trained with image-text pairs.

This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very different from pre-training tasks. In this paper, we present UNITER, a large-scale pre-trained model providing UNiversal Image-TExt Representations for Vision-and-Language tasks.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions.

The remaining images become the COCO subset within our full dataset, as shown in Figure 2 bottom row. We apply the same rules to Visual Genome, Conceptual Captions, and SBU Captions.

The'modality embedding' is extended to help distinguish question, answer and rationale.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.

Natural Language for Visual Reasoning for Real (NLVR2) NLVR2 is a new challenging task for visual reasoning. The goal is to determine whether a natural language statement is true about the given image pair.

Since UNITER only handles one image and one text input at pre-training, the'modality embedding' is extended to help distinguish the additional image presented in the NLVR2 task.

Two datasets are considered for this task: COCO and Flickr30K. COCO consists of \(123\)K images, each accompanied with five human-written captions.

Flickr30K dataset contains 31K images collected from the Flickr website, with five textual descriptions per image.

Visual Entailment (SNLI-VE) Visual Entailment is a task derived from Flickr30K images and Stanford Natural Language Inference (SNLI) dataset, where the goal is to determine the logical relationship between a natural language statement and an image. Similar to BERT for Natural Language Inference (NLI), we treat SNLI-VE as a three-way classification problem and apply an MLP Transform on [CLS] output. The UNITER model is finetuned using cross-entropy loss.

We use three referring expression datasets: RefCOCO, RefCOCO+, and RefCOCOg for the evaluation, all collected on COCO images. To finetune UNITER on this task, we add a MLP layer on top of the region outputs from Transformer, to compute the alignment score between the query phrase/sentence and each region.

The finetuning is efficient - we train the model with a batch size of 64 examples and a learning rate of \(5e-5\) for only 5 epochs, and achieve state-of-the-art performance.

Similar to Kovaleva et al. (2019), we observe several patterns in the attention maps of the UNITER model, as shown in Fig. 3.

Note that _Reversed Block_ (Fig. 3f) shows cross-modality alignment between tokens and regions.

Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

Table. 7 shows the comparison between VLBERT, ViLBERT and UNITER on VCR. The \(Q\to AR\) accuracy of our ensemble model outperforms ViLBERT (Lu et al., 2019) ensemble by a large margin of 7.0%. Note even single UNITER-large already outperforms ViLBERT ensemble and VLBERT-large by 3.0%. Besides, we also compare our UNITER-large with LXMERT (Tan and Bansal, 2019) and VisualBERT (Li et al., 2019) on an additional testing split of NLVR2 in Table. 8. Our results consistently outperform the previous SOTA on all metrics19 by a large margin of \(\sim\)4.0%.

To further demonstrate our idea, we conduct a direct comparison to ViLBERT (Lu et al., 2019) and VLBERT (Su et al., 2019), trained on Conceptual Captions (Sharma et al., 2018). We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR). Table. 9 shows that UNITER still consistently outperforms the other models by a visible margin on VQA and RefCOCO+. \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline \multicolumn{2}{c}{Task} & \multicolumn{2}{c}{Datasets} & \multicolumn{2}{c}{Image Src.} & \multicolumn{2}{c}{\#Images} & \multicolumn{2}{c}{\#Text} & \multicolumn{1}{l}{Metric} \\ \hline
1 & VQA & VQA & COCO & 204K & 1.1M & VQA-score \\ \hline
2 & VCR & VCR & Movie Clips & 110K & 290K & Accuracy \\ \hline
3 & NLVR2  & NUVR2  \\ \hline
4 & Visual Entailment & SNLI-VE & Flickr30K & 31K & 507K & Accuracy \\ \hline
5 & Image-Text Retrieval & COCO & COCO & 92K & 460K & Recall@1,5,10 \\  & & Flickr30K & Flickr30K & 32K & 160K & \\ \hline \multirow{3}{*}{6} & RefCOCO & \multirow{3}{*}{COCO} & 20K & 142K & \multirow{3}{*}{Accuracy} \\  & & RefCOCO+ & & 20K & 142K & \\ \cline{1-1}  & RefCOCOg & & 26K & 95K & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics on the datasets of downstream tasks.

 \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input: Figure 2: Different data splits from downstream tasks based on COCO images. Our UNITER pretraining avoids seeing any downstream evaluation images.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 2: Statistics on the datasets of downstream tasks.

Input: Figure 4: Text-to-image attention visualization example.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
The claim appears to be well-supported by the extensive evidence provided. The paper demonstrates:
- Clear architecture details of UNITER and its components
- Comprehensive pre-training on 4 major datasets (COCO, Visual Genome, Conceptual Captions, SBU Captions)
- Strong empirical results showing SOTA performance across 6 diverse V+L tasks
- Thorough ablation studies examining contributions of different pre-training tasks
- Detailed comparison with other models showing UNITER's advantages

The claim is measured in scope, accurately reflecting UNITER's capabilities and the evidence presented. It avoids exaggeration while highlighting the model's key contributions and versatility across downstream tasks.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.

The review comment to be evaluated is:

This paper presents a novel method for image-text representations called UNITER. The proposed method has been subsequently tested in many downstream tasks. A detailed ablation study helps to understand the role of each pretrained task in the proposed model.

Although the empirical results are nice, performing the intensive set of experiments on many different tasks is definitely time-consuming and needs a lot of engineering efforts, the technical contribution does not seem significant to me. The paper modifies an existing pre-training procedure by conditional masking (Section 2). I agree this is well-motivated but it has little novelty and a similar idea is there in VQA (See “Dynamic fusion with intra and inter-modality attention flow for visual question answering”). MLM and MRM are not new training procedure either, they are basically extending the BERT’s training procedure with the consideration of multiple modalities.

I have some questions for the authors:
(1) What are the advantages of using single-stream transformer over two-stream transformer (page 2). I guess it leads to fewer parameters but I don’t think this is a big problem.
(2) Some visualization of attention weights would be helpful.
Minor
•	In “m \e N^M” (equation 1), what is N and M?

The evidence to be evaluated is:More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training on large-scale image/video and text pairs, then finetuning on downstream tasks.

Our UNITER model belongs to the second family. One key difference between UNITER and the other methods is the masking approach on pre-training tasks.

In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

Compared with Gao et al. (2019), UNITER learns a relatively more generic V+L representation via pre-training. In this section, we first introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).

Given a pair of image and sentence, UNITER takes the visual regions of the image and textual tokens of the sentence as the input. We design an Image Embedder and a Text Embedder to extract their respective embeddings. These embeddings are then feed it into a multi-layer self-attention Transformer to learn a cross-modality contextualized embedding between visual regions and textual tokens.

Specifically, in _Image Embedder_, we first use Faster R-CNN1 to extract the visual features (pooled ROI features) for each region. We also encode the location features for each region via a 7-dimensional vector2. Both visual and location features are then fed through a fully-connected (FC) layer, to be projected into the same embedding space. The final visual embedding for each region is obtained by summing up the two FC outputs and then passing through a layer normalization (LN) layer. For _Text Embedder_, we follow BERT (Devlin et al., 2019) and tokenize the input sentence into WordPieces (Wu et al., 2016). The final representation for each sub-word token3 is obtained via summing up its word embedding and position embedding, followed by another LN layer4.

We also use a special modality embedding to help the model distinguish between textual and visual input, which is similar to the ‘segment embedding’ in BERT.

We introduce three main tasks to pre-train our model: Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM). As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer. Specifically, word masking is realized by replacing the token with a special token [MASK], and region masking is implemented by replacing the visual feature vector with all zeros. Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word.

Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores. To pre-train UNITER with the aforementioned different tasks, we randomly sample one pre-training task for each mini-batch and train on only one objective per SGD update. Masked Language Modeling (MLM) We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

\[\mathcal{L}_{\text{MLM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}\log P_{ \theta}(\mathbf{w_{m}}|\mathbf{w_{\setminus m}},\mathbf{v}). \tag{1}\] where \(\theta\) is the trainable parameters. Each pair \((\mathbf{w},\mathbf{v})\) is sampled from the whole training set \(D\).

In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1.

During training, we sample a positive or negative pair \((\mathbf{w},\mathbf{v})\) from the dataset \(D\) at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples. We denote the label as \(y\in\{0,1\}\), indicating if the sampled pair is a match.

We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011). Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.

In this way, we obtain 5.6M image-text pairs for training and 131K image-text pairs for our internal validation, which is half the size of the dataset used in LXMERT (Tan and Bansal, 2019), due to the filtering of overlapping images and the use of image-text pairs only. We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training9.

We evaluate UNITER on six V+L tasks (listed in Table 2), by transferring the pre-trained model to each target task and finetuning through end-to-end training. We report experimental results on two model sizes: UNITER-base with 12 layers and UNITER-large with 24 layers10. In VQA, VCR and NLVR\({}^{2}\) tasks, given an input image (or a pair of images) and a natural language question (or description), the model predicts an answer (or judges the correctness of the description) based on the visual content in the image.

For Image-Text Retrieval, we consider two datasets (COCO and Flickr30K) and evaluate the model in two settings: Image Retrieval (IR) and Text Retrieval (TR). Referring Expression (RE) Comprehension requires the model to select the target from a set of image region proposals given the query description.

For VQA, VCR, NLVR\({}^{2}\), Visual Entailment and Image-Text Retrieval, we extract the joint embedding of the input image-text pairs via a multi-layer perceptron (MLP) from the representation of the [CLS] token. For RE Comprehension, we use the MLP to compute the region-wise alignment scores. These MLP layers are learned during the finetuning stage. Specifically, we formulate VQA, VCR, NLVR\({}^{2}\), Visual Entailment and RE Comprehension as classification problems and minimize the cross-entropy loss over the ground-truth answers/responses. For Image-Text Retrieval, we formulate it as a ranking problem. During finetuning, we sample three pairs of image and text, one positive pair from the dataset and two negative pairs by randomly replacing its sentence/image with others. We compute the similarity scores (based on the joint embedding) for both positive and negative pairs, and maximize the margin between them through triplet loss. We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks.

In this study, we pre-train our model on Out-of-domain data (Conceptual Captions Table 4 presents the results of UNITER on all downstream tasks. Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

SOTA task-specific models include: MCAN (Yu et al., 2019) for VQA, MaxEnt (Suhr et al., 2019) for NLVR2, 2BT2 (Albert et al., 2019) for VCR, SCAN (Lee et al., 2018) for Image-Text Retrieval, EVE-Image (Xie et al., 2019) for SNLI-VE, and MAttNet for RE Comprehension (RefCOCO, ReFCOCO+ and ReFCOCOg)12. Concurrent pre-trained models include: ViLBERT, LXBERT, Unicoder-VL, VisualBERT and VLBERT.

Results show that our UNITER-large model achieves new state of the art across all the benchmarks. UNITER-base model also outperforms the others by a large margin across all tasks except VQA. Specifically, our UNITER-base model outperforms SOTA by approximately \(+2.8\%\) for VCR on Q\(\rightarrow\)AR, \(+2.5\%\) for NLVR2, \(+7\%\) for SNLI-VE, \(+4\%\) on R@1 for Image-Text Retrieval (\(+15\%\) for zero-shot setting), and \(+2\%\) for RE Comprehension.

In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

For VCR, we propose a two-stage pre-training approach: (\(i\)) pre-train on standard pre-training datasets; and then (\(ii\)) pre-train on downstream VCR dataset. Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

We hypothesize that it is due to the fact that our UNITER is pre-trained with image-text pairs.

This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very different from pre-training tasks. In this paper, we present UNITER, a large-scale pre-trained model providing UNiversal Image-TExt Representations for Vision-and-Language tasks.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions.

The remaining images become the COCO subset within our full dataset, as shown in Figure 2 bottom row. We apply the same rules to Visual Genome, Conceptual Captions, and SBU Captions.

The'modality embedding' is extended to help distinguish question, answer and rationale.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.

Natural Language for Visual Reasoning for Real (NLVR2) NLVR2 is a new challenging task for visual reasoning. The goal is to determine whether a natural language statement is true about the given image pair.

Since UNITER only handles one image and one text input at pre-training, the'modality embedding' is extended to help distinguish the additional image presented in the NLVR2 task.

Two datasets are considered for this task: COCO and Flickr30K. COCO consists of \(123\)K images, each accompanied with five human-written captions.

Flickr30K dataset contains 31K images collected from the Flickr website, with five textual descriptions per image.

Visual Entailment (SNLI-VE) Visual Entailment is a task derived from Flickr30K images and Stanford Natural Language Inference (SNLI) dataset, where the goal is to determine the logical relationship between a natural language statement and an image. Similar to BERT for Natural Language Inference (NLI), we treat SNLI-VE as a three-way classification problem and apply an MLP Transform on [CLS] output. The UNITER model is finetuned using cross-entropy loss.

We use three referring expression datasets: RefCOCO, RefCOCO+, and RefCOCOg for the evaluation, all collected on COCO images. To finetune UNITER on this task, we add a MLP layer on top of the region outputs from Transformer, to compute the alignment score between the query phrase/sentence and each region.

The finetuning is efficient - we train the model with a batch size of 64 examples and a learning rate of \(5e-5\) for only 5 epochs, and achieve state-of-the-art performance.

Similar to Kovaleva et al. (2019), we observe several patterns in the attention maps of the UNITER model, as shown in Fig. 3.

Note that _Reversed Block_ (Fig. 3f) shows cross-modality alignment between tokens and regions.

Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

Table. 7 shows the comparison between VLBERT, ViLBERT and UNITER on VCR. The \(Q\to AR\) accuracy of our ensemble model outperforms ViLBERT (Lu et al., 2019) ensemble by a large margin of 7.0%. Note even single UNITER-large already outperforms ViLBERT ensemble and VLBERT-large by 3.0%. Besides, we also compare our UNITER-large with LXMERT (Tan and Bansal, 2019) and VisualBERT (Li et al., 2019) on an additional testing split of NLVR2 in Table. 8. Our results consistently outperform the previous SOTA on all metrics19 by a large margin of \(\sim\)4.0%.

To further demonstrate our idea, we conduct a direct comparison to ViLBERT (Lu et al., 2019) and VLBERT (Su et al., 2019), trained on Conceptual Captions (Sharma et al., 2018). We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR). Table. 9 shows that UNITER still consistently outperforms the other models by a visible margin on VQA and RefCOCO+. \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline \multicolumn{2}{c}{Task} & \multicolumn{2}{c}{Datasets} & \multicolumn{2}{c}{Image Src.} & \multicolumn{2}{c}{\#Images} & \multicolumn{2}{c}{\#Text} & \multicolumn{1}{l}{Metric} \\ \hline
1 & VQA & VQA & COCO & 204K & 1.1M & VQA-score \\ \hline
2 & VCR & VCR & Movie Clips & 110K & 290K & Accuracy \\ \hline
3 & NLVR2  & NUVR2  \\ \hline
4 & Visual Entailment & SNLI-VE & Flickr30K & 31K & 507K & Accuracy \\ \hline
5 & Image-Text Retrieval & COCO & COCO & 92K & 460K & Recall@1,5,10 \\  & & Flickr30K & Flickr30K & 32K & 160K & \\ \hline \multirow{3}{*}{6} & RefCOCO & \multirow{3}{*}{COCO} & 20K & 142K & \multirow{3}{*}{Accuracy} \\  & & RefCOCO+ & & 20K & 142K & \\ \cline{1-1}  & RefCOCOg & & 26K & 95K & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics on the datasets of downstream tasks.

 \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input: Figure 2: Different data splits from downstream tasks based on COCO images. Our UNITER pretraining avoids seeing any downstream evaluation images.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 2: Statistics on the datasets of downstream tasks.

Input: Figure 4: Text-to-image attention visualization example.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that UNITER's introduction is generally well-supported by extensive experimental evidence and clear explanations of its architecture, pre-training approach, and performance across multiple tasks. The evidence shows:

- Comprehensive pre-training on 4 datasets with clear methodology
- State-of-the-art results across 6 downstream tasks
- Detailed ablation studies validating design choices
- Performance improvements over existing models with fewer parameters
- Clear explanations of technical contributions around conditional masking

While the paper makes bold claims about UNITER's capabilities, these are largely substantiated by thorough empirical results and detailed technical analysis. The claim appropriately reflects the scope and significance of the work without major exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.

The review comment to be evaluated is:

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).

I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.

Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.

# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.

The evidence to be evaluated is:More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training on large-scale image/video and text pairs, then finetuning on downstream tasks.

Our UNITER model belongs to the second family. One key difference between UNITER and the other methods is the masking approach on pre-training tasks.

In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

Compared with Gao et al. (2019), UNITER learns a relatively more generic V+L representation via pre-training. In this section, we first introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).

Given a pair of image and sentence, UNITER takes the visual regions of the image and textual tokens of the sentence as the input. We design an Image Embedder and a Text Embedder to extract their respective embeddings. These embeddings are then feed it into a multi-layer self-attention Transformer to learn a cross-modality contextualized embedding between visual regions and textual tokens.

Specifically, in _Image Embedder_, we first use Faster R-CNN1 to extract the visual features (pooled ROI features) for each region. We also encode the location features for each region via a 7-dimensional vector2. Both visual and location features are then fed through a fully-connected (FC) layer, to be projected into the same embedding space. The final visual embedding for each region is obtained by summing up the two FC outputs and then passing through a layer normalization (LN) layer. For _Text Embedder_, we follow BERT (Devlin et al., 2019) and tokenize the input sentence into WordPieces (Wu et al., 2016). The final representation for each sub-word token3 is obtained via summing up its word embedding and position embedding, followed by another LN layer4.

We also use a special modality embedding to help the model distinguish between textual and visual input, which is similar to the ‘segment embedding’ in BERT.

We introduce three main tasks to pre-train our model: Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM). As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer. Specifically, word masking is realized by replacing the token with a special token [MASK], and region masking is implemented by replacing the visual feature vector with all zeros. Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word.

Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores. To pre-train UNITER with the aforementioned different tasks, we randomly sample one pre-training task for each mini-batch and train on only one objective per SGD update. Masked Language Modeling (MLM) We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

\[\mathcal{L}_{\text{MLM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}\log P_{ \theta}(\mathbf{w_{m}}|\mathbf{w_{\setminus m}},\mathbf{v}). \tag{1}\] where \(\theta\) is the trainable parameters. Each pair \((\mathbf{w},\mathbf{v})\) is sampled from the whole training set \(D\).

In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1.

During training, we sample a positive or negative pair \((\mathbf{w},\mathbf{v})\) from the dataset \(D\) at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples. We denote the label as \(y\in\{0,1\}\), indicating if the sampled pair is a match.

We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011). Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.

In this way, we obtain 5.6M image-text pairs for training and 131K image-text pairs for our internal validation, which is half the size of the dataset used in LXMERT (Tan and Bansal, 2019), due to the filtering of overlapping images and the use of image-text pairs only. We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training9.

We evaluate UNITER on six V+L tasks (listed in Table 2), by transferring the pre-trained model to each target task and finetuning through end-to-end training. We report experimental results on two model sizes: UNITER-base with 12 layers and UNITER-large with 24 layers10. In VQA, VCR and NLVR\({}^{2}\) tasks, given an input image (or a pair of images) and a natural language question (or description), the model predicts an answer (or judges the correctness of the description) based on the visual content in the image.

For Image-Text Retrieval, we consider two datasets (COCO and Flickr30K) and evaluate the model in two settings: Image Retrieval (IR) and Text Retrieval (TR). Referring Expression (RE) Comprehension requires the model to select the target from a set of image region proposals given the query description.

For VQA, VCR, NLVR\({}^{2}\), Visual Entailment and Image-Text Retrieval, we extract the joint embedding of the input image-text pairs via a multi-layer perceptron (MLP) from the representation of the [CLS] token. For RE Comprehension, we use the MLP to compute the region-wise alignment scores. These MLP layers are learned during the finetuning stage. Specifically, we formulate VQA, VCR, NLVR\({}^{2}\), Visual Entailment and RE Comprehension as classification problems and minimize the cross-entropy loss over the ground-truth answers/responses. For Image-Text Retrieval, we formulate it as a ranking problem. During finetuning, we sample three pairs of image and text, one positive pair from the dataset and two negative pairs by randomly replacing its sentence/image with others. We compute the similarity scores (based on the joint embedding) for both positive and negative pairs, and maximize the margin between them through triplet loss. We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks.

In this study, we pre-train our model on Out-of-domain data (Conceptual Captions Table 4 presents the results of UNITER on all downstream tasks. Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

SOTA task-specific models include: MCAN (Yu et al., 2019) for VQA, MaxEnt (Suhr et al., 2019) for NLVR2, 2BT2 (Albert et al., 2019) for VCR, SCAN (Lee et al., 2018) for Image-Text Retrieval, EVE-Image (Xie et al., 2019) for SNLI-VE, and MAttNet for RE Comprehension (RefCOCO, ReFCOCO+ and ReFCOCOg)12. Concurrent pre-trained models include: ViLBERT, LXBERT, Unicoder-VL, VisualBERT and VLBERT.

Results show that our UNITER-large model achieves new state of the art across all the benchmarks. UNITER-base model also outperforms the others by a large margin across all tasks except VQA. Specifically, our UNITER-base model outperforms SOTA by approximately \(+2.8\%\) for VCR on Q\(\rightarrow\)AR, \(+2.5\%\) for NLVR2, \(+7\%\) for SNLI-VE, \(+4\%\) on R@1 for Image-Text Retrieval (\(+15\%\) for zero-shot setting), and \(+2\%\) for RE Comprehension.

In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

For VCR, we propose a two-stage pre-training approach: (\(i\)) pre-train on standard pre-training datasets; and then (\(ii\)) pre-train on downstream VCR dataset. Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

We hypothesize that it is due to the fact that our UNITER is pre-trained with image-text pairs.

This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very different from pre-training tasks. In this paper, we present UNITER, a large-scale pre-trained model providing UNiversal Image-TExt Representations for Vision-and-Language tasks.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions.

The remaining images become the COCO subset within our full dataset, as shown in Figure 2 bottom row. We apply the same rules to Visual Genome, Conceptual Captions, and SBU Captions.

The'modality embedding' is extended to help distinguish question, answer and rationale.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.

Natural Language for Visual Reasoning for Real (NLVR2) NLVR2 is a new challenging task for visual reasoning. The goal is to determine whether a natural language statement is true about the given image pair.

Since UNITER only handles one image and one text input at pre-training, the'modality embedding' is extended to help distinguish the additional image presented in the NLVR2 task.

Two datasets are considered for this task: COCO and Flickr30K. COCO consists of \(123\)K images, each accompanied with five human-written captions.

Flickr30K dataset contains 31K images collected from the Flickr website, with five textual descriptions per image.

Visual Entailment (SNLI-VE) Visual Entailment is a task derived from Flickr30K images and Stanford Natural Language Inference (SNLI) dataset, where the goal is to determine the logical relationship between a natural language statement and an image. Similar to BERT for Natural Language Inference (NLI), we treat SNLI-VE as a three-way classification problem and apply an MLP Transform on [CLS] output. The UNITER model is finetuned using cross-entropy loss.

We use three referring expression datasets: RefCOCO, RefCOCO+, and RefCOCOg for the evaluation, all collected on COCO images. To finetune UNITER on this task, we add a MLP layer on top of the region outputs from Transformer, to compute the alignment score between the query phrase/sentence and each region.

The finetuning is efficient - we train the model with a batch size of 64 examples and a learning rate of \(5e-5\) for only 5 epochs, and achieve state-of-the-art performance.

Similar to Kovaleva et al. (2019), we observe several patterns in the attention maps of the UNITER model, as shown in Fig. 3.

Note that _Reversed Block_ (Fig. 3f) shows cross-modality alignment between tokens and regions.

Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

Table. 7 shows the comparison between VLBERT, ViLBERT and UNITER on VCR. The \(Q\to AR\) accuracy of our ensemble model outperforms ViLBERT (Lu et al., 2019) ensemble by a large margin of 7.0%. Note even single UNITER-large already outperforms ViLBERT ensemble and VLBERT-large by 3.0%. Besides, we also compare our UNITER-large with LXMERT (Tan and Bansal, 2019) and VisualBERT (Li et al., 2019) on an additional testing split of NLVR2 in Table. 8. Our results consistently outperform the previous SOTA on all metrics19 by a large margin of \(\sim\)4.0%.

To further demonstrate our idea, we conduct a direct comparison to ViLBERT (Lu et al., 2019) and VLBERT (Su et al., 2019), trained on Conceptual Captions (Sharma et al., 2018). We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR). Table. 9 shows that UNITER still consistently outperforms the other models by a visible margin on VQA and RefCOCO+. \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline \multicolumn{2}{c}{Task} & \multicolumn{2}{c}{Datasets} & \multicolumn{2}{c}{Image Src.} & \multicolumn{2}{c}{\#Images} & \multicolumn{2}{c}{\#Text} & \multicolumn{1}{l}{Metric} \\ \hline
1 & VQA & VQA & COCO & 204K & 1.1M & VQA-score \\ \hline
2 & VCR & VCR & Movie Clips & 110K & 290K & Accuracy \\ \hline
3 & NLVR2  & NUVR2  \\ \hline
4 & Visual Entailment & SNLI-VE & Flickr30K & 31K & 507K & Accuracy \\ \hline
5 & Image-Text Retrieval & COCO & COCO & 92K & 460K & Recall@1,5,10 \\  & & Flickr30K & Flickr30K & 32K & 160K & \\ \hline \multirow{3}{*}{6} & RefCOCO & \multirow{3}{*}{COCO} & 20K & 142K & \multirow{3}{*}{Accuracy} \\  & & RefCOCO+ & & 20K & 142K & \\ \cline{1-1}  & RefCOCOg & & 26K & 95K & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics on the datasets of downstream tasks.

 \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input: Figure 2: Different data splits from downstream tasks based on COCO images. Our UNITER pretraining avoids seeing any downstream evaluation images.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 2: Statistics on the datasets of downstream tasks.

Input: Figure 4: Text-to-image attention visualization example.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, the claim appears to be well-stated and accurately reflects what is demonstrated in the paper. The evidence shows that:

1. UNITER was indeed pre-trained on four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions)
2. The results demonstrate strong performance across multiple downstream V+L tasks including VQA, VCR, NLVR2, Visual Entailment, Image-Text Retrieval and RE Comprehension
3. The model architecture and training approach are clearly described and supported by experimental results
4. Performance improvements over SOTA are quantified and validated

The claim avoids overstating capabilities while accurately describing the model's core contribution and empirical validation. It sticks to factual statements that are directly supported by the evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training on large-scale image/video and text pairs, then finetuning on downstream tasks.

Our UNITER model belongs to the second family. One key difference between UNITER and the other methods is the masking approach on pre-training tasks.

In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

Compared with Gao et al. (2019), UNITER learns a relatively more generic V+L representation via pre-training. In this section, we first introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).

Given a pair of image and sentence, UNITER takes the visual regions of the image and textual tokens of the sentence as the input. We design an Image Embedder and a Text Embedder to extract their respective embeddings. These embeddings are then feed it into a multi-layer self-attention Transformer to learn a cross-modality contextualized embedding between visual regions and textual tokens.

Specifically, in _Image Embedder_, we first use Faster R-CNN1 to extract the visual features (pooled ROI features) for each region. We also encode the location features for each region via a 7-dimensional vector2. Both visual and location features are then fed through a fully-connected (FC) layer, to be projected into the same embedding space. The final visual embedding for each region is obtained by summing up the two FC outputs and then passing through a layer normalization (LN) layer. For _Text Embedder_, we follow BERT (Devlin et al., 2019) and tokenize the input sentence into WordPieces (Wu et al., 2016). The final representation for each sub-word token3 is obtained via summing up its word embedding and position embedding, followed by another LN layer4.

We also use a special modality embedding to help the model distinguish between textual and visual input, which is similar to the ‘segment embedding’ in BERT.

We introduce three main tasks to pre-train our model: Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM). As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer. Specifically, word masking is realized by replacing the token with a special token [MASK], and region masking is implemented by replacing the visual feature vector with all zeros. Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word.

Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores. To pre-train UNITER with the aforementioned different tasks, we randomly sample one pre-training task for each mini-batch and train on only one objective per SGD update. Masked Language Modeling (MLM) We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

\[\mathcal{L}_{\text{MLM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}\log P_{ \theta}(\mathbf{w_{m}}|\mathbf{w_{\setminus m}},\mathbf{v}). \tag{1}\] where \(\theta\) is the trainable parameters. Each pair \((\mathbf{w},\mathbf{v})\) is sampled from the whole training set \(D\).

In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1.

During training, we sample a positive or negative pair \((\mathbf{w},\mathbf{v})\) from the dataset \(D\) at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples. We denote the label as \(y\in\{0,1\}\), indicating if the sampled pair is a match.

We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014), Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and SBU Captions (Ordonez et al., 2011). Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.

In this way, we obtain 5.6M image-text pairs for training and 131K image-text pairs for our internal validation, which is half the size of the dataset used in LXMERT (Tan and Bansal, 2019), due to the filtering of overlapping images and the use of image-text pairs only. We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training9.

We evaluate UNITER on six V+L tasks (listed in Table 2), by transferring the pre-trained model to each target task and finetuning through end-to-end training. We report experimental results on two model sizes: UNITER-base with 12 layers and UNITER-large with 24 layers10. In VQA, VCR and NLVR\({}^{2}\) tasks, given an input image (or a pair of images) and a natural language question (or description), the model predicts an answer (or judges the correctness of the description) based on the visual content in the image.

For Image-Text Retrieval, we consider two datasets (COCO and Flickr30K) and evaluate the model in two settings: Image Retrieval (IR) and Text Retrieval (TR). Referring Expression (RE) Comprehension requires the model to select the target from a set of image region proposals given the query description.

For VQA, VCR, NLVR\({}^{2}\), Visual Entailment and Image-Text Retrieval, we extract the joint embedding of the input image-text pairs via a multi-layer perceptron (MLP) from the representation of the [CLS] token. For RE Comprehension, we use the MLP to compute the region-wise alignment scores. These MLP layers are learned during the finetuning stage. Specifically, we formulate VQA, VCR, NLVR\({}^{2}\), Visual Entailment and RE Comprehension as classification problems and minimize the cross-entropy loss over the ground-truth answers/responses. For Image-Text Retrieval, we formulate it as a ranking problem. During finetuning, we sample three pairs of image and text, one positive pair from the dataset and two negative pairs by randomly replacing its sentence/image with others. We compute the similarity scores (based on the joint embedding) for both positive and negative pairs, and maximize the margin between them through triplet loss. We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR\({}^{2}\), Flickr30K and RefCOCO+ as representative V+L benchmarks.

In this study, we pre-train our model on Out-of-domain data (Conceptual Captions Table 4 presents the results of UNITER on all downstream tasks. Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

SOTA task-specific models include: MCAN (Yu et al., 2019) for VQA, MaxEnt (Suhr et al., 2019) for NLVR2, 2BT2 (Albert et al., 2019) for VCR, SCAN (Lee et al., 2018) for Image-Text Retrieval, EVE-Image (Xie et al., 2019) for SNLI-VE, and MAttNet for RE Comprehension (RefCOCO, ReFCOCO+ and ReFCOCOg)12. Concurrent pre-trained models include: ViLBERT, LXBERT, Unicoder-VL, VisualBERT and VLBERT.

Results show that our UNITER-large model achieves new state of the art across all the benchmarks. UNITER-base model also outperforms the others by a large margin across all tasks except VQA. Specifically, our UNITER-base model outperforms SOTA by approximately \(+2.8\%\) for VCR on Q\(\rightarrow\)AR, \(+2.5\%\) for NLVR2, \(+7\%\) for SNLI-VE, \(+4\%\) on R@1 for Image-Text Retrieval (\(+15\%\) for zero-shot setting), and \(+2\%\) for RE Comprehension.

In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

For VCR, we propose a two-stage pre-training approach: (\(i\)) pre-train on standard pre-training datasets; and then (\(ii\)) pre-train on downstream VCR dataset. Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

We hypothesize that it is due to the fact that our UNITER is pre-trained with image-text pairs.

This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very different from pre-training tasks. In this paper, we present UNITER, a large-scale pre-trained model providing UNiversal Image-TExt Representations for Vision-and-Language tasks.

Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a significant margin.

As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions.

The remaining images become the COCO subset within our full dataset, as shown in Figure 2 bottom row. We apply the same rules to Visual Genome, Conceptual Captions, and SBU Captions.

The'modality embedding' is extended to help distinguish question, answer and rationale.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.

Natural Language for Visual Reasoning for Real (NLVR2) NLVR2 is a new challenging task for visual reasoning. The goal is to determine whether a natural language statement is true about the given image pair.

Since UNITER only handles one image and one text input at pre-training, the'modality embedding' is extended to help distinguish the additional image presented in the NLVR2 task.

Two datasets are considered for this task: COCO and Flickr30K. COCO consists of \(123\)K images, each accompanied with five human-written captions.

Flickr30K dataset contains 31K images collected from the Flickr website, with five textual descriptions per image.

Visual Entailment (SNLI-VE) Visual Entailment is a task derived from Flickr30K images and Stanford Natural Language Inference (SNLI) dataset, where the goal is to determine the logical relationship between a natural language statement and an image. Similar to BERT for Natural Language Inference (NLI), we treat SNLI-VE as a three-way classification problem and apply an MLP Transform on [CLS] output. The UNITER model is finetuned using cross-entropy loss.

We use three referring expression datasets: RefCOCO, RefCOCO+, and RefCOCOg for the evaluation, all collected on COCO images. To finetune UNITER on this task, we add a MLP layer on top of the region outputs from Transformer, to compute the alignment score between the query phrase/sentence and each region.

The finetuning is efficient - we train the model with a batch size of 64 examples and a learning rate of \(5e-5\) for only 5 epochs, and achieve state-of-the-art performance.

Similar to Kovaleva et al. (2019), we observe several patterns in the attention maps of the UNITER model, as shown in Fig. 3.

Note that _Reversed Block_ (Fig. 3f) shows cross-modality alignment between tokens and regions.

Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

Table. 7 shows the comparison between VLBERT, ViLBERT and UNITER on VCR. The \(Q\to AR\) accuracy of our ensemble model outperforms ViLBERT (Lu et al., 2019) ensemble by a large margin of 7.0%. Note even single UNITER-large already outperforms ViLBERT ensemble and VLBERT-large by 3.0%. Besides, we also compare our UNITER-large with LXMERT (Tan and Bansal, 2019) and VisualBERT (Li et al., 2019) on an additional testing split of NLVR2 in Table. 8. Our results consistently outperform the previous SOTA on all metrics19 by a large margin of \(\sim\)4.0%.

To further demonstrate our idea, we conduct a direct comparison to ViLBERT (Lu et al., 2019) and VLBERT (Su et al., 2019), trained on Conceptual Captions (Sharma et al., 2018). We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR). Table. 9 shows that UNITER still consistently outperforms the other models by a visible margin on VQA and RefCOCO+. \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline \multicolumn{2}{c}{Task} & \multicolumn{2}{c}{Datasets} & \multicolumn{2}{c}{Image Src.} & \multicolumn{2}{c}{\#Images} & \multicolumn{2}{c}{\#Text} & \multicolumn{1}{l}{Metric} \\ \hline
1 & VQA & VQA & COCO & 204K & 1.1M & VQA-score \\ \hline
2 & VCR & VCR & Movie Clips & 110K & 290K & Accuracy \\ \hline
3 & NLVR2  & NUVR2  \\ \hline
4 & Visual Entailment & SNLI-VE & Flickr30K & 31K & 507K & Accuracy \\ \hline
5 & Image-Text Retrieval & COCO & COCO & 92K & 460K & Recall@1,5,10 \\  & & Flickr30K & Flickr30K & 32K & 160K & \\ \hline \multirow{3}{*}{6} & RefCOCO & \multirow{3}{*}{COCO} & 20K & 142K & \multirow{3}{*}{Accuracy} \\  & & RefCOCO+ & & 20K & 142K & \\ \cline{1-1}  & RefCOCOg & & 26K & 95K & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics on the datasets of downstream tasks.

 \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input: Figure 2: Different data splits from downstream tasks based on COCO images. Our UNITER pretraining avoids seeing any downstream evaluation images.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 2: Statistics on the datasets of downstream tasks.

Input: Figure 4: Text-to-image attention visualization example.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the provided evidence and review comment:

The evidence shows UNITER achieving state-of-the-art results across multiple vision-language tasks through pre-training on 4 image-text datasets. The experimental results clearly demonstrate strong performance improvements over baseline models and prior work. However, as noted in the review comment, there are significant gaps in explaining WHY and HOW these results are achieved - the paper lacks deeper analysis of what is being learned and represented in the pre-trained network.

While the empirical results support the claim about UNITER's effectiveness, the claim could better acknowledge these limitations in understanding the underlying mechanisms. The lack of mechanistic insights makes the claim slightly overstated, though the strong empirical evidence prevents it from being severely overstated.

<score>2</score>