Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Besides, our masked language/region modeling is conditioned on full observation of image/text, different from other concurrent pre-trained models that apply joint random masking to both modalities.

The review comment to be evaluated is:

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).

I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.

Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.

# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.

The evidence to be evaluated is:More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training on large-scale image/video and text pairs, then finetuning on downstream tasks.

ViLBERT (Lu et al., 2019) and LXMERT (Tan & Bansal, 2019) introduced the two-stream architecture, where two Transformers are applied to images and text independently, which will be fused by a third Transformer in a later stage. On the other hand, VisualBERT (Li et al., 2019), Unicoder-VL (Li et al., 2019), VL-BERT (Su et al., 2019) and B2T2 (Alberti et al., 2019) proposed the single-stream architecture, where a single Transformer is applied to both image and text.

One key difference between UNITER and the other methods is the masking approach on pre-training tasks. Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim about conditional masking being different from concurrent models is well-stated and accurately supported by the evidence. The paper:

1. Clearly demonstrates their conditional masking approach (masking one modality while keeping the other intact) differs from ViLBERT and LXMERT's joint random masking
2. Provides empirical evidence showing better performance through:
- Higher Meta-Sum scores (399.97 vs 396.51)
- Faster convergence and higher accuracy on MLM and MRC-kl validation (Fig 8)
- Better downstream task performance (Table 3)
3. Explains the conceptual advantage of preventing misalignment between masked regions and words

The claim reflects the evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Besides, our masked language/region modeling is conditioned on full observation of image/text, different from other concurrent pre-trained models that apply joint random masking to both modalities.

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training on large-scale image/video and text pairs, then finetuning on downstream tasks.

ViLBERT (Lu et al., 2019) and LXMERT (Tan & Bansal, 2019) introduced the two-stream architecture, where two Transformers are applied to images and text independently, which will be fused by a third Transformer in a later stage. On the other hand, VisualBERT (Li et al., 2019), Unicoder-VL (Li et al., 2019), VL-BERT (Su et al., 2019) and B2T2 (Alberti et al., 2019) proposed the single-stream architecture, where a single Transformer is applied to both image and text.

One key difference between UNITER and the other methods is the masking approach on pre-training tasks. Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2).

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim that their "masked language/region modeling is conditioned on full observation of image/text, different from other concurrent pre-trained models that apply joint random masking to both modalities."

This claim appears well-stated (1) based on the following evidence:

1. The paper clearly demonstrates their conditional masking approach where they only mask one modality while keeping the other intact, which is explicitly different from other methods like ViLBERT and LXMERT that use joint random masking.

2. The evidence provides concrete empirical validation:
- Higher Meta-Sum score (399.97 vs 396.51) with conditional vs joint masking
- Better convergence and final accuracy shown in Figure 8 plots
- Superior downstream task performance (Table 3, rows 10 & 11)
- Clear explanation of the masking probability (15%) and process

The reviewer's comment about "why it works" doesn't invalidate the factual claim about their masking approach being different. The claim strictly describes their technical approach without overstating its benefits.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We show that the conditional masking strategy can successfully ease the missing-alignment between images and text, and obtain better joint embeddings for downstream tasks.

The evidence to be evaluated is:Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM).

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2). Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores.

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\).

MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\).

Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

Thus, in MRC-kl, we avoid this assumption by using soft label as supervision signal, which is the raw output from the detector (i.e., a distribution of object classes \(\tilde{c}(\mathbf{v}_{m}^{(i)})\)). MRC-kl aims to distill such knowledge into UNITER as Hinton et al. (2015), by minimizing the KL divergence between two distributions: \(f_{\theta}(\mathbf{v}_{\mathbf{m}}|\mathbf{v}_{\mathbf{m}},\mathbf{w})=\sum_{i= 1}^{M}D_{KL}(\tilde{c}(\mathbf{v}_{\mathbf{m}}^{(i)})||g_{\theta}(\mathbf{v}_{ \mathbf{m}}^{(i)}))\).

On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.

When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

Note that _Reversed Block_ (Fig. 3f) shows cross-modality alignment between tokens and regions. In Fig. 4, 5, and 6, we visualize several examples of text-to-image attention to demonstrate the local cross-modality alignment between regions and tokens. We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR). \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Figure 4: Text-to-image attention visualization example.

Input: Figure 5: Text-to-image attention visualization example.

Input: Figure 6: Text-to-image attention visualization example.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the conditional masking strategy's effectiveness is supported by multiple pieces of empirical evidence:

1. Direct performance comparisons show conditional masking achieves higher accuracy than joint random masking on both MLM and MRC-kl tasks (Fig 8 graphs)
2. Meta-Sum scores dropped from 399.97 to 396.51 without conditional masking (Table 3, rows 10 vs 11)
3. Visualization examples (Figs 4-6) demonstrate successful cross-modal alignment
4. Validation curves show faster convergence and better final performance with conditional masking
5. Downstream task improvements across multiple benchmarks

The claim is well-stated as it specifically references "easing missing-alignment" and "better joint embeddings" - both of which are directly supported by the comprehensive empirical results provided.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We show that the conditional masking strategy can successfully ease the missing-alignment between images and text, and obtain better joint embeddings for downstream tasks.

The review comment to be evaluated is:

This paper presents a novel method for image-text representations called UNITER. The proposed method has been subsequently tested in many downstream tasks. A detailed ablation study helps to understand the role of each pretrained task in the proposed model.

Although the empirical results are nice, performing the intensive set of experiments on many different tasks is definitely time-consuming and needs a lot of engineering efforts, the technical contribution does not seem significant to me. The paper modifies an existing pre-training procedure by conditional masking (Section 2). I agree this is well-motivated but it has little novelty and a similar idea is there in VQA (See “Dynamic fusion with intra and inter-modality attention flow for visual question answering”). MLM and MRM are not new training procedure either, they are basically extending the BERT’s training procedure with the consideration of multiple modalities.

I have some questions for the authors:
(1) What are the advantages of using single-stream transformer over two-stream transformer (page 2). I guess it leads to fewer parameters but I don’t think this is a big problem.
(2) Some visualization of attention weights would be helpful.
Minor
•	In “m \e N^M” (equation 1), what is N and M?

The evidence to be evaluated is:Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM).

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2). Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores.

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\).

MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\).

Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

Thus, in MRC-kl, we avoid this assumption by using soft label as supervision signal, which is the raw output from the detector (i.e., a distribution of object classes \(\tilde{c}(\mathbf{v}_{m}^{(i)})\)). MRC-kl aims to distill such knowledge into UNITER as Hinton et al. (2015), by minimizing the KL divergence between two distributions: \(f_{\theta}(\mathbf{v}_{\mathbf{m}}|\mathbf{v}_{\mathbf{m}},\mathbf{w})=\sum_{i= 1}^{M}D_{KL}(\tilde{c}(\mathbf{v}_{\mathbf{m}}^{(i)})||g_{\theta}(\mathbf{v}_{ \mathbf{m}}^{(i)}))\).

On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.

When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

Note that _Reversed Block_ (Fig. 3f) shows cross-modality alignment between tokens and regions. In Fig. 4, 5, and 6, we visualize several examples of text-to-image attention to demonstrate the local cross-modality alignment between regions and tokens. We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR). \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Figure 4: Text-to-image attention visualization example.

Input: Figure 5: Text-to-image attention visualization example.

Input: Figure 6: Text-to-image attention visualization example.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I'll evaluate the claim "We show that the conditional masking strategy can successfully ease the missing-alignment between images and text, and obtain better joint embeddings for downstream tasks."

The evidence strongly supports this claim through multiple angles:
1. Quantitative results show higher Meta-Sum scores with conditional masking (399.97) vs without (396.51)
2. Convergence graphs in Fig 8 demonstrate faster and better convergence with conditional masking
3. Downstream task performance improvements are documented in Table 3
4. Visual examples and explanations demonstrate how conditional masking helps with cross-modal alignment
5. The claim's wording is measured and supported by concrete evidence

The claim is well-stated as it accurately reflects the evidence without overreaching conclusions.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We show that the conditional masking strategy can successfully ease the missing-alignment between images and text, and obtain better joint embeddings for downstream tasks.

The review comment to be evaluated is:

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).

I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.

Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.

# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.

The evidence to be evaluated is:Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM).

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2). Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores.

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\).

MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\).

Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

Thus, in MRC-kl, we avoid this assumption by using soft label as supervision signal, which is the raw output from the detector (i.e., a distribution of object classes \(\tilde{c}(\mathbf{v}_{m}^{(i)})\)). MRC-kl aims to distill such knowledge into UNITER as Hinton et al. (2015), by minimizing the KL divergence between two distributions: \(f_{\theta}(\mathbf{v}_{\mathbf{m}}|\mathbf{v}_{\mathbf{m}},\mathbf{w})=\sum_{i= 1}^{M}D_{KL}(\tilde{c}(\mathbf{v}_{\mathbf{m}}^{(i)})||g_{\theta}(\mathbf{v}_{ \mathbf{m}}^{(i)}))\).

On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.

When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

Note that _Reversed Block_ (Fig. 3f) shows cross-modality alignment between tokens and regions. In Fig. 4, 5, and 6, we visualize several examples of text-to-image attention to demonstrate the local cross-modality alignment between regions and tokens. We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR). \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Figure 4: Text-to-image attention visualization example.

Input: Figure 5: Text-to-image attention visualization example.

Input: Figure 6: Text-to-image attention visualization example.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the comprehensive evidence provided, I will evaluate the claim "We show that the conditional masking strategy can successfully ease the missing-alignment between images and text, and obtain better joint embeddings for downstream tasks."

The evidence strongly supports this claim through:
- Empirical results showing conditional masking outperforms joint random masking in convergence speed and final accuracy (Fig. 8)
- Higher Meta-Sum scores with conditional masking (399.97) vs without (396.51)
- Better performance on downstream tasks (Table 3, rows 10 vs 11)
- Visualization examples demonstrating successful cross-modal alignment (Figs 4-7)
- Theoretical justification explaining why avoiding masked region-word pairs prevents misalignment

The claim is measured and well-supported by multiple pieces of concrete evidence, making it well-stated rather than overstated.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We show that the conditional masking strategy can successfully ease the missing-alignment between images and text, and obtain better joint embeddings for downstream tasks.

The review comment to be evaluated is:

This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks.
The major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?
This is really the major obscure point of this impressive paper.

The evidence to be evaluated is:Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM).

Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential miss-alignment when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings(in Section 4.2). Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores.

We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros.

MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\).

MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\).

Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\).

Thus, in MRC-kl, we avoid this assumption by using soft label as supervision signal, which is the raw output from the detector (i.e., a distribution of object classes \(\tilde{c}(\mathbf{v}_{m}^{(i)})\)). MRC-kl aims to distill such knowledge into UNITER as Hinton et al. (2015), by minimizing the KL divergence between two distributions: \(f_{\theta}(\mathbf{v}_{\mathbf{m}}|\mathbf{v}_{\mathbf{m}},\mathbf{w})=\sum_{i= 1}^{M}D_{KL}(\tilde{c}(\mathbf{v}_{\mathbf{m}}^{(i)})||g_{\theta}(\mathbf{v}_{ \mathbf{m}}^{(i)}))\).

On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.

When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by \(>\)\(1.5\%\) on VQA.

It is also worth mentioning that both ViLBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, ViLBERT: 221M)14.

Interestingly, while VLBERT and B2TZ observed that pre-training is not very helpful on VCR, we find that the second-stage pre-training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table 5). This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

Note that _Reversed Block_ (Fig. 3f) shows cross-modality alignment between tokens and regions. In Fig. 4, 5, and 6, we visualize several examples of text-to-image attention to demonstrate the local cross-modality alignment between regions and tokens. We further discuss the advantage of our proposed conditional masking over joint random masking used in (Tan and Bansal, 2019; Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

With conditional masking, when the region of dog is masked, our model should be able to infer that the region is dog, based on the context of both surrounding regions and the full sentence (Fig. 7(a)), and vice versa.

Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER.

In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR). \begin{table}
\begin{tabular}{c c c} \hline \hline Stage I & Stage II & Q\(\rightarrow\)A & QA\(\rightarrow\) R & Q \(\rightarrow\) AR \\ \hline N & N & 72.44 & 73.71 & 53.52 \\ N & Y & 73.52 & 75.34 & 55.6 \\ \hline Y & N & 72.83 & 75.25 & 54.94 \\ Y & Y & **74.56** & **77.03** & **57.76** \\ \hline \hline \end{tabular}
\begin{tabular}{c c} \hline \hline Setting & dev & test-P \\ \hline Triplet & 72.76 & 73.55 \\ Pair & 75.37 & 75.97 \\ \hline Pair-bitation & **77.14** & **77.87** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Table 5: Experiments on two-stage pre-training for VCR. Results are from UNITER-base on VCR val split. Stage I and Stage II denote first-stage and second-stage pre-training.

Input: Figure 4: Text-to-image attention visualization example.

Input: Figure 5: Text-to-image attention visualization example.

Input: Figure 6: Text-to-image attention visualization example.

Input: Table 4: Results on downstream V+L tasks from UNITER model, compared with task-specific state-of-the-art (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.

Input: Figure 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our proposed conditional masking.

Input: Figure 7: Example showing difference between conditional masking and joint random masking.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, the claim stating that "the conditional masking strategy can successfully ease the missing-alignment between images and text, and obtain better joint embeddings for downstream tasks" appears well-supported by multiple pieces of empirical evidence:

1. Table 3 shows higher Meta-Sum scores (399.97 vs 396.51) when using conditional masking vs random masking
2. Figure 8 demonstrates faster convergence and higher validation accuracy for conditional masking on both MLM and MRC-kl tasks
3. Visualization examples (Figures 4-6) show effective cross-modal alignment
4. The strategy leads to state-of-the-art results on multiple downstream tasks with fewer parameters than competing models
5. The evidence systematically demonstrates both the mechanism and benefits

The claim is measured and accurately reflects the evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We design three pre-training tasks: Masked Language Modeling (MLM), Image-Text Matching (ITM), and Masked Region Modeling (MRM, with three variants).

The evidence to be evaluated is:In this section, we first introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).

We introduce three main tasks to pre-train our model: Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM). As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer.

Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores.

Masked Language Modeling (MLM) We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

\[\mathcal{L}_{\text{MLM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}\log P_{ \theta}(\mathbf{w_{m}}|\mathbf{w_{\setminus m}},\mathbf{v}). \tag{1}\] where \(\theta\) is the trainable parameters. Each pair \((\mathbf{w},\mathbf{v})\) is sampled from the whole training set \(D\). Image-Text Matching (ITM) In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1. We denote the output score as \(s_{\theta}(\mathbf{w},\mathbf{v})\). The ITM supervision is over the [CLS] token.7 During training, we sample a positive or negative pair \((\mathbf{w},\mathbf{v})\) from the dataset \(D\) at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples. We denote the label as \(y\in\{0,1\}\), indicating if the sampled pair is a match.

\[\mathcal{L}_{\text{ITM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}[y\log s_{ \theta}(\mathbf{v},\mathbf{w})+(1-y)\log(1-s_{\theta}(\mathbf{v},\mathbf{w})) ]). \tag{2}\] Masked Region Modeling (MRM) Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros. Unlike textual tokens that are represented as discrete labels, visual features are high-dimensional and continuous, thus cannot be supervised via class likelihood. Instead, we propose three variants for Masked Region Modeling, which share the same objective base: \[\mathcal{L}_{\text{MRM}}(\theta)=E_{(\mathbf{w},\mathbf{v})\sim D}f_{\theta}( \mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w}). \tag{3}\] 1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). Note that there is no ground-truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\). 3) Masked Region Classification with KL-Divergence (MRC-kl) MRC takes the most likely object class from the object detection model as the hard label (w.p. 0 or 1), which assumes the detected object class is the ground-truth label for the region. However, this may not be true, as no ground-truth label is provided for the detected region. Thus, in MRC-kl, we avoid this assumption by using soft label as supervision signal, which is the raw output from the detector (i.e., a distribution of object classes \(\tilde{c}(\mathbf{v}_{m}^{(i)})\)). MRC-kl aims to distill such knowledge into UNITER as Hinton et al. (2015), by minimizing the KL divergence between two distributions: \(f_{\theta}(\mathbf{v}_{\mathbf{m}}|\mathbf{v}_{\mathbf{m}},\mathbf{w})=\sum_{i= 1}^{M}D_{KL}(\tilde{c}(\mathbf{v}_{\mathbf{m}}^{(i)})||g_{\theta}(\mathbf{v}_{ \mathbf{m}}^{(i)}))\).

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study.

On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks. ITM is discarded because the text in VCR does not explicitly describe the image.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence, this claim about designing three pre-training tasks is well-stated and directly supported. The evidence clearly describes MLM, ITM, and MRM (with its three variants: MRFR, MRC, and MRC-kl) in detail. Figure 1 provides a visual overview of these tasks. The evidence shows extensive ablation studies evaluating each task's effectiveness (Table 3), and demonstrates how these tasks work together. The claim is precise and matches what is actually presented in the methodology, without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We design three pre-training tasks: Masked Language Modeling (MLM), Image-Text Matching (ITM), and Masked Region Modeling (MRM, with three variants).

The review comment to be evaluated is:

This paper presents a novel method for image-text representations called UNITER. The proposed method has been subsequently tested in many downstream tasks. A detailed ablation study helps to understand the role of each pretrained task in the proposed model.

Although the empirical results are nice, performing the intensive set of experiments on many different tasks is definitely time-consuming and needs a lot of engineering efforts, the technical contribution does not seem significant to me. The paper modifies an existing pre-training procedure by conditional masking (Section 2). I agree this is well-motivated but it has little novelty and a similar idea is there in VQA (See “Dynamic fusion with intra and inter-modality attention flow for visual question answering”). MLM and MRM are not new training procedure either, they are basically extending the BERT’s training procedure with the consideration of multiple modalities.

I have some questions for the authors:
(1) What are the advantages of using single-stream transformer over two-stream transformer (page 2). I guess it leads to fewer parameters but I don’t think this is a big problem.
(2) Some visualization of attention weights would be helpful.
Minor
•	In “m \e N^M” (equation 1), what is N and M?

The evidence to be evaluated is:In this section, we first introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).

We introduce three main tasks to pre-train our model: Masked Language Modeling _conditioned on image regions_ (MLM), Masked Region Modeling _conditioned on input text_ (with three variants) (MRM), and Image-Text Matching (ITM). As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer.

Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores.

Masked Language Modeling (MLM) We denote the image regions as \(\mathbf{v}=\{v_{1},...,v_{K}\}\), the input words as \(\mathbf{w}=\{w_{1},...,w_{T}\}\), and the mask indices as \(\mathbf{m}\in\mathbb{N}^{M}\). 5 In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones \(\mathbf{w_{m}}\) with special token [MASK] 6. The goal is to predict these masked words based on the observation of their surrounding words \(\mathbf{w}_{\setminus m}\) and all image regions \(\mathbf{v}\), by minimizing the negative log-likelihood:

\[\mathcal{L}_{\text{MLM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}\log P_{ \theta}(\mathbf{w_{m}}|\mathbf{w_{\setminus m}},\mathbf{v}). \tag{1}\] where \(\theta\) is the trainable parameters. Each pair \((\mathbf{w},\mathbf{v})\) is sampled from the whole training set \(D\). Image-Text Matching (ITM) In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1. We denote the output score as \(s_{\theta}(\mathbf{w},\mathbf{v})\). The ITM supervision is over the [CLS] token.7 During training, we sample a positive or negative pair \((\mathbf{w},\mathbf{v})\) from the dataset \(D\) at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples. We denote the label as \(y\in\{0,1\}\), indicating if the sampled pair is a match.

\[\mathcal{L}_{\text{ITM}}(\theta)=-E_{(\mathbf{w},\mathbf{v})\sim D}[y\log s_{ \theta}(\mathbf{v},\mathbf{w})+(1-y)\log(1-s_{\theta}(\mathbf{v},\mathbf{w})) ]). \tag{2}\] Masked Region Modeling (MRM) Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions \(\mathbf{v_{m}}\) given the remaining regions \(\mathbf{v_{\setminus m}}\) and all the words \(\mathbf{w}\). The visual features \(v_{m}\) of the masked region are replaced by zeros. Unlike textual tokens that are represented as discrete labels, visual features are high-dimensional and continuous, thus cannot be supervised via class likelihood. Instead, we propose three variants for Masked Region Modeling, which share the same objective base: \[\mathcal{L}_{\text{MRM}}(\theta)=E_{(\mathbf{w},\mathbf{v})\sim D}f_{\theta}( \mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w}). \tag{3}\] 1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region \(\mathbf{v_{m}^{(i)}}\) to its visual features. Specifically, we apply an FC layer to convert its Transformer output into a vector \(h_{\theta}(\mathbf{v_{m}^{(i)}})\) of same dimension as the input ROI pooled feature \(r(\mathbf{v_{m}^{(i)}})\). Then we apply L2 regression between the two: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M }\|h_{\theta}(\mathbf{v_{m}^{(i)}})-r(\mathbf{v_{m}^{(i)}})\|_{2}^{2}\). 2) Masked Region Classification (MRC) MRC learns to predict the object semantic class for each masked region. We first feed the Transformer output of the masked region \(\mathbf{v_{m}^{(i)}}\) into an FC layer to predict the scores of \(K\) object classes, which further goes through a softmax function to be transformed into a normalized distribution \(g_{\theta}(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). Note that there is no ground-truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R-CNN, and take the detected object category (with the highest confidence score) as the label of the masked region, which will be converted into a one-hot vector \(c(\mathbf{v_{m}^{(i)}})\in\mathbb{R}^{K}\). The final objective minimizes the cross-entropy (CE) loss: \(f_{\theta}(\mathbf{v_{m}}|\mathbf{v_{\setminus m}},\mathbf{w})=\sum_{i=1}^{M} \operatorname{CE}(c(\mathbf{v_{m}^{(i)}}),g_{\theta}(\mathbf{v_{m}^{(i)}}))\). 3) Masked Region Classification with KL-Divergence (MRC-kl) MRC takes the most likely object class from the object detection model as the hard label (w.p. 0 or 1), which assumes the detected object class is the ground-truth label for the region. However, this may not be true, as no ground-truth label is provided for the detected region. Thus, in MRC-kl, we avoid this assumption by using soft label as supervision signal, which is the raw output from the detector (i.e., a distribution of object classes \(\tilde{c}(\mathbf{v}_{m}^{(i)})\)). MRC-kl aims to distill such knowledge into UNITER as Hinton et al. (2015), by minimizing the KL divergence between two distributions: \(f_{\theta}(\mathbf{v}_{\mathbf{m}}|\mathbf{v}_{\mathbf{m}},\mathbf{w})=\sum_{i= 1}^{M}D_{KL}(\tilde{c}(\mathbf{v}_{\mathbf{m}}^{(i)})||g_{\theta}(\mathbf{v}_{ \mathbf{m}}^{(i)}))\).

Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study.

On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a significant improvement across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks. Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combining MRC-kl and MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score.

Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting: MLM+ITM+MRC-kl+MRFR.

Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks. ITM is discarded because the text in VCR does not explicitly describe the image.

To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8. Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy18 measures how well UNITER can classify the masked regions. In both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.

We pre-train UNITER on Conceptual Captions only (instead of 4 datasets in 3.3) using our proposed conditional masking and the best pre-training tasks (MLM + \(\mathrm{ITM}\) + MRC-kl + MRFR).

Input: Table 3: Evaluation on pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ as benchmarks. All results are obtained from UNITER-base. Averages of R@1, R@5 and R@10 on Flickr30K for Image Retrieval (IR) and Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.

Input: Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and a multi-layer self-attention Transformer, learned through three pre-training tasks.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We design three pre-training tasks: MLM, ITM, and MRM (with three variants)" appears to be well-stated and accurately reflects what is presented in the paper. The evidence thoroughly details each of these tasks:

1. MLM (Masked Language Modeling) - masking words and predicting them based on context
2. ITM (Image-Text Matching) - binary classification of image-text pairs
3. MRM (Masked Region Modeling) with three variants:
   - MRFR (Masked Region Feature Regression)
   - MRC (Masked Region Classification)
   - MRC-kl (MRC with KL-divergence)

The claim makes no exaggerated statements and directly corresponds to what is implemented and tested in the ablation studies shown in Table 3. Even the reviewer's critical comments don't dispute the existence or implementation of these tasks.

<score>1</score>