Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark

Dongping Chenequal,lehigh
Ruoxi Chenequal,lehigh
Shilin Zhangequal,lehigh
Yinuo Liuequal,lehigh
Yaochen Wangequal,lehigh
Huichi Zhouequal,lehigh
Qihui Zhangequal,lehigh

Pan Zhouhust
Yao Wanhust
Lichao Sunlehigh

lehighLAIR Lab, Lehigh University
hustHuazhong University of Science and Technology

Dongping<EMAIL_ADDRESS>Lichao<EMAIL_ADDRESS>Yao<EMAIL_ADDRESS>
Machine Learning, ICML

Multimodal Large Language Models (MLLMs) have gained significant attention recently, showing remarkable potential in artificial general intelligence.

However, assessing the utility of MLLMs presents considerable challenges, primarily due to the absence multimodal benchmarks that align with human preferences.

Inspired by LLM-as-a-Judge in LLMs, this paper introduces a novel benchmark, termed MLLM-as-a-Judge, to assess the ability of MLLMs in assisting judges including three distinct tasks: Scoring Evaluation, Pair Comparison, and Batch Ranking.
Our study reveals that, while MLLMs demonstrate remarkable human-like discernment in Pair Comparisons, there is a significant divergence from human preferences in Scoring Evaluation and Batch Ranking tasks.

Furthermore, MLLMs still face challenges in judgment, including diverse biases, hallucinatory responses, and inconsistencies, even for advanced models such as GPT-4V.

These findings emphasize the pressing need for enhancements and further research efforts regarding MLLMs as fully reliable evaluators.

Code and dataset are available at <https://github.com/Dongping-Chen/MLLM-as-a-Judge>.

§ INTRODUCTION

The advent of Large Language Models (LLMs), such as GPT-3 [OpenAI, 2023] and Llama [Touvron et al., 2023], has achieved substantial progress in content generation,
including text generation [OpenAI, 2023], code generation [Roziere et al., 2023], and video synthesis [Wu et al., 2023].

The emergent abilities of LLMs, as demonstrated by the Chain-of-Thought (CoT) framework [Wei et al., 2022], present a promising avenue for their utilization as evaluators, also referred to as the LLM-as-a-Judge [Zheng et al., 2023].

Initial explorations indicate a better alignment with human preferences, emphasizing the considerable potential inherent in this approach.

Recently, building upon LLMs, Multimodal Large Language Models (MLLMs) like GPT-4V [OpenAI, 2023] and LLaVA [Liu et al., 2023] exhibit exceptional proficiency by incorporating multiple modalities (e.g., text, charts, images, and videos) and showcasing remarkable performance in multimodal applications, including text-to-video [Wu et al., 2023] and visual dialog [Cai et al., 2023].

Despite this,
assessing the effectiveness of MLLMs remains challenging due to the limitations of traditional metrics, which hinge on text-based exact matches or embedding distances. These metrics fall short in adhering to the granular evaluation criteria of interest and fail to capture the rich context within the generated outputs.

Drawing inspiration from the concept of LLM-as-a-Judge within LLMs, a pertinent research question arises: “Can MLLMs effectively serve as judges in the multimodal domain, and how closely do their evaluations align with human preferences?”

Comparative performance of different MLLMs across three judging settings, each is the average of three iterations. As the CogVLM is unable to perform the batch ranking task, we show the other three MLLMs only.

To answer this question, this paper undertakes an extensive study, introducing a groundbreaking benchmark, MLLM-as-a-Judge, specifically crafted to evaluate the efficacy of MLLMs in assisting judges across diverse modalities.

For the purpose of realizing this goal, we first thoughtfully curate a selection of 10 datasets across various tasks, including image captioning, math reasoning, text reading, and infographics understanding, culminating in the acquisition of a dataset comprising 3,300 image-instruction pairs.

Subsequently, we utilize four well-established MLLMs: GPT-4V [OpenAI, 2023], Gemini [GeminiTeam, 2023][For conciseness, we refer to  GPT-4V(ision) as GPT-4V, and Gemini-Pro-Vision as Gemini throughout this paper.], LLaVA [Liu et al., 2023], and CogVLM [Wang et al., 2023], to generate responses to instructions across three distinct evaluation settings.

The produced responses are subsequently gathered and undergo additional annotation by human evaluators, who apply stringent criteria to ensure an impartial and thorough assessment of the judgments made by the MLLMs.

Furthermore, we assess the ability of MLLMs as judges in multimodal tasks by calculating the similarity between human and MLLMs judgment and measuring human agreement on the analysis and judgment made by those MLLMs.
In particular, we target four widely-used MLLMs, i.e, GPT-4V and Gemini, CogVLM, and LLaVA, across two settings (with, or without vision input), over three distinct tasks (i.e., Scoring Evaluation, Pair Comparison, and Batch Ranking).
Figure <ref> shows the comparison performance of those MLLMs in various datasets and settings.

Our findings reveal that, although MLLMs demonstrate proficiency in aligning with human preferences during pair comparison tasks, discernible disparities emerge in their performance when tasked with scoring evaluation and batch ranking, particularly in the realm of reasoning.
The detection of hallucinations, biases, and inconsistent judgments within MLLM judgments underscores the formidable challenges that should be tackled for these models to emerge as a promising and alternative to traditional human evaluations.

As a benchmark, we also release two curated datasets to facilitate further studies: MLLM-as-as-Judge-HQ, showcasing responses with a high level of concordance with human judgments, and MLLM-as-a-Judge-Hard, encompassing responses marked by inconsistency with human preferences and instances of hallucination. Furthermore, we have addressed the limitations of MLLMs in judgment, encompassing egocentric bias, position bias, length bias, and hallucination. We demonstrate that the integration of Chain-of-Thought (CoT) [Wei et al., 2022] and a vision expert system can effectively mitigate some of these biases.

To summarize, our work provides three key contributions:

* A Benchmark.
We are the first to develop a comprehensive benchmark MLLM-as-a-Judge in multimodal domains, with human annotations to assess the judging capability of MLLMs in tasks of Scoring Evaluation, Pair Comparison and Batch Ranking.

* Two Datasets.
We curate two human preference datasets with high-quality questions MLLM-as-a-Judge-HQ and MLLM-as-a-Judge-Hard dataset with hallucination instances. They can serve as a rigorous testing ground to facilitate the development of MLLMs.

* Findings and Implications.
Our evaluation of mainstream MLLMs reveals that while MLLMs exhibit alignment with human judgments in pair comparison tasks, notable discrepancies can be found in scoring evaluation and batch ranking. Furthermore, our findings reveal that MLLMs exhibit a range of biases and hallucinations, along with inconsistent judgments during the evaluation process, representing significant hurdles in establishing MLLMs as reliable judges.

An overview of MLLM-as-a-Judge.

§ MLLM-AS-A-JUDGE: A BENCHMARK TO ASSESS VISION-LANGUAGE JUDGING ABILITY

The statistics of responses in different steps for MLLM judging. In Step 3, under the w.o. vision input settings, we sample 10% from the original data and mainly proceed with GPT-4V and Gemini. M-I: Image-Instruction.

Step               2cSetting                                       Input                                                                         Num.              Output                                                                        Num.
2*1      2c2*/                                     Image                                                                                3,000                      2*c]@l@M-I Pairs     2*3,300

2c                                                       Instruction                                                                            33,00
4*2      2c4*/                                     4*c]@l@ M-I Pairs     4*3,300     Gemini                                                                                2,918

2c                                                                                GPT-4V                                                                    3,294

2c                                                                                CogVLM                                                                                3,300

2c                                                                                LLaVA                                                                                3,300
25*3     2c12*90w. Vision Input                        4*Batch                                                                 4*1,100     Gemini                                                                                983

2c                                                                                GPT-4V                                                                                1,085

2c                                                                                CogVLM                                                                                1,100

2c                                                                                LLaVA                                                                                1,015
2c                                                       4*Pair                                                                  4*6,179     Gemini                                                                                5,751

2c                                                                                GPT-4V                                                                                6,151

2c                                                                                CogVLM                                                                                4,288

2c                                                                                LLaVA                                                                                6174

2c                                                       4*Score                                                                 4*4,430     Gemini                                                                                3,893

2c                                                                                GPT-4V                                                                                4,277

2c                                                                                CogVLM                                                                                3,220

2c                                                                                LLaVA                                                                                4,266

12*90w.o. Vision Input      6*90No Vision      2*Batch                                                                 2*110      Gemini                                                                                107

GPT-4V                                                                                110
2*Pair                                                                  2*425      Gemini                                                                                385

GPT-4V                                                                                355
2*Score                                                                 2*612      Gemini                                                                                582

GPT-4V                                                                                584
6*90Vision Experts       2*Batch                                                                 2*110      Gemini                                                                                107

GPT-4V                                                                                110
2*Pair                                                                  2*425      Gemini                                                                                396

GPT-4V                                                                                425
2*Score                                                                 2*612      Gemini                                                                                576

GPT-4V                                                                                612

Figure <ref> shows an overview of our proposed MLLM-as-a-Judge, consisting of three steps: 1) image-instruction pair collection, 2) MLLM response collection, and 3) comparison with human annotation.

Initially, we collect a dataset $\mathcal{P} = \{(M_1, I_1), \ldots, (M_n, I_n)\}$, containing pairs of images $(M)$ and their corresponding instructions $(I)$ sourced from 10 diverse domains (e.g., math, chart, diffusion), ensuring comprehensive coverage for a wide array of downstream tasks.
Subsequently, each pair $(M_i, I_i)$ is processed through several MLLMs,
generating a set of responses $\mathcal{R}_i = \{r_1, r_2, \ldots, r_n\}$ for each pair. This process contributes to the formation of the dataset of image-instruction-responses pairs, denoted as $\mathcal{D} = \{(M_i, I_i, \mathcal{R}_i) | (M_i, I_i) \in \mathcal{P}\}$.

Finally, the dataset $\mathcal{D}$ is partitioned into three distinct subsets to facilitate diverse task evaluations: $\mathcal{D}_{\text{score}}$ for Scoring Evaluation, $\mathcal{D}_{\text{pair}}$ for Pair Comparison, and $\mathcal{D}_{\text{batch}}$ for Batch Ranking.

Each subset will be employed for specific judging tasks, with each of them being configured as follows.

* Scoring Evaluation: Each individual response is evaluated on a scale from 1 to 5, with the specific criteria for this rating system detailed in Appendix <ref>.

* Pair Comparison: It involves a direct comparison between two responses,

culminating in the identification of the superior one. Following the principles outlined by [Deutsch et al., 2023], a tie option is incorporated to ensure a more equitable assessment.

* Batch Ranking:

The responses are systematically arranged in descending order of quality based on a given instruction, without any tie option.

§.§ Step 1: Image-Instruction Pair Collection

We meticulously curate a dataset consisting of 3,000 image-text pairs, gathered from a variety of downstream task datasets, as detailed in Table <ref> in Appendix <ref>. These pairs are carefully tailored into image-instruction pairs to suit a free-form response format. To illustrate, within the domain of diffusion tasks, our dataset incorporated pairs challenging models to adeptly recognize and articulate connections between provided images and user-specified keywords.

§.§ Step 2: MLLM Response Collection

We employ four widely-used MLLMs – GPT-4V [OpenAI, 2023], Gemini [GeminiTeam, 2023], LLaVA [Liu et al., 2023], and CogVLM [Wang et al., 2023] – to generate responses based on the image-instruction pairs, obtaining approximately 12,000 responses.
Responses that are either too brief or non-compliant with security regulations (e.g., “I'm sorry, but I cannot assist with this request”) from GPT-4V and Gemini are excluded.
The number of responses and the length distributions for different MLLMs are shown in Table <ref> and Figure <ref>, respectively.
We show specific hyper-parameter settings in Appendix <ref>. Besides, we segment these responses into three non-overlapping groups, to prevent response overlap.

Length distribution in responses for different MLLMs. Horizontal axis: length; Vertical axis: density.

§.§ Step 3: Comparison with Human Annotation

The annotation is conducted by 6 authors of this paper independently. These annotators are proficient in this domain, with different genders, ages, and educational backgrounds to ensure diversity [Sun et al., 2020]. They are required to give objective judgments without considering answer lengths, and certain names or positions of the response to minimize human bias. More details are referred to Appendix <ref>.

The overall performance of different MLLMs in judging, compared with human annotations on different datasets. We sample all the data three times and took the average to mitigate the casualty. w. and w.o. tie represents tie and non-tie situations respectively. We omit Gemini's results on the diffusion task for its challenges in processing AI-generated images.
All presented data of Pearson similarity exhibit a p-value below 0.05, indicating a statistically significant level of confidence.

Settings     MLLM      COCO      C.C.     Diffusion     Graphics     Math     Text     WIT     Chart     VisIT     CC-3M     Average
LLaVA    0.247     0.227     0.060     0.242     0.093     0.245     0.109     0.237     0.177     0.071     0.171

CogVLM    0.107     -0.048     0.049     -0.158     0.065     0.097     -0.131     -0.135     0.278     0.157     0.028

Gemini    0.262     0.408     -     0.400     0.228     0.222     0.418     0.343     0.336     0.374     0.332

GPT-4V    0.454     0.507     0.458     0.645     0.606     0.624     0.579     0.645     0.620     0.431     0.557
4*Pair w. Tie
LLaVA    0.273     0.478     0.286     0.273     0.657     0.510     0.369     0.383     0.456     0.484     0.417

CogVLM    0.548     0.409     0.562     0.613      0.412     0.250     0.273     0.262     0.324     0.433     0.409

Gemini    0.616     0.787       -       0.650     0.436     0.664     0.605     0.500     0.660     0.560     0.609

GPT-4V    0.696     0.824     0.847     0.639     0.564     0.673     0.679     0.657     0.640     0.612     0.683
4*Pair w.o. Tie
LLaVA    0.327     0.537     0.302     0.300     0.726     0.684     0.600     0.610     0.648     0.583     0.532

CogVLM    0.654     0.450     0.643     0.704     0.481     0.292     0.500     0.423     0.500     0.591     0.524

Gemini    0.717     0.840      -        0.770     0.678     0.793     0.688     0.658     0.711     0.652     0.723

GPT-4V    0.804     0.870     0.922     0.807     0.801     0.805     0.734     0.849     0.761     0.703     0.806
LLaVA    0.577     0.492     0.562     0.535     0.598     0.650     0.616     0.644     0.620     0.563     0.586

Gemini    0.287     0.299     -     0.473     0.462     0.430     0.344     0.520     0.426     0.357     0.400

GPT-4V    0.318     0.353     0.070     0.385     0.348     0.319     0.290     0.347     0.300     0.402     0.313

§ EXPERIMENT SETTINGS

§.§ Settings of MLLM-as-a-Judge

We evaluate the judging performance of four leading MLLMs – GPT-4V [OpenAI, 2023], Gemini [GeminiTeam, 2023], LLaVA [Liu et al., 2023], and CogVLM [Wang et al., 2023] – across three distinct evaluation settings. Adapting the `Analyze-then-Judge' paradigm from Chiang & Lee, 2023, which is a one-step CoT approach [Wei et al., 2022], we first asked MLLMs to analyze responses and then provide a judgment based on their analysis. However, due to capability limitations to perform the `Analyze-then-Judge' setting for LLaVA and CogVLM, we prompt them to directly output their judgment.

Furthermore, to extensively explore MLLMs judging capabilities, we conduct experiments on various settings, including scenarios without vision input, replacing vision input with a detailed description generated by GPT-4V as a vision expert, and employing multi-step COT.
Considering that the first two settings do not involve image inputs, we also include tests on the latest GPT-4 [OpenAI, 2023] and Gemini [GeminiTeam, 2023], to assess whether LLMs can effectively perform judging tasks without vision perception. Comprehensive details of these experimental setups are available in Appendix <ref>, and the prompts can be found in Appendix <ref>.

§.§ Judging Metrics

After collecting responses from MLLM judgments, we quantify their alignment with human annotations across three settings, employing distinct metrics as follows:

$\triangleright$Scoring Evaluation: Following LLM-as-a-Judge [Zheng et al., 2023], we compute the Pearson similarity [Lee Rodgers & Nicewander, 1988] between the MLLMs' judgments and human ratings across different sub-datasets.

$\triangleright$Pair Comparison: We assess the similarity between the MLLM judgments and human decisions using accuracy, F1-score [Goutte & Gaussier, 2005], and recall [Goutte & Gaussier, 2005] to gauge the judging abilities of models.

$\triangleright$Batch Evaluation: We consolidate the ranking results into a singular sequence and employ the Normalized Levenshtein distance [Levenshtein et al., 1966] to evaluate the similarity between judgments from MLLMs and human annotation.

§.§ Human Agreement in MLLM Judgment

Apart from traditional metrics for similarity assessment between judgments from MLLMs and humans, we further evaluate the judgments provided by MLLMs to uncover latent bias and hallucination. We enlisted human annotators for further validation, focusing on the following aspects:

$\triangleright$ Human Agreement: This involves a simple `yes' or `no' response to assess agreement with the MLLM judgments. While some judgments might appear reasonable, they may still be considered incorrect due to unique human perspectives. Hence, we conduct experiments on human agreement to address situations that traditional metrics may not adequately capture.

$\triangleright$ Analysis Grading: Each MLLM analysis is assigned a score from 1 to 5, considering relevance, accuracy, creativity, and response granularity, detailed in Appendix <ref>.

$\triangleright$ Hallucination Detection: Given the propensity for hallucination issues in the complex reasoning chains and long-term vision-language contexts of MLLMs, we tasked human annotators with identifying any hallucinations in the analyses of MLLM judgments, adhering to established definitions of vision and language hallucination [Sun et al., 2024].

§ EMPIRICAL RESULTS AND ANALYSIS

Human agreement percentage on MLLMs in judging. Each judgment is independently reviewed three times by different annotators and consensus results are recorded. We omit Gemini's results on the diffusion task for its challenges in processing AI-generated images.

Settings     MLLM      COCO      C.C.     Diffusion     Graphics     Math     Text     WIT     Chart     VisIT     CC-3M     Average

Gemini     0.783     0.739     -     0.618     0.536     0.621     0.749     0.630     0.712     0.702     0.677

GPT-4V    0.799     0.725     0.506     0.688     0.638     0.706     0.714     0.676     0.779     0.754     0.699
Gemini    0.705     0.833     -     0.733     0.520     0.717     0.827     0.620     0.853     0.703     0.724

GPT-4V     0.821     0.926     0.873     0.794     0.618     0.752     0.790     0.796     0.797     0.766     0.793

Gemini     0.642     0.639     -     0.333     0.330     0.473     0.511     0.315     0.422     0.554     0.469

GPT-4V     0.663     0.639     0.912     0.536     0.475     0.615     0.641     0.640     0.622     0.467     0.621

Pair preference density (Left) and scoring density (Right) of different MLLM judgments and human annotations.

§.§ MLLM Judgment vs Human Annotation

As shown in Figure <ref> and Table <ref>, judgments made by GPT-4V are closer to human annotations among all settings, while Gemini is far different, and LLaVA and CogVLM is even worse. Overall, MLLM judgments perform better on Pair Comparison, while falling short in Scoring Evaluation and Batch Ranking, showing a huge gap between the model and human preferences. Under the “Analyze-then-Judge” setting, GPT4-V prefers to give a longer Judge in all settings, convincing its ability to reason on long-term text.

$\triangleright$ Scoring Evaluation: GPT-4V demonstrated the highest similarity to human scoring with a similarity score of 0.557. In contrast, Gemini achieved only 0.332, with LLaVA and CogVLM scoring even lower. This discrepancy is primarily due to Gemini's tendency to assign scores around 4 points as depicted in Figure <ref>, seldom giving 1 or 2 points. LLaVA and CogVLM show a similar pattern to Gemini, predominantly assigning scores around 4 points. We attribute this to a `High-Score' Bias, akin to the `Yes/No' bias identified by Liu et al., 2023, which may result from an imbalance in positive and negative judging instructions in their training data [Liu et al., 2023], severely limits their ability to provide just and varied scores in scoring settings. In comparison, GPT-4V's scores are more evenly distributed and align closely with human preferences.

$\triangleright$ Pair Comparison: As illustrated in Figure <ref>, GPT-4V outshines other MLLMs in pair comparison tasks, achieving 0.683 in tie settings and 0.806 in non-tie settings, surpassing 0.8 in many datasets, which indicate a strong alignment with human preferences. Gemini, LLaVA, and CogVLM show a marked preference for declaring a clear winner, possibly due to a lack of tie situations in their training, leading to biased judgments. It's also interesting that the frequency of ties given by GPT-4V closely mirrors that of human judges, suggesting similar thresholds for tie decisions.

$\triangleright$ Batch Ranking: GPT-4V aligns more closely with human ranking results, indicating a significant lead with a mean Levenshtein Distance of 0.313. However, there is still substantial room for improvement in this task for all MLLMs. Notably, CogVLM is unable to provide a full ranking in this context, offering only the top choice; so it was excluded from this comparison; LLaVA also exhibits position bias influenced by prompt structure, often replicating judgments seen in example prompts, which complicates its ability to produce fair judgments.

Consistency Comparisons of GPT-4V and Gemini. `Average' means weighted average for consistency times, `MCC' means `Majority Consistency Criterion', which deems responses consistent if over half of them are identical across 6 repeated experiments.

2*MLLM    2c|Score      2c|Pair     2cBatch

Average     MCC     Average     MCC     Average     MCC

Gemini                0.531     0.054     0.781     0.547     0.629     0.338

GPT-4V               0.796     0.611     0.836     0.675     0.679     0.418

§.§ MLLM Judging Consistency

To be a reliable judge, consistent decision-making across repeated evaluations of the same query is crucial. For this purpose, we conducted six repeated tests with MLLM judgments and calculated the weighted average consistency scores and Majority Consistency Criterion ratios for GPT-4V and Gemini, as shown in Table <ref> and Figure <ref>. Despite a higher temperature setting, GPT-4V substantially outperforms Gemini across all tasks. Particularly in Pair Comparison, GPT-4V achieves a higher consistency score of 0.675, but it encounters difficulties in maintaining similar levels of consistency in Scoring and Batch Ranking tasks, with scores dropping to 0.611 and 0.418, indicating the challenge of producing qualified and convincing judgments.

Consistency checking on 6 repetitions of experiments on GPT-4V (Left) and Gemini (Right). GPT-4V outperforms Gemini with a relatively higher ratio for high consistency.

§.§ Vision Perception benefits MLLM Judging

We explore the feasibility of using LLMs for judging text-based responses without directly analyzing the original images. This involves two approaches: omitting vision information entirely and providing a detailed description of the picture. Surprisingly, as illustrated in Table <ref>, we find that LLMs' performance in multimodal judging tasks significantly improved with picture descriptions, achieving a Pearson similarity of 0.435 in Scoring Evaluation tasks, markedly outperformed judgments made without any vision perception. Notably, in non-tie Pair Comparison, MLLMs with detailed vision descriptions even exceed the standard performance of MLLMs in judging. This suggests that MLLMs may lack certain human-like judging capabilities, while LLMs can effectively judge multimodal tasks when provided with comprehensive task-related descriptions.

How vision perception significantly enhances multimodal judging performance in traditional LLM-as-a-Judge setting, slightly outperforming MLLMs in judging. Vision Exp. stands for judging with a detailed image description.

2*MLLM      2*Settings     Score       2c|Pair               Batch

Pearson     1cw. Tie      w.o. Tie     Edit Dis.
3*Gemini     Vision                        0.044     0.435                         0.769           0.412

No Vision                     0.108       0.433                         0.758       0.470

Vision Exp.                0.120       0.438                         0.785       0.472
3*GPT-4V     Vision                        0.311       1l0.527     0.878           0.369

No Vision                     0.299       1l0.491     0.868       0.394

Vision Exp.                0.435       1l0.544     0.878       0.400

§.§ Human Agreement

Our manual evaluation of MLLMs in judging, focusing on agreement and scoring, revealed notable findings. Table <ref> shows that GPT-4V achieved around 70% human agreement across all settings, excelling in the Pair Comparison task with 79.3% agreement. Specifically, GPT-4V reached 78% in human agreement for Pair Comparison, with Gemini close at 72%, indicating strong performance in most sample pairs and supporting the idea that large models excel in pairwise distinctions [Zheng et al., 2023], though improvements are needed in other judging settings.

In the Scoring Evaluation task, GPT-4V achieved a 70% human agreement rate, peaking at 79.9% in MS-COCO [Lin et al., 2014], while Gemini maintained an average rate of 67.7%. To assess the consistency of MLLM judging quality across multiple responses to a single image-instruction pair, we employed the Mean Absolute Deviation (MAD) metric. This measures the average absolute variance between individual scores and the mean, thereby gauging quality variability. Figure <ref> demonstrates that GPT-4V exhibits lower variation in quality assessments, indicating more consistent and reliable judgment compared to Gemini, which is further evidenced by its superior performance.

However, in Batch Ranking, both models showed reduced human performance. GPT-4V managed 69% human agreement, and Gemini only 47%. Additionally, their analyses received lower scores, especially in complex tasks like Math [Lu et al., 2023] and graphics information [Mathew et al., 2021]. This suggests that the models' inherent capabilities may not fully support understanding and completing intricate user instructions to provide accurate judgments.

§.§ Bias and Hallucination

Length distributions for GPT-4 and Gemini on Scoring Evaluation (left) and Pair Comparison (right).

Experiments on length bias in Scoring Evaluation for GPT-4 (left) and Gemini (right) demonstrate that both models assign higher scores to more verbose responses.

Egocentric Bias.
It means models assign higher scores to their own responses while scoring others lower [Zheng et al., 2023, Li et al., 2024]. As illustrated in Figure <ref> and <ref>, GPT-4V exhibits a slight degree of Egocentricity. This bias contrasts with Gemini, which tends to judge each response more equitably, displaying a similar scoring distribution across different sources. Further investigation into the rationale behind GPT-4V's self-favoring behavior indicated that its judgments align closely with its own ethical guidelines. For instance, when faced with questions involving user privacy, GPT-4V's responses typically emphasize privacy preservation and refuse to engage, leading to higher self-scoring in these scenarios. Despite efforts in prompt engineering to encourage impartiality, these models inherently rely on their built-in judgment criteria retained from post-alignment [Ouyang et al., 2022], which can lead to a divergence from human preferences. Such a discrepancy highlights the complexity of aligning MLLM judgments with human standards.

Position Bias.
It means a model consistently favors answers in specific positions, often influenced by training data that typically places correct responses at the beginning or end of prompts [Liu et al., 2023]. Figure <ref> illustrates this bias in LLaVA and CogVLM, showing a distinct preference for one particular option in Pair Comparison tasks, habitually selecting the answer in their favored position. Such bias might arise from their restricted instruction-following capabilities, making their judgments disproportionately influenced by the structure of prompts. For example, when a Batch Ranking prompt includes a sample answer sequence like `ABCD', LLaVA tends to replicate this sequence in its responses with a high frequency of 88.2%, significantly more than other sequences. However, introducing multiple examples in the prompt appears to lessen this bias, as evidenced by a reduced Position Bias score of 53.3% when two examples are provided. This suggests that augmenting prompts with more examples might help guide these models to adhere more closely to the given instructions.

Length Bias.
Length bias means models prefer longer answers over concise but correct ones [Li et al., 2024], also known as verbosity bias [Zheng et al., 2023]. As illustrated in Figure <ref>, both GPT-4V and Gemini are inclined to award higher scores and preference to longer content. To delve deeper into this bias, we conducted an expanded scoring experiment using GPT-4 [OpenAI, 2023], which lacks vision perception, to semantically increase the length of answers without altering their original meaning. As shown in Figure <ref>, the results showed a noticeable increase in the scores assigned by GPT-4V and Gemini, averaging gains of 0.6 and 0.75 points, respectively. This finding conclusively demonstrates the presence of Verbosity Bias, suggesting that MLLMs might exploit extended text as a backdoor method to achieve higher scores.

Hallucination Detection and Mitigation.
We observe a higher incidence of hallucinations in Batch Ranking tasks compared to Pair Comparison and Score Evaluation, which may stem from misunderstandings of the long-term context. Delving deeper, we encounter more severe language hallucinations, including miscomprehensions of textual meanings or errors in text retrieval, which significantly impact the accuracy and reliability of the final judgments. To mitigate hallucination, we perform multi-step CoT [Wei et al., 2022] on MLLM-as-a-Judge-Hard by telling MLLMs to judge step-by-step, perform extra reasoning steps before normal “Analyze-then-Judge” setting, following: 1) image-instruction 2) image 3) instruction.
As shown in Table <ref>, hallucinations are mitigated across all settings, with extra reasoning on image information showing the most notable improvement in both score and pair tasks. Notably, in the Batch Ranking task, which involves analyzing longer texts, more reasoning steps significantly reduce hallucinations.

Reduction of hallucinations in MLLM-as-a-Judge-Hard through additional CoT steps compared to normal setting.

Setting     Score     Pair     Batch
Figure-Instruction     46.15%     28.21%     43.59%

Figure     48.72%     35.90%     35.90%

Instruction     33.33%     33.33%     35.90%

§.§ Two High-Quality Subset: HQ & HARD

Utilizing additional Human Agreement evaluations, we compiled judgments with human preference score of 5, to form the MLLM-as-a-Judge-HQ dataset. In the 'Image-Instruction-Response-Judgment' format, this dataset comprises samples from various original datasets. Each sample has undergone cross-validation by human annotators to ensure the absence of hallucinations, as well as the focus on judgment accuracy.

The total sample count across HQ and Hard subsets.

Subset     Score     Pair      Batch
HQ         142                    133                 133

HARD       23                     116                 114

Conversely, we established the MLLM-as-a-Judge-Hard dataset from 'Image-Instruction-Response' samples identified through manual scoring and hallucination detection, where analysis and judgments of MLLMs deviate from human preferences, with hallucinations instances. This dataset makes it challenging for mainstream models like GPT-4 and Gemini in the `Analyze-then-Judge' setting.

This dataset poses a more formidable challenge for future MLLM-as-a-Judge endeavors, prompting the advancement of MLLM capabilities. For the sizes of subsets, see Table <ref>.

§ RELATED WORK

LLM as a Judge.

The evolution of LLMs has made them increasingly effective evaluators in Natural Language Processing (NLP) tasks. Zhu et al., 2023 introduced JudgeLM for LLM evaluation, followed by AUTO-J [Li et al., 2023], a system that evaluates LLMs through both pairwise comparisons and single-response assessments, demonstrating close alignment with human judgment [Bai et al., 2023, Li et al., 2023, Kim et al., 2023]. Advancements in CoT reasoning [Wei et al., 2022, Chu et al., 2023] and training-free instruction following [Brown et al., 2020, Wei et al., 2021] further extend LLMs' judging capability in diverse tasks like translation quality assessment [Kocmi & Federmann, 2023] and story generation [Chiang & Lee, 2023].

Hallucination and Bias in Judgments.
MLLMs suffer from vision and language hallucinations [Ji et al., 2023, Huang et al., 2023, Cui et al., 2023, Wang et al., 2023], often due to vision-language misalignments in training phase [Sun et al., 2024, Huang et al., 2023]. Recent research focuses on hallucination evaluation [Liu et al., 2023], detection [Li et al., 2023, Wang et al., 2023], and mitigation [Yin et al., 2023, Gunjal et al., 2023, Zhou et al., 2023], noting that even GPT-4V suffer from these issues [Shi et al., 2023, Liu et al., 2023, Cui et al., 2023]. Besides, biases in MLLM-as-a-Judge, similar to those in human decision-making [Blunch, 1984, Raghubir & Valenzuela, 2006] and other ML domains [Wang et al., 2018, Liu et al., 2023], such as position [Zheng et al., 2023], egocentric [Li et al., 2024], and verbosity biases [Saito et al., 2023], are compounded by the integration of visual perception, necessitating further investigation.

§ CONCLUSION

In this paper, we have presented a new benchmark, termed MLLM-as-a-Judge, to assess the judging capabilities of MLLMs across three critical evaluation settings in the multimodal domain: Scoring Evaluation, Pair Comparison, and Batch Ranking. We further evaluate their agreement with humans. Our results reveal that advanced MLLMs can win significant human recognition in Pair Comparisons, but perform poorly in Scoring Evaluation and Batch Ranking tasks. Our work highlights potential areas for future refinement and improvement of MLLMs.
We advocate for additional efforts dedicated to supporting the continuous development of MLLMs as judges.

§ IMPACT STATEMENTS

In this paper, we introduce a novel benchmark, termed MLLM-as-a-Judge, designed to propel the evolution of MLLMs toward achieving judgments that align more closely with human perspectives. This benchmark establishes a heightened criterion for assessing MLLMs, emphasizing their proficiency in comprehending and processing information in a manner reflective of human cognitive processes.  One limitation of our work lies in the bias in human annotation and MLLMs. We leave the exploration of more objectives, ethically principled, and socially beneficial MLLM-as-a-Judge systems as our future work.

[Antol et al., 2015]
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D.
Vqa: Visual question answering.
In Proceedings of the IEEE international conference on computer vision, pp.  2425–2433, 2015.

[Bai et al., 2023]
Bai, S., Yang, S., Bai, J., Wang, P., Zhang, X., Lin, J., Wang, X., Zhou, C., and Zhou, J.
Touchstone: Evaluating vision-language models by language models.
arXiv preprint arXiv:2308.16890, 2023.

[Banerjee & Lavie, 2005]
Banerjee, S. and Lavie, A.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.
In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp.  65–72, 2005.

[Bitton et al., 2023]
Bitton, Y., Bansal, H., Hessel, J., Shao, R., Zhu, W., Awadalla, A., Gardner, J., Taori, R., and Schimdt, L.
Visit-bench: A benchmark for vision-language instruction following inspired by real-world use.
ArXiv, abs/2308.06595, 2023.
URL <https://api.semanticscholar.org/CorpusID:260887670>.

[Blunch, 1984]
Blunch, N. J.
Position bias in multiple-choice questions.
Journal of Marketing Research, 210 (2):0 216–220, 1984.

[Brown et al., 2020]
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.
Language models are few-shot learners.
Advances in neural information processing systems, 33:0 1877–1901, 2020.

[Cai et al., 2023]
Cai, Y., Mao, S., Wu, W., Wang, Z., Liang, Y., Ge, T., Wu, C., You, W., Song, T., Xia, Y., et al.
Low-code llm: Visual programming over llms.
arXiv preprint arXiv:2304.08103, 2023.

[Cao et al., 2023]
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., and Sun, L.
A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt.
arXiv preprint arXiv:2303.04226, 2023.

[Chen, 2023]
Chen, D.
Aggregate, decompose, and fine-tune: A simple yet effective factor-tuning method for vision transformer, 2023.

[Chiang & Lee, 2023]
Chiang, C.-H. and Lee, H.-y.
Can large language models be an alternative to human evaluations?
arXiv preprint arXiv:2305.01937, 2023a.

[Chiang & Lee, 2023]
Chiang, C.-H. and Lee, H.-y.
A closer look into automatic evaluation using large language models.
arXiv preprint arXiv:2310.05657, 2023b.

[Chu et al., 2023]
Chu, Z., Chen, J., Chen, Q., Yu, W., He, T., Wang, H., Peng, W., Liu, M., Qin, B., and Liu, T.
A survey of chain of thought reasoning: Advances, frontiers and future.
arXiv preprint arXiv:2309.15402, 2023.

[Cui et al., 2023]
Cui, C., Zhou, Y., Yang, X., Wu, S., Zhang, L., Zou, J., and Yao, H.
Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges.
arXiv preprint arXiv:2311.03287, 2023.

[Deutsch et al., 2023]
Deutsch, D., Foster, G., and Freitag, M.
Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration.
In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.  12914–12929, 2023.

[Gao et al., 2024]
Gao, C., Chen, D., Zhang, Q., Huang, Y., Wan, Y., and Sun, L.
Llm-as-a-coauthor: The challenges of detecting llm-human mixcase.
arXiv preprint arXiv:2401.05952, 2024.

[GeminiTeam, 2023]
Gemini: A family of highly capable multimodal models, 2023.

[Goutte & Gaussier, 2005]
Goutte, C. and Gaussier, E.
A probabilistic interpretation of precision, recall and f-score, with implication for evaluation.
In European conference on information retrieval, pp.  345–359. Springer, 2005.

[Gunjal et al., 2023]
Gunjal, A., Yin, J., and Bas, E.
Detecting and preventing hallucinations in large vision language models.
arXiv preprint arXiv:2308.06394, 2023.

[Huang et al., 2023]
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al.
A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.
arXiv preprint arXiv:2311.05232, 2023a.

[Huang et al., 2023]
Huang, Y., Shi, J., Li, Y., Fan, C., Wu, S., Zhang, Q., Liu, Y., Zhou, P., Wan, Y., Gong, N. Z., et al.
Metatool benchmark for large language models: Deciding whether to use tools and which to use.
arXiv preprint arXiv:2310.03128, 2023b.

[Huang et al., 2023]
Huang, Y., Zhang, Q., Sun, L., et al.
Trustgpt: A benchmark for trustworthy and responsible large language models.
arXiv preprint arXiv:2306.11507, 2023c.

[Ji et al., 2023]
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P.
Survey of hallucination in natural language generation.
ACM Computing Surveys, 550 (12):0 1–38, 2023.

[Kim et al., 2023]
Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., et al.
Prometheus: Inducing fine-grained evaluation capability in language models.
arXiv preprint arXiv:2310.08491, 2023.

[Kocmi & Federmann, 2023]
Kocmi, T. and Federmann, C.
Large language models are state-of-the-art evaluators of translation quality.
arXiv preprint arXiv:2302.14520, 2023.

[Lee Rodgers & Nicewander, 1988]
Lee Rodgers, J. and Nicewander, W. A.
Thirteen ways to look at the correlation coefficient.
The American Statistician, 420 (1):0 59–66, 1988.

[Levenshtein et al., 1966]
Levenshtein, V. I. et al.
Binary codes capable of correcting deletions, insertions, and reversals.
In Soviet physics doklady, volume 10, pp.  707–710. Soviet Union, 1966.

[Li et al., 2023]
Li, J., Sun, S., Yuan, W., Fan, R.-Z., Zhao, H., and Liu, P.
Generative judge for evaluating alignment.
arXiv preprint arXiv:2310.05470, 2023a.

[Li et al., 2023]
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B.
Alpacaeval: An automatic evaluator of instruction-following models.
GitHub repository, 2023b.

[Li et al., 2023]
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R.
Evaluating object hallucination in large vision-language models.
arXiv preprint arXiv:2305.10355, 2023c.

[Li et al., 2024]
Li, Y., Huang, Y., Lin, Y., Wu, S., Wan, Y., and Sun, L.
I think, therefore i am: Awareness in large language models.
arXiv preprint arXiv:2401.17882, 2024a.

[Li et al., 2024]
Li, Z., Xu, X., Shen, T., Xu, C., Gu, J.-C., and Tao, C.
Leveraging large language models for nlg evaluation: A survey.
arXiv preprint arXiv:2401.07103, 2024b.

[Lin, 2004]
Lin, C.-Y.
Rouge: A package for automatic evaluation of summaries.
In Text summarization branches out, pp.  74–81, 2004.

[Lin et al., 2014]
Lin, T.-Y., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.
Microsoft coco: Common objects in context.
In European Conference on Computer Vision, 2014.
URL <https://api.semanticscholar.org/CorpusID:14113767>.

[Liu et al., 2023]
Liu, F., Guan, T., Li, Z., Chen, L., Yacoob, Y., Manocha, D., and Zhou, T.
Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models.
arXiv preprint arXiv:2310.14566, 2023a.

[Liu et al., 2023]
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L.
Aligning large multi-modal model with robust instruction tuning.
arXiv preprint arXiv:2306.14565, 2023b.

[Liu et al., 2023]
Liu, H., Li, C., Wu, Q., and Lee, Y. J.
Visual instruction tuning, 2023c.

[Liu et al., 2023]
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P.
Lost in the middle: How language models use long contexts.
arXiv preprint arXiv:2307.03172, 2023d.

[Lu et al., 2023]
Lu, P., Bansal, H., Xia, T., Liu, J., yue Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J.
Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models.
ArXiv, abs/2310.02255, 2023.
URL <https://api.semanticscholar.org/CorpusID:264491155>.

[Masry et al., 2022]
Masry, A., Long, D., Tan, J. Q., Joty, S., and Hoque, E.
ChartQA: A benchmark for question answering about charts with visual and logical reasoning.
In Findings of the Association for Computational Linguistics: ACL 2022, pp.  2263–2279, Dublin, Ireland, May 2022. Association for Computational Linguistics.
URL <https://aclanthology.org/2022.findings-acl.177>.

[Mathew et al., 2021]
Mathew, M., Bagal, V., Tito, R. P., Karatzas, D., Valveny, E., and Jawahar, C.
2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.  2582–2591, 2021.
URL <https://api.semanticscholar.org/CorpusID:233394125>.

[OpenAI, 2023]
Gpt-4 technical report.

[OpenAI, 2023]
Openai models - gpt-4-vision.
<https://openai.com/research/gpt-4v-system-card>, 2023.

[Ouyang et al., 2022]
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.
Training language models to follow instructions with human feedback.
Advances in Neural Information Processing Systems, 35:0 27730–27744, 2022.

[Papineni et al., 2002]
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.
Bleu: a method for automatic evaluation of machine translation.
In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.  311–318, 2002.

[Prendki, 2023]
Prendki, J.
Are you spending too much money labeling data?
<https://towardsdatascience.com/are-you-spending-too-much-money-labeling-data-70a712123df1>, 2023.

[Raghubir & Valenzuela, 2006]
Raghubir, P. and Valenzuela, A.
Center-of-inattention: Position biases in decision-making.
Organizational Behavior and Human Decision Processes, 990 (1):0 66–80, 2006.

[Roziere et al., 2023]
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al.
Code llama: Open foundation models for code.
arXiv preprint arXiv:2308.12950, 2023.

[Saito et al., 2023]
Saito, K., Wachi, A., Wataoka, K., and Akimoto, Y.
Verbosity bias in preference labeling by large language models.
arXiv preprint arXiv:2310.10076, 2023.

[Sharma et al., 2018]
Sharma, P., Ding, N., Goodman, S., and Soricut, R.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.
In Annual Meeting of the Association for Computational Linguistics, 2018.
URL <https://api.semanticscholar.org/CorpusID:51876975>.

[Shi et al., 2023]
Shi, Y., Peng, D., Liao, W., Lin, Z., Chen, X., Liu, C., Zhang, Y., and Jin, L.
Exploring ocr capabilities of gpt-4v (ision): A quantitative and in-depth evaluation.
arXiv preprint arXiv:2310.16809, 2023.

[Singh et al., 2019]
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M.
Towards vqa models that can read.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  8309–8318, 2019.
URL <https://api.semanticscholar.org/CorpusID:85553602>.

[Srinivasan et al., 2021]
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M.
Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.
Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
URL <https://api.semanticscholar.org/CorpusID:232092726>.

[Sun et al., 2024]
Sun, L., Huang, Y., Wang, H., Wu, S., Zhang, Q., Gao, C., Huang, Y., Lyu, W., Zhang, Y., Li, X., et al.
Trustllm: Trustworthiness in large language models.
arXiv preprint arXiv:2401.05561, 2024.

[Sun et al., 2020]
Sun, W., Nasraoui, O., and Shafto, P.
Evolution and impact of bias in human and machine learning algorithm interaction.
Plos one, 150 (8):0 e0235502, 2020.

[Touvron et al., 2023]
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023.

[Vedantam et al., 2015]
Vedantam, R., Lawrence Zitnick, C., and Parikh, D.
Cider: Consensus-based image description evaluation.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4566–4575, 2015.

[Wang et al., 2023]
Wang, J., Zhou, Y., Xu, G., Shi, P., Zhao, C., Xu, H., Ye, Q., Yan, M., Zhang, J., Zhu, J., et al.
Evaluation and analysis of hallucination in large vision-language models.
arXiv preprint arXiv:2308.15126, 2023a.

[Wang et al., 2023]
Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., Xu, J., Xu, B., Li, J., Dong, Y., Ding, M., and Tang, J.
Cogvlm: Visual expert for pretrained language models, 2023b.

[Wang et al., 2018]
Wang, X., Golbandi, N., Bendersky, M., Metzler, D., and Najork, M.
Position bias estimation for unbiased learning to rank in personal search.
In Proceedings of the eleventh ACM international conference on web search and data mining, pp.  610–618, 2018.

[Wang et al., 2022]
Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., and Chau, D. H.
Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models.
ArXiv, abs/2210.14896, 2022.
URL <https://api.semanticscholar.org/CorpusID:253116574>.

[Wei et al., 2021]
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V.
Finetuned language models are zero-shot learners.
arXiv preprint arXiv:2109.01652, 2021.

[Wei et al., 2022]
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al.
Chain-of-thought prompting elicits reasoning in large language models.
Advances in Neural Information Processing Systems, 35:0 24824–24837, 2022.

[Wu et al., 2023]
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S.
Next-gpt: Any-to-any multimodal llm.
arXiv preprint arXiv:2309.05519, 2023a.

[Wu et al., 2023]
Wu, Y., Wang, S., Yang, H., Zheng, T., Zhang, H., Zhao, Y., and Qin, B.
An early evaluation of gpt-4v (ision).
arXiv preprint arXiv:2310.16534, 2023b.

[Yan et al., 2023]
Yan, Z., Zhang, K., Zhou, R., He, L., Li, X., and Sun, L.
Multimodal chatgpt for medical applications: an experimental study of gpt-4v.
arXiv preprint arXiv:2310.19061, 2023.

[Yang et al., 2023]
Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang, L.
The dawn of lmms: Preliminary explorations with gpt-4v (ision).
arXiv preprint arXiv:2309.17421, 90 (1):0 1, 2023.

[Yin et al., 2023]
Yin, S., Fu, C., Zhao, S., Xu, T., Wang, H., Sui, D., Shen, Y., Li, K., Sun, X., and Chen, E.
Woodpecker: Hallucination correction for multimodal large language models.
arXiv preprint arXiv:2310.16045, 2023.

[Yu et al., 2023]
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L.
Mm-vet: Evaluating large multimodal models for integrated capabilities.
ArXiv, abs/2308.02490, 2023.
URL <https://api.semanticscholar.org/CorpusID:260611572>.

[Zhang et al., 2023]
Zhang, K., Yu, J., Yan, Z., Liu, Y., Adhikarla, E., Fu, S., Chen, X., Chen, C., Zhou, Y., Li, X., et al.
Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks.
arXiv preprint arXiv:2305.17100, 2023.

[Zheng et al., 2023]
Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M.
On large language models' selection bias in multi-choice questions.
arXiv preprint arXiv:2309.03882, 2023a.

[Zheng et al., 2023]
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
arXiv preprint arXiv:2306.05685, 2023b.

[Zhou et al., 2023]
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al.
A comprehensive survey on pretrained foundation models: A history from bert to chatgpt.
arXiv preprint arXiv:2302.09419, 2023a.

[Zhou et al., 2023]
Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H.
Analyzing and mitigating object hallucination in large vision-language models.
arXiv preprint arXiv:2310.00754, 2023b.

[Zhu et al., 2023]
Zhu, L., Wang, X., and Wang, X.
Judgelm: Fine-tuned large language models are scalable judges.
arXiv preprint arXiv:2310.17631, 2023.

§ COMPREHENSIVE RELATED WORKS

§.§ Large Model as Judge

The rapid development of LLMs has significantly enhanced their capabilities in long-term context perception and reasoning [Cao et al., 2023, Zhou et al., 2023], increasingly popularizing their use as evaluators in various Natural Language Processing (NLP) tasks [Huang et al., 2023, Gao et al., 2024, Li et al., 2024]. Zhu et al., 2023 were pioneers in this area, introducing JudgeLM, a fine-tuned LLM designed for evaluating other LLMs. Building on this, Li et al., 2023 introduced AUTO-J, a system that evaluates LLMs through both pairwise comparisons and single-response assessments, demonstrating close alignment with human judgment [Bai et al., 2023, Li et al., 2023, Kim et al., 2023]. Further advancements in LLMs, such as the development of Chain-of-Thought reasoning [Wei et al., 2022, Chu et al., 2023], training-free instruction following [Brown et al., 2020, Wei et al., 2021], and enhanced alignment with human preferences [Ouyang et al., 2022], have solidified their role in diverse tasks like translation quality assessment [Kocmi & Federmann, 2023] and story generation [Chiang & Lee, 2023].

§.§ Hallucination and Bias in Judge

MLLMs are known to exhibit both vision hallucination and hallucination originating from LMs, a phenomenon typically characterized by responses containing information not present in the visual or natural language context [Ji et al., 2023, Huang et al., 2023, Cui et al., 2023, Wang et al., 2023]. This issue often stems from misalignments in vision-language training [Sun et al., 2024, Huang et al., 2023]. Recent studies have begun to address these hallucination issues, focusing on evaluation [Liu et al., 2023], detection [Li et al., 2023, Wang et al., 2023], and mitigation strategies [Yin et al., 2023, Gunjal et al., 2023, Zhou et al., 2023]. Notably, GPT-4V [OpenAI, 2023], despite being a leading model in many fields [Yang et al., 2023, Wu et al., 2023], has also demonstrated susceptibility to hallucinations [Shi et al., 2023, Liu et al., 2023, Cui et al., 2023]. This raises concerns about the reliability of MLLMs in evaluative roles, especially in finance and medical field [Yan et al., 2023, Zhang et al., 2023].

In terms of bias, MLLM judging is subject to issues not exclusive to our context of evaluation but also observed in human decision-making [Blunch, 1984, Raghubir & Valenzuela, 2006] and Machine Learning (ML) domains [Wang et al., 2018, Liu et al., 2023] such as position bias [Zheng et al., 2023], egocentric bias [Li et al., 2024], and verbosity bias [Saito et al., 2023]. The integration of visual perception in MLLMs introduces additional complexities, resulting in biases unique to the fusion of dual perceptions, an area that still demands thorough exploration.

§.§ Evaluating Large Multimodal Models

Evaluating MLLMs typically involves diverse tasks and corresponding metrics, which reflect the models' ability to comprehend and generate content based on both visual and textual information. For instance, in image captioning tasks, models are tasked with generating descriptive text for a given image. The effectiveness of these models is measured using metrics such as BLEU [Papineni et al., 2002], METEOR [Banerjee & Lavie, 2005], ROUGE [Lin, 2004], and CIDEr [Vedantam et al., 2015]. In the context of Visual Question Answering (VQA), models are evaluated based on their ability to answer questions on an image’s content. Here, the accuracy of model responses is compared against human-annotated answers, serving as the primary metric to ensure alignment with human preferences [Antol et al., 2015, Chen, 2023].

However, when tackling sophisticated visual-language tasks, conventional evaluation metrics often fail to accurately capture the nuanced responses generated by these models, especially in complex or subjective scenarios that involve both visual elements and extended textual content [Liu et al., 2023]. Additionally, while manual annotation offers a more comprehensive and human-like evaluation, it comes with significant challenges. These include high costs [Prendki, 2023], potential biases [Zheng et al., 2023], and the difficulty of ensuring consistent replication [Chiang & Lee, 2023]. These limitations highlight the need for a more holistic approach to evaluation, one that combines human-like calibration with more fine-grained assessment methods.

§ DETAILED BENCHMARK CONSTRUCTION

§.§ Step 1: Image-Instruction Collection

To attain the outlined objectives, our approach begins with a detailed analysis of the capabilities of MLLMs. Specifically, we focus on the following abilities within MLLMs:

* Recognition Ability[Yu et al., 2023]: This encompasses general visual recognition capabilities, such as object recognition, Optical Character Recognition (OCR), and other high-level tasks in computer vision.

* Comprehension Ability: This pertains to the model's proficiency in spatial understanding and scenario comprehension.

* Inferential Ability: This involves the model's capacity to process information and reasoning, a critical component in processing charts, graphs, and mathematics.

* Multilingual Ability: This assesses the model's competence in understanding and processing multiple languages, especially focusing on their appearance in visual tasks such as text reading on images[Singh et al., 2019].

To ensure a robust and comprehensive assessment, we meticulously identified and incorporated 10 diverse datasets <ref> into our evaluation framework. This strategic selection aims to enrich the diversity of our assessment tasks and enhance the breadth and depth of our evaluation capabilities, as well as prevent biases. These datasets were chosen based on their ability to effectively challenge the various aspects of MLLMs, via different downstream tasks, ensuring a thorough and nuanced understanding of their performance and potential.

To construct a robust and unbiased set of image-instruction pairs, we randomly select 300 images from each dataset, ensuring a diverse representation. Specifically, for the MathVista dataset, which includes the provision of hints, we extract 600 corresponding instructions, encompassing both scenarios: with and without hints. For the remaining datasets, we align 300 instructions with the sampled images. This process culminates in a comprehensive collection comprising 3000 images corresponding with 3300 instructions.

Datasets and corresponding tasks in benchmark construction, each task is matched with several required abilities (Rec.-Recognition, Comp.-Comprehension, Inf.-Inferential, Mul.-Multilingual)

2*Dataset     Image     2*#Images     2*#Questions     #Selected     2*Task     Ability

Type             Pairs         Required

2*Conceptual Captions     2*Web image     2*3.3M     2*–     2*300     2*Captioning     2*Rec.&Comp.

[Sharma et al., 2018]

2*ChartQA     2*Chart     2*21K     2*33K     2*300     2*Chart reasoning     2*Rec.&Comp.

[Masry et al., 2022]

2*InfographicVQA     2*Infographics     2*5.4K     2*30K     2*300     2*Graph reasoning     2*Rec.&Comp.

[Mathew et al., 2021]

2*MathVista     2*Mathematics     2*6K     2*6K     2*300     2*Math reasoning     2*Rec.&Comp.&Inf.

[Lu et al., 2023]

2*TextVQA     2*Text     2*28K     2*45K     2*300     2*Text reading     2*Rec.&Comp.

[Singh et al., 2019]

2*WIT     2*Multilingual text     2*11.5M     2*–     2*300     2*Transcription     2*Rec.&Mul.

[Srinivasan et al., 2021]

2*MS COCO     2*Real-life scene     2*328K     2*2.5M(labels)     2*300     2*Image Segmentation     2*Rec.&Comp.

[Lin et al., 2014]

2*DiffusionDB     2*Diffusion     2*14M     2*1.8M(prompts)     2*300     2*Comprehensive     2*Rec.&Comp.&Inf.

[Wang et al., 2022]

2*CC-3M Concept-balanced     2*Comprehensive     2*595K     2*595K     2*300     2*Comprehensive     2*Rec.&Comp.&Inf.

[Liu et al., 2023]

2*VisIT-Bench     2*Comprehensive     2*1K     2*592     2*300     2*Instruction following     2*Rec.&Comp.&Inf.

[Bitton et al., 2023]

§.§ Step 2: VLM Responses Collection

We engage with 4 mainstream MLLMs (GPT4-V, Gemini, LLaVA, CogVLM) by providing them with our assembled image-instruction pairs. For the set of 3,300 instructions, each VLM generated a response, resulting in a comprehensive collection of 13,200 answers, with each of the 3300 instructions receiving a distinct response from each of the four MLLMs.
Upon collecting a total of 13,200 responses from the MLLMs, we proceed to analyze the distribution of response lengths for each model. Figure <ref> is a detailed illustration of length distribution in corresponding datasets.

Response length distribution in 10 datasets. The horizontal and vertical axis represents length and density respectively.

§ DETAILED EXPERIMENT SETTINGS

§.§ Response VLM Settings

We use GPT-4V, Gemini, LLaVA, and CogVLM to answer the image-instruction pair. We talk about their hyperparameter settings and problems encounter during inference respectively:

* GPT-4V [OpenAI, 2023]. For the hyper-parameter, we set the temperature and top-k as 0.9, max_tokens as 2048. However, we encounter some situations where it cannot answer accurately or refuses to answer due to ethical issues like Unfortunately, due to my programming, I'm unable to ..., which brings some difficulties to us in defining its judging capability.

* Gemini [GeminiTeam, 2023]. For the hyper-parameter, we use the default settings, which set temperature as 0.4, top-p as 1, and max_token as 2048. It should be noted that Gemini will receive more ethical limitations than GPT-4V, and will refuse to answer on the diffusion data set. But for some more difficult questions, it can't answer the questions, but it will "forcibly answer" the user's questions. In this case, GPT-4V will sincerely admit its shortcomings and give a possible answer.

* LLaVA [Liu et al., 2023]. For the hyper-parameter, we set temperature as 0, tok-$p$ as 1, max_tokens as 2048, and beam search number as 3. The reason why we select such a low temperature is that LLaVA cannot correctly output its judge in a specific format. We collect responses by inference on a dual-4090 local server.

* CogVLM [Wang et al., 2023]. For the hyper-parameter, we use the default setting and set max_tokens as 2048. We collect responses by inference on a dual-4090 local server.

§.§ GPT-4V as Judge

We adopt GPT-4V without using JSON_Mode mod based on our preliminary experiment in Appendix <ref>, but required it to output in JSON format in our prompt. Following the hyper-parameter set in [Chiang & Lee, 2023], we set the temperature to 0.9, top-k to 0.9, and max_tokens to 2048 in both cases with and without pictures. When there is a format error in the output or ethical settings are triggered, we will sample again. If it is still wrong, we will skip this piece of data.

§.§ Gemini-Vision-Pro as Judge

We call Gemini's API on the Google Cloud Server and use the default settings for temperature and top-k.
It should be noted that even though Gemini is currently the only VLM that can perform Judge, it will occasionally speak Chinese or Japanese during the output process when using English text and English pictures, shown as Unicode in Figure <ref>. Although it is semantically continuous, that is, it can be used after translation, and it greatly affects the continuity of the judge. We believe that this is a capability issue, and it is speculated that it is affected by the multi-linguist in some training sets.

[colback=gray!0!white,colframe=gray!50!black,   colbacktitle=gray!75!black]

Instruction: Here is the caption of this figure: human hands touch an alchemical circle. According to this figure and the caption, please give a detailed analysis of this figure:

Analysis: The assistant's response is in Japanese. It provides a detailed analysis of the figure, including the symbols on the alchemical circle and the possible meaning of the hands touching it. The response is well-written and easy to understand."

Judgment: 5

Gemini speaks Japanese.

§.§ Other MLLMs as Judge

We also use other mainstream MLLMs such as CogVLM [Wang et al., 2023] and LLaVA [Liu et al., 2023] for judging, which hyper-parameters follow the response generation settings. Some issues arise when we use the basic “Analyze-then-Judge” setting, where they cannot follow the judging instructions very well. Therefore we do not put their results into the results of the text. Recently, we have also noticed the advent of GLM-4V [<https://open.bigmodel.cn/>], which has shown good performance on some benchmarks and can receive long-term text and follow the instructions for judging. However, due to the time constraints, we have not completed tests on GLM-4V in our work.

§ ADDITIONAL EXPERIMENTAL RESULTS

§.§ Preliminary Experiment

Human Agreement on GPT-4V Output Mode

The recently introduced `Json Mode'[<https://openai.com/blog/new-models-and-developer-products-announced-at-devday>] in GPT-4V represents a significant advancement, particularly in structuring outputs in JSON format while restricting token usage. This mode has been observed to regularize responses, a feature particularly advantageous when dealing with structured data. However, this structuring tends to compartmentalize responses, potentially leading to a loss in the natural flow and contextual linkage typically inherent in human-like responses. This segmentation might inadvertently affect the readability and perceived coherence of the generated text.

To quantitatively assess the impact of Json Mode on output quality and its alignment with human preferences, we meticulously construct a test set. This set comprises 50 data instances, randomly selected from three distinct datasets used for evaluation purposes. The objective is to discern human evaluators' predilection for the outputs generated in Json Mode by GPT-4V.

For a comprehensive analysis, we engage three annotators, each responsible for labeling the data. Their assessments aim to discern the balance between structured, JSON-formatted responses and the inherently fluid nature of human judgment and preference in textual content, as shown in Figure <ref>.

Json Mode Preference Analysis.

Human Agreement Bias Checking
Acknowledging the inherent variability in human annotations, we embark on an empirical study involving ten annotators to ascertain the reliability of derived statistical patterns, notwithstanding the subjective nature of human judgment. This study aims to mitigate the individual biases that might skew the evaluation of GPT-4's outputs. A dataset comprising 50 entries, processed using the GPT-4 pair comparison setting, serves as the foundation for this investigation.

The results, detailed in Figure <ref>, underscore a noteworthy observation: while the annotators exhibit minimal variance in determining the correctness of GPT-4's judgments, a discernible divergence emerged in the scoring of analytical responses. This divergence presumably stems from individual perceptual differences and inherent biases. However, it's crucial to note that these discrepancies in scoring did not significantly compromise the overall integrity of the annotations.

A remarkable consensus is observed in the labeling of hallucinations. The employment of a meticulously defined decision tree for identifying hallucinations ensures a high degree of uniformity across the annotations. This structured approach substantially minimizes errors, underscoring the effectiveness of well-defined criteria in achieving consistent and reliable annotations across different individuals.

[The distribution of Human Annotators' ratings for the data.]

[Human Labeling and Agreement Bias Checking.]

Human Labeling and Agreement Bias.

§.§ Length Distribution on MLLM Judgments Analysis

In our analysis, we have included length distribution diagrams that showcase the differences in the responses provided by GPT-4V and Gemini during their judgment tasks as illustrated in Figure <ref>. These diagrams reveal that GPT-4V typically generates longer responses than Gemini in both the Scoring Evaluation (Figure <ref>) and Pair Comparison (Figure <ref>), whereas in Batch Ranking (Figure <ref>), the output lengths from both models are comparatively similar.

Length distribution in analysis collections.

Response length distribution in Scoring Evaluation. The horizontal axis represents length, and the vertical axis represents density.

Response length distribution in Pair Comparison.

Response length distribution in Batch Ranking.

§.§ Results on Human Scoring and Ego Bias

We employed the Mean Absolute Deviation (MAD) metric to assess the consistency of MLLM judging quality across multiple responses to a single image-instruction pair, as shown in <ref>.

MAD of Human Scoring on MLLM Judgments Analysis.

The Egocentric Bias of different models are shown in Figures <ref> and <ref>.

Scoring Density of Different MLLMs in Judging.

The proportion of different responses chosen by humans and different MLLMs in Tie Scenario and Non-Tie Scenario.

§ HUMAN LABELING AND AGREEMENT COLLECTION

The annotation is conducted by 6 authors of this paper independently. As acknowledged, the diversity of annotators plays a crucial role in reducing bias and enhancing the reliability of the benchmark. These annotators have knowledge in this domain, with different genders, ages, and educational backgrounds. To ensure the annotators can proficiently mark the data, we provide them with detailed tutorials, teaching them how to evaluate model responses more objectively. Specifically, they are required to give judgments without considering answer lengths, and certain names or positions of the response. Besides, we implement cross-validation between different annotators and conduct continuous monitoring to ensure they are maintaining objectivity and fairness.

We provide screenshots of the labeling pages involved in the human labeling (Figure <ref>) and human agreement experiments (Figure <ref>) respectively.

Human labeling

Human agreement

arc=1.5mm, auto outer arc,
left=0.9mm, right=0.9mm,
Your assessment should identify whether the assistant effectively adheres to the user's instructions and addresses the user's inquiry.

Do not allow the length of the responses to influence your evaluation.

Do not favor certain names or positions of the assistants. Be as objective as possible.

In your evaluation, weigh factors such as relevance, accuracy, comprehensiveness, creativity, and the granularity of the responses:

Relevance: The judge's decisions directly correspond to the provided instructions or criteria. Every judgment made is pertinent to the case at hand, without deviation into unrelated areas.

Accuracy: The judge's decisions are consistently in line with the established rules or guidelines. There is a clear understanding and correct application of these guidelines in every judgment.

Comprehensiveness: The judge considers all necessary aspects and evidence related to each case. Every relevant point in the guidelines is addressed in the judge's evaluation.

Creativity: The judge demonstrates the ability to approach complex or ambiguous situations with innovative thinking. This includes providing insightful, constructive feedback or solutions not explicitly covered in the guidelines.

Granularity of Responses: The judge offers detailed and specific reasoning for each decision. This entails a thorough breakdown of how each aspect of the guidelines applies to the case or situation at hand.

Prompts for Human Agreement Experiment.

§ PROMPT TEMPLATES

For step 2, which is to obtain the response of MLLM to the user instruction, the prompt we use is shown in Figure <ref>. Following Chiang & Lee, 2023 and Li et al., 2024, we have designed prompts and presented the prompt template of VLM's operation including score, pair comparison, and batch ranking judgments in a prompt template as system prompt, instruction, criteria, noticement, and desired output form, as shown in Figure <ref>, <ref>, and <ref> respectively.

arc=1.5mm, auto outer arc,
left=0.9mm, right=0.9mm,
You are a helpful assistant proficient in analyzing vision reasoning problems.

[The Start of User Instruction]

[The End of User Instruction]

Please provide a detailed explanation for your response.

Query prompts of MLLMs in giving responses.

arc=1.5mm, auto outer arc,
left=0.9mm, right=0.9mm,
(System Prompt)

You are a helpful assistant proficient in analyzing vision reasoning problems.

Please examine the provided image attentively and serve as an unbiased judge in assessing the quality of the response from an AI assistants regarding the user's instruction. You will receive a single response from the AI assistant to user's instruction.

Your assessment should identify whether the assistant effectively adheres to the user's instructions and addresses the user's inquiry.

In your evaluation, weigh factors such as relevance, accuracy, comprehensiveness, creativity, and the granularity of the responses.

Do not allow the length of the responses to influence your evaluation.

Do not favor certain names or positions of the assistants. Be as objective as possible.

Use scores to show the quality of the response. Here is the detailed scoring rubric for evaluating the quality of responses from AI assistants:

Poor (1): The response significantly deviates from the user's instruction and fails to address the query effectively. It shows a lack of relevance, accuracy, and comprehensiveness. Creativity and granularity are absent or poorly executed.

Fair (2): The response addresses the user's instruction partially, with evident shortcomings in relevance, accuracy, or comprehensiveness. It lacks depth in creativity and granularity, indicating a superficial understanding of the user's inquiry.

Average (3): The response adequately addresses the user's instruction, showing a fair level of relevance, accuracy, and comprehensiveness. It reflects a basic level of creativity and granularity but may lack sophistication or depth in fully capturing the user's inquiry.

Good (4): The response is well-aligned with the user's instruction, demonstrating a high degree of relevance, accuracy, and comprehensiveness. It shows creativity and a nuanced understanding of the topic, with a detailed granularity that enhances the response quality.

Excellent (5): The response perfectly adheres to the user's instruction, excelling in relevance, accuracy, comprehensiveness, creativity, and granularity. It provides an insightful, detailed, and thorough answer, indicating a deep and nuanced understanding of the user's inquiry.

(Desired Output Format)

Use "[[1]]", "[[2]]", "[[3]]", "[[4]]", "[[5]]" to indicate your evaluate score in the key `Judgement'.

[The Start of User Instruction]

[The End of User Instruction]

[The Start of Assistant’s Answer]

[The End of Assistant’s Answer]

Template prompts of scoring evaluation

arc=1.5mm, auto outer arc,
left=0.9mm, right=0.9mm,
(System Prompt)

You are a helpful assistant proficient in analyzing vision reasoning problems.

Please examine the provided image attentively and serve as an unbiased judge in assessing the quality of responses from two AI assistants regarding the user's question shown beneath the image.

Your assessment should identify the assistant that more effectively adheres to the user's instruction and aptly addresses the user's inquiry.

In your evaluation, weigh factors such as relevance, accuracy, comprehensiveness, creativity, and the granularity of the responses.

Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision.

Do not allow the length of the responses to influence your evaluation.

Do not favor certain names of the assistants. Be as objective as possible.

Present your verdict in a JSON format, with the key 'analysis' for a short reason of your judgement and the key 'judgment' to indicate your decision: use “[[A]]" if assistant A prevails, "[[B]]" if assistant B does, and "[[C]]" for a tie.

(Desired Output Format)

[The Start of User Instruction].

[The End of User Instruction]

[The Start of Assistant A’s Answer]

[The End of Assistant A’s Answer]

[The Start of Assistant B’s Answer]

[The End of Assistant B’s Answer]

Template prompts of pair comparison.

arc=1.5mm, auto outer arc,
left=0.9mm, right=0.9mm,
(System Prompt)

You are a helpful assistant proficient in analyzing vision reasoning problems.

Please serve as an unbiased judge in assessing the quality of the responses from AI assistants regarding the user's instruction and a figure.

You will be presented with several responses from different assistants to the same user instruction.
Your task is to assess and compare these responses based on how effectively they adhere to the user's original instruction and how aptly they address the user's inquiry.
After your assessment and comparison, you should RANK the responses from best to worst as the following template. If Assistant A is the best response, Assistant D is the worst response, you should output like [[A]], [[B]], [[C]], [[D]]"

Your assessment should identify whether the assistant effectively adheres to the user's instructions and addresses the user's inquiry.

In your evaluation, weigh factors such as relevance, accuracy, comprehensiveness, creativity, and the granularity of the responses.

Do not allow the length of the responses to influence your evaluation.

Do not favor certain names or positions of the assistants. Be as objective as possible.

Indicate your final rank in the key 'Judgement'.

(Desired Output Format)

[The Start of User Instruction].

[The End of User Instruction]

Template prompts of batch ranking

§ CASE STUDY

In the pair comparison task, the result 'C' indicates a tie. We shuffled the order of ABCD in the experiments to maintain anonymity of the assistants, mirroring the anonymization process used in our actual experiments. For examples of Scoring Evaluation, Pair Comparison, and Batch Ranking see Figures <ref>, <ref>, and <ref> respectively.

arc=1.5mm, auto outer arc,
left=0.9mm, right=0.9mm,
boxrule=0.9pt, colbacktitle = black!65!black

Instruction: Please analyze this figure in detail and answer the following question based on this figure: Who was the train  built for?

answer1.name: Assistant A

answer1.answer: The train was built for children to play with in a park. The image shows a small red and black toy train parked on a dirt surface in a park, surrounded by trees. The train appears to be designed for children to ride on and enjoy, making it an ideal attraction for families visiting the park.

result1-1.name: Gemini

result1-1.analysis:  The assistant's response is relevant to the user's instruction and addresses the user's inquiry. The response is accurate and comprehensive, providing a clear and concise answer to the question. The response is also creative and uses detailed granularity to enhance the quality of the response. Judgement: 5

result1-2.name: GPT-4V

result1-2.analysis: The response provided by the AI assistant correctly identifies the purpose of the toy train in the image, stating that it was built for children to play with in a park. The description of the train as small, red and black, and parked on a dirt surface surrounded by trees is accurate based on the visual evidence in the image. Furthermore, the assistant correctly deduces that the train is designed for children to ride on and asserts that it is an ideal attraction for families, which is a logical conclusion given the context of the setting. The response is relevant to the user's instruction and demonstrates accuracy and a reasonable level of comprehensiveness. There is a minor display of creativity in the assistant's description of the train being an ideal attraction for families. Judgement: 4

An example of score comparison.

arc=1.5mm, auto outer arc,
left=0.9mm, right=0.9mm,
boxrule=0.9pt, colbacktitle = black!65!black

Instruction: This is a task of reasoning on chart data. Please analyze this figure in detail and answer the following question ith reason based on this figure. Who is the career touchdown leader of the Seattle Seahawks?

answer1.name: Assistant A

answer1.answer: The career touchdown leader of the Seattle Seahawks is Shaun Alexander with 112 touchdowns as illustrated by the bar chart, where his bar is the longest and the numerical value of 112 is the highest among all the listed players.

answer2.name: Assistant B

answer2.answer: The answer is Shaun Alexander. Shaun Alexander has 112 career touchdowns, the most of any player on the Seattle Seahawks. Steve Largent is second with 101 career touchdowns.

result1-1.name: Gemini

result1-1.judge: C

result1-1.analysis: Both assistants correctly answer the question and provide the same answer with sufficient supporting details.

result1-2.name: GPT-4V

result1-2.judge: B

result1-2.analysis: Both Assistant A and Assistant B correctly identified Shaun Alexander as the career touchdown leader for the Seattle Seahawks with 112 touchdowns, as shown in the bar chart. Assistant B provided additional context by mentioning that Steve Largent is second with 101 career touchdowns, offering a comparative perspective. While Assistant A's answer is accurate, Assistant B's answer adds a layer of detail that enriches the response without detracting from its relevance or accuracy.

An example of pair comparison.

arc=1.5mm, auto outer arc,
left=0.9mm, right=0.9mm,
boxrule=0.9pt, colbacktitle = black!65!black

Instruction: This is a task of text reading on natural image. Please analyze this figure in detail and answer the following  question with reason based on this figure. what year is at the top of the magazine?

answer1.name: Assistant A

answer1.answer: The year at the top of the magazine is 2014.

answer2.name: Assistant B

answer2.answer: The year at the top of the magazine is 2014. It is prominently displayed in large, bold letters, making it easily readable and the focal point of the image.

answer3.name: Assistant C

answer3.answer: The year at the top of the magazine is 2014. This can be determined by reading the text that says '9 Maggio – 10 Giugno 2014,' which indicates that the magazine or event was from May 9 to June 10, 2014. This text is found in the upper section of the image, in a red-colored font, just above the image of the two monsters.

answer4.name: Assistant D

answer4.answer: The year at the top of the magazine is 2014. The top of the magazine says “9 MAGGIO - 1 GIUGNO 2014". “9 MAGGIO - 1 GIUGNO" means “May 9 - June 1". “2014" is the year.

result1-1.name: Gemini

Assistant A: The assistant provides a concise and straightforward response, stating that the year at the top of the magazine is 2014. However, the response lacks any further explanation or supporting details.
Assistant B: Similar to Assistant A, this assistant also briefly states that the year is 2014, but it adds a bit more detail by mentioning that the year is displayed prominently in large, bold letters, making it easily readable and the focal point of the image.
Assistant C: Assistant C offers a more comprehensive response by not only stating the year as 2014 but also providing a clear explanation. The assistant points out the specific text on the magazine that indicates the date range of May 9 to June 10, 2014, and correctly concludes that the year at the top is 2014.
Assistant D: Assistant D's response is comparable to that of Assistant C, as it includes both the identification of the year as 2014 and an explanation. However, Assistant D's explanation is slightly different, as it mentions the Italian translation of the date range, which is 9 MAGGIO - 1 GIUGNO 2014. The response also includes additional information about the event, mentioning that it took place in Milan and was organized by WOW Spazio Fumetto.

Judgement: Based on the analysis, I rank the responses as follows: [[C]], [[D]], [[B]], [[A]]

result1-2.name: GPT-4V

result1-2.analysis: Assistant A: The response correctly identifies the year at the top of the magazine as 2014. However, it provides no additional information or context to support the answer., Assistant B: This response also correctly identifies the year as 2014 and goes further to describe the appearance of the text, which supports the user's instruction for detail and provides reasoning for the answer., Assistant C: The response is accurate in identifying the year as 2014 and provides a detailed description of where the date is located and what the text around it indicates. It adds context by mentioning the text color and the image below the date, giving a more comprehensive understanding., Assistant D: Assistant D accurately identifies the year as 2014 and gives a translation of the other text around the year, providing additional context. However, it slightly misquotes the text, stating '1 GIUGNO' instead of `10 Giugno,' which could be seen as a minor error in accurately reading the text. Judgement: Rank: [[C], [B], [A], [D]]

An example of batch ranking.