Face-Human-Bench: A Comprehensive Benchmark of Face and Human Understanding for Multi-modal Assistants
Under review as a conference paper at ICLR 2025

FACE-HUMAN-BENCH: A COMPREHENSIVE
BENCHMARK OF FACE AND HUMAN UNDERSTANDING
FOR MULTI-MODAL ASSISTANTS

Anonymous authors
Paper under double-blind review

ABSTRACT

Faces and humans are crucial elements in social interaction and are widely in-
cluded in everyday photos and videos. Therefore, a deep understanding of faces
and humans will enable multi-modal assistants to achieve improved response qual-
ity and broadened application scope. Currently, the multi-modal assistant com-
munity lacks a comprehensive and scientific evaluation of face and human under-
standing abilities. In this paper, we first propose a hierarchical ability taxonomy
that includes three levels of abilities. Then, based on this taxonomy, we collect im-
ages and annotations from publicly available datasets in the face and human com-
munity and build a semi-automatic data pipeline to produce problems for the new
benchmark. Finally, the obtained Face-Human-Bench comprises a development
set with 900 problems and a test set with 1800 problems, supporting both English
and Chinese. We conduct evaluations over 25 mainstream multi-modal large lan-
guage models (MLLMs) with our Face-Human-Bench, focusing on the correlation
between abilities, the impact of the relative position of targets on performance, and
the impact of Chain of Thought (CoT) prompting on performance. Moreover, in-
spired by multi-modal agents, we also explore which abilities of MLLMs need
to be supplemented by specialist models. The data and evaluation code of the
Face-Human-Bench will be made publicly available.

1

INTRODUCTION

Faces and humans are always the most crucial elements of photos and videos in our everyday
lives. Consequently, they are also critical focuses in multi-modal AI applications. In the past two
years, ChatGPT (OpenAI, 2023a) and GPT-4 (OpenAI, 2023b) have achieved great success with
impressive instruction-following and multi-modal understanding capabilities respectively. Numer-
ous excellent works (Liu et al., 2023b; Zhu et al., 2024; Dai et al., 2023; Bai et al., 2023) from
the open-source community have followed, collectively presenting the immense potential of multi-
modal assistants. Since faces and humans are central to social interaction, a deep understanding of
this information can make multi-modal assistants achieve improved response quality and broadened
application scope. For instance, in movie understanding (Yue et al., 2023; Han et al., 2023; Wang
et al., 2024), identifying characters is a prerequisite for multi-modal assistants to describe the plot
accurately. In multi-modal human-computer interaction (Fu et al., 2024), perceiving expressions and
body language can help multi-modal assistants accurately understand the context, generating more
personalized and humanized responses. In media forensics (Liu et al., 2024b;c; Jia et al., 2024),
determining whether deepfake artifacts exist on a face is crucial for multi-modal assistants to detect
misinformation.

Comprehensive and scientific evaluation is the foundation for researching applications of multi-
modal assistants related to “faces and humans.” Existing benchmarks Fu et al. (2023); Li et al.
(2023a); Liu et al. (2023c) for large multi-modal models typically involve limited abilities of face
and human understanding, such as celebrity recognition, action recognition, identity reasoning, and
social relation, leaving many important abilities unexplored. On the other hand, since face and hu-
man understanding is one of the earliest research topics in artificial intelligence, there are numerous
datasets available for evaluating the performance of specialist models. The images and annotations
from these datasets can serve as original material to evaluate multi-modal assistants.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: The hierarchical ability taxonomy for evaluating face and human understanding abilities.
We construct the Face-Human-Bench based on this taxonomy. The proportion of the sectors repre-
sents the weight of the corresponding abilities in the overall score on the Face-Human-Bench.

As the starting point of our evaluation, we propose a hierarchical ability taxonomy, as shown in
Figure 1. This taxonomy consists of three levels. Level-1 (L1) has two perspectives to study:
from the target perspective, L1 includes face understanding and human understanding; from the
cognitive process perspective, L1 includes perception and reasoning. Subsequently, we incorporate
finer-grained abilities into the taxonomy and categorize them into 10 Level-2 (L2) and 18 Level-
3 (L3) ability dimensions. Then, based on this taxonomy, we collect datasets from the face and
human community and use a semi-automatic data pipeline to transform original images and anno-
tations into multi-modal QAs. The final obtained benchmark called Face-Human-Bench, including
a development set with 900 problems and a test set with 1800 problems, supporting evaluations in
both English and Chinese. For ease of evaluation, we adopt multiple-choice as the problem format
following MMBench (Liu et al., 2023c) and SEED-Bench (Li et al., 2023a).

In the literature, multi-modal assistants can be broadly categorized into two types: (1) Multi-modal
large language models (MLLMs), which achieve end-to-end output by aligning visual information
to the language domain with visual instruction-tuning (Liu et al., 2023b). (2) Multi-modal agents
(Wu et al., 2023; Yang et al., 2023), where LLMs decide when to call specialist models to solve
particular problems and then integrate the outputs of these specialist models. Compared to multi-
modal agents, MLLMs generally have better multi-modal perception and reasoning abilities with
more effective relationship modeling across modalities. In this study, the first research question
(RQ1) is: “How do existing MLLMs perform in face and human understanding?” In this question,
we focus on (a) the performance of 25 mainstream MLLMs, (b) the correlation between abilities at
different levels, (c) the impact of the relative position of targets on performance, and (d) the impact of
Chain of Thought (CoT) prompting on performance. Meanwhile, for face and human understanding
tasks in which specialist models significantly outperform MLLMs, we can draw inspiration from
multi-modal agents by utilizing the output of these specialist models to enhance the responses of
multi-modal assistants. Thus, the second research question emerges (RQ2): In the field of face and
human understanding, which tasks’ specialist models can achieve significantly better performance
than current MLLMs?

In response to RQ1, our main findings are as follows: (a) The Face-Human-Bench effectively dis-
tinguishes the abilities of MLLMs in faces and human understanding. Under the zero-shot setting,

2

Relative Position UnderstandingBasic Expression RecognitionHuman Attribute RecognitionAction RecognitionSpatial Relation UnderstandingSocial Relation UnderstandingPersonRe-IdentificationFace RecognitionFace Attack DetectionFacial Expression RecognitionAge EstimationFacial Attribute RecognitionAction RecognitionCrowd CountingSocial Relationship RecognitionIdentity ReasoningCompound Expression RecognitionDeepfake DetectionFace Anti-SpoofingBasicCross-PoseCross-Age Similar-LookingOccludedWhich description best matches the person in the picture?What is the most likely age?Is the face digitally manipulated?What emotional state is the person in?Is the face a physical spoof?Are the individuals in the left and right images identical?What is the occupation of the person in the red box?Do the two photographs showcase the same individual?What is the relationship between the two people in the red boxes?How many individuals are in this picture?How to describe the positional relationship of the girl relative to the boy?What is the person in the red box doing in the picture?Which description best matches the person in the red box?Face-HumanBench👀🤔👀👀👀👀👀👀🤔🤔🤔🤔🤔AgeEstimationFacialAttributeRecognitionHumanAttributeRecognitionPersonRe-Identification👀🤔PerceptionReasoningFaceHumanUnder review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

the best-performing closed-source model, GPT-4o (OpenAI, 2024), does not perform as well as the
best open-source model, InternVL-Chat-v1.2-Plus (Chen et al., 2023). (b) The correlation coeffi-
cients can reveal correlations between abilities at different levels. At L2 and L3, there are some
ability groups in which the ability dimensions exhibit significant positive correlations between each
pair. (c) Many models show substantial performance differences on the same task with different
relative positions of targets. We design a new metric called the relative position sensitivity score
(RPSS) to measure this phenomenon. On this metric, InternLM-XComposer2-VL-7B (Dong et al.,
2024) performs the best, indicating that its performance is almost unaffected by the relative posi-
tion of targets. (d) Introducing hints and CoT instructions into the prompts significantly improves
the performance of the closed-source model GPT-4o, but has no effect on the open-source model,
InternVL-Chat-v1.2-Plus. In response to RQ2, we find that in tasks of deepfake detection, crowd
counting, and face recognition (under challenging scenarios), the performance of MLLMs is signif-
icantly inferior to that of corresponding specialist models. Therefore, we recommend incorporating
specialist models in applications requiring these abilities to help improve the response quality of
multi-modal assistants.

Our contributions can be summarized as follows:

• We propose the Face-Human-Bench, the first benchmark dedicated to evaluating multimodal as-
sistants’ face and human understanding abilities. The Face-Human-Bench is based on a three-
level ability taxonomy and supports both English and Chinese.

• Utilizing the Face-Human-Bench, we conduct a comprehensive evaluation of mainstream
MLLMs, revealing the correlation between abilities, and exploring the impact of the relative po-
sition of targets and CoT prompting on the performance of MLLMs.

• We explore which specialist models significantly outperform MLLMs in certain face and human
understanding tasks. Based on this, we provide suggestions for enhancing the response quality of
multi-modal assistants.

2 FACE-HUMAN-BENCH

2.1 HIERARCHICAL ABILITY TAXONOMY

As shown in Figure 1, the proposed ability taxonomy includes three levels. Level-1 (L1) has two
research perspectives. From the target perspective, L1 includes face understanding and human un-
derstanding. From the cognitive process perspective, L1 includes perception and reasoning. In our
evaluation, perception involves direct comprehension of only one target, while reasoning requires
synthesizing information from multiple targets and environments to conclude. There are ten abilities
in total at Level-2 (L2). Five are focused on faces: facial attribute recognition, age estimation, facial
expression recognition, face attack detection, and face recognition, and five are focused on humans:
human attribute recognition, action recognition, spatial relation understanding, social relation un-
derstanding, and person re-identification. It should be noted that at L2, there are 6 abilities under
perception and 4 abilities under reasoning. Level-3 (L3) further refines the ability dimensions at L2.
Facial expression recognition can be categorized into basic and compound types. Face attack detec-
tion includes deepfake detection and face anti-spoofing. Face recognition involves five scenarios:
basic, cross-pose, cross-age, similar-looking, and occluded. Spatial relation understanding concerns
relative position and count. Social relation understanding includes social relationship recognition
and identity reasoning. Please refer to Appendix A.1 for detailed definitions and examples of these
abilities.

2.2 SEMI-AUTOMATIC DATA PIPELINE

Based on the hierarchical ability taxonomy defined in Section 2.1, we collect 16 public datasets from
the face and human community, covering each L3 ability. Then, we employ a semi-automatic data
pipeline to produce problems for the Face-Human-Bench.

An original sample Si from public datasets can be represented as a binary tuple (Ii, Li), where Ii
denotes an original image set and Li denotes an original label set. Note that we use “image set” and
“label set” to describe the composition of one sample because, in some datasets, a single sample may
consist of multiple images or labels. For instance, in face recognition, a sample includes a pair of

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

face images to verify identity, and in facial attribute recognition, a sample may involve 40 attribute
labels.

For ease of evaluation, we adopt multiple-choice as the problem format in our Face-Human-Bench.
Each problem Pi corresponds to a quadruple (Vi, Qi, Oi, Ai). Here, Vi refers to the images obtained
via the image processing pipeline pimage : I → V. pimage performs an operation such as cropping,
concatenating, adding boxes, or leaving the original images unchanged, depending on the ability to
test. Qi denotes the question. Each L3 ability includes a set of pre-written questions that share the
same semantics but exhibit diversity. When producing samples, a question Qi is randomly selected
from this question set. Oi is the set of n options (o1, o2, ..., on), where 2 ≤ n ≤ 4. These options are
obtained through the text processing pipeline ptext : L → O. ptext converts the original labels into
one correct option and n − 1 incorrect options. For some tasks, ChatGPT (OpenAI, 2023a) is used
within ptext to assist in generating incorrect options or adjusting options at the sentence level (fixing
grammar or re-wording sentences for fluency. Ai is the correct answer to the problem. The produced
Pi will be checked by data reviewers to ensure that the options are unambiguous and there is one
and only one correct answer. The problems that do not meet the requirements will be removed.

In summary, our semi-automatic data pipeline leverages image and text processing pipelines, pimage
and ptext, to transform original samples into multiple-choice format problems. These problems are
then manually checked to ensure quality. We obtain a benchmark with a development set of 900
problems for the MLLM community to evaluate during training iterations and a test set of 1800
problems for the formal evaluation in our paper. Additionally, the English problems are translated
into Chinese to create a Chinese version of the benchmark. For more details on data sources, statis-
tics, and the semi-automatic data pipeline, please refer to Appendices A.2 and A.3.

3 EXPERIMENT

3.1 EXPERIMENTAL SETUP

Evaluation Protocols. We use the weighted accuracy of multiple-choice problems as the evaluation
score. As shown in Figure 1, the proportion of the sectors represents the weight of the corresponding
abilities in the overall score on the Face-Human-Bench. Note that we set equal weights for each L2
ability. 1 To prevent models from favoring certain option letters over others, we shuffle the options
to ensure the correct answers are evenly distributed across all option letters. During the testing, we
add some constraint instructions to ensure MLLMs output only option letters as much as possible.
2 After obtaining the MLLM’s response, we use regular expressions to extract the option letters. If
this fails, we follow the implementation of MMBench (Liu et al., 2023c) using ChatGPT (OpenAI,
2023a) to extract the choices. 3

Models. We evaluate 25 MLLMs in different sizes from 13 model families. For open-source models,
we select LLaVA-13B (Liu et al., 2023b), LLaVA-1.5-7B/13B (Liu et al., 2023a), LLaVA-Next-
7B/13B/34B (Liu et al., 2024a), MiniGPT-4-7B/13B (Zhu et al., 2024), InstructBLIP-7B/13B (Dai
et al., 2023), Qwen-VL-Chat (Bai et al., 2023), InternLM-XComposer2-VL-7B (Dong et al., 2024),
Yi-VL-6B (Young et al., 2024), InternVL-Chat-v1.2-Plus (Chen et al., 2023), InternVL-Chat-v1.5
(Chen et al., 2023), DeepSeek-VL-1.3B/7B-Chat (Lu et al., 2024), CogVLM2-19B-Chat (Hong
et al., 2024), GLM-4V-9B (Hong et al., 2024), LLaVA-OneVison-0.5B/7B (Li et al., 2024). For
closed-source models, we use Gemini-1.5-Pro (Reid et al., 2024), Claude-3.5-Sonnet (Anthropic,
2024a), GPT-4V (OpenAI, 2023b), and GPT-4o OpenAI (2024). For more details on these models,
please refer to Appendix B.1.

3.2 MAIN RESULTS

Table 1 shows the performance of all evaluated MLLMs at different levels of abilities on the Human-
Face-Bench (English) 4 under the zero-shot setting. Overall scores range from 27.9% to 76.4%,
demonstrating the effectiveness of the Face-Human-Bench in distinguishing the abilities of MLLMs

1For detailed weights of each subset in Face-Human-Bench, please refer to Appendix A.2.
2For the prompt template under zero-shot setting, please refer to Appendix B.2.1.
3For the prompt for choice extraction, please refer to Appendix B.2.2.
4For the results of the Chinese version, please refer to Appendix C.2.

4

Under review as a conference paper at ICLR 2025

Table 1: Zero-shot scores of MLLMs on the hierarchical Face-Human-Bench (EN). The highest
scores for open-source and closed-source MLLMs are marked in blue and green respectively.

Model

Random
LLaVA

-OneVision-0.5B

DeepSeek

-VL-1.3B-Chat

Yi-VL-6B
MiniGPT-4-7B
InstructBLIP-7B
Qwen-VL-Chat
DeepSeek

-VL-7B-Chat

LLaVA-1.5-7B
LLaVA-NeXT-7B
InternLM
-XComposer2-VL-7B
LLaVA

-OneVision-7B
CogVLM2-19B-Chat
GLM-4V-9B
MiniGPT-4-13B
InstructBLIP-13B
LLaVA-13B
LLaVA-1.5-13B
LLaVA-NeXT-13B
InternVL-Chat-v1.5
LLaVA-NeXT-34B
InternVL

-Chat-v1.2-Plus

Gemini-1.5-Pro
Claude-3.5-Sonnet
GPT-4V
GPT-4o

Model

Random
LLaVA

-OneVision-0.5B

DeepSeek

-VL-1.3B-Chat

Yi-VL-6B
MiniGPT-4-7B
InstructBLIP-7B
Qwen-VL-Chat
DeepSeek

-VL-7B-Chat

LLaVA-1.5-7B
LLaVA-NeXT-7B
InternLM
-XComposer2-VL-7B
LLaVA

-OneVision-7B
CogVLM2-19B-Chat
GLM-4V-9B
MiniGPT-4-13B
InstructBLIP-13B
LLaVA-13B
LLaVA-1.5-13B
LLaVA-NeXT-13B
InternVL-Chat-v1.5
LLaVA-NeXT-34B
InternVL

-Chat-v1.2-Plus

Gemini-1.5-Pro
Claude-3.5-Sonnet
GPT-4V
GPT-4o

Attr.

25.0

36.0

36.5

75.5
24.0
39.5
55.5

57.5

61.0
69.5

92.0

90.5

75.0
79.5
20.5
25.5
32.0
75.5
77.5
92.0
95.0

86.0

66.0
83.5
77.5
77.0

Age

25.0

43.0

49.0

51.7
17.7
36.7
49.7

52.3

49.3
50.0

53.0

60.3

57.3
55.7
24.3
38.3
40.7
58.7
46.7
61.7
58.7

59.7

40.0
54.0
53.7
61.0

Attr. Action

25.0

47.0

40.5

67.0
15.5
31.0
49.5

64.0

62.0
62.0

87.5

90.5

70.5
85.5
19.5
33.5
27.0
60.5
69.5
89.5
91.5

90.0

50.0
71.5
73.0
63.5

25.0

78.0

66.0

73.0
27.0
46.0
83.0

78.0

71.0
80.0

87.0

92.0

93.0
94.0
46.0
71.0
66.0
72.0
74.0
89.0
88.0

92.0

75.0
90.0
78.0
81.0

Expression

Face Understanding

Attack Detection

Face Recognition

Basic Comp. Mean DFD FAS mean
50.0
25.0

50.0

25.0

25.0

50.0

Basic
50.0

71.0

60.0

65.5

46.0

55.0

50.5

50.0

57.0

65.0
26.0
38.0
65.0

68.0

62.0
72.0

76.0

74.0

71.0
79.0
35.0
50.0
56.0
72.0
71.0
72.0
80.0

74.0

72.0
73.0
75.0
83.0

50.0

52.0
24.0
40.0
50.0

58.0

58.0
62.0

68.0

62.0

70.0
74.0
26.0
42.0
30.0
54.0
52.0
68.0
62.0

60.0

48.0
32.0
48.0
62.0

53.5

58.5
25.0
39.0
57.5

63.0

60.0
67.0

72.0

68.0

70.5
76.5
30.5
46.0
43.0
63.0
61.5
70.0
71.0

67.0

60.0
52.5
61.5
72.5

50.0

34.0
31.5
50.5
51.0

46.0

55.5
59.5

41.0

35.0

37.0
46.0
49.5
57.5
55.0
51.0
50.0
71.5
63.5

65.5

31.0
55.0
50.5
53.0

50.0

43.0
40.5
53.0
54.0

53.0

55.0
58.5

54.0

56.0

51.0
50.0
37.5
52.0
54.0
54.0
54.0
67.0
60.5

65.0

21.0
45.0
58.5
64.0

50.0

38.5
36.0
51.8
52.5

49.5

55.3
59.0

47.5

45.5

44.0
48.0
43.5
54.8
54.5
52.5
52.0
69.2
62.0

65.3

26.0
50.0
54.5
58.5

50.0

50.0
38.0
52.0
66.0

54.0

54.0
62.0

54.0

58.0

66.0
68.0
52.0
48.0
52.0
54.0
58.0
90.0
92.0

94.0

98.0
92.0
96.0
96.0

Human Understanding

C.P.
50.0

42.0

50.0

48.0
56.0
58.0
52.0

52.0

52.0
50.0

54.0

42.0

36.0
54.0
46.0
52.0
60.0
48.0
54.0
60.0
70.0

74.0

82.0
64.0
72.0
72.0

C.A.
50.0

44.0

50.0

48.0
44.0
48.0
54.0

50.0

50.0
48.0

50.0

34.0

44.0
54.0
42.0
52.0
52.0
54.0
54.0
60.0
70.0

62.0

86.0
76.0
92.0
74.0

S.L. Occ. Mean
50.0
50.0
50.0

50.0

38.0

44.8

50.0

50.0
48.0
52.0
58.0

48.0

56.0
56.0

56.0

42.0

46.0
62.0
46.0
50.0
40.0
48.0
56.0
60.0
72.0

72.0

90.0
74.0
82.0
76.0

50.0

44.0
34.0
54.0
54.0

50.0

50.0
50.0

36.0

34.0

48.0
52.0
48.0
52.0
52.0
50.0
56.0
52.0
56.0

52.0

72.0
66.0
64.0
50.0

50.0

48.0
44.0
52.8
56.8

50.8

52.4
53.2

50.0

42.0

48.0
58.0
46.8
50.8
51.2
50.8
55.6
64.4
72.0

70.8

85.6
74.4
81.2
73.6

Spatial Relation
CC
25.0

Mean
25.0

RPU
25.0

Social Relation

SRR
25.0

IR Mean
25.0
25.0

44.0

22.7

33.3

62.0

94.0

78.0

40.0

54.0
18.0
34.0
54.0

52.0

54.0
62.0

58.0

58.0

68.0
62.0
42.0
38.0
36.0
44.0
46.0
62.0
64.0

66.0

52.0
54.0
38.0
50.0

26.0

24.0
16.7
0.7
34.0

35.3

30.0
24.7

41.3

48.0

33.3
32.0
17.3
28.0
30.7
26.0
28.0
50.7
59.3

58.7

25.3
42.7
71.3
58.7

33.0

39.0
17.3
17.3
44.0

43.7

42.0
43.3

49.7

53.0

50.7
47.0
29.7
33.0
33.3
35.0
37.0
56.3
61.7

62.3

38.7
48.3
54.7
54.3

64.0

48.0
24.0
16.0
64.0

70.0

68.0
62.0

64.0

66.0

74.0
68.0
30.0
52.0
38.0
60.0
58.0
70.0
64.0

76.0

74.0
74.0
68.0
66.0

72.0

66.0
34.0
28.0
70.0

76.0

78.0
86.0

86.0

86.0

92.0
88.0
50.0
86.0
76.0
60.0
70.0
74.0
86.0

96.0

84.0
80.0
84.0
94.0

68.0

57.0
29.0
22.0
67.0

73.0

73.0
74.0

75.0

76.0

83.0
78.0
40.0
69.0
57.0
60.0
64.0
72.0
75.0

86.0

79.0
77.0
76.0
80.0

Re-ID

Face Human

Per. Rea. Overall

50.0

45.0

50.0

47.0
44.0
51.0
50.0

57.0

63.0
56.0

59.0

61.0

56.0
67.0
48.0
51.0
55.0
54.0
63.0
77.0
88.0

85.0

82.0
74.0
83.0
79.0

35.0

48.0

47.8

54.4
29.3
43.9
54.4

54.6

55.6
59.7

62.9

61.3

59.0
63.5
33.1
43.1
44.3
60.1
58.7
71.5
71.7

69.7

55.6
62.9
65.7
68.5

30.0

56.3

51.5

56.6
26.6
33.5
58.7

63.1

62.2
63.1

71.6

74.5

70.6
74.3
36.6
51.5
47.7
56.3
61.5
76.8
80.8

83.1

64.9
72.2
72.9
71.6

29.2

37.5

53.3

50.3

49.3

60.7
24.2
40.7
57.9

60.7

59.8
64.6

73.2

74.5

68.4
73.2
30.7
44.9
43.9
63.7
63.5
78.6
77.7

76.7

52.8
70.0
66.4
68.9

50.3

47.8
33.6
35.8
54.5

56.1

57.6
56.6

58.4

58.0

59.4
62.5
41.1
51.0
49.1
50.0
54.9
67.4
74.2

76.0

71.3
68.4
73.7
71.7

32.5

52.1

49.7

55.5
27.9
38.7
56.5

58.9

58.9
61.4

67.3

67.9

64.8
68.9
34.9
47.3
46.0
58.2
60.1
74.1
76.3

76.4

60.3
67.5
69.3
70.0

in face and human understanding. We visualize the overall scores of MLLMs in Figure 2. Our
findings can be summarized as follows.

Overall Performance.
(1) The top three performing open-source models in terms of the over-
all score are InternvL-Chat-v1.2-Plus, LLaVA-Next-34B, and InternVL-Chat-v1.5. These models’
LLMs have the largest number of parameters among all open-source models we evaluate. (2) Gen-
erally, open-source models within the same series tend to show improved performance with increas-
ing parameter scale. However, there are exceptions; for instance, the 13B version of LLaVA-1.5

5

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Figure 2: The leaderboard of MLLMs on our proposed Face-Human-Bench (English).

and LLaVA-Next perform slightly worse than their 7B counterparts. (3) Under the zero-shot set-
ting, the best closed-source model, GPT-4o, does not surpass the performance of the top-performing
open-source models. We believe this is because GPT-4o does not fully realize its potential under the
zero-shot setting. The experiments in Section 3.5 confirm our hypothesis. (4) Newer models show
significant improvements compared to earlier models. Among MLLMs with 7B parameters within
LLM, the recently released LLaVA-OneVision-7B performs best. Impressively, LLaVA-OneVision-
0.5B, with only 0.5B parameters within LLM, outperforms the earlier InstructBLIP-13B.

L2 and L3 Performance5 (1) At L2 and L3, the
best performance among open-source models is usu-
ally achieved by one of InternvL-Chat-v1.2-Plus,
LLaVA-Next-34B, and InternVL-Chat-v1.5. Specif-
ically, GLM-4V-9B achieves the best results in com-
pound expression recognition (L3), facial expression
recognition (L2), and action recognition (L2) and
CogVLM2-19B-Chat achieves the best result in rel-
ative position understanding (L3).
(2) At L2 and
L3, the best performance among closed-source mod-
els is usually achieved by GPT-4o or GPT-4v. No-
tably, Gemini-1.5-Pro demonstrates outstanding face
recognition ability (L2), achieving the best perfor-
mance among all models with a score of 85.6%.

3.3 CORRELATION BETWEEN ABILITIES

In this section, we examine whether improving one
ability in a model will enhance another by calcu-
lating the Pearson Correlation Coefficient between
levels, using the evaluation
abilities at different
scores from Section 3.2. At L1, the correlation coef-
ficient of face and human understanding is 0.94 and
the correlation coefficient of perception and reason-
ing is 0.79, both indicating significant positive correlations, as shown in Figure 3(a) and Figure 3(b).
We further investigate the correlations between L2 abilities, resulting in the correlation coefficient
matrix shown in Figure 3(c). For clarity, we have drawn this as a lower triangular matrix. Our find-
ings can be summarized as follows: (1) For the three face understanding abilities—facial attribute
recognition, age estimation, and facial expression recognition—there are high positive correlations
between each pair. (2) For the four human understanding abilities—human attribute recognition,
action recognition, spatial relation understanding, and social relation understanding—there are high
positive correlations between each pair. (3) For the three face understanding abilities and four hu-

Figure 3: Correlation between abilities.

5For the visualization of L2 and L3 results, please refer to the Appendix C.1.

6

0.5B1B7B13B20B34BClosed-SourceMLLMScale of LLM within MLLM (Billion Parameters)304050607080Overall Score (%) on Face-Human-Bench (EN)LLaVA-OneVision-0.5BDeepSeek-VL-1.3B-ChatYi-VL-6BMiniGPT-4-7BInstructBLIP-7BQwen-VL-ChatDeepseek-VL-7B-ChatLLaVA-1.5-7BLLaVA-NeXT-7BInternLM-XComposer2-VL-7BLLaVA-OneVision-7BCogVLM2-19B-ChatGLM-4V-9BMiniGPT-4-13BInstructBLIP-13BLLaVA-13BLLaVA-1.5-13BLLaVA-NeXT-13BInternVL-Chat-v1.5LLaVA-NeXT-34BInternVL-Chat-v1.2-PlusGemini-1.5-ProClaude-3.5-SonnetGPT-4VGPT-4o3040506070Overall  Score (%)1020304050607080Perception (%)20304050607080Reasoning (%)maxminmaxmin3040506070Overall  Score (%)F. Attr.AgeExpr.AttackFRH. Attr.ActionSpatialSocialRe-IDRe-IDSocialSpatialActionH. Attr.FRAttackExpr.AgeF. Attr.(c)0.00.20.40.60.81.0  Correlation  Coefficient1020304050607080Face Understanding (%)(a)20304050607080Human Understanding (%)maxminmaxmin(1)(2)(3)(5)(4)(b)Under review as a conference paper at ICLR 2025

man understanding abilities mentioned above, there are high positive correlations between each pair.
(4) The two identity recognition tasks—face recognition and person re-identification—show a high
positive correlation. (5) The correlation between face attack detection and any other ability is low.
In Appendix C.3, we further present the correlations between L3 abilities.

3.4 RELATIVE POSITION OF TARGETS

Figure 4: (a) The versions used for the three face understanding abilities. (b) The versions used for
human attribute recognition. (c) When MLLMs are evaluated with different versions, the wording
of the questions varies slightly.

Figure 5: The performance differences between the two versions across various models. For the
three face understanding abilities, we show the performance of the original version minus that of
the cropped version. For human attribute recognition, we show the performance of the box-added
version minus that of the cropped version.

We investigate the impact of the relative position of targets on performance in four L3 abilities: facial
attribute recognition, age estimation, basic expression recognition, and human attribute recognition.
As shown in Figure 4, for the three face understanding abilities, we provide both the original and
cropped versions, where only one person is included but the relative position varies. For human
attribute recognition, we offer box-added and cropped versions. In the box-added version, multiple
people are included, with the target to be discussed indicated by a red box. Figure 5 illustrates
the performance differences between the two versions across various models. Our findings can be
summarized as follows.

Face Understanding Abilities.
(1) Preferences for either version depend on the model and the
ability, with no overarching trend observed. (2) A model’s preference can vary across different face
understanding abilities. For example, Yi-VL-6B shows no significant preference for facial attribute
recognition, prefers the original images for age estimation, and favors cropped images for basic
expression recognition. We think that this phenomenon may occur because MLLMs have been
trained using images with different target relative positions when aligning visual information for
different facial features.

Human Attribute Recognition. The majority of models perform better on the cropped version.
This indicates that these models still struggle to accurately understand a specific individual when
there are multiple people in the image.

We define the relative position sensitivity score (RPSS) as the sum of the absolute differences in
scores between the two versions across the four tasks. This metric can serve as an effective reference

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Question for Cropped/Original Version: Which of the following descriptions best matches the person in the picture?Question for Box-Added Version: Which of the following descriptions best matches the person in the red boxof the picture?OriginalCroppedBox-AddedCropped(a)(b)(c)LLaVA-OneVision-0.5BDeepSeek-VL-1.3B-ChatYi-VL-6BMiniGPT-4-7BInstructBLIP-7BQwen-VL-ChatDeepSeek-VL-7B-ChatLLaVA-1.5-7BLLaVA-NeXT-7BInternLM-XComposer2-VL-7BLLaVA-OneVision-7BCogVLM2-19B-ChatGLM-4V-9BMiniGPT-4-13BInstructBLIP-13BLLaVA-13BLLaVA-1.5-13BLLaVA-NeXT-13BInternVL-Chat-v1.5LLaVA-NeXT-34BInternVL-Chat-v1.2-PlusGemini-1.5-ProClaude-3.5-SonnetGPT-4VGPT-4o−30−20−1001020Performance DifferenceFacial Attribute RecognitionAge EstimationBasic Expression RecognitionHuman Attribute RecognitionUnder review as a conference paper at ICLR 2025

Table 2: Scores of the best open-source model, InternVL-Chat-v1.2-Plus, and the best closed-source
model, GPT-4o, under different settings. ZS is short for Zero-Shot, H is short for Hints, VCoT is
short for Vanilla CoT, 1TCoT is short for 1-stage Task-specific CoT. 2TCoT is short for 2-stage
Task-specific CoT. Q is short for Question. O is short for Options. A is short for Answer. R is short
for Relevant Analysis. The highest scores for open-source and closed-source MLLMs are marked
in blue and green respectively.

Setting

Format

Open-Source: InternVL-Chat-v1.2-Plus
Per. Rea. Overall
Face Human
QO→A
76.7
69.7
ZS
QOH→A
76.4
68.4
H
QOH→RA
75.9
69.1
H+VCoT
H+1TCoT QOH→RA
75.6
68.6
H+2TCoT QOH→R, QOHR→A 69.1
75.8

76.0
75.9
74.8
74.3
71.8

76.4
75.9
75.7
75.0
74.1

83.1
83.2
82.5
81.4
79.1

Close-Source: GPT-4o

Face Human
68.5
72.2
76.4
77.9
77.0

71.6
74.6
80.7
81.9
81.2

Per. Rea. Overall
68.9
70.4
78.2
79.0
78.4

70.0
73.4
78.6
79.9
79.1

71.7
78.0
77.2
81.2
77.2

for training MLLMs with more robust visual alignment for face and human understanding. We
observe that InternLM-XComposer2-VL-7B, LLaVA-OneVision-7B, InternVL-Chat-v1.5, LLaVA-
NeXT-34B, and InternVL-Chat-v1.2-Plus not only perform well in the four tasks but also exhibit low
sensitivity scores. Among them, InternLM-XComposer2-VL-7B has the lowest sensitivity score of
only 3.7%.6

3.5 COT PROMPTING

In this section, we select InternVL-Chat-v1.2-
Plus and GPT-4o to explore whether incorporat-
ing hints and Chain-of-Thought (CoT) instruc-
tions in the prompts can enhance the MLLMs’
performance. These two models have achieved
the best overall performance in the main exper-
iment among open-source models and closed-
source models respectively. A hint involves tips
on how to answer the question. For example,
the hint for person re-identification is “if two
people have significant differences in posture
and their faces are relatively blurry, the main
basis for determining whether they are the same
person is their clothing characteristics.” CoT in-
structions, on the other hand, guide MLLMs
to articulate the reasoning process leading to
the answer. The vanilla CoT instruction sim-
ply requires the model to “analyze the question
and options step by step”, whereas task-specific
CoT instructions provide more tailored guidance based on the task. For example, for the deepfake
detection task, the prompt might instruct the model to “analyze whether there are any artifacts in the
facial image.” Following Multi-modal CoT (Zhang et al., 2024), we also conduct ablation experi-
ments with both 1-stage and 2-stage frameworks. In the 1-stage framework, MLLMs are required to
sequentially output the relevant analysis (rationale) and the answer in one round of dialogue. In the
2-stage framework, MLLMs first output the relevant analysis (rationale) in the first round and then
provide the answer in the second round. Hints and task-specific CoT instructions for each L3 ability
can be found in Appendix B.2.3.

Figure 6: Main reasons of performance improve-
ments for each L2 ability are highlighted in red.

Table 2 presents the performance of InternVL-Chat-v1.2-Plus and GPT-4o after incorporating hints
and three different CoT settings. The results indicate that including hints and CoT instructions
does not improve the performance of the open-source model; in fact, it may even cause a slight
performance decline. By analyzing the outputs, we find that the open-source model does not provide
rationales in its responses after adding CoT instructions to prompts. We believe this could be due
to the model’s insufficient generalization capabilities, preventing it from understanding the CoT
instructions. In contrast, the closed-source GPT-4o shows significant performance improvements.

6For more models’ RPSS, please refer to the Appendix C.4.

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

ZSHH+VCoTH+1TCoTH+2TCoT5075100Score (%)(a) Facial Attribute RecognitionZSHH+VCoTH+1TCoTH+2TCoT5075100Score (%)(f) Human Attribute RecognitionZSHH+VCoTH+1TCoTH+2TCoT2550  75Score (%)(b) Age EstimationZSHH+VCoTH+1TCoTH+2TCoT5075100Score (%)(g) Action RecognitionZSHH+VCoTH+1TCoTH+2TCoT5075100Score (%)(c) Facial Expression RecognitionZSHH+VCoTH+1TCoTH+2TCoT2550  75Score (%)(h) Spatial Relation UnderstandingZSHH+VCoTH+1TCoTH+2TCoT5075100Score (%)(d) Face Attack DetectionZSHH+VCoTH+1TCoTH+2TCoT5075100Score (%)(i) Social Relation UnderstandingZSHH+VCoTH+1TCoTH+2TCoT5075100Score (%)(e) Face RecognitionZSHH+VCoTH+1TCoTH+2TCoT5075100Score (%)(j) Person Re-IdentificationUnder review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Table 3: Comparison between MLLMs and specialist models on 13 L3 abilities. The best-
performing MLLMs are highlighted in blue, while abilities where MLLMs perform significantly
worse than specialist models are marked in orange.

L3 Ability

Dataset

Matric
Random
InternVL-Chat-v1.5
LLaVA-NeXT-34B
InternVL-Chat-v1.2-Plus
Best of The Above 3
Early Specialist Model
Relative Score
Need Specialist Model
L3 Ability
Dataset
Matric
Random
InternVL-Chat-v1.5
LLaVA-NeXT-34B
InternVL-Chat-v1.2-Plus
Best of The Above 3
Early Specialist Model
Relative Score
Need Specialist Model

Age

UTKFace

MAE ↓
27.89
6.43
6.01
5.21
5.21
5.47
1.01
No.
Basic FR
LFW
ACC ↑
50.05
83.68
91.32
92.57
92.57
99.50
0.86
No.

Expression

Deepfake

Spoofing

Action

Counting

RAF-DB
(Basic)
ACC ↑
13.85
72.23
77.71
76.40
77.71
74.20
1.06
No.
C.P. FR
CPLFW
ACC ↑
49.75
58.13
65.87
67.98
67.98
87.47
0.48
Yes.

RAF-DB
(Compound)
ACC ↑
8.08
42.93
41.04
30.56
42.93
44.55
0.96
No.
C.A. FR
CALFW
ACC ↑
50.12
61.40
62.07
66.50
66.50
92.43
0.39
Yes.

FF++

SiW-Mv2 HICO-DET

ShTech-A

ACC ↑
50.84
56.21
53.42
52.89
56.21
82.01
0.17
Yes.
S.L. FR
SLLFW
ACC ↑
50.18
56.72
70.25
68.50
70.25
98.40
0.42
Yes.

ACER ↓
50.05
14.84
22.38
19.92
14.84
9.40
0.87
No.
Occ. FR
MLFW
ACC ↑
50.05
52.15
53.73
58.65
58.65
82.87
0.26
Yes.

mAP ↑
9.32
22.29
13.74
12.25
22.29
19.81
1.24
No.

MAE ↓
1512.65
2195.69
1592.55
2518.25
1592.55
110.20
-0.06
Yes.
Re-ID
Market1501
ACC ↑7
49.47
77.53
85.67
88.73
88.73
95.26
0.86
No.

Adding hints leads to a 3.4% improvement compared to the zero-shot setting. Building upon this,
vanilla CoT, 1-stage task-specific CoT, and 2-stage task-specific CoT further improve performance
by 5.2%, 6.5%, and 5.7%, respectively. Ultimately, the combination of hints and 1-stage task-
specific CoT instructions emerge as the best setting for overall performance.

In Figure 6, we further explore the main reasons for the performance improvements of GPT-4o in
each ability at L2. Hints significantly improve performance in face attack detection, face recog-
nition, and person re-identification, while CoT instructions significantly improve performance in
facial attribute recognition, face attack detection, human attribute recognition, and action recogni-
tion. For the reasons behind the performance improvements in each ability at L3, please refer to
Appendix C.5.

3.6 SPECIALIST MODELS SIGNIFICANTLY OUTPERFORMING MLLMS

In this section, we explore whether specialist models corresponding to 13 L3 abilities can be used
to enhance MLLMs. 8 We directly test the performance of MLLMs using original datasets from
the face and human community to facilitate comparison with specialist models. We design a set
of prompt templates to transform the classification problems into multiple-choice problems and the
regression problems (age estimation and crowd counting) into fill-in-the-blank problems. 9 Special-
ist models are generally trained and tested on data from the same distribution. They can achieve
high performance even if the test labels contain noise. However, the visual information learned by
MLLMs and the original datasets used for testing may exhibit data distribution bias. To enable an ef-
fective comparison, we utilize early specialist models (which emerged after the widespread adoption
of deep learning) as a reference to judge the performance of MLLMs on these tasks.10

We further define the relative performance score S to normalize performances across different tasks:
S = (Pm − Pr)/(Ps − Pr), where Pm is the performance of the MLLM. Here, we take the highest-
performing model among InternVL-Chat-v1.2-Plus, LLaVA-Next-34B , and InternVL-Chat-v1.5
(the top three models in the main experiment). Pr is the performance of random responses, and Ps
is the performance of the early specialist model. This metric typically ranges from 0 to 1, where a
higher relative score indicates stronger abilities of MLLMs on the corresponding task. A relative

7The original metric for Market1501 is mAP. For easier comparison, we create a new testing protocol
consisting of 750 positive pairs and 750 negative pairs. The ACC can be calculated in the same way as for
LFW. We re-evaluate the early specialist model for Re-ID using the new protocol.

8We explain the reasons for not conducting experiments on the remaining 5 L3 abilities in Appendix B.3.1.
9For prompt templates, please refer to Appendix B.3.2.
10For the early specialist models used for comparison, please refer to Appendix C.6.

9

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

score below 0 stands for even worse results than random responses, whereas a score above 1 indicates
the performance surpassing the corresponding specialist models for reference. As shown in Table 3,
there is no need for MLLMs to introduce specialist models to enhance the response quality when the
abilities of age estimation, facial expression recognition, face anti-spoofing, action recognition, and
person re-identification are needed. In contrast, for deepfake detection and crowd counting tasks,
the MLLM significantly underperforms specialist models. Moreover, for face recognition, MLLMs
can approach the specialist model under the basic scenario but indicate poor performance under
more challenging scenarios, such as cross-pose, cross-age, similar-looking, and occluded. To sum
up, we recommend incorporating the corresponding specialist models into multi-modal assistants
for applications where deepfake detection, crowd counting, and accurate face recognition are re-
quired. Appendix F provides a demonstration of how to enhance multi-modal assistant responses
with specialist models.

4 RELATED WORK

Evaluation of MLLMs about Face and Human Understanding. Currently, there is no dedicated
benchmark evaluating the face and human understanding abilities of MLLMs. Some efforts aim at
comprehensively benchmarking MLLMs, containing some ability dimensions about face and human
understanding. LAMM (Yin et al., 2023) evaluates 9 different 2D vision tasks using 11 existing
public datasets. Among these, the facial classification task utilizes the CelebA (Liu et al., 2015)
dataset to evaluate the accuracy of smile detection and hair attribute classification. MME (Fu et al.,
2023) includes the celebrity recognition ability, requiring MLLMs to respond with Yes/No answers.
SEED-Bench (Li et al., 2023a) includes the action recognition ability, where the inputs consist of
multiple frames taken from a video, and MLLMs are required to choose the correct answer from
four descriptions. MMBench (Liu et al., 2023c) includes the most extensive set of abilities related to
faces and humans: celebrity recognition, action recognition, identity reasoning, and social relation,
all of which are tested using multiple-choice problems. Considering the importance of faces and
humans in multimedia, these evaluations are insufficient.

Face and Human Understanding. Face and human understanding is among the earliest research
topics in artificial intelligence with successful applications. During the 2010s, the introduction of
deep learning, particularly convolutional neural networks, significantly advanced face and human
perception.
In that era, numerous high-quality datasets were proposed for training and evaluat-
ing tasks of face attribute recognition (Liu et al., 2015), age estimation (Rothe et al., 2015; Escalera
et al., 2015; Zhang et al., 2017), facial expression recognition (Barsoum et al., 2016; Li et al., 2017b;
Mollahosseini et al., 2019), deepfake detection (R¨ossler et al., 2019; Dolhansky et al., 2019), face
anti-spoofing (Liu et al., 2018; 2019), face recognition (Yi et al., 2014; Guo et al., 2016; Zheng
et al., 2017; Deng et al., 2017; Zheng & Deng, 2018), human attribute recognition (Li et al., 2016;
Liu et al., 2017), human-object interaction detection (Gupta & Malik, 2015; Xu et al., 2019), crowd
counting (Zhang et al., 2016), social relationship recognition Sun et al. (2017); Li et al. (2017a) and
person re-ideitification Li et al. (2014); Zheng et al. (2015). Entering the 2020s, a new paradigm
emerged, which initially pre-trains a task-agnostic backbone and then based on this, trains a uni-
fied face or human model (Ci et al., 2023; Wang et al., 2023b; Qin et al., 2024) to simultaneously
handle multiple face and human understanding tasks within a unified structure. In our evaluation,
we observe that in certain tasks, MLLMs do not perform as well as specialist models. Utilizing
these unified face or human models as the specialist models to help MLLMs can greatly facilitate
deployment.

5 CONCLUSION

In this work, we propose the hierarchical Face-Human-Bench, the first benchmark specifically de-
signed to evaluate MLLMs’ face and human understanding abilities. We comprehensively and sci-
entifically assess the performance of 25 mainstream MLLMs with our benchmark. We reveal the
correlations between abilities and explore the impact of the relative position of targets and CoT
prompting on the performance of MLLMs. Inspired by multimodal agents, we investigate which
abilities of MLLMs need to be supplemented by specialist models. Our work will provide the face
and human community valuable insights on how to more effectively leverage multi-modal assistants
in applications related to “faces and humans.”

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2023. URL https://www-cdn.

anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_
Card_Claude_3.pdf.

Anthropic.

Claude 3.5 sonnet, 2024a. URL https://www.anthropic.com/news/

claude-3-5-sonnet.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang
Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local-
ization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.

Emad Barsoum, Cha Zhang, Cristian Canton-Ferrer, and Zhengyou Zhang. Training deep networks
for facial expression recognition with crowd-sourced label distribution. In ICMI, pp. 279–283.
ACM, 2016. doi: 10.1145/2993148.2993165.

Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural
networks with application to age estimation. Pattern Recognit. Lett., 140:325–331, 2020. doi:
10.1016/J.PATREC.2020.11.008.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong
Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl:
Scaling up vision foundation models and aligning for generic visual-linguistic tasks. CoRR,
abs/2312.14238, 2023.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong,
Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to
commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024.

Franc¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, pp.

1800–1807. IEEE Computer Society, 2017.

Yuanzheng Ci, Yizhou Wang, Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei
Yu, Donglian Qi, and Wanli Ouyang. Unihcp: A unified model for human-centric perceptions. In
CVPR, pp. 17840–17852. IEEE, 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang,
Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-
language models with instruction tuning. In NeurIPS, 2023.

Weihong Deng, Jiani Hu, Nanhai Zhang, Binghui Chen, and Jun Guo. Fine-grained face verification:
FGLFW database, baselines, and human-dcmn partnership. Pattern Recognit., 66:63–73, 2017.
doi: 10.1016/J.PATCOG.2016.11.023.

Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton-Ferrer. The deep-

fake detection challenge (DFDC) preview dataset. CoRR, abs/1910.08854, 2019.

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei,
Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-
form text-image composition and comprehension in vision-language large model. arXiv preprint
arXiv:2401.16420, 2024.

Sergio Escalera, Junior Fabian, Pablo Pardo, Xavier Bar´o, Jordi Gonz`alez, Hugo Jair Escalante,
Dusan Misevic, Ulrich Steiner, and Isabelle Guyon. Chalearn looking at people 2015: Apparent
age and cultural event recognition datasets and results. In ICCV Workshops, pp. 243–251. IEEE
Computer Society, 2015. doi: 10.1109/ICCVW.2015.40.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei
Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive
evaluation benchmark for multimodal large language models. CoRR, abs/2306.13394, 2023.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang,
Di Yin, Long Ma, Xiawu Zheng, Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing
Sun. VITA: towards open-source interactive omni multimodal LLM. CoRR, abs/2408.05211,
2024.

Xiao Guo, Yaojie Liu, Anil K. Jain, and Xiaoming Liu. Multi-domain learning for updating face
anti-spoofing models. In ECCV (13), volume 13673 of Lecture Notes in Computer Science, pp.
230–249. Springer, 2022.

Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset
and benchmark for large-scale face recognition. In ECCV (3), volume 9907 of Lecture Notes in
Computer Science, pp. 87–102. Springer, 2016. doi: 10.1007/978-3-319-46487-9\ 6.

Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. CoRR, abs/1505.04474, 2015.

Tengda Han, Max Bain, Arsha Nagrani, G¨ul Varol, Weidi Xie, and Andrew Zisserman. Autoad:

Movie description in context. In CVPR, pp. 18930–18940. IEEE, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. In CVPR, pp. 770–778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90.

Fabian Herzog, Xunbo Ji, Torben Teepe, Stefan H¨ormann, Johannes Gilg, and Gerhard Rigoll.
Lightweight multi-branch network for person re-identification. In 2021 IEEE international con-
ference on image processing (ICIP), pp. 1129–1133. IEEE, 2021.

Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng,
Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video
understanding. arXiv preprint arXiv:2408.16500, 2024.

Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild:
A database forstudying face recognition in unconstrained environments. In Workshop on faces
in’Real-Life’Images: detection, alignment, and recognition, 2008.

Shan Jia, Reilin Lyu, Kangran Zhao, Yize Chen, Zhiyuan Yan, Yan Ju, Chuanbo Hu, Xin Li,
Baoyuan Wu, and Siwei Lyu. Can chatgpt detect deepfakes? a study of using multimodal large
language models for media forensics. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 4324–4333, 2024.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei
Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint
arXiv:2408.03326, 2024.

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench-

marking multimodal llms with generative comprehension. CoRR, abs/2307.16125, 2023a.

Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. Dual-glance model for decipher-

ing social relationships. In ICCV, pp. 2669–2678. IEEE Computer Society, 2017a.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image
pre-training with frozen image encoders and large language models. In International conference
on machine learning, pp. 19730–19742. PMLR, 2023b.

Shan Li, Weihong Deng, and Junping Du. Reliable crowdsourcing and deep locality-preserving
learning for expression recognition in the wild. In CVPR, pp. 2584–2593. IEEE Computer Society,
2017b. doi: 10.1109/CVPR.2017.277.

Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. Deepreid: Deep filter pairing neural network

for person re-identification. In CVPR, pp. 152–159. IEEE Computer Society, 2014.

Yining Li, Chen Huang, Chen Change Loy, and Xiaoou Tang. Human attribute recognition by deep
In ECCV (6), volume 9910 of Lecture Notes in Computer Science, pp.

hierarchical contexts.
684–700. Springer, 2016.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction

tuning. CoRR, abs/2310.03744, 2023a.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS,

2023b.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.
Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://
llava-vl.github.io/blog/2024-01-30-llava-next/.

Xihui Liu, Haiyu Zhao, Maoqing Tian, Lu Sheng, Jing Shao, Shuai Yi, Junjie Yan, and Xiaogang
Wang. Hydraplus-net: Attentive deep features for pedestrian analysis. In ICCV, pp. 350–359.
IEEE Computer Society, 2017.

Xuannan Liu, Pei Pei Li, Huaibo Huang, Zekun Li, Xing Cui, Weihong Deng, Zhaofeng He, et al.
Fka-owl: Advancing multimodal fake news detection through knowledge-augmented lvlms. In
ACM Multimedia 2024, 2024b.

Xuannan Liu, Zekun Li, Peipei Li, Shuhan Xia, Xing Cui, Linzhi Huang, Huaibo Huang, Weihong
Deng, and Zhaofeng He. Mmfakebench: A mixed-source multimodal misinformation detection
benchmark for lvlms. CoRR, abs/2406.08772, 2024c.

Yaojie Liu, Amin Jourabloo, and Xiaoming Liu. Learning deep models for face anti-spoofing:
In CVPR, pp. 389–398. Computer Vision Foundation / IEEE

Binary or auxiliary supervision.
Computer Society, 2018.

Yaojie Liu, Joel Stehouwer, Amin Jourabloo, and Xiaoming Liu. Deep tree learning for zero-shot

face anti-spoofing. In CVPR, pp. 4680–4689. Computer Vision Foundation / IEEE, 2019.

Ye Liu, Junsong Yuan, and Chang Wen Chen. Consnet: Learning consistency graph for zero-shot

human-object interaction detection. In ACM Multimedia, pp. 4235–4243. ACM, 2020.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan,
Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal
model an all-around player? CoRR, abs/2307.06281, 2023c.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In
ICCV, pp. 3730–3738. IEEE Computer Society, 2015. URL https://doi.org/10.1109/
ICCV.2015.425.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng
Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong
Ruan. Deepseek-vl: Towards real-world vision-language understanding. CoRR, abs/2403.05525,
2024.

Ali Mollahosseini, Behzad Hassani, and Mohammad H. Mahoor. Affectnet: A database for facial
IEEE Trans. Affect. Comput., 10(1):

expression, valence, and arousal computing in the wild.
18–31, 2019. doi: 10.1109/TAFFC.2017.2740923.

OpenAI. Chatgpt. https://openai.com/blog/chatgpt/, 2023a.

OpenAI. Gpt-4v(ision) system card, 2023b.

OpenAI. Hello gpt-4o, 2024. URL https://openai.com/index/hello-gpt-4o/.

Lixiong Qin, Mei Wang, Xuannan Liu, Yuhang Zhang, Wei Deng, Xiaoshuai Song, Weiran Xu, and
Weihong Deng. Faceptor: A generalist model for face perception. CoRR, abs/2403.09500, 2024.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-
baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem-
ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint
arXiv:2403.05530, 2024.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Joseph P Robinson, Gennady Livitz, Yann Henon, Can Qin, Yun Fu, and Samson Timoner. Face
recognition: too bias, or not too bias? In Proceedings of the ieee/cvf conference on computer
vision and pattern recognition workshops, pp. 0–1, 2020.

Andreas R¨ossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias
Nießner. Faceforensics++: Learning to detect manipulated facial images. In ICCV, pp. 1–11.
IEEE, 2019.

Rasmus Rothe, Radu Timofte, and Luc Van Gool. DEX: deep expectation of apparent age from a
single image. In ICCV Workshops, pp. 252–257. IEEE Computer Society, 2015. doi: 10.1109/
ICCVW.2015.41.

Qianru Sun, Bernt Schiele, and Mario Fritz. A domain based approach to social relation recognition.

In CVPR, pp. 435–444. IEEE Computer Society, 2017.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu,
Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly
capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.

InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities,

2023.

Chengrui Wang, Han Fang, Yaoyao Zhong, and Weihong Deng. MLFW: A database for face recog-
In CCBR, volume 13628 of Lecture Notes in Computer Science, pp.

nition on masked faces.
180–188. Springer, 2022.

Hanlin Wang, Zhan Tong, Kecheng Zheng, Yujun Shen, and Limin Wang. Contextual AD narration

with interleaved multimodal sequence. CoRR, abs/2403.12922, 2024.

Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei
Liu. Cosface: Large margin cosine loss for deep face recognition. In CVPR, pp. 5265–5274.
Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00552.

Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang. Racial faces in the wild:
In Proceedings of the

Reducing racial bias by information maximization adaptation network.
ieee/cvf international conference on computer vision, pp. 692–702, 2019.

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang,
Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang.
Cogvlm: Visual expert for pretrained language models. CoRR, abs/2311.03079, 2023a.

Yizhou Wang, Yixuan Wu, Shixiang Tang, Weizhen He, Xun Guo, Feng Zhu, Lei Bai, Rui Zhao,
Jian Wu, Tong He, and Wanli Ouyang. Hulk: A universal knowledge translator for human-centric
tasks. CoRR, abs/2312.01697, 2023b.

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual
chatgpt: Talking, drawing and editing with visual foundation models. CoRR, abs/2303.04671,
2023.

Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mohan S. Kankanhalli. Learning to detect
human-object interactions with knowledge. In CVPR, pp. 2019–2028. Computer Vision Founda-
tion / IEEE, 2019.

Kaiyu Yang, Olga Russakovsky, and Jia Deng. Spatialsense: An adversarially crowdsourced bench-

mark for spatial relation recognition. In ICCV, pp. 2051–2060. IEEE, 2019.

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng
Liu, Ce Liu, Michael Zeng, and Lijuan Wang. MM-REACT: prompting chatgpt for multimodal
reasoning and action. CoRR, abs/2303.11381, 2023.

Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv

preprint arXiv:1411.7923, 2014.

14

Under review as a conference paper at ICLR 2025

Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang,
Zhiyong Wang, Lu Sheng, Lei Bai, Jing Shao, and Wanli Ouyang. LAMM: language-assisted
multi-modal instruction-tuning dataset, framework, and benchmark. In NeurIPS, 2023.

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng
Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang,
Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu,
Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and
Zonghong Dai. Yi: Open foundation models by 01.ai. CoRR, abs/2403.04652, 2024.

Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, and Qin Jin. Movie101: A new
In ACL (1), pp. 4669–4684. Association for Computational

movie understanding benchmark.
Linguistics, 2023.

Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan-
grui Ding, Songyang Zhang, Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei
Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang.
Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and
composition. CoRR, abs/2309.15112, 2023.

Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting
via multi-column convolutional neural network. In CVPR, pp. 589–597. IEEE Computer Society,
2016.

Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regression by conditional adversarial
autoencoder. In CVPR, pp. 4352–4360. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017.
463.

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal

chain-of-thought reasoning in language models. Trans. Mach. Learn. Res., 2024, 2024.

Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person

re-identification: A benchmark. In ICCV, pp. 1116–1124. IEEE Computer Society, 2015.

Tianyue Zheng and Weihong Deng. Cross-pose lfw: A database for studying cross-pose face recog-
nition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech.
Rep, 5:7, 2018.

Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age LFW: A database for studying cross-age

face recognition in unconstrained environments. CoRR, abs/1708.08197, 2017.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing
vision-language understanding with advanced large language models. In ICLR. OpenReview.net,
2024.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

Appendix

CONTENTS

1 Introduction

2 Face-Human-Bench

2.1 Hierarchical Ability Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2 Semi-Automatic Data Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 Experiment

3.1 Experimental Setup .

3.2 Main Results

.

.

.

.

.

.

.

.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.3 Correlation Between Abilities

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.4 Relative Position of Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.5 CoT prompting .

.

.

.

.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.6 Specialist Models Significantly Outperforming MLLMs . . . . . . . . . . . . . . .

4 Related Work

5 Conclusion

A More Details on Face-Human-Bench

A.1 Definition about Each Leaf Ability . . . . . . . . . . . . . . . . . . . . . . . . . .

A.2 Data Sources and Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.3 More Details on the Semi-Automatic Data Pipeline . . . . . . . . . . . . . . . . .

A.3.1 Details on Image Processing Pipeline . . . . . . . . . . . . . . . . . . . .

A.3.2 Details on Text Processing Pipeline . . . . . . . . . . . . . . . . . . . . .

B More Details on Experiment Setup

B.1 Overviews of Involved MLLMs

. . . . . . . . . . . . . . . . . . . . . . . . . . .

B.2 More Details on the Experiments for RQ1 . . . . . . . . . . . . . . . . . . . . . .

B.2.1 Prompt Templates for Different Settings . . . . . . . . . . . . . . . . . . .

B.2.2 Prompt Used for Choice Extraction . . . . . . . . . . . . . . . . . . . . .

B.2.3 Hints and Task-specific CoT Instructions

. . . . . . . . . . . . . . . . . .

B.3 More Details on the Experiments for RQ2 . . . . . . . . . . . . . . . . . . . . . .

B.3.1 Unexplored L3 Abilities . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.3.2 Explored L3 Abilities .

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

C Additional Results

C.1 Face-Human-Bench (English)

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2 Face-Human-Bench (Chinese)

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

1

3

3

3

4

4

4

6

7

8

9

10

10

18

18

25

27

27

27

29

29

30

30

31

31

34

34

35

37

37

38

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

C.3 Correlation Between Abilities

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.4 Relative Position of Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.5 CoT prompting .

.

.

.

.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.6 Specialist Models Significantly Outperforming MLLMs . . . . . . . . . . . . . . .

D Potential Bias for Demographic Characteristics

E Privacy Protection

F A demonstration of How to Enhance Multi-Modal Assistant Responses with Specialist

Models

G Limitations

H Ethics Statement

39

40

40

48

48

48

49

49

50

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

A MORE DETAILS ON FACE-HUMAN-BENCH

A.1 DEFINITION ABOUT EACH LEAF ABILITY

We will sequentially describe the definitions of L2 abilities and the L3 abilities they encompass. We
provide examples of problems in Face-Human-Bench in Tables 4 to 11.

Facial Attribute Recognition: Recognize various characteristics and traits from facial images.

Age Estimation: Estimate the age of the person in the image based on facial information.

Facial Expression Recognition: Recognize the emotions of the person in the image, categorized
into basic and compound types. Basic expressions include surprised, fearful, disgusted, happy, sad,
angry, and neutral. Compound expressions provide more nuanced emotional descriptions, including:
happily surprised, happily disgusted, sadly fearful, sadly angry, sadly surprised, sadly disgusted,
fearfully angry, fearfully surprised, angrily surprised, angrily disgusted, and disgustedly surprised.

Face Attack Detection: Determine whether the face in the image involves digital manipulation or
physical spoofing. The corresponding sub-abilities are referred to as Deepfake Detection and Face
Anti-Spoofing, respectively.

Face Recognition Identify and verify individuals’ identities in images according to facial infor-
mation. In our tests, this ability is mainly to determine whether two photos showcase the same
individual. Five scenarios are involved: basic, cross-pose, cross-age, similar-looking, and occluded.

Human Attribute Recognition Recognize various characteristics and traits from human images.

Action Recognition Recognize human actions, including interactions with objects.

Spatial Relation Understanding Understand the spatial positions of people in the image, including
relative position understanding (comprehending the relative positions of one person to others and
objects) and crowd counting (counting the number of people in the image).

Social Relation Understanding Including social relationship recognition (inferring social relation-
ships between people through their interactions) and identity reasoning (deducing social identity
based on a person’s attributes, actions, interactions with others, and environmental information).

Person Re-Identification Identify and verify individuals’ identities in images based on full-body
attributes (usually excluding the face, as facial features are often blurry).

Table 4: Examples of problems in Face-Human-Bench.

Ability

Example

Facial Attribute Recognition

Image:

Question:
Please select the description that best applies to the person in the
picture.
A. not wearing necktie, not wearing lipstick, not wearing earrings.
B. without eyeglasses, bald, with mouth slightly open.
C. male, with black hair, wearing earrings.
D. with eyeglasses, not wearing hat, with bangs.
Answer: A.

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

Table 5: Examples of problems in Face-Human-Bench.

Ability

Example

Image:

Age Estimation
(5-Year Interval)

Age Estimation
(10-Year Interval)

Age Estimation
(15-Year Interval)

Facial Expression
Recognition
(Basic Expression
Recognition)

Facial Expression
Recognition
(Compound Expression
Recognition)

Question:
Which age do you believe is most likely for the person in the
photo?
A. 10. B. 15. C. 20. D. 25.
Answer: D.

Image:

Question:
Which of the following ages is the most likely for the person in
the picture?
A. 20. B. 30. C. 40. D. 50.
Answer: A.

Image:

Question:
Which of the following ages is the most likely for the person in
the picture?
A. 47. B. 62. C. 77. D. 92.
Answer: B.

Image:

Question:
What is the expression of the person in this photo?
A. Neutral.
B. Sadness.
C. Disgust.
D. Fear.
Answer: A.

Image:

Question:
Based on this picture, what is the person’s expression?
A. Happily Disgusted.
B. Fearfully Surprised.
C. Sadly Disgusted.
D. Sadly Fearful.
Answer: A.

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

Table 6: Examples of problems in Face-Human-Bench.

Ability

Example

Image:

Face Attack Detection
(Deepfake Detection)
(Bona Fide)

Face Attack Detection
(Deepfake Detection)
(Face-Swapping)

Face Attack Detection
(Deepfake Detection)
(Face-Reenactment)

Face Attack Detection
(Face Anti-Spoofing)
(Bona Fide)

Face Attack Detection
(Face Anti-Spoofing)
(Print)

Face Attack Detection
(Face Anti-Spoofing)
(Replay)

Question:
Has the facial image undergone digital alteration?
A. yes.
B. no.
Answer: B.

Image:

Question:
Was the facial image digitally modified in any way?
A. yes.
B. no.
Answer: A.

Image:

Question:
Was the facial appearance digitally changed?
A. yes.
B. no.
Answer: A.

Image:

Question:
Has the facial image been compromised by a presentation attack?
A. yes.
B. no.
Answer: B.

Image:

Question:
Is there a spoofing attempt visible in the facial image?
A. yes.
B. no.
Answer: A.

Image:

Question:
Is the facial recognition being deceived by a presentation attack?
A. yes.
B. no.
Answer: A.

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

Table 7: Examples of problems in Face-Human-Bench.

Ability

Example

Image:

Face Recognition
(Basic Face Recognition)

Face Recognition
(Basic Face Recognition)

Question:
Are the people portrayed in the two pictures identical?
A. yes.
B. no.
Answer: A.

Image:

Question:
Are the individuals in both images one and the same?
A. yes.
B. no.
Answer: B.

Image:

Face Recognition
(Cross-Pose Face Recognition)

Face Recognition
(Cross-Pose Face Recognition)

Face Recognition
(Cross-Age Face Recognition)

Question:
Do the individuals appearing in the two images happen to be
identical?
A. yes.
B. no.
Answer: A.

Image:

Question:
Do the people shown in both pictures happen to be one and the
same person?
A. yes.
B. no.
Answer: B.

Image:

Question:
Are the people portrayed in the two pictures identical?
A. yes.
B. no.
Answer: A.

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

21

Under review as a conference paper at ICLR 2025

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Table 8: Examples of problems in Face-Human-Bench.

Ability

Example

Image:

Face Recognition
(Cross-Age Face Recognition)

Face Recognition
(Similar-Looking
Face Recognition)

Face Recognition
(Similar-Looking
Face Recognition)

Face Recognition
(Occluded Face Recognition)

Face Recognition
(Occluded Face Recognition)

Question:
Do the individuals in both images happen to be the same person?
A. yes.
B. no.
Answer: B.

Image:

Question:
Are the persons depicted in the photos on the left and right sides
identical?
A. yes.
B. no.
Answer: A.

Image:

Question:
Are the persons depicted in the photos on the left and right sides
identical?
A. yes.
B. no.
Answer: B.

Image:

Question:
Is the individual captured in both the left and right photographs
one and the same person?
A. yes.
B. no.
Answer: A.

Image:

Question:
Do the individuals appearing in the two images happen to be
identical?
A. yes.
B. no.
Answer: B.

22

Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Table 9: Examples of problems in Face-Human-Bench.

Ability

Example

Human Attribute Recognition

Image:

Question:
Which statement best describes the individual highlighted in the
red box in the picture?
A. She is wearing a long-sleeve shirt and is not wearing a hat or a
skirt.
B. She is wearing a T-shirt and a hat, but her clothes do not have
any logos.
C. She is dressed informally in a skirt and wearing sunglasses.
D. She has long hair and is wearing a short-sleeved shirt along
with a face mask.
Answer: A.

Image:

Action Recognition

Question:
Which of these options best describes what the person in the red
box is doing in the picture?
A. Washing the motorcycle.
B. Waxing the motorcycle.
C. Polishing the motorcycle.
D. Repairing the motorcycle.
Answer: A.

Spatial Relation Understanding
(Relative Position Understanding)

Image:

Question:
Among the following options, what is the most fitting way to
characterize the subject (marked with a red box)’s location in
relation to the object (marked with a green box)?
A. The child is behind the sofa.
B. The child is to the right of the sofa.
C. The child is to the left of the sofa.
D. The child is under the sofa.
Answer: A.

23

Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Table 10: Examples of problems in Face-Human-Bench.

Ability

Example

Spatial Relation Understanding
(Crowd Counting)
(Less than 10)

Image:

Question:
What’s the number of individuals in this picture?
A. 2. B. 3. C. 4. D. 5.
Image: D.

Spatial Relation Understanding
(Crowd Counting)
(10-100)

Image:

Question:
Among the options, which numeral is closest to the total count of
humans in the picture?
A. 10. B. 30. C. 90. D. 140.
Image: B.

Spatial Relation Understanding
(Crowd Counting)
(More than 100)

Image:

Question:
What is the closest numerical value among the options to the
number of individuals in the image?
A. 400. B. 1100. C. 3200. D. 5300.
Answer: B.

Social Relation Understanding
(Social Relationship Recognition)

Social Relation Understanding
(Identity Reasoning)

Image:

Question:
Which relationship do the two people in the red box in the photo
most likely have?
A. Couple. B. No Relation. C. Family. D. Friends.
Answer: A.

Image:

What is the most likely occupation of the person highlighted in
red in the picture?
A. basketball player.
B. basketball team manager.
C. basketball coach.
D. sports commentator.
Answer: A.

24

Under review as a conference paper at ICLR 2025

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Table 11: Examples of problems in Face-Human-Bench.

Ability

Example

Image:

Person Re-Identification

Question:
Is the person in the first picture the same as the person in the
second picture?
A. yes.
B. no.
Answer: A.

Image:

Person Re-Identification

Is the individual captured in both the left and right photographs
one and the same person?
A. yes.
B. no.
Image: B.

A.2 DATA SOURCES AND STATISTICS

Table 12 provides information on the data sources for Face-Human-Bench, as well as the image
processing pipeline, the number of problems in the development and test sets, and the weights, for
each subset.

We set the weights of all 10 L2 abilities to be equal. For L2 abilities that encompass multiple L3
abilities, each L3 ability equally shares the weight of the corresponding L2 ability. For L3 abilities
that encompass multiple image versions, each image version subset equally shares the weight of
the corresponding L3 ability. Finally, we obtain the detailed weights of each subset, as shown in
Table 12.

We sequentially provide overviews of the public datasets we used for original samples.

CelebA (Liu et al., 2015) is a large-scale facial attributes dataset released by the Multimedia Labora-
tory of Chinese University of Hong Kong. It contains over 200,000 celebrity images, each annotated
with 40 attributes. The dataset includes a wide range of body pose variations and complex, di-
verse background information. It comprises 10,177 identities, 202,599 face images, and 5 landmark
positions, with 40 binary attribute annotations for each image.

UTKFace (Zhang et al., 2017) dataset is a large-scale facial dataset with a wide age range, spanning
from 0 to 116 years. It contains over 20,000 face images, annotated with age, gender, and ethnicity
labels.

RAF-DB (Li et al., 2017b) is a large-scale facial expression database consisting of 29,672 real-
world images, each accompanied by a 7-dimensional expression distribution vector. It includes two
different subsets: a single-label subset with 7 basic expressions (RAF-DB Basic) and a two-tab
subset with 12 compound expressions (RAF-DB Compound). Additionally, the dataset provides 5
precise landmark locations, 37 automatic landmark positions, bounding boxes, and annotations for
ethnicity, age range, and gender attributes for each image.

FF++ (R¨ossler et al., 2019) consists of 1,000 original video sequences processed using four different
automated facial manipulation methods: Deepfakes, Face2Face, FaceSwap, and NeuralTextures.
The data in FaceForensics++ comes from 977 YouTube videos, all featuring trackable frontal faces
without occlusions, allowing the automated manipulation methods to generate realistic forgeries.

25

Under review as a conference paper at ICLR 2025

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Level-1

Table 12: Data sources and statistics of the Face-Human-Bench.
Level-2

Data Source

Level-3

pimage

Dev.
Num.
50
50
75
75
25
25

Identity
Crop
Identity
Crop
Identity
Crop

Identity

Facial Attribute
Recognition

Facial Attribute
Recognition

CelebA

Age Estimation

Age Estimation

UTKFace

Facial Expression
Recognition

Face

Face Attack
Detection

Face Recognition

Human Attribute
Recognition

Basic Expression
Recognition
Compound
Expression
Recognition
Deepfake
Detection
Face
Anti-Spoofing
Basic Face
Recognition
Cross-Pose
Face Recognition
Cross-Age
Face Recognition
Similar-Looking
Face Recognition
Occluded
Face Recognition
Human Attribute
Recognition

Human

Action Recognition Action Recognition

Spatial Relation
Understanding

Social Relation
Understanding

Person
Re-Identification

Relative Position
Understanding

Crowd Counting

Social
Relationship
Recognition
Identity Reasoning
Person
Re-Identification

RAF-DB
(Basic)

RAF-DB
(Compound)

FF++

Identity

SiW-Mv2

Identity

LFW

CPLFW

CALFW

SLLFW

MLFW

Cat

Cat

Cat

Cat

Cat

WIDER
Attribute
HICO-DET

AddBox
Crop
AddBox

SpatialSense

Identity

PISC
ShTech

PISC

PISC

Identity

AddBox

AddBox

Market-1501

Cat

25

50

50

25

25

25

25

25

50
50
50

25

75

25

25

50

Test
Num.
100
100
150
150
50
50

Weight

5.0%
5.0%
5.0%
5.0%
2.5%
2.5%

50

5.0%

100

100

50

50

50

50

50

100
100
100

50

150

50

50

5.0%

5.0%

2.0%

2.0%

2.0%

2.0%

2.0%

5.0%
5.0%
10.0%

5.0%

5.0%

5.0%

5.0%

100

10.0%

SiW-Mv2 (Guo et al., 2022) collects 785 videos from 493 subjects, and 915 spoof videos from 600
subjects. The dataset includes 14 types of spoofing, ranging from typical print and replay attack, to
various masks, impersonation makeup and physical material coverings. SiW-Mv2 exhibits a good
variance in spoofing modes, with each mode specified and validated by the IARPA project.

LFW (Huang et al., 2008) is a commonly used test set for face recognition, comprising 13,233
face images sourced from natural scenes in everyday life. Each image is associated with a name,
representing 5,749 individuals, with most people having only one image. The database randomly
selected 6,000 pairs of faces to create face recognition image pairs to test the accuracy of face
recognition systems, with 3,000 pairs containing two images of the same person and 3,000 pairs
featuring one image of different individuals.

CPLFW (Zheng & Deng, 2018) builds upon LFW by considering the impact of pose variations. It
specifically searches for and selects 3,000 pairs of positive faces with differing poses, adding pose
variation to the intra-class variance. Additionally, it includes negative pairs with the same gender
and ethnicity to minimize the influence of attribute differences between positive and negative pairs.

CALFW (Zheng et al., 2017) builds upon LFW by considering the impact of age variations. It
specifically searches for and selects 3,000 pairs of positive faces with age differences to increase the
intra-class variance associated with the aging process. Negative pairs are chosen to have the same
gender and ethnicity to reduce the influence of attribute differences.

SLLFW (Deng et al., 2017) intentionally selects 3,000 pairs of visually similar faces through human
crowdsourcing from the original image folder, replacing the random negative pairs in LFW.

26

Under review as a conference paper at ICLR 2025

MLFW (Wang et al., 2022) dataset is created based on CALFW and focuses on masked faces. The
masks generated for the faces in the dataset maintain good visual consistency with the original faces.
It includes a variety of mask templates that cover most common styles encountered in everyday life,
achieving diversity of the samples.

WIDER Attribute (Li et al., 2016) is a large-scale human attributes dataset containing 13,789
images across 30 scene categories, with 57,524 human bounding boxes. Each bounding box is an-
notated with 14 binary attributes, including male, long hair, sunglasses, hat, long shirt, long sleeves,
formal, shorts, jeans, long pants, skirt, mask, logo, and checkered or striped patterns.

HICO-DET Xu et al. (2019) is a commonly used dataset in the Human Object Interaction (HOI)
domain, consisting of 47,776 images, with 38,118 in the training set and 9,658 in the testing set.
The dataset includes 117 action (verb) categories, 80 object categories, and 600 verb-object combi-
nations.

SpatialSense Yang et al. (2019) is a dataset for spatial relation recognition, where the task is to
determine whether a specific spatial relation holds between two given objects. The dataset contains
17,498 relations on 11,569 images, involving 3,679 unique object classes, with 2,139 of these classes
appearing only once, presenting a challenging long-tail distribution.

PISC Li et al. (2017a) is focused on the task of social relationship recognition in still images. It is
used to benchmark models that analyze the relationships between people based on contextual and
individual features. It contains 22,670 images with 76,568 annotated samples representing 9 types
of social relationships.

ShTech Zhang et al. (2016) is focused on the task of crowd counting, where the goal is to accurately
estimate the number of people in an image with varying crowd density and perspective. It contains
1,198 images with approximately 330,000 annotated heads. The dataset aims to address challenges
in crowd counting that were not covered by previous datasets.

Market-1501 Zheng et al. (2015) is designed for the task of person re-identification. This dataset
addresses the limitations of scale and realistic conditions found in previous datasets. The large-scale
data supports training and testing models effectively for person re-identification. It includes over
32,000 annotated bounding boxes and a distractor set of more than 500,000 images.

A.3 MORE DETAILS ON THE SEMI-AUTOMATIC DATA PIPELINE

A.3.1 DETAILS ON IMAGE PROCESSING PIPELINE

Figure 7 illustrates four operations of the image processing pipeline: cropping, concatenating,
adding boxes, or leaving the original images unchanged. For simplicity, these four operations are
denoted as Crop, Cat, AddBox, and Identity, respectively. The image processing pipeline used for
each L3 ability is shown in Table 12.

Figure 7: Four operations of the image processing pipeline.

A.3.2 DETAILS ON TEXT PROCESSING PIPELINE

We introduce the text processing pipeline for each L3 ability as follows. Facial Attribute Recog-
nition Each option involves three attributes. At least two of the three attribute descriptions are
incorrect in the incorrect options.

Age Estimation Add incorrect options at intervals of 5 years, 10 years, and 15 years, with each
interval accounting for one-third of the total.

27

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

CropAddBoxCatIdentityUnder review as a conference paper at ICLR 2025

Basic Expression Recognition Incorrect options are randomly selected from the remaining 6 cate-
gories of expressions after removing the correct option.

Compound Expression Recognition Incorrect options are randomly selected from the remaining
10 categories of expressions after removing the correct option.

Deepfake Detection Set the options to “Yes” and “No”. “Yes” indicates the presence of digital
manipulations, while “No” indicates their absence.

Face Anti-Spoofing Set the options to “Yes” and “No”. “Yes” indicates the presence of physical
spoofs, while “No” indicates their absence.

Basic/Cross-Pose/Cross-Age/Similar-Looking/Occluded Face Recognition Set the options to
“Yes” and “No”. “Yes” indicates that the two photos are of the same person, while “No” indicates
that the two photos are not of the same person.

Human Attribute Recognition Each option involves three attributes combined into a complete
sentence using ChatGPT. At least two of the three attribute descriptions are incorrect in the incorrect
options.

Action Recognition The incorrect options are actions generated by ChatGPT related to but not the
same as the correct option.

Relative Position Understanding Each option is a sentence formed by connecting the subject and
the object with a preposition. Incorrect options are generated by randomly selecting prepositions
from the remaining 8 categories of relative positions after removing the correct preposition.

Crowd Counting The set includes three equally sized subsets, with the number of people in each
subset being within the ranges of less than 10, 10-100, and more than 100, respectively. In the first
subset, the incorrect options are also numbers within 10. In the latter two subsets, the incorrect
options are numbers that are half, three times, and five times the correct option, respectively, with
all options rounded to the nearest 10 and 100.

Social Relationship Recognition Incorrect options are randomly selected from the remaining 5
categories of social relations after removing the correct option.

Identity Reasoning The incorrect options are occupations generated by GPT related to but not the
same as the correct option.

Person Re-Identification Set the options to “Yes” and “No”. “Yes” indicates that the two photos
are of the same person, while “No” indicates that the two photos are not of the same person.

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

28

Under review as a conference paper at ICLR 2025

B MORE DETAILS ON EXPERIMENT SETUP

B.1 OVERVIEWS OF INVOLVED MLLMS

GPT-4V and GPT-4o: GPT-4V (OpenAI, 2023b), released by OpenAI in September 2023, is a
vision-enabled variant of the GPT-4 model, utilizing the same training process as GPT-4 for its vi-
sual capabilities. It is first trained on a large dataset of text and images, followed by fine-tuning
through Reinforcement Learning with Human Feedback (RLHF). GPT-4V demonstrates the excep-
tional performance of a language-only system augmented with new modalities. The API we applied
in our experiments is “gpt-4-turbo-2024-04-09”. GPT-4o OpenAI (2024) is released by OpenAI in
May 2024. It accepts any combination of text, image, audio and video as input and generates any
combination of text, image, and audio output. GPT-4o attains GPT-4 Turbo-level performance in
text, inference, and code, while also demonstrating strong capabilities in multilingual, audio, and
visual tasks. The API we applied in our experiments is “gpt-4o-2024-05-13”.

Gemini (Team et al., 2023): Gemini is a multimodal large model developed by Google, available
in three scales: Ultra, Pro, and Nano. From its inception, Gemini was designed with a multimodal
focus, excelling in tasks across image, audio, video, and text domains. In February 2024, Google
released Gemini 1.5 (Reid et al., 2024), which includes Gemini 1.5 Pro and the more lightweight
Gemini 1.5 Flash. In our work, we employ Gemini 1.5 Pro to conduct experiments.

Claude (Anthropic, 2023): The Claude model is developed by Anthropic and is intended to be a
useful, honest and harmless assistant. The version we applied in this paper, Claude 3.5 Sonnet
(Anthropic, 2024a), was released on June 2024. It is the most powerful visual model in the Claude
series to date.

LLaVA (Liu et al., 2023b): LLaVA is an open-source large multimodal model that leverages mul-
timodal language-image instruction-following data for instruction tuning. It was released in April
2023. LLaVA-1.5 (Liu et al., 2023a), released in October 2023, introduced the following key im-
provements: the use of MLP as a vision-language connector, the use of prompt data with explic-
itly specified output formats, and the addition of task-specific datasets for training. Following that,
LLaVA-1.6 (LLaVA-NeXT) (Liu et al., 2024a) was released in January 2024, featuring improved in-
put image resolution and enhanced visual reasoning and OCR capabilities. The model also supports
better visual conversation on different scenarios and applications. SGLang was utilized for efficient
deployment and inference. We apply LLaVA-13B, LLaVA-1.5-7B, LLaVA-1.5-13B, LLaVA-NeXT-
7B, LLaVA-NeXT-13B, and LLaVA-NeXT-34B in our experiments.

MiniGPT-4 (Zhu et al., 2024): MiniGPT-4, released in April 2023, uses a projection layer to align
a frozen vision encoder with the frozen LLM Vicuna. The authors trained MiniGPT-4 in two stages:
the first stage involved using a low-level dataset, and in the second stage, they curated a detailed
image description dataset to fine-tune the model. In our experiments, we use MiniGPT-4-7B and
MiniGPT-4-13B.

InstructBLIP (Dai et al., 2023): InstructBLIP, released in May 2023, applies its instruction-tuning
paradigm to the BLIP-2 (Li et al., 2023b) model. To be specific, InstructBLIP performs instruction
fine-tuning on visual tasks to enhance model performance. In our experiments, InstructBLIP-7B and
InstructBLIP-13B are used.

Qwen-VL (Bai et al., 2023): Qwen-VL, released in August 2023, accepts images, text, and bound-
ing boxes as inputs, and outputs text and bounding boxes. It supports multilingual and multi-image
interleaved dialogue, as well as open-domain localization in Chinese. Qwen-VL is also capable of
relatively fine-grained recognition and understanding. We adapt Qwen-VL-Chat in our experiments.

InternLM-XComposer2-VL (Zhang et al., 2023): InternLM-XComposer-VL, released in Septem-
ber 2023, is a multimodal large language model built with InternLM (Team, 2023) as the lan-
guage model. Later, in January 2024, InternLM-XComposer2-VL (Dong et al., 2024) was re-
leased, supporting free-form text and image composition. The authors proposed the Partial LoRA
(PLoRA) method, which balances precise visual understanding with literary-inspired text genera-
tion. InternLM-XComposer2-VL-7B is used in our experiments.

Yi-VL (Young et al., 2024): Yi-VL, released in May 2024, excels in image-text understanding
and chat generation, supporting multi-turn image-text conversations, bilingual text, and fine-grained

29

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

Under review as a conference paper at ICLR 2025

image comprehension. Yi-VL adopts the LLaVA architecture and employs a three-stage training
process to align visual information with the semantic space of Yi LLM (Young et al., 2024).

InternVL (Chen et al., 2023): InternVL, released in December 2023, extends its visual model
It progressively aligns with the LLM using web-scale image-text data.
to 6 billion parameters.
InternVL-Chat-V1.2 was released in February 2024, expanding the LLM to 34 billion parameters.
Shortly after, InternVL-Chat-v1.2-Plus was introduced, utilizing more supervised fine-tuning (SFT)
data to further enhance its performance. Subsequently, InternVL-Chat-v1.5 (Chen et al., 2024) was
released in April 2024, with improvements primarily focused on a stronger visual encoder, dynamic
high-resolution capability, and a high-quality bilingual dataset. The model we use in the experiments
includes InternVL-Chat-v1.2-Plus and InternVL-Chat-v1.5.

DeepSeek-VL (Lu et al., 2024): DeepSeek-VL, released in March 2024, is designed for general
multimodal understanding. It is built for real-world applications in visual and language comprehen-
sion, capable of handling tasks such as logical diagrams, web pages, formula recognition, scientific
literature, natural images, etc. In the experiments, we apply DeepSeek-VL-1.3B and DeepSeek-VL-
7B.

CogVLM2 and GLM-4V (Wang et al., 2023a; Hong et al., 2024): CogVLM, released in October
2023, enables deep fusion of visual and language features without sacrificing performance on NLP
tasks. In May 2024, the next generation, CogVLM2, was introduced. It inherited the visual expert
architecture and improved training recipes in the pre-training and post-training stages, supporting
high input resolutions. Shortly after, in June 2024, GLM-4V was released. It used the same data
and training recipes as CogVLM2 but employed GLM-4-9B as the language models and removed
the visual expert to reduce the model size. In our experiments, we utilize CogVLM2-19B-Chat and
GLM-4V-9B.

LLaVA-OneVision (Li et al., 2024): LLaVA-OneVision, released in August 2024, supports three
major computer vision scenarios: single image, multi-image, and video scenes.
It also exhibits
strong transfer learning capabilities across different modalities and scenarios. We use LLaVA-
OneVision-0.5B and LLaVA-OneVision-7B in our experiments.

Table 13 summarizes the LLMs and vision encoders used in involved MLLMs.

Table 13: The LLMs and vision encoders used in involved MLLMs.

LLM
Qwen2-0.5B
DeepSeek-LLM-1.3B-Base
Yi-6B
Vicuna-7B
Vicunad-7B
Qwen-7B
DeepSeek-LLM-7B-Base
Vicuna-v1.5-7B
Vicuna-v1.5-7B

Model
LLaVA-OneVision-0.5B
DeepSeek-VL-1.3B-Chat
Yi-VL-6B
MiniGPT-4-7B
InstructBLIP-7B
Qwen-VL-Chat
DeepSeek-VL-7B-Chat
LLaVA-1.5-7B
LLaVA-NeXT-7B
InternLM-XComposer2-VL-7B InternLM-7B
LLaVA-OneVision-7B
CogVLM2-19B-Chat
GLM-4V-9B
MiniGPT-4-13B
InstructBLIP-13B
LLaVA-13B
LLaVA-1.5-13B
LLaVA-NeXT-13B
InternVL-Chat-v1.5
LLaVA-NeXT-34B
InternVL-Chat-v1.2-Plus

Qwen2-7B
Llama-3-8B-Instruct
GLM-4-9B
Vicuna-13B
Vicuna-13B
LLaMA-2-13B-Chat
Vicuna-v1.5-13B
Vicuna-v1.5-13B
InternLM2-20B
Yi-34B
Nous-Hermes-2-Yi-34B

Params.
400M
400M
632M
1.0B
1.0B
1.8B

Params. Vision Encoder
0.5B
1.3B
6B
7B
7B
7B
7B
7B
7B
7B
7B
8B
9B
13B
13B
13B
13B
13B
20B
34B
34B

SigLIP ViT-L/16
SigLIP ViT-L/16
CLIP ViT-H/14
EVA-CLIP-g/14
EVA-CLIP-g/14
Open CLIP-G/14
SigLIP ViT-L/16 + SAM ViT-B 400M + 86M
CLIP-L/14
CLIP-L/14
EVA-CLIP-g/14
SigLIP ViT-L/16
EVA-02-CLIP-E/14
EVA-02-CLIP-E/14
EVA-CLIP-g/14
EVA-CLIP-g/14
CLIP-L/14
CLIP-L/14
CLIP-L/14
InternViT-6B
CLIP-L/14
InternViT-6B

304M
304M
1.0B
400M
4.4B
4.4B
1.0B
1.0B
304M
304M
304M
6B
304M
6B

B.2 MORE DETAILS ON THE EXPERIMENTS FOR RQ1

B.2.1 PROMPT TEMPLATES FOR DIFFERENT SETTINGS

Zero-Shot (ZS) The prompt template used for the zero-shot setting is shown in Table 14.

Hints (H) The prompt template for experiments with hints is shown in Table 15.

30

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Under review as a conference paper at ICLR 2025

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Table 14: The prompt template used for the zero-shot setting.

Question: [Question]
[Options]
Please provide the answer to the multiple-choice question, using only the option’s letter to indicate
your choice. Note: Only one option is correct. For questions you are unsure about, please choose
the answer you think is most likely.

Table 15: The prompt template used for experiments with hints.

Question: [Question]
[Options]
Hint: [Hint]
Please provide the answer to the multiple-choice question based on the hint, using only the option’s
letter to indicate your choice. Note: Only one option is correct. For questions you are unsure about,
please choose the answer you think is most likely.

Hints and Vanilla CoT Instructions (H+VCoT) The prompt template for experiments with hints
and vanilla CoT instructions is shown in Table 16.

Table 16: The prompt template used for experiments with hints and vanilla CoT instructions.

Question: [Question]
[Options]
Hint: [Hint]
First, please analyze the question and options step by step in conjunction with the input image.
Then, please provide the answer to the multiple-choice question based on the hint and relevant
analysis. Note: Only one option is correct. For questions you are unsure about, please choose the
answer you think is most likely.

Table 17: The prompt template used for one-stage experiments with hints and task-specific CoT
instructions.

Question: [Question]
[Options]
Hint: [Hint]
First, [Task-specific CoT instruction]
Then, please provide the answer to the multiple-choice question based on the hint and relevant
analysis. Note: Only one option is correct. For questions you are unsure about, please choose the
answer you think is most likely.

Hints and Task-Specific Instructions With One-Stage Framework (H+1TCoT) The prompt tem-
plate for one-stage experiments with hints and task-specific CoT instructions is shown in Table 17.

Hints and Task-Specific Instructions With Two-Stage Framework (H+2TCoT) The prompt tem-
plate for two-stage experiments with hints and task-specific CoT instructions is shown in Table 18.

B.2.2 PROMPT USED FOR CHOICE EXTRACTION

The prompt used for choice extraction is shown in Table 19.

B.2.3 HINTS AND TASK-SPECIFIC COT INSTRUCTIONS

Hints and task-specific CoT instructions for each L3 ability are shown in Table 20.

31

Under review as a conference paper at ICLR 2025

Table 18: The prompt template used for two-stage experiments with hints and task-specific CoT
instructions.

Stage 1
Question: [Question]
[Options]
Hint: [Hint]
[Task-specific CoT instruction]
Stage 2
Question: [Question]
[Options]
Hint: [Hint]
Relevant Analysis: [Output from stage 1]
Please provide the answer to the multiple-choice question based on the hint and relevant analysis.
Note: Only one option is correct. For questions you are unsure about, please choose the answer you
think is most likely.

Table 19: The prompt template used for choice extraction.

You are an AI assistant to help me match an answer with several options of a multiple-choice
problem. You are provided with a question, several options, and an answer, and you need to find
which option is most similar to the answer. If the meaning of all options is significantly different
from the answer, output X. You should output a single uppercase character in A, B, C, D (if they are
valid options), and X.

Question: Please select the description that best matches the individual depicted.
Options:
A. He is wearing a face mask but is not wearing a hat or a skirt.
B. He is wearing a face mask, a hat, and shorts.
C. He has short hair and is not wearing a face mask or a T-shirt.
D. He is not wearing clothes with a logo or stripes, and he isn’t wearing sunglasses.
Answer: He is wearing a face mask, a hat, and shorts.
Your Output: B

Question: Which description best represents the person in the image?
Options:
A. She is wearing a T-shirt and sunglasses, and her clothes do not have a logo.
B. She is wearing a face mask and sunglasses but is not wearing long pants.
C. She is without sunglasses, not wearing a hat, and not wearing a T-shirt.
D. She is dressed informally in a short-sleeved top and is not wearing a T-shirt.
Answer: None of the provided descriptions accurately represent the person in the image.
Your Output: X

Question: [Question]
Options: [Options]
Answer: [Answer]
Your Output:

Table 20: Hints and task-specific CoT instructions.

L3 Ability
F. Attr.

Hint
/

Task-specific CoT instruction
Please analyze whether the characteristics
described in the multiple-choice options
match the attributes of the face in the
image, one by one.

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

32

Under review as a conference paper at ICLR 2025

L3 Ability
Age

Hint
/

Basic Expr.
Comp. Expr.

/

Deepfake

Spoofing

Basic FR
C.P. FR

C.A. FR

S.L. FR

Occ. FR

H. Attr.

A forged face may be
generated by
face-swapping, which is a
technique that replaces one
person’s facial features with
those of another person.
A forged face may be
generated by
face-reenactment, which is
a technique that transfers
the facial expressions and
movements of one person
onto another person’s face
in real-time or in a recorded
video.
A spoof face image may be
printed on paper and then
re-photographed.

A spoof face image may be
re-photographed after being
played on a video playback
device.
/
Even if the two images are
of the same person, there
may be differences in
posture.
Even if the two images are
of the same person, there
may be differences in age,
meaning the two photos
were taken at different ages
of this person.
Even if the two photos are
not of the same person, they
may still have similar facial
features.
To determine whether the
two partially obscured
photos are of the same
person, it is necessary to
analyze other unobscured
facial areas.
/

33

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

Task-specific CoT instruction
Please (1) analyze the facial age
characteristics of the person in the image
and (2) provide a possible age number that
you think is appropriate. Note: Please do
not respond with ”I can’t determine the
exact age”; just provide the number you
think is closest.
Please describe the facial emotional fea-
tures of the person in the image.
Please analyze whether there are any
artifacts indicating face-swapping in the
facial image.

Please analyze whether there are any
artifacts indicating face-reenactment in the
facial image.

Please analyze whether there are any clues
in the facial image that indicate it was
printed on paper and then
re-photographed.
Please analyze whether there are any clues
in the facial image that indicate it was
re-photographed from a video playback
device.

Please analyze whether the two people in
the images are the same person by explain-
ing the similarities and differences in their
facial features.

Please analyze whether the characteristics
described in each option of the
multiple-choice question match the person
in the red box, one by one.

Under review as a conference paper at ICLR 2025

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

L3 Ability
Action

Hint
/

Position

/

Counting

Social Rel.

There are fewer than 10
people in the image.
There are fewer than 100
people in the image.
There are more than 100
people in the image, but
fewer than 4,000.
/

Identity

/

Re-ID

If two people have
significant differences in
posture and their faces are
relatively blurry, the main
basis for determining
whether they are the same
person is their clothing
characteristics.

Task-specific CoT instruction
Please analyze the actions of the person in
the red box.
Please analyze the relative positional
relationship between the subject (marked
with a red box) and the object (marked
with a green box).
Please estimate the number of people ap-
pearing in the image, including those who
are occluded or incomplete. Note: Please
do not say ’I cannot determine the exact
number of people’; just provide the num-
ber you think is approximate.

Please analyze the possible social
relationship between the two people in the
red boxes from the perspectives of relative
position, posture, and facial expressions.
Please analyze the occupation of the
person in the red box from the
perspectives of clothing, actions,
background, etc.
Please analyze whether the two people in
the images are the same person by
explaining the similarities and differences
in their full-body features.

B.3 MORE DETAILS ON THE EXPERIMENTS FOR RQ2

B.3.1 UNEXPLORED L3 ABILITIES

We explain the reasons for not conducting experiments on the remaining 5 L3 abilities as follows.

Face/Human Attribute Recognition These two tasks include a large number of binary classifica-
tion labels (40 labels in CelebA for face and 14 labels in WIDER Attribute for human). Using eval-
uation protocols designed for specialist models to fully assess the performance of MLLMs would
result in huge computational costs. Additionally, many attribute labels have ambiguous semantics
that are difficult to define accurately, such as ”attractive,” ”big lips,” and ”big nose.”

Relative Position Understanding In the face and human understanding community, there are no
specialist models specifically constructed to perceive the spatial relationships between one person to
others and objects.

Identity Reasoning There is a lack of publicly available specialist models that perform occupation
classification.

Social Relationship Recognition Existing evaluation datasets for specialist models, such as PIPA
and PISC, are not suitable for directly evaluating the social relationship recognition abilities of
MLLMs. Because many annotations in these datasets are ambiguous (it is often impossible to assert
that people in an image belong to one relationship category rather than another) and semantically
overlapping (for example, “couple” is a separate category, but a married couple is semantically also
part of “family”). Fine-tuned specialist models can still learn to classify under unclear standards,
but for zero-shot learning MLLMs, the lack of clear definitions greatly reduces performance. It is
worth noting that in Face-Human-Bench, when we use PISC to construct problems for evaluating
the social relationship recognition ability of MLLMs, we manually review and remove problems
with ambiguous options or potentially non-unique answers, allowing us to assess accurately.

34

Under review as a conference paper at ICLR 2025

B.3.2 EXPLORED L3 ABILITIES

We provide the prompt templates for directly evaluating L3 abilities on public datasets from the face
and human community in Tables 21 to 29. During testing, the options will be randomly shuffled.

Table 21: Prompt for Age Estimation on UTKFace.

What is the age of the person in the image? Please answer with a number between 0 and 100.
Answer the question with a single number, and don’t provide other additional explanations.

Table 22: Prompt for Basic Expression Recognition on RAF-DB (Basic).

What expression is on the face in the image?
A. Surprise B. Fear C. Disgust D. Happiness E. Sadness F. Anger G. Neutral
Answer with the option’s letter from the given choices directly, and don’t provide other additional
explanations.

Table 23: Prompt for Compound Expression Recognition on RAF-DB (Compound).

What expression is on the face in the image?
A. Happily Surprised B. Happily Disgusted C. Sadly Fearful D. Sadly Angry
E. Sadly Surprised F. Sadly Disgusted G. Fearfully Angry H. Fearfully Surprised
I. Angrily Surprised J. Angrily Disgusted K. Disgustedly Surprised
Answer with the option’s letter from the given choices directly, and don’t provide other additional
explanations.

Table 24: Prompt for Deepfake Detection on FF++.

Is there any evidence of face forgery artifacts in the picture?
A. Yes B. No
Answer with the option’s letter from the given choices directly, and don’t provide other additional
explanations.

Table 25: Prompt for Face Anti-Spoofing on SiW-Mv2.

Is the face in the picture a spoof face?
A. Yes B. No
Answer with the option’s letter from the given choices directly, and don’t provide other additional
explanations.

Table 26: Prompt for Basic/Cross-Pose/Cross-Age/Similar-Looking/Occluded Face Recognition
on LFW/CPLFW/CALFW/SLLFW/MLFW.

Are the people in the two photos the same person?
A. Yes B. No
Answer with the option’s letter from the given choices directly, and don’t provide other additional
explanations.

35

1836
1837

1838
1839
1840
1841
1842
1843

1844
1845
1846
1847
1848
1849

1850
1851
1852
1853
1854

1855
1856
1857
1858
1859
1860

1861
1862
1863
1864
1865
1866

1867
1868
1869
1870
1871
1872

1873
1874
1875
1876
1877

1878
1879
1880
1881
1882
1883

1884
1885
1886
1887
1888
1889

Under review as a conference paper at ICLR 2025

Table 27: Prompt for Action Recognition on HICO-DET.

Which of the following words best describes the interaction between the person in the red box and
the object in the green box?
[Opions, include all actions involving the same object extracted from HICO-DET.]
Answer with the option’s letter from the given choices directly, and don’t provide other additional
explanations.

Table 28: Prompt for Crowd Counting on ShTech-A.

How many people are there in the picture approximately? Please answer with a number between 0
and 4000.
Answer the question with a single number, and don’t provide other additional explanations.

Table 29: Prompt for Person Re-Identification on Market-1501.

Are the people in the two photos the same person?
A. Yes B. No
Answer with the option’s letter from the given choices directly, and don’t provide other additional
explanations. Only one option is correct.

1890
1891

1892
1893
1894
1895
1896
1897

1898
1899
1900
1901
1902
1903

1904
1905
1906
1907
1908

1909
1910
1911
1912
1913
1914

1915
1916
1917
1918
1919
1920

1921
1922
1923
1924
1925
1926

1927
1928
1929
1930
1931

1932
1933
1934
1935
1936
1937

1938
1939
1940
1941
1942
1943

36

Under review as a conference paper at ICLR 2025

C ADDITIONAL RESULTS

C.1 FACE-HUMAN-BENCH (ENGLISH)

We provide the visualization of the L2 and L3 results in Figures 8 to 10.

Figure 8: The performance of open-source MLLMs with LLM parameter scales below 10B on L2
and L3 abilities.

Figure 9: The performance of open-source MLLMs with LLM parameter scales above 10B on L2
and L3 abilities.

Figure 10: The performance of closed-source MLLMs on L2 and L3 abilities.

37

1944
1945

1946
1947
1948
1949
1950
1951

1952
1953
1954
1955
1956
1957

1958
1959
1960
1961
1962

1963
1964
1965
1966
1967
1968

1969
1970
1971
1972
1973
1974

1975
1976
1977
1978
1979
1980

1981
1982
1983
1984
1985

1986
1987
1988
1989
1990
1991

1992
1993
1994
1995
1996
1997

H. Attr.ActionSpatialSocialRe-IDFace  RecognitionAttackExpr.AgeF. Attr.20406080(a)LLaVA-OneVision-0.5BDeepSeek-VL-1.3B-ChatYi-VL-6BMiniGPT-4-7BInstructBLIP-7BQwen-VL-ChatDeepSeek-VL-7B-ChatLLaVA-1.5-7BLLaVA-NeXT-7BInternLM-XComposer2-VL-7BLLaVA-OneVision-7BCogVLM2-19B-ChatGLM-4V-9BF. Attr.AgeBasic  Expr. Comp.  Expr.DeepfakeSpoofingBasic FRC.P. FRC.A. FRS.L. FROcc. FR20406080(b)H. Attr.ActionPositionCountingSocial  Rel.IdentityRe-ID20406080(c)H. Attr.ActionSpatialSocialRe-IDFace  RecognitionAttackExpr.AgeF. Attr.20406080(a)MiniGPT-4-13BInstructBLIP-13BLLaVA-13BLLaVA-1.5-13BLLaVA-NeXT-13BLLaVA-NeXT-34BInternVL-Chat-v1.5InternVL-Chat-v1.2-PlusF. Attr.AgeBasic  Expr. Comp.  Expr.DeepfakeSpoofingBasic FRC.P. FRC.A. FRS.L. FROcc. FR20406080(b)H. Attr.ActionPositionCountingSocial  Rel.IdentityRe-ID20406080(c)H. Attr.ActionSpatialSocialRe-IDFace  RecognitionAttackExpr.AgeF. Attr.20406080(a)Gemini-1.5-ProClaude-3.5-SonnetGPT-4VGPT-4oF. Attr.AgeBasic  Expr. Comp.  Expr.DeepfakeSpoofingBasic FRC.P. FRC.A. FRS.L. FROcc. FR20406080(b)H. Attr.ActionPositionCountingSocial  Rel.IdentityRe-ID20406080(c)Under review as a conference paper at ICLR 2025

C.2 FACE-HUMAN-BENCH (CHINESE)

Table 30 shows the performance of all evaluated MLLMs at different levels of abilities on the
Human-Face-Bench (Chinese). We further compare the performance of different MLLMs on En-
glish and Chinese versions of the Face-Human-Bench, as shown in Figure 11. Models are sorted
with the ascending order of average performance.

Table 30: Zero-shot scores of MLLMs on the hierarchical Face-Human-Bench (CN). The highest
scores for open-source and closed-source MLLMs are marked in blue and green respectively.

Model

Random
LLaVA

-OneVision-0.5B

DeepSeek

-VL-1.3B-Chat

Yi-VL-6B
MiniGPT-4-7B
InstructBLIP-7B
Qwen-VL-Chat
DeepSeek

-VL-7B-Chat

LLaVA-1.5-7B
LLaVA-NeXT-7B
InternLM
-XComposer2-VL-7B
LLaVA

-OneVision-7B
CogVLM2-19B-Chat
GLM-4V-9B
MiniGPT-4-13B
InstructBLIP-13B
LLaVA-13B
LLaVA-1.5-13B
LLaVA-NeXT-13B
InternVL-Chat-v1.5
LLaVA-NeXT-34B
InternVL

-Chat-v1.2-Plus

Gemini-1.5-Pro
Claude-3.5-Sonnet
GPT-4V
GPT-4o

Model

Random
LLaVA

-OneVision-0.5B

DeepSeek

-VL-1.3B-Chat

Yi-VL-6B
MiniGPT-4-7B
InstructBLIP-7B
Qwen-VL-Chat
DeepSeek

-VL-7B-Chat

LLaVA-1.5-7B
LLaVA-NeXT-7B
InternLM
-XComposer2-VL-7B
LLaVA

-OneVision-7B
CogVLM2-19B-Chat
GLM-4V-9B
MiniGPT-4-13B
InstructBLIP-13B
LLaVA-13B
LLaVA-1.5-13B
LLaVA-NeXT-13B
InternVL-Chat-v1.5
LLaVA-NeXT-34B
InternVL

-Chat-v1.2-Plus

Gemini-1.5-Pro
Claude-3.5-Sonnet
GPT-4V
GPT-4o

Attr.

25.0

29.0

37.0

60.0
21.0
24.0
54.5

67.5

48.0
39.5

87.0

91.0

77.5
84.5
18.5
7.0
24.5
62.0
54.5
89.0
93.5

87.0

58.5
79.5
68.5
77.5

Age

25.0

34.3

48.7

49.3
21.7
28.3
49.0

54.7

49.7
40.0

53.0

61.0

55.7
58.3
26.0
29.0
37.7
53.0
44.0
61.3
55.3

57.3

29.0
54.0
55.0
57.0

Attr. Action

25.0

37.5

35.0

56.5
25.0
30.0
44.0

55.5

35.0
33.0

75.0

84.5

66.5
77.0
28.5
5.0
22.5
38.0
47.5
80.5
87.5

80.0

46.0
55.0
51.0
51.0

25.0

62.0

60.0

68.0
29.0
24.0
72.0

81.0

65.0
70.0

78.0

89.0

86.0
91.0
32.0
41.0
59.0
70.0
74.0
87.0
83.0

88.0

79.0
83.0
59.0
74.0

Expression

Face Understanding

Attack Detection

Face Recognition

Basic Comp. Mean DFD FAS mean
50.0
25.0

50.0

25.0

25.0

50.0

Basic
50.0

67.0

58.0

62.5

38.0

56.0

47.0

50.0

61.0

67.0
28.8
39.0
68.0

65.0

51.0
66.0

74.0

75.0

76.0
80.0
35.4
37.2
56.6
72.0
69.1
82.0
83.0

73.0

70.0
74.0
75.0
82.0

62.0

46.0
25.0
34.0
40.0

52.0

56.0
68.0

68.0

60.0

68.0
78.0
35.4
31.3
29.4
60.0
37.5
70.0
58.0

52.0

36.0
38.0
54.0
70.0

61.5

56.5
24.0
36.5
54.0

58.5

53.5
67.0

71.0

67.5

72.0
79.0
33.5
21.0
34.0
66.0
51.5
76.0
70.5

62.5

53.0
56.0
64.5
76.0

47.0

25.0
50.9
49.0
55.0

49.0

54.5
55.5

45.0

35.0

40.0
37.0
50.8
59.5
50.8
51.5
53.1
61.0
63.0

61.5

11.0
55.0
50.0
52.0

50.0

28.0
45.5
47.0
53.3

51.0

51.0
50.0

51.0

52.0

45.0
52.0
43.9
47.4
54.5
53.5
56.0
62.0
63.0

60.5

16.0
57.0
54.5
56.0

48.5

26.5
39.3
48.0
53.8

50.0

52.8
52.0

48.0

43.5

42.5
44.5
29.0
27.2
44.0
52.5
54.0
61.5
63.0

61.0

13.5
56.0
52.3
54.0

50.0

36.0
60.4
48.0
66.0

58.0

64.0
56.0

58.0

60.0

60.0
72.0
52.1
7.1
52.1
62.0
58.0
94.0
92.0

96.0

98.0
90.0
90.0
78.0

Human Understanding

C.P.
50.0

44.0

50.0

34.0
57.8
50.0
52.0

52.0

46.0
52.0

46.0

38.0

40.0
60.0
50.0
9.5
54.0
54.0
50.0
68.0
68.0

78.0

74.0
74.0
58.0
60.0

C.A.
50.0

50.0

48.0

34.0
46.7
50.0
68.0

40.0

46.0
52.0

48.0

20.0

56.0
68.0
60.0
12.2
52.0
50.0
60.0
62.0
78.0

68.0

84.0
82.0
84.0
68.0

S.L. Occ. Mean
50.0
50.0
50.0

52.0

52.0

49.6

44.0

24.0
35.4
48.0
54.0

42.0

62.0
52.0

66.0

36.0

68.0
70.0
39.5
12.8
56.0
50.0
50.0
66.0
70.0

72.0

88.0
72.0
84.0
80.0

50.0

38.0
45.7
48.0
50.0

42.0

46.0
46.0

34.0

28.0

48.0
64.0
51.0
25.0
46.0
50.0
50.0
48.0
58.0

48.0

68.0
60.0
68.0
54.0

48.4

33.2
45.6
48.8
58.0

46.8

52.8
51.6

50.4

36.4

54.4
66.8
46.8
10.8
51.6
53.2
53.6
67.6
73.2

72.4

82.4
75.6
76.8
68.0

Re-ID

Face Human

Per. Rea. Overall

50.0

51.0

50.0

44.0
36.0
51.0
64.0

50.0

64.0
55.0

70.0

61.0

60.0
62.0
44.0
8.0
55.0
61.0
58.0
87.0
94.0

88.0

49.0
78.0
74.0
69.0

35.0

44.5

48.8

45.1
30.3
37.1
53.9

55.5

51.3
50.0

61.9

59.9

60.4
66.6
30.8
19.0
38.4
57.3
51.5
71.1
71.1

68.0

47.3
64.2
63.4
66.5

30.0

50.9

50.5

53.1
29.6
32.0
55.5

61.6

54.5
51.5

69.7

73.0

67.2
71.0
31.2
22.4
43.6
53.0
56.4
76.1
78.8

78.4

58.1
66.1
61.9
65.5

29.2

37.5

45.4

51.2

48.4

52.8
26.7
31.8
54.5

61.2

50.7
50.3

68.7

72.8

66.7
72.4
27.9
21.7
36.9
56.9
54.3
75.9
75.5

72.6

46.5
63.9
58.4
64.9

51.4

43.6
34.9
38.7
54.9

54.5

56.3
51.5

61.5

56.9

59.5
63.5
35.5
19.2
47.1
52.6
53.6
70.2
74.1

74.1

61.9
67.0
69.1
67.7

32.5

47.7

49.6

49.1
30.0
34.6
54.7

58.5

52.9
50.7

65.8

66.4

63.8
68.8
31.0
20.7
41.0
55.2
54.0
73.6
74.9

73.2

52.7
65.1
62.7
66.0

Spatial Relation
CC
25.0

Mean
25.0

RPU
25.0

Social Relation

SRR
25.0

IR Mean
25.0
25.0

42.0

20.0

31.0

64.0

82.0

73.0

44.0

46.0
37.2
28.0
46.0

54.0

30.0
28.0

60.0

48.0

56.0
62.0
24.5
17.0
26.5
24.0
40.0
50.0
64.0

52.0

52.0
50.0
48.0
54.0

24.7

24.0
28.2
10.0
26.8

40.7

32.9
25.2

45.3

46.7

29.3
32.0
26.6
7.0
31.1
18.0
33.0
50.0
44.7

50.0

24.7
36.7
65.3
51.3

34.3

35.0
25.0
17.0
35.7

47.3

31.3
26.3

52.7

47.3

42.7
47.0
23.3
10.0
26.7
21.0
35.7
50.0
54.3

51.0

38.3
43.3
56.7
52.7

82.0

74.0
38.1
45.8
81.6

82.0

88.0
92.0

84.0

92.0

98.0
90.0
40.4
65.2
73.5
88.0
84.0
82.0
88.0

98.0

78.0
78.0
78.0
92.0

73.0

62.0
33.0
38.0
62.0

74.0

77.0
73.0

73.0

83.0

81.0
78.0
28.0
48.0
55.0
75.0
67.0
76.0
75.0

85.0

78.0
71.0
69.0
81.0

64.0

50.0
38.6
32.7
46.8

66.0

66.0
54.0

62.0

74.0

64.0
66.0
18.4
42.9
38.0
62.0
51.0
70.0
62.0

72.0

78.0
64.0
60.0
70.0

38

1998
1999

2000
2001
2002
2003
2004
2005

2006
2007
2008
2009
2010
2011

2012
2013
2014
2015
2016

2017
2018
2019
2020
2021
2022

2023
2024
2025
2026
2027
2028

2029
2030
2031
2032
2033
2034

2035
2036
2037
2038
2039

2040
2041
2042
2043
2044
2045

2046
2047
2048
2049
2050
2051

Under review as a conference paper at ICLR 2025

Figure 11: Comparation for the performance of different MLLMs on English and Chinese versions
of the Face-Human-Bench.

C.3 CORRELATION BETWEEN ABILITIES

The correlation coefficient matrix for L3 is shown in Figure 12. Pay particular attention to the ability
correlations highlighted in the red boxes.

Figure 12: Correlation coefficient matrix for L3.

39

2052
2053

2054
2055
2056
2057
2058
2059

2060
2061
2062
2063
2064
2065

2066
2067
2068
2069
2070

2071
2072
2073
2074
2075
2076

2077
2078
2079
2080
2081
2082

2083
2084
2085
2086
2087
2088

2089
2090
2091
2092
2093

2094
2095
2096
2097
2098
2099

2100
2101
2102
2103
2104
2105

MiniGPT-4-7BMiniGPT-4-13BInstructBLIP-13BInstructBLIP-7BLLaVA-13BDeepSeek-VL-1.3B-ChatLLaVA-OneVision-0.5BYi-VL-6BQwen-VL-ChatLLaVA-1.5-7BLLaVA-NeXT-7BGemini-1.5-ProLLaVA-1.5-13BLLaVA-NeXT-13BDeepSeek-VL-7B-ChatCogVLM2-19B-ChatGPT-4VClaude-3.5-SonnetInternLM-XComposer2-VL-7BLLaVA-OneVision-7BGPT-4oGLM-4V-9BInternVL-Chat-v1.5InternVL-Chat-v1.2-PlusLLaVA-NeXT-34B203040506070ScoreChineseEnglishAverageF. Attr.AgeBasic Expr.Comp. Expr.DeepfakeSpoofingBasic FRC.P FRC.A FRS.L FROcc. FRH. Attr.ActionPositionCountingSocial Rel.IdentityRe-IDRe-IDIdentitySocial Rel.CountingPositionActionH. Attr.Occ. FRS.L FRC.A FRC.P FRBasic FRSpoofingDeepfakeComp. Expr.Basic Expr.AgeF. Attr.−0.20.00.20.40.60.81.0  Correlation  CoefficientUnder review as a conference paper at ICLR 2025

C.4 RELATIVE POSITION OF TARGETS

Table 31 presents the performance differences of MLLMs across different relative positions of tar-
gets, under the three face understanding abilities and human attribute recognition.

Table 31: The impact of the relative position of targets on performance in four L3 abilities. Models
with absolute performance differences greater than 5 between the two versions are highlighted in
orange. Models with the smallest RPSS are marked in green.

Facial Attribute

Age

Basic Expression

Model

Ori. Crop. Dif. Ori. Crop. Dif. Ori. Crop.
74.0
37.0
LLaVA-OneVision-0.5B
56.0
35.0
DeepSeek-VL-1.3B-Chat
70.0
77.0
Yi-VL-6B
24.0
23.0
MiniGPT-4-7B
40.0
46.0
InstructBLIP-7B
64.0
57.0
Qwen-VL-Chat
DeepSeek-VL-7B-Chat
74.0
57.0
64.0
59.0
LLaVA-1.5-7B
LLaVA-NeXT-7B
76.0
68.0
76.0
InternLM-XComposer2-VL-7B 91.0
LLaVA-OneVision-7B
76.0
91.0
72.0
75.0
CogVLM2-19B-Chat
78.0
83.0
GLM-4V-9B
36.0
19.0
MiniGPT-4-13B
50.0
28.0
InstructBLIP-13B
60.0
35.0
LLaVA-13B
74.0
74.0
LLaVA-1.5-13B
68.0
77.0
LLaVA-NeXT-13B
72.0
93.0
InternVL-Chat-v1.5
78.0
96.0
LLaVA-NeXT-34B
76.0
86.0
InternVL-Chat-v1.2-Plus
66.0
65.0
Gemini-1.5-Pro
68.0
86.0
Claude-3.5-Sonnet
74.0
79.0
GPT-4V
80.0
80.0
GPT-4o

42.0
47.3
48.0
19.3
34.7
50.7
52.7
50.7
48.0
53.3
59.3
55.3
51.3
26.0
36.0
43.3
60.0
40.7
60.0
58.0
58.0
28.0
50.7
52.7
58.7

68.0
58.0
60.0
28.0
36.0
66.0
62.0
60.0
68.0
76.0
72.0
70.0
80.0
34.0
50.0
52.0
70.0
74.0
72.0
82.0
72.0
78.0
78.0
76.0
86.0

44.0
50.7
55.3
16.0
38.7
48.7
52.0
48.0
52.0
52.7
61.3
59.3
60.0
22.7
40.7
38.0
57.3
52.7
63.3
59.3
61.3
52.7
57.3
54.7
63.3

2.0
3.3
7.3
-3.3
4.0
-2.0
-0.7
-2.7
4.0
-0.7
2.0
4.0
8.7
-3.3
4.7
-5.3
-2.7
12.0
3.3
1.3
3.3
24.7
6.7
2.0
4.7

2.0
-3.0
3.0
-2.0
13.0
3.0
-1.0
-4.0
-3.0
-2.0
1.0
0.0
7.0
-3.0
5.0
6.0
-3.0
-1.0
2.0
2.0
0.0
-2.0
5.0
3.0
6.0

35.0
38.0
74.0
25.0
33.0
54.0
58.0
63.0
71.0
93.0
90.0
75.0
76.0
22.0
23.0
29.0
77.0
78.0
91.0
94.0
86.0
67.0
81.0
76.0
74.0

Dif.
-6.0
2.0
-10.0
4.0
-4.0
2.0
-12.0
-4.0
-8.0
0.0
-4.0
-2.0
2.0
-2.0
0.0
-8.0
-4.0
6.0
0.0
4.0
-4.0
12.0
10.0
2.0
6.0

Human Attribute
Boxed Crop. Diff.
6.0
44.0
50.0
-13.0
47.0
34.0
-16.0
75.0
59.0
5.0
13.0
18.0
-8.0
35.0
27.0
-3.0
51.0
48.0
-18.0
73.0
55.0
-14.0
69.0
55.0
-8.0
66.0
58.0
-1.0
88.0
87.0
-1.0
91.0
90.0
-7.0
74.0
67.0
1.0
85.0
86.0
7.0
16.0
23.0
11.0
28.0
39.0
2.0
26.0
28.0
-29.0
75.0
46.0
-11.0
75.0
64.0
-5.0
92.0
87.0
-3.0
93.0
90.0
-4.0
92.0
88.0
-14.0
57.0
43.0
9.0
67.0
76.0
-12.0
79.0
67.0
-19.0
73.0
54.0

RPSS

16.0
21.3
36.3
14.3
29.0
10.0
31.7
24.7
23.0
3.7
8.0
13.0
18.7
15.3
20.7
21.3
38.7
30.0
10.3
10.3
11.3
52.7
30.7
19.0
35.7

C.5 COT PROMPTING

Based on Table 32, we explore the main reasons for the performance improvements of GPT-4o in
each ability at L3, as shown in Figure 13.

Table 32: Scores of the best open-source model, InternVL-Chat-v1.2-Plus, and the best closed-
source model, GPT-4o, under different settings on the hierarchical Face-Human-Bench. The highest
scores for open-source and closed-source MLLMs are marked in blue and green respectively.

Model

Setting

InternVL
-Chat-v1.2-Plus

GPT-4o

ZS
H
H+VCoT
H+1TCoT
H+2TCoT
ZS
H
H+VCoT
H+1TCoT
H+2TCoT

Attr.

86.0
87.0
86.0
89.0
88.0
77.0
77.0
85.0
89.5
89.5

Age

59.7
60.0
58.3
61.0
62.3
61.0
61.0
59.3
60.7
63.0

Model

Setting

Attr. Action

InternVL
-Chat-v1.2-Plus

GPT-4o

ZS
H
H+VCoT
H+1TCoT
H+2TCoT
ZS
H
H+VCoT
H+1TCoT
H+2TCoT

90.0
90.0
87.0
89.0
87.0
63.5
63.5
81.0
81.0
79.5

92.0
95.0
94.0
92.0
92.0
81.0
81.0
91.0
87.0
88.0

Expression

Face Understanding

Attack Detection

Face Recognition

Basic Comp. Mean DFD FAS Mean
65.3
74.0
65.0
71.0
63.3
70.0
62.0
71.0
62.3
72.0
58.5
83.0
67.5
83.0
81.5
85.0
80.5
84.0
75.0
79.0

65.5
67.0
60.0
66.0
61.5
52.0
65.5
67.0
64.0
58.0
60.5
50.0
58.0
63.0
54.0
53.0
72.5
62.0
52.0
72.5
62.0
70.0
71.5
58.0
67.0
75.0
66.0
61.0
75.5
72.0
Human Understanding

65.0
64.0
61.0
66.0
66.5
64.0
83.0
93.0
94.0
89.0

Spatial Relation
CC
58.7
60.6
65.6
51.0
51.3
58.7
55.3
55.3
62.7
61.3

Mean
62.3
60.3
56.3
54.3
54.6
54.3
52.7
56.7
61.3
59.7

RPU
66.0
60.0
48.0
58.0
58.0
50.0
50.0
58.0
60.0
58.0

Social Relation

SRR
76.0
76.0
78.0
74.0
72.0
66.0
66.0
72.0
74.0
78.0

IR Mean
86.0
96.0
85.0
94.0
87.0
86.0
84.0
94.0
82.0
92.0
80.0
94.0
80.0
94.0
77.0
82.0
82.0
90.0
83.0
88.0

Basic
94.0
92.0
92.0
90.0
94.0
96.0
96.0
94.0
98.0
78.0

Re-ID

85.0
86.0
88.0
88.0
80.0
79.0
96.0
98.0
98.0
96.0

C.P.
74.0
66.0
68.0
68.0
66.0
72.0
80.0
76.0
76.0
90.0

C.A.
62.0
56.0
58.0
64.0
56.0
74.0
86.0
86.0
84.0
78.0

S.L. Occ. Mean
70.8
52.0
72.0
68.0
52.0
74.0
70.8
56.0
80.0
70.4
54.0
76.0
70.0
56.0
78.0
73.6
50.0
76.0
83.2
64.0
90.0
84.8
78.0
90.0
83.6
72.0
88.0
82.0
76.0
88.0

Face Human

Per. Rea. Overall

69.7
68.4
69.1
68.6
69.1
68.5
72.2
76.4
77.9
77.0

83.1
83.2
82.5
81.4
79.1
71.6
74.6
80.7
81.9
81.2

76.7
76.4
75.9
75.6
75.8
68.9
70.4
78.2
79.0
78.4

76.0
75.9
74.8
74.3
71.8
71.7
78.0
77.2
81.2
77.2

76.4
75.9
75.7
75.0
74.1
70.0
73.4
78.6
79.9
79.1

40

2106
2107

2108
2109
2110
2111
2112
2113

2114
2115
2116
2117
2118
2119

2120
2121
2122
2123
2124

2125
2126
2127
2128
2129
2130

2131
2132
2133
2134
2135
2136

2137
2138
2139
2140
2141
2142

2143
2144
2145
2146
2147

2148
2149
2150
2151
2152
2153

2154
2155
2156
2157
2158
2159

Under review as a conference paper at ICLR 2025

2160
2161

2162
2163
2164
2165
2166
2167

2168
2169
2170
2171
2172
2173

2174
2175
2176
2177
2178

2179
2180
2181
2182
2183
2184

2185
2186
2187
2188
2189
2190

2191
2192
2193
2194
2195
2196

2197
2198
2199
2200
2201

2202
2203
2204
2205
2206
2207

2208
2209
2210
2211
2212
2213

Figure 13: Main reasons of performance improvements for each L3 ability are highlighted in red.
Abilities with performance improvements mainly due to hints include face anti-spoofing, cross-
pose face recognition, cross-age face recognition, similar-looking face recognition, occluded face
recognition, and person re-identification.

Abilities with performance improvements mainly due to vanilla CoT instructions include facial at-
tribute recognition, deepfake detection, face anti-spoofing, occluded face recognition, human at-
tribute recognition, action recognition, relative position understanding, and social relationship recog-
nition. Comparison of outputs from H and H + VCoT settings is shown in Tables 33 to 36.

Abilities with performance improvements mainly due to 1-stage task-specific CoT instructions in-
clude compound expression recognition and crowd Counting. Comparison of outputs from H +
VCoT and H + 1TCoT settings is shown in Table 37.

Abilities with performance improvements mainly due to 2-stage task-specific CoT instructions in-
clude compound expression recognition and cross-pose face recognition. Comparison of outputs
from H + 1TCoT and H + 2TCoT settings is shown in Table 38.

41

ZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(a) Facial Attribute RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(g) Basic Face RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(m) Action RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(b) Age EstimationZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(h) Cross-Pose Face RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(n) Relative Position UnderstandingZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(c) Basic Expression RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(i) Cross-Age Face RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(o) Crowd CountingZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(d) Compound Expression RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(j) Similar-Looking Face RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(p) Social Relationship RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(e) Deepfake DetectionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(k) Occluded Face RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(q) Identity ReasoningZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(f) Face Anti-SpoofingZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(l) Human Attribute RecognitionZSHH+VCoTH+1TCoTH+2TCoT0255075100Score (%)(r) Person Re-IdentificationUnder review as a conference paper at ICLR 2025

Table 33: Comparison of outputs from H and H + VCoT settings.

Ability

Example

Facial Attribute Recognition

Deepfake Detection

Image:

Question:
Are the individuals in both images one and the same?
A. male, with black hair, wearing earrings..
B. with eyeglasses, not wearing hat, with bangs.
C. not wearing necktie, not wearing lipstick, not wearing earrings.
D. D. without eyeglasses, bald, with mouth slightly open.
Answer: C.
Output from the H setting:
B
Output from the H + VCoT setting:
To determine which description best matches the person in the picture, let’s analyze
each option based on the characteristics of the person depicted: A. male, with black
hair, wearing earrings. The person in the image does not have black hair; the hair
appears to be blonde or light brown. Therefore, this option does not match. B. with
eyeglasses, not wearing hat, with bangs. The person in the image is not wearing
eyeglasses and does not have bangs. Therefore, this option does not match. C. not
wearing necktie, not wearing lipstick, not wearing earrings The person in the image
is not wearing a necktie, and there is no visibility of lipstick or earrings. Hence, this
description fits. D. without eyeglasses, bald, with mouth slightly open. The person in
the image is without eyeglasses, but the person is not bald. He has medium-length
hair. Therefore, this option does not match. Based on the analysis, the correct
description that best applies to the person in the picture is: C. not wearing necktie,
not wearing lipstick, not wearing earrings.

Image:

Question:
Is the presented face digitally manipulated?
A. yes.
B. no.
Answer: B.
Output from the H setting:
A
Output from the H + VCoT setting:
To determine if the face presented in the image is digitally manipulated, we need to
analyze several aspects of the image for indicators of face-swapping or other digital
alterations. The hint provided states that signs of a forged face may include
inconsistencies in the blending of facial features.Here’s a step-by-step analysis based
on the image: 1. Facial Symmetry: Check for unnatural asymmetry which could
indicate manipulation.2. Skin Tone and Texture: Look for differences in skin tone
and texture that might suggest different sources for parts of the face.3. Edges and
Blending: Examine the areas around the jawline, neck, and facial features for any
inconsistencies or unnatural blending. 4. Lighting and Shadows: Confirm that
lighting and shadows are consistent across the entire face. 5. Facial Feature
Alignment: See if the eyes, nose, and mouth appear naturally aligned with regards to
the facial structure.Upon examining the image: The facial features (eyes, nose,
mouth) appear naturally aligned. The skin tone seems consistent across the face.
There are no obvious signs of unnatural blending or asymmetry. Based on this
analysis, the face does not show clear indications of being digitally manipulated. The
most likely answer is: B. no.

42

2214
2215

2216
2217
2218
2219
2220
2221

2222
2223
2224
2225
2226
2227

2228
2229
2230
2231
2232

2233
2234
2235
2236
2237
2238

2239
2240
2241
2242
2243
2244

2245
2246
2247
2248
2249
2250

2251
2252
2253
2254
2255

2256
2257
2258
2259
2260
2261

2262
2263
2264
2265
2266
2267

Under review as a conference paper at ICLR 2025

Table 34: Comparison of outputs from H and H + VCoT settings.

Ability

Example

Face Anti-Spoofing

Occluded Face Recogntion

Image:

Question:
Is the facial recognition being deceived by a presentation attack?
A. yes.
B. no.
Answer: B.
Output from the H setting:
A. yes. The image appears to exhibit characteristics of a presentation attack, such as
a potentially re-photographed printed face, which can deceive facial recognition
systems.
Output from the H + VCoT setting:
To determine if the facial recognition is being deceived by a presentation attack, we
should consider the hint that a spoof face image may be printed on paper and then
re-photographed. Describe significant indicators that may suggest whether the image
may be printed and re-photographed: 1. Quality of the image: Low-quality features
such as blurriness, low resolution, or pixelation might be signs of a re-photographed
image. 2. Reflection or shadows: Unnatural reflections or shadows may indicate that
the image is a photograph printed on paper. 3. Distortion: Warped or distorted facial
features can also be indicative of re-photographing. 4. Edges: Visible edges or
borders might suggest a physical photograph in the frame.Upon observing the
provided image: The image seems to be of good quality, showing clear and sharp
facial features. There are no obvious signs of reflections or unnatural shadows. The
facial features appear natural, without noticeable distortion. No visible edges or
borders that would indicate it is a paper photograph within the frame. Based on the
provided image and considerations, it does not show clear signs of being a
presentation attack. Therefore, the answer to the multiple-choice question is: B. no.

Image:

Question:
Question: Are the individuals portrayed in each of the two pictures one and the same?
A. no.
B. yes.
Answer: B.
Output from the H setting:
A
Output from the H + VCoT setting:
To determine if the individuals in the two photos are the same, we need to look at
some key factors that remain visible despite the face masks.
1. **Hair color and style**: Both individuals have similar hair color and styles,
though this can change over time and with age.
2. **Facial features**: Look at the shape of the brow, eyes, forehead, and ears.
These areas are partially visible and can offer indications of similarity.
3. **Age difference**: Keep in mind the possible age difference and how facial
features can mature over the years.
Let’s analyze each area:
- The individual on the left appears younger with similar hair.
- The forehead, eyes, and brow seem consistent between the two images.
- The ear shape visible in both photos appears similar.
Based on this analysis, it is most likely that the individuals portrayed in the two
pictures are the same person.
Thus, the answer is:
B. yes

43

2268
2269

2270
2271
2272
2273
2274
2275

2276
2277
2278
2279
2280
2281

2282
2283
2284
2285
2286

2287
2288
2289
2290
2291
2292

2293
2294
2295
2296
2297
2298

2299
2300
2301
2302
2303
2304

2305
2306
2307
2308
2309

2310
2311
2312
2313
2314
2315

2316
2317
2318
2319
2320
2321

Under review as a conference paper at ICLR 2025

2322
2323

2324
2325
2326
2327
2328
2329

2330
2331
2332
2333
2334
2335

2336
2337
2338
2339
2340

2341
2342
2343
2344
2345
2346

2347
2348
2349
2350
2351
2352

2353
2354
2355
2356
2357
2358

2359
2360
2361
2362
2363

2364
2365
2366
2367
2368
2369

2370
2371
2372
2373
2374
2375

Table 35: Comparison of outputs from H and H + VCoT settings.

Ability

Example

Human Attribute Recognition

Image:

Question:
Is the facial recognition being deceived by a presentation attack?
A. He has long hair and is wearing sunglasses but is not wearing a hat.
B. He is wearing a face mask and clothes without stripes, and he is not wearing
sunglasses.
C. He has long hair and is wearing a hat with a short-sleeved shirt.
D. He has short hair and is wearing a short-sleeved shirt with stripes.
Answer: D.
Output from the H setting:
C. He has long hair and is wearing a hat with a short-sleeved shirt.
Output from the H + VCoT setting:
To determine which description best matches the person in the red box in the picture,
let’s analyze the image step by step. The person in the red box: 1. Is wearing a hat or
something on his head. 2. Has short hair. 3. Is wearing a short-sleeved shirt. 4. Is not
wearing sunglasses. 5. Is not wearing a face mask. 6. His shirt is plain (not
striped).Based on these observations: A. This option is incorrect because the person
is wearing a hat and not sunglasses. B. This option is incorrect because the person
has short hair, not long hair. C. This option is incorrect because the person is not
wearing a face mask. D. This option is correct because the person has short hair and
is wearing a short-sleeved shirt with stripes. Therefore, the best matching description
is: D. He has short hair and is wearing a short-sleeved shirt with stripes.

Image:

Action Recognition
Hint v.s. Vanilla CoT

Question:
Which choice best depicts the actions of the individual in the red box in the image?
A. holding the bicycle.
B. riding the bicycle.
C. repairing the bicycle.
D. renting the bicycle.
Answer: A.
Output from the H setting:
B. riding the bicycle.
Output from the H + VCoT setting:
Upon analyzing the image, the individual in the red box is seen with one foot on the
ground and holding the handlebar of the bicycle, which is leaning to one side. This
indicates that the person is not riding, repairing, or renting the bicycle at that
moment. Therefore, the best choice that depicts the actions of the individual in the
red box is: A. holding the bicycle.

44

Under review as a conference paper at ICLR 2025

Table 36: Comparison of outputs from H and H + VCoT settings.

Ability

Example

Relative Position Understanding

Image:

Question:
Among the following options, what is the most fitting way to characterize the
subject?
A. The woman is on the table.
B. The woman is behind the table.
C. The woman is under the table.
D. The woman is in front of the table.
Answer: B.
Output from the H setting:
D. The woman is in front of the table.
Output from the H + VCoT setting:
To determine the most fitting way to characterize the woman’s location (indicated by
a red box) in relation to the table (marked with a green box), let’s analyze the image
step by step. The options are: A. The woman is on the table. B. The woman is
behind the table.C. The woman is under the table.D. The woman is in front of the
table. From the image analysis: 1. Option A suggests that the woman is on the table.
The image clearly shows that she is not physically on top of the table, so this option
is incorrect. 2. Option B suggests that the woman is behind the table. The image
shows the woman standing further away from the camera than the table, implying
that the table is closer to the viewpoint than she is. This would make the woman
behind the table. 3. Option C suggests that the woman is under the table. It is evident
from the image that she is standing and not underneath the table, so this option is
incorrect. 4. Option D suggests that the woman is in front of the table. Given that the
woman is further from the camera than the table, this is incorrect because, from the
perspective of the camera, she would be behind the table. Based on the analysis, the
correct answer is: B. The woman is behind the table.

Image:

Social Relationship Recognition

Question:
The two people in the red box in the image are most likely to have what relationship?
A. Couple.
B. Friends.
C. Professional Relation.
D. No Relation.
Answer: C.
Output from the H setting:
B. Friends.
Output from the H + VCoT setting:
The two people in the red box are both wearing sports uniforms, indicating that they
are likely participating in the same team activity. Since they are on the same
volleyball court, it suggests they have coordinated roles within the same context,
typically associated with professional relations in a sports setting. Therefore, the
most likely answer is: C. Professional Relation.

45

2376
2377

2378
2379
2380
2381
2382
2383

2384
2385
2386
2387
2388
2389

2390
2391
2392
2393
2394

2395
2396
2397
2398
2399
2400

2401
2402
2403
2404
2405
2406

2407
2408
2409
2410
2411
2412

2413
2414
2415
2416
2417

2418
2419
2420
2421
2422
2423

2424
2425
2426
2427
2428
2429

Under review as a conference paper at ICLR 2025

Table 37: Comparison of outputs from H + VCoT and H + 1TCoT settings. Incorrect and correct
analysis contents are marked in red and blue, respectively.

Ability

Example

Compound Expression Recognition

Crowd Counting

Image:

Question:
What emotion is the person in the photo displaying?
A. Angrily Disgusted.
B. Happily Disgusted.
C. Disgustedly Surprised.
D. Sadly Surprised.
Answer: D.
Output from the H + VCoT setting:
To determine which emotion the baby in the photo is displaying, let’s analyze each of
the options given:- Happily Disgusted: An emotion combining happiness and
disgust, likely would show signs of smiling with a look of dislike. Sadly Surprised:
An emotion combining sadness and surprise, would show signs of tears or
downturned lips with wide eyes or raised eyebrows. Disgustedly Surprised: An
emotion combining disgust and surprise, would show signs of wrinkled nose,
possibly pulled back lips, and wide eyes. Angrily Disgusted: An emotion combining
anger and disgust, would show furrowed brows, clenched teeth, and a wrinkled nose.
Analyzing the baby’s face: The baby’s mouth is slightly open. Eyes are wide open.
Eyebrows are raised.- There are no signs of a smile or frown. These features suggest
a combination of surprise and possibly a hint of discomfort or unease, but there are
no clear signs of happiness or anger. Given these observations, the most likely
emotion the baby is displaying is: C. Disgustedly Surprised.
Output from the H + 1TCoT setting:
The person in the photo has their eyebrows raised and slightly together, their eyes
open wide, and their mouth slightly open. These facial features generally indicate a
surprised expression, which can be mixed with another emotion. Based on the
available options and the facial features observed:- Eyebrows are not furrowed or
lowered, which rules out anger. The eyes are wide open, suggesting surprise or
sadness rather than disgust. The mouth is not forming a clear smile or frown,
maintaining a neutral or surprised shape . Given these observations, the most likely
emotion is D. Sadly Surprised. The other combinations do not fit well with the facial
features depicted. The correct answer is: D. Sadly Surprised

Image:

Question:
How many figures are shown in this pic?
A. 7.
B. 6.
C. 5.
D. 4.
Answer: D.
Output from the H + VCoT setting:
To determine the number of people in the image, let’s examine it closely: 1. There is
one person wearing a white shirt at the back left. 2. There is one person wearing a
black shirt with text and shorts, slightly to the right of the first person. 3. There is
one person in a light gray shirt, squatting and reaching for the ball. 4. There is one
person in a turquoise shirt actively engaging with the person in gray. 5. There is
another person, partially obscured by the trees and goalpost, sitting on the ground in
the background. Counting all these, there are 5 people in the image. The correct
answer is: C. 5
Output from the H + 1TCoT setting:
Based on the provided image, there are a total of 4 people visible. My rationale: 1.
One person in the foreground with sunglasses, actively engaging with the ball. 2.
One person near the goalpost, in a teal shirt and white shorts. 3. Another person
standing, wearing a black shirt and shorts. 4. Another person further back, near the
goalpost and the other player. Answer: D. 4.

46

2430
2431

2432
2433
2434
2435
2436
2437

2438
2439
2440
2441
2442
2443

2444
2445
2446
2447
2448

2449
2450
2451
2452
2453
2454

2455
2456
2457
2458
2459
2460

2461
2462
2463
2464
2465
2466

2467
2468
2469
2470
2471

2472
2473
2474
2475
2476
2477

2478
2479
2480
2481
2482
2483

Under review as a conference paper at ICLR 2025

Table 38: Comparison of outputs from H + 1TCoT and H + 2TCoT settings. Incorrect and correct
analysis contents are marked in red and blue, respectively.

Ability

Example

Compound Expression Recognition

Cross-Pose Face Recognition

Image:

Question:
What is the facial expression of the person in this image?
A. Sadly Fearful.
B. Disgustedly Surprised.
C. Angrily Disgusted.
D. Happily Surprised.
Answer: A.
Output from the H + 1TCoT setting:
The facial expression of the person in the image includes wide eyes and an open
mouth showing teeth, indicative of surprise, with a general positive emotional tone.
This suggests D. Happily Surprised. The facial emotional features described match
the expression shown in the image, with eyebrows raised and drawn together, eyes
wide open, and mouth slightly open with a downturn indicating sadness and fear.
The person is showing a mixture of happiness and surprise. Thus, the correct answer
is: D. Happily Surprised.
Output from the H + 2TCoT setting:
A. Sadly Fearful. The facial emotional features described match the expression
shown in the image, with eyebrows raised and drawn together, eyes wide open, and
mouth slightly open with a downturn indicating sadness and fear.

Image:

Question:
Are the subjects in the two photographs the identical individual?
A. yes.
B. no.
Answer: A.
Output from the H + 1TCoT setting:
Based on the analysis of the two photographs, here are the similarities and
differences in their facial features: Similarities: Both individuals have light-colored
hair. Both have similar styles in terms of hair length and general appearance.
Differences: The person in the first image has a more mature appearance and their
hair is a lighter shade, almost blond. The person on the right appears younger and
has a hairstyle that frames their face differently. The facial structures and expressions
differ; the person on the left has a more elongated face, whereas the person on the
right has a more rounded face. Taking these observations into account, it appears that
they have notable differences in their appearance which suggest that they are not the
same person. Therefore, the answer is: B. no.
Output from the H + 2TCoT setting:
Based on the analysis of the similarities and differences: Similarities 1. Hair Color
and Style: Both individuals have light-colored hair with a similar longer length and
some layering. 2. Facial Structure: Both have similar facial structures, particularly
around the cheeks and jawlines. Differences: 1. Age Appearance: The individual in
the left photograph appears older compared to the one on the right. 2. Facial
Expression and Angle: The left photograph shows a front-facing angle, while the
right is a side profile, which can affect the perception of facial features. 3. Makeup
and Lighting: The lighting and possible use of makeup in the left photograph make it
difficult to see the natural contours of the face compared to the right photograph.
Taking all these factors into account, the differences in age appearance and the
angles make it challenging to definitively determine if they are the same individual.
However, based on the overall similarity in facial structure and hair, it seems
plausible that they could be the same person at different times in their life or captured
differently. Answer: A. yes.

47

2484
2485

2486
2487
2488
2489
2490
2491

2492
2493
2494
2495
2496
2497

2498
2499
2500
2501
2502

2503
2504
2505
2506
2507
2508

2509
2510
2511
2512
2513
2514

2515
2516
2517
2518
2519
2520

2521
2522
2523
2524
2525

2526
2527
2528
2529
2530
2531

2532
2533
2534
2535
2536
2537

Under review as a conference paper at ICLR 2025

C.6 SPECIALIST MODELS SIGNIFICANTLY OUTPERFORMING MLLMS

We list the early specialist models used for comparison in Table 39.

Table 39: Early specialist models used for comparison.
Performance
5.47

Early Specialist Model
CORAL (Cao et al., 2020)

Metric
MAE

Ability
Age

Basic Expr.

Comp. Expr.

Deepfake
Spoofing
Basic FR
C.P. FR
C.A. FR
S.L. FR
Occ. FR
Action
Counting
Re-ID

Dataset
UTKFace
RAF-DB
(Basic)
RAF-DB
(Compound)
FF++
SiW-Mv2
LFW
CPLFW
CALFW
SLLFW
MLFW
HICO-DET
ShTech-A
Market1501

ACC

74.20

DLP-CNN (Li et al., 2017b)

ACC

ACC
ACER
ACC
ACC
ACC
ACC
ACC
mAP
MAE
ACC

44.55

82.01
9.40
99.50
87.47
92.43
98.40
82.87
19.81
110.20
95.26

DLP-CNN (Li et al., 2017b)

XceptionNet Chollet (2017)
SRENet Guo et al. (2022)

R50 (He et al., 2016)
+ CosFace (Wang et al., 2018)
+ CASIA-WebFace (Yi et al., 2014)

ConsNet (Liu et al., 2020)
MCNN (Zhang et al., 2016)
LightMBN (Herzog et al., 2021)

D POTENTIAL BIAS FOR DEMOGRAPHIC CHARACTERISTICS

Do MLLMs contain potential biases? Specifically, do their performances vary based on the demo-
graphic characteristics of the input faces? Existing works, such as constructing the RFW (Wang
et al., 2019) and BFW (Robinson et al., 2020) datasets, have explored racial biases in face recog-
nition systems.
Inspired by these works, we investigate whether MLLMs exhibit different face
recognition abilities across different racial groups.

We transform face pairs from the Caucasian, African, Asian, and Indian subsets of the RFW dataset
into face recognition problems similar to those in Face-Human-Bench. The test results of the three
best-performing open-source models in our main experiments are presented in Table 40, revealing
the racial bias of MLLMs in face recognition ability. The performance of Caucasians is the best for
each model, significantly surpassing that of other racial groups. In our future work, we will sys-
tematically evaluate the performance variations of MLLMs on samples with different demographic
characteristics.

Table 40: Racial bias of MLLMs. The evaluation metric used is ACC.

Model
ResNet34+CASIA-WebFace+ArcFace
InternVL-Chat-v1.5
LLaVA-NeXT-34B
InternVL-Chat-v1.2-Plus

Caucasian African Asian
83.98
69.67
66.35
70.38

92.15
76.62
71.12
76.68

84.93
60.75
62.23
67.97

Indian Mean
87.27
88.00
69.65
71.58
66.71
67.15
71.90
72.55

E PRIVACY PROTECTION

Face-Human-Bench can also be used to evaluate privacy protection. In some scenarios, we want
MLLMs to refuse to answer certain questions related to faces and humans. In such cases, lower
performance on the Face-Human-Bench indicates a higher success rate in privacy protection on
this information. Table 41 presents a comparison of the performance between APIs provided by
OpenAI and Azure OpenAI. Note that Azure OpenAI primarily offers security and enterprise-grade
services. GPT-4V and GPT-4o from Azure OpenAI show significant performance degradation in
age estimation and expression recognition. Here are some example outputs:

• “I cannot determine the age of the person in the photo with the information provided.”

48

2538
2539

2540
2541
2542
2543
2544
2545

2546
2547
2548
2549
2550
2551

2552
2553
2554
2555
2556

2557
2558
2559
2560
2561
2562

2563
2564
2565
2566
2567
2568

2569
2570
2571
2572
2573
2574

2575
2576
2577
2578
2579

2580
2581
2582
2583
2584
2585

2586
2587
2588
2589
2590
2591

Under review as a conference paper at ICLR 2025

Table 41: Scores of GPT-4o and GPT-4V APIs from OpenAI and Azure OpenAI.

Model

GPT-4V (Azure OpenAI)
GPT-4V (OpenAI)
GPT-4o (Azure OpenAI)
GPT-4o (OpenAI)

Model

Attr.

64.5
77.5
56.0
77.0

Age

34.7
53.7
41.3
61.0

Attr. Action

GPT-4V (Azure OpenAI)
GPT-4V (OpenAI)
GPT-4o (Azure OpenAI)
GPT-4o (OpenAI)

52.0
73.0
64.0
63.5

82.0
78.0
78.0
81.0

Expression

Face Understanding

Attack Detection

Basic Comp. Mean DFD FAS mean
50.0
27.0
54.5
75.0
52.5
17.0
58.5
83.0

48.0
13.5
0.0
50.5
61.5
48.0
46.0
8.5
0.0
62.0
53.0
72.5
Human Understanding

52.0
58.5
59.0
64.0

Spatial Relation
CC
48.7
71.3
45.3
58.7

Mean
55.3
54.7
45.7
54.3

RPU
62.0
38.0
46.0
50.0

Social Relation

SRR
64.0
68.0
68.0
66.0

IR Mean
69.0
74.0
76.0
84.0
76.0
84.0
80.0
94.0

Basic
76.0
96.0
88.0
96.0

Re-ID

73.0
83.0
79.0
79.0

Face Recognition

C.P.
54.0
72.0
62.0
72.0

C.A.
62.0
92.0
60.0
74.0

S.L. Occ. Mean
66.0
72.0
66.0
81.2
64.0
82.0
72.4
72.0
80.0
73.6
50.0
76.0

Face Human

Per. Rea. Overall

45.7
65.7
46.1
68.5

66.3
72.9
68.5
71.6

49.4
66.4
50.1
68.9

65.8
73.7
68.3
71.7

56.0
69.3
57.3
70.0

• “I’m sorry, but the image is too blurry to make an accurate assessment of the person’s age.”
• “I don’t have enough visual information from the image provided to accurately determine the

emotion being expressed by the person.”

• “I’m unable to determine the person’s expression due to the blurred face. Based on the available

data, I cannot select a correct answer from the provided options.”

From these outputs, it can be observed that Azure OpenAI might employ security strategies such as
refusing to answer or blurring images.

F A DEMONSTRATION OF HOW TO ENHANCE MULTI-MODAL ASSISTANT

RESPONSES WITH SPECIALIST MODELS

In Figure 14, we use media forensics as an application scenario to demonstrate how specialist models
can improve the response quality of a multimodal assistant. Path 1 directly uses the MLLM to
generate responses, while Path 2 introduces a well-trained specialist model for deepfake detection
to determine whether there are digital artifacts on the faces in the image. By using the output of
the specialist model to enhance the prompt, Path 2 ultimately allows the MLLM to provide more
accurate responses.

Figure 14: A demonstration of how to enhance multi-modal assistant responses with specialist mod-
els in media forensics.

G LIMITATIONS

Despite the rich findings, there are still some limitations in this study. (1) This is the first work to
comprehensively evaluate the face and human understanding abilities of MLLMs, mainly focusing
on perception and simple reasoning. It does not involve tasks that require complex reasoning by
integrating multiple face and human information. We plan to explore this in future work. (2) Con-
sidering the languages supported by existing mainstream MLLMs, Face-Human-Bench currently
includes only English and Chinese. The capabilities of MLLMs in understanding face and human
information in more languages remain to be further explored.

49

2592
2593

2594
2595
2596
2597
2598
2599

2600
2601
2602
2603
2604
2605

2606
2607
2608
2609
2610

2611
2612
2613
2614
2615
2616

2617
2618
2619
2620
2621
2622

2623
2624
2625
2626
2627
2628

2629
2630
2631
2632
2633

2634
2635
2636
2637
2638
2639

2640
2641
2642
2643
2644
2645

OriginalPrompt:Please determine whether the following content is misinformation:Gordon Brown is forced to resign EU meeting by Nicolas Sarkozy the French president in Paris.Enhanced Prompt by Specialist Model:Please determine whether the following content is misinformation:Gordon Brown is forced to resign EU meeting by Nicolas Sarkozy the French president in Paris.Note: There are deepfake artifacts on the face of the person on the left.FakeMLLMSpecialistModelForDeepfakeDetection①②②②②②①&②Under review as a conference paper at ICLR 2025

H ETHICS STATEMENT

Our work does not involve reproducing, duplicating, copying, selling, trading, reselling, or exploit-
ing any images from the original public datasets of the face and human community for any commer-
cial purposes. Additionally, our work does not involve further copying, publishing, or distributing
any portion of the images from the original public datasets. We fully comply with the agreements of
all used original public datasets.

We will only open-source the JSON files containing our test problems and the data preprocessing
scripts. You need to download all the original images from the involved public datasets yourself
and organize the folders according to our instructions. The data preprocessing scripts will produce
images for multi-modal QAs only during testing.

In our semi-automatic data pipeline, we provide adequate compensation to all participating data re-
viewers and ensure that this process complies with laws and ethical guidelines. Data reviewers only
remove erroneous problems and thus do not involve the impact of regional or cultural differences
among reviewers.

Face-Human-Bench is intended solely for academic and research purposes. Any commercial use or
other misuse that deviates from this purpose is strictly prohibited. We urge all users to respect this
provision to maintain the integrity and ethical use of this valuable resource.

2646
2647

2648
2649
2650
2651
2652
2653

2654
2655
2656
2657
2658
2659

2660
2661
2662
2663
2664

2665
2666
2667
2668
2669
2670

2671
2672
2673
2674
2675
2676

2677
2678
2679
2680
2681
2682

2683
2684
2685
2686
2687

2688
2689
2690
2691
2692
2693

2694
2695
2696
2697
2698
2699

50

