MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
Under review as a conference paper at ICLR 2025

MMKE-BENCH: A MULTIMODAL EDITING BENCH-
MARK FOR DIVERSE VISUAL KNOWLEDGE

Anonymous authors
Paper under double-blind review

ABSTRACT

Knowledge editing techniques have emerged as essential tools for updating the
factual knowledge of large language models (LLMs) and multimodal models
(LMMs), allowing them to correct outdated or inaccurate information without
retraining from scratch. However, existing benchmarks for multimodal knowledge
editing primarily focus on entity-level knowledge represented as simple triplets,
which fail to capture the complexity of real-world multimodal information. To
address this issue, we introduce MMKE-Bench, a comprehensive MultiModal
Knowledge Editing Benchmark, designed to evaluate the ability of LMMs to edit
diverse visual knowledge in real-world scenarios. MMKE-Bench addresses these
limitations by incorporating three types of editing tasks: visual entity editing, visual
semantic editing, and user-specific editing. Besides, MMKE-Bench uses free-form
natural language to represent and edit knowledge, offering a more flexible and
effective format. The benchmark consists of 2,940 pieces of knowledge and 7,229
images across 110 fine-grained types, with evaluation questions automatically
generated and human-verified. We assess five state-of-the-art knowledge editing
methods on three prominent LMMs, revealing that no method excels across all
criteria, and that visual and user-specific edits are particularly challenging. MMKE-
Bench sets a new standard for evaluating the robustness of multimodal knowledge
editing techniques, driving progress in this rapidly evolving field.

1

INTRODUCTION

Large language models (LLMs) and multimodal models (LMMs) have demonstrated remarkable
success across various tasks due to their powerful understanding and reasoning abilities, grounded
in vast amounts of knowledge (Brown et al., 2020; Zhao et al., 2023; Liu et al., 2024b). However,
the knowledge within these models can become outdated or inaccurate over time due to evolving
real-world information and changes in factual data. To address this, knowledge editing techniques
have been developed to correct inaccuracies and inject new knowledge into pre-trained models
with minimal cost, without affecting unrelated content (Mitchell et al., 2022b; Yao et al., 2023). In
recent years, several datasets have been introduced to benchmark the progress of knowledge editing
methods in both the textual (Yao et al., 2023; Onoe et al., 2023; Cao et al., 2021; Li et al., 2023b) and
multimodal domains (Cheng et al., 2023; Huang et al., 2024; Li et al., 2024; Zhang et al., 2024).

However, most existing benchmarks focus on editing entity-level knowledge, typically formatted as a
triplet (subject, relation, object). While effective in certain tasks, this format lacks the complexity
required for real-world applications, particularly in multimodal domains where visual knowledge
must also encompass actions, body gestures, and object relationships. Furthermore, knowledge
editing techniques have quickly saturated on these benchmarks, achieving near-perfect performance.
For example, simply fine-tuning the LLaVA model achieved 99.59%, 99.43%, and 95.48% accuracies
for reliability, text generalization, and image generalization, respectively, on the VLKEB bench-
mark Huang et al. (2024). This highlights the urgent need for a more challenging benchmark to foster
the development of multimodal knowledge editing techniques.

To address these issues, we introduce MMKE-Bench, a comprehensive multimodal knowledge editing
benchmark designed to evaluate diverse semantic editing in real-world scenarios. MMKE-Bench
represents multimodal knowledge using free-form natural language descriptions paired with images,
providing a richer and more flexible expression of interconnected information. Reflecting real-world

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: Comparison between the existing benchmark and MMKE-Bench with a detailed example.
In this example, the texts in red represent the edited counterfactual content. T/I-Rel represents
text and image reliability, T/I-Gen represents text and image generalization and Port represents
portability. Previous benchmarks mainly focus on entity recognition editing using a triplet-based
knowledge representation format, which does not align with actual scenarios. MMKE-Bench focuses
on evaluating diverse semantic editing in realistic scenarios in a natural language format.

needs, MMKE-Bench includes three types of editing: visual entity editing, visual semantic editing,
and user-specific editing. Visual entity editing updates entity-centric visual knowledge, while visual
semantic editing targets complex object behaviors and relationships, such as referee gestures and
traffic signals. Lastly, user-specific editing evaluates the model’s ability to integrate individualized
knowledge. The first two types modify existing knowledge, while the third adds new knowledge.
Comparisons with existing benchmarks are shown in Fig.1 and Tab.1.

To construct MMKE-Bench, we first collect original knowledge from various images and knowledge
sources (e.g., multimodal knowledge graphs, demo videos, Google, and LLM generation). Next,
we create editing knowledge by applying counterfactual editing for the text modality and image
replacement for the image modality. User-specific editing involves adding entirely new, personalized
knowledge to the model and does not need counterfactual editing. Following previous works (Zheng
et al., 2023; Huang et al., 2024), we adhere to four evaluation principles: reliability, locality, gener-
alization, and portability, generating evaluation questions and answers automatically. Finally, all
questions and answers undergo human verification and are revised where necessary. The resulting
benchmark contains 2,940 pieces of knowledge and 7,229 images across 110 fine-grained types.

We evaluate five of the most prominent multimodal knowledge editing methods on three representative
LMMs, assessing their performance in both single and sequential editing tasks. Empirically, we
find that (i) no single editing method excels across all evaluation criteria; (ii) visual knowledge and
user-specific knowledge are more difficult for LMMs to edit; (iii) modern LMMs excel in producing
and applying edited knowledge; and (iv) the proposed benchmark proves more challenging than
previous benchmarks.

To sum up, our contribution can be summarized as follows:

• We propose MMKE-Bench, a challenging benchmark for evaluating diverse semantic editing
in real-world scenarios. It adopts free-form natural language-based knowledge representation
and includes three types of editing aligned with real-world contexts.

2

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Table 1: Overall comparison with existing multimodal knowledge editing benchmarks.

Benchmark

Knowledge Representation

Visual Entity
Editing

Visual Semantic
Editing

User-Specific
Editing

Evaluation Principle

MMEdit
MIKE
MC-MKE
VLKEB
MMKE-Bench

Short-Text
Triplet
Triplet
Triplet
Free-Form Natural Language

✓
✓
✓
✓
✓

✗
✗
✗
✗
✓

✗
✗
✗
✗
✓

Reliability, Locality, and Generalization
Reliability, Locality, and Generalization
Reliability, Locality, and Generalization
Reliability, Locality, Generalization, and Portability
Reliability, Locality, Generalization, and Portability

• We introduce a novel pipeline for benchmark construction that collects original knowledge,
generates editing knowledge, and produces evaluation questions guided by four principles.
• Extensive experiments with various baseline methods and LMMs in both single and se-
quential editing settings are conducted, revealing several limitations in existing knowledge
editing approaches.

2 RELATED WORK

2.1 LARGE MULTIMODAL MODEL

Large multimodal models have achieved excellent performance in various multimodal understanding
tasks due to vast knowledge and effective cross-modality alignment. Typically, such models integrate
a vision encoder with a pertained large language model, linking the two components by an alignment
module. Notably, BLIP-2 (Li et al., 2023a) adopts Q-Former, a lightweight Transformer, as the
alignment module. Inspired by the instruction tuning in LMM, MiniGPT-4 (Zhu et al., 2023) and
InstructBLIP (Dai et al., 2023) enhance this structure with multimodal instruction tuning. In contrast,
LLaVA (Liu et al., 2024b) utilizes an MLP layer for alignment and proposes to generate an instruction-
tuning dataset by self-instruct strategy (Wang et al., 2022). Qwen-VL (Bai et al., 2023) introduces a
novel module, the visual receptor, as its alignment module and proposes a three-stage training pipeline,
achieving excellent performance across various multimodal tasks. Besides, several notable LMMs,
such as mPLUG-DocOw 1.5 (Hu et al., 2024), InternVL-2 (Chen et al., 2024), and MiniCPM-V
2.5 (Yao et al., 2024), have also achieved comparable or even superior results compared with GPT-4o.

2.2 KNOWLEDGE EDITING FOR LARGE LANGUAGE MODEL

Existing methods for LLM can be divided into three categories: resorting to external knowledge,
incorporating knowledge into the model, and editing internal knowledge. Resorting to external
knowledge typically involves maintaining memory and retrieving the most relevant cases for each
input. For instance, IKE Zheng et al. (2023) provides in-context learning example support by building
three types of demo examples: copy, update, and retain. SERAC Mitchell et al. (2022b) builds a new
counterfactual model by keeping the base model and using a scope classifier to determine whether
to answer with a counterfactual model. The category of merging the knowledge into the model
aims to learn representations of the new knowledge and incorporate this information into the model.
Eva-KELLM Wu et al. (2023a) employs LoRA for knowledge editing, while GRACE (Hartvigsen
et al., 2023) adopts a novel approach by maintaining a discrete codebook functioning as an adapter.
Lastly, editing intrinsic knowledge works on directly modifying the model’s weight using knowledge-
specific methods through meta-learning and localization editing. The meta-learning method trains
a hypernetwork to learn how to adjust the model. KE De Cao et al. (2021) utilizes new knowledge
representations directly to train the model to update the matrix, while MEND Mitchell et al. (2022a)
applies rank-one decomposition to divide the model into two rank matrices. Additionally, localization
approaches, like ROME Meng et al. (2022) and MEMIT, Meng et al. (2024) employ a causal analysis
method to detect which parts of the hidden state are more important by treating editing as minimal
optimization, ensuring its reliability and non-circumvention.

2.3 KNOWLEDGE EDITING FOR LARGE MULTIMODAL MODEL

Recently, several benchmarks have been proposed to evaluate the performance of editing LMMs.
The MMEdit benchmark (Cheng et al., 2023) systematically defines the first evaluation framework

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

for multimodal knowledge editing based on visual question answering and image caption tasks. As
the MMEdit could not assess fine-grained entity knowledge, subsequent evaluation benchmarks
focus on fine-grained entity recognition editing. MIKE (Li et al., 2024) evaluates recognizing new
entities while VLKEB (Huang et al., 2024) targets editing known entities and introduces a portability
evaluation principle. MC-MKE (Zhang et al., 2024) further extends fine-grained entity recognition by
emphasizing modality consistency. However, these benchmarks mainly represent editing knowledge
through triples and overlook diverse semantic editing in realistic scenarios.

3 PROBLEM DEFINITION

3.1 KNOWLEDGE REPRESENTATION AND EDITING

MMKE-Bench is distinctive in evaluating diverse semantic editing in realistic scenarios, leveraging
natural language-based knowledge representation. It includes three types of editing: visual entity
editing, visual semantic editing, and user-specific editing. Each piece of knowledge is represented
in a unified format, k = (i, d), where i refers to the image and d represents the natural language
description of the main object, visual content, or a user-personalized item. For example, in the case
of a referee’s gesture, the image captures the action performed by the referee, while the description
explains how the gesture is executed and its impact on the match. During knowledge editing, the
original knowledge is transformed into ke = (ie, de) in both visual entity and visual semantic editing,
while it remains ke = (i, d) for user-specific editing. This is because user-specific editing introduces
entirely new personalized knowledge into LMMs without needing to alter the image or description.

3.2 EDITING TYPE OF MMKE-BENCH

Considering real-world needs, MMKE-Bench includes three types of editing as follows.

Visual Entity Editing This type targets entity-centric modifications and the description covers
multiple aspects of an entity. In realistic scenarios, models may misidentify or retain incorrect or
outdated information about the entity. Visual entity editing addresses this issue by allowing for
simultaneous correction of all related content. To simulate such scenarios, we propose replacing the
original image of the entity with that of another entity of the same type and modifying key information
into counterfactual content. As shown in Fig.1, Zlatan Ibrahimovi´c’s image is replaced with that of
Wayne Rooney, and related information (e.g., nationality, club) is altered to counterfactual details.

Visual Semantic Editing This type focuses on complex visual semantics-centric modifications,
encompassing body gestures, actions, object relationships, and so on. The description provides de-
tailed information about the semantic action and its rules or meanings. The LMMs may misrecognize
and misunderstand these semantics, but visual semantic editing can address this issue by modifying
both actions images, and meanings simultaneously. To simulate this, this type of editing also involves
replacing the image of one semantic action with that of another action of the same type and altering
the rule or meaning to counterfactual content. As shown in Fig.1, the offside gesture in soccer is
replaced with that of substitution, and the associated rule (e.g. kick-off location) is modified to
counterfactual contents.

User-Specific Editing This type focuses on injecting personalized user information into LMMs, and
the description details the relationship between the user and the object, as well as their experiences.
As there is a growing demand for LMMs to function as personalized AI assistants that can remember
relevant user information, user-specific editing is designed to meet this need. Pre-trained LMMs
serve as general models, so all user-specific information is treated as new knowledge for LMM. Thus,
counterfactual editing is unnecessary, and original knowledge is used as editing knowledge. For
example, Fig.1 describes the relationship between the toy puppet and the user’s habits.

4 BENCHMARK

As shown in Fig. 2, we construct the benchmark through four steps: i) Original Knowledge Collection;
ii) Editing Knowledge Generation; iii) Evaluation Question Generation; and iv) Human Verification.

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 2: The construction pipeline of MMKE-Bench.

4.1 ORIGINAL KNOWLEDGE COLLECTION

In gathering original knowledge, we first list candidate fine-grained entities, visual semantics, or
user-specific items, and then collect their corresponding images and descriptions.

For visual entity editing, we source candidates from two datasets: the multimodal knowledge graph,
MMpedia Wu et al. (2023b), and the visual entity recognition dataset, OVEN Hu et al. (2023).
For each entity selected from the existing dataset, we get their images from the datasets and then
manually review the images by removing the entities that cannot uniquely identify the main entity
from images and noise images. For entities with less than two images, we recollect additional images
by crawling from Google. Next, we retrieve entity descriptions from the Wikipedia summary dumps1
and summarize the description by an LLM to generate the final descriptions. As shown in Fig. 3, this
type covers 10 broad categories and 70 types.

For visual semantic editing, as shown in
Fig. 3, we define the candidates across 14
broad categories of semantic knowledge,
including single-person behaviors, single-
object behaviors or attributes, object rela-
tionships, and global structures. These cat-
egories are further divided into 25 types.
For certain types of visual knowledge that
have corresponding datasets, such as ob-
ject relationships, textures, and art styles,
we collect both the candidate semantics
and associated images from these datasets.
For other cases, we extract images from
demonstration videos or gather them via
Google, applying human verification for
quality control. Descriptions of the visual
semantic actions, along with the rules or
meanings conveyed by these behaviors, are
generated with the assistance of LLM or hu-
man writers. Details of the image sources
are provided in the appendix.

Figure 3: The types of samples in MMKE-Bench.

For user-specific editing, we consider 9 broad categories of personalized information sources, such as
favorite singers, owned pets, and alma maters. For personal items and pets, we gather candidates and
images from the existing personalized research works Nguyen et al. (2024); Alaluf et al. (2024). For
singers, actors, and cartoon characters, we first generate a candidate list and then crawl images from
Google. For other categories, including company, university, sports club, and organization, we source
candidates from MMpedia, manually verifying and removing noise images. Finally, we employ an
LLM to generate personalized relationships and experiences between the user and these objects.

1https://dumps.wikimedia.org/enwiki/20240620/

5

Under review as a conference paper at ICLR 2025

4.2 EDITING KNOWLEDGE GENERATION

Considering the multimodal nature of large multimodal models (LMMs), we propose editing both
text and visual modalities when constructing the benchmark. Specifically, we focus on editing visual
entities and visual semantic knowledge while leaving user-specific knowledge unchanged. The former
is treated as knowledge editing, while the latter is regarded as knowledge insertion.

For the visual modality, we follow the image-replacement-based editing approach from previous
work Huang et al. (2024), where an image of the entity or semantic action is randomly replaced
with another of the same type. For example, as illustrated in Fig. 1 and Fig. 2, the assistant referee’s
offside penalty gesture is replaced with a substitution gesture in the edited visual content. In the text
modality, we modify key information about the entity and the rule or meaning into counterfactual
content for visual entity editing and visual semantic editing, respectively. Additionally, we update
the action description to align with the new visual content. In the example of the offside gesture, the
original action description is replaced with that of the substitution gesture, and the kick-off location
is edited from the foul position to the penalty spot.

4.3 EVALUATION QUESTION GENERATION

We adhere to four key evaluation principles to generate both the questions and answers. The reliability
and portability questions are generated by prompting LLM and we show the prompts in the appendix.

Reliability Question Generation The reliability criterion assesses whether the edited knowledge is
correctly produced after the editing process. When generating questions and answers, we prompt the
LLM with a requirement that the question must ask one aspect of the edited counterfactual content
(e.g., the kick-off location of the offside penalty). To evaluate this, we consider both text reliability
and image reliability, measuring the LMM’s ability to edit across text and visual modalities. Text
reliability questions are crafted to be answerable without images, while image reliability questions
use the format {the type in the image} to reference the main object, behavior, or personalized item.
An example is provided in Fig. 2. We denote the reliability question sets as Qrel = (ie, qr, ar), where
ie represents the edited image, qr the question, and ar the answer. Let Mθ and M ′
θ denote the original
and edited LMMs, respectively, and I[·] denoted indicator function, reliability is then evaluated as:
I [M ′

E(ie,qr,ar)∼Qrel

θ(ie, qr) = ar]

(1)

Locality Question Generation The locality criterion evaluates how much unrelated knowledge
remains unchanged in the edited model by comparing its outputs before and after the editing process.
For locality, we assess both text and image locality, which tests the model’s stability when dealing
with out-of-scope knowledge from each modality. Following prior work, we source locality questions
and answers from the VLKEB benchmark Huang et al. (2024), where the text questions are drawn
from the NQ dataset Kwiatkowski et al. (2019), and the image questions are specifically designed by
VLKEB. We represent the locality question set as Qloc = (il, ql), and locality is evaluated as:

E(il,ql)∼Qloc

I [Mθ(il, ql) = M ′

θ(il, ql)]

(2)

Generalization Question Generation The generalization criterion evaluates how effectively the
model responds to neighboring samples. Unlike triplet-based knowledge editing, we focus exclusively
on image generalization, as text generalization is not considered due to the free-form knowledge
format. For image generalization, we randomly select another image ig
e from the multiple available
images of an entity, visual behavior, or personalized item, and reuse the same question and answer
from the image reliability, with an example shown in Fig. 2. We define the generalization question as
Qgen = (ig
e, qg, ag), where qg = qr and ag = ar for the same object. Generalization is evaluated as:
θ(ig

e, qg) = ag]

e ,qg,ag)∼Qgen

I [M ′

E(ig

(3)

Portability Question Generation The portability criterion evaluates whether the edited knowledge
can be successfully applied to related content. Following prior work Huang et al. (2024), we adopt
text portability evaluation for visual entity editing and image modality portability for visual semantic
and user-specific editing to enhance visual modality evaluation.

For visual entity editing, we generate questions about the edited content, utilizing supplementary
information from Wikipedia for question generation. For example, if the current entity is the Eiffel

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

Tower and the edited content refers to the building’s designer, we might create a question like, “Who
is the designer of the Eiffel Tower?” We can then generate another question about the edited content,
such as asking for the designer’s birth year. By combining these two questions, we can formulate the
final probability question: “In which year was the builder of the Eiffel Tower born?”

In the case of visual semantic and user-specific editing, we first combine the image of the main
behavior or item with another image of the same type to create a new image, denoted as ip
e. We then
pose a question focusing on the differences between the two images, such as hair color or object shape.
By integrating this question with one related to the edited content, we derive the final portability
question. For instance, as shown in Fig. 2, given an image that includes the offside penalty gesture
and the corner-kick gesture made by two assistant referees, we might ask, “What color is the tops
of the referee who is making the offside gesture in the image?”. Denote the portability question as
Qport = (ip

e, qp, ap), portability is evaluated as:

E(ip

e ,qp,ap)∼Qport

I [M ′

θ(ip

e, qp) = ap]

(4)

4.4 HUMAN CHECK & BENCHMARK STATISTICS
During benchmark construction, we manually collected, reviewed, and filtered the samples multiple
times. In the original knowledge collection stage, we conducted a thorough manual review of
the images associated with each entity, behavior, and object to ensure the quality of the collected
visuals. Furthermore, after counterfactual editing and question generation, we manually reviewed the
questions, revised unsuitable questions, and corrected wrong answers.

The statistics of MMKE-Bench are shown in Tab.2.
MMKE-Bench encompasses 3 classes of edited
knowledge, totaling 2,940 knowledge pieces and
7,229 images. The knowledge spans 110 types,
highlighting the diversity of MMKE-Bench. We
split the dataset into training and validation sets at
4:6, with the training set reserved solely for specific
knowledge editing methods (e.g., SERAC Mitchell et al. (2022b) and MEND Mitchell et al. (2022a)).

Table 2: The statistics of MMKE-Bench.

Visual Entity Editing
Visual Semantic Editing
User-Specific Editing

Types Train Test

3,182
1,521
2,526

955
293
511

636
214
331

75
42
24

Images

5 EXPERIEMENT
5.1 EXPERIMENTAL SETUP

LMMs and Editing Methods To evaluate our benchmark, we conduct experiments on three
representative LMMs: BLIP-2 (Li et al., 2023a), MiniGPT-4 (Zhu et al., 2023), and LLaVA-1.5 (Liu
et al., 2024a). Besides, following the previous benchmarks, we select five representative multimodal
knowledge editing methods: 1) Fine-tuning (FT). We focus on finetuning the LLM (FT-LLM) or
the vision-language alignment module (FT-Alignment), where only the last layer of the LLM is fine-
tuned.2) Knowledge Editor (KE) (De Cao et al., 2021). KE uses a hyper-network with constrained
optimization to predict the weight update at test time. 3) MEND (Mitchell et al., 2022a): MEND
learns a low-rank decomposition of the gradient of standard fine-tuning. 4) SERAC (Mitchell et al.,
2022b): SERAC is a memory-based method and it stores edits in explicit memory. 5) In-context
Knowledge Editing (IKE) (Zheng et al., 2023): IKE is inspired by in-context learning, and a new
demonstration formatting and organization strategies are to construct for guiding knowledge editing.

Experiments settings We perform experiments under both single editing and sequential editing.
Single editing is mostly adopted and it updates the base model for each piece of knowledge and then
evaluates the editing performance. The sequential editing continuously updates the base model with
multiple pieces of knowledge and then evaluates the first piece of knowledge. We follow the previous
benchmark and adopt the token-level editing accuracy.

5.2 REULTS
5.2.1 SINGLE EDITING RESULTS

The results of the existing multimodal knowledge editing methods on MMKE-Bench are shown in
Tab. 3, Tab. 4, and Tab. 5. Based on the results, we have several observations.

1) FT-LLM is a strong baseline, while IKE demonstrates the best reliability and generalization.
FT-LLM serves as a strong baseline, with other multimodal knowledge editing methods like SERAC,
MEND, and KE performing similarly or even worse than FT-LLM. Notably, IKE achieves the best

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Under review as a conference paper at ICLR 2025

Table 3: The results of single editing for BLIP-2 on MMKE-Bench.

Visual Entity
Editing

Visual Semantic
Editing

User-Specific
Editing

Average

Method

T-Loc

I-Loc T-Rel

I-Rel

I-Gen

Port

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

66.72
100.00
65.41
99.98
96.36
78.43

63.69
100.00
74.63
99.99
97.37
69.15

62.90
100.00
74.64
99.90
96.91
67.23

64.44
100.00
71.56
99.96
96.88
71.60

19.55
8.65
12.31
63.18
68.42
17.86

20.01
9.46
12.24
76.96
75.02
15.68

21.32
8.61
12.39
93.39
73.03
17.48

20.29
8.91
12.31
77.84
72.16
17.01

30.88
20.21
34.82
20.23
29.69
28.00

32.16
15.83
32.55
16.13
26.38
27.57

12.34
7.37
12.82
7.37
11.15
13.3

25.13
14.47
26.73
14.58
22.41
22.96

28.37
23.23
34.04
23.05
28.50
26.93

31.01
28.91
32.73
17.92
27.18
20.55

26.70
17.28
31.39
14.07
25.66
20.45

28.69
23.14
32.72
18.35
27.11
22.64

28.72
22.84
33.99
23.12
28.49
27.52

31.17
26.11
32.90
18.92
27.56
21.30

26.95
16.99
31.10
14.39
25.45
20.21

28.95
21.98
32.66
18.81
27.17
23.01

22.06
16.90
20.17
16.36
16.97
28.74

2.47
4.92
4.84
3.56
3.64
5.76

5.18
6.29
5.84
4.91
4.92
10.83

9.90
9.37
10.28
8.28
8.51
15.11

Table 4: The results of single editing for MiniGPT4 on MMKE-Bench.

Visual Entity
Editing

Visual Semantic
Editing

User-Specific
Editing

Average

Method

T-Loc

I-Loc T-Rel

I-Rel

I-Gen

Port

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

81.42
100.00
54.80
99.99
97.36
81.93

85.20
100.00
60.79
99.95
97.84
80.61

81.81
100.00
61.51
100.00
97.51
78.04

82.81
100.00
59.03
99.98
97.57
80.19

29.97
24.60
10.50
85.26
77.86
20.47

31.54
25.20
11.13
70.14
80.52
21.50

34.19
28.33
11.37
99.90
81.09
21.67

31.90
26.04
11.00
85.10
79.82
21.21

43.44
31.93
60.56
31.92
41.77
39.53

44.55
23.08
61.49
23.25
36.79
37.77

39.79
21.28
84.09
21.30
28.12
21.89

42.59
25.43
68.71
25.49
35.56
33.06

36.88
31.11
55.46
32.02
37.97
39.00

45.08
41.62
53.44
26.52
43.08
35.32

38.83
33.86
62.05
30.48
40.82
36.29

40.26
35.53
56.98
29.67
40.62
36.87

36.83
32.06
44.14
32.18
38.01
38.89

45.31
38.45
53.18
25.40
42.83
35.20

38.56
34.69
61.89
30.08
40.23
36.36

40.23
35.07
53.07
29.22
40.36
36.82

34.79
27.22
43.15
28.73
30.66
36.70

6.71
8.25
10.92
7.25
6.85
13.25

10.24
11.56
13.89
10.50
11.19
19.97

17.25
15.68
22.65
15.49
16.23
23.31

results across nearly all knowledge editing tasks for three LMMs, excelling in text reliability, image
reliability, and image generalization. These results indicate that in-context examples significantly
enhance the model’s understanding of how knowledge is edited, leading to superior performance.

2) Image locality is more challenging than text locality, and SERAC and MEND perform best in
maintaining locality. Most knowledge editing methods deliver better text locality results compared
to image locality, suggesting that editing LMMs tends to compromise visual knowledge more severely,
resulting in lower image locality scores. SERAC and MEND stand out by achieving high locality
results. It may owe to the good retrieval accuracy of SERAC and fewer parameter updates by MEND.

3) All knowledge editing methods generalize well but struggle with portability. The I-gen results
mirror those of I-rel, indicating that current large multimodal models can extract invariant features
across different image variants of the same object. However, all existing multimodal methods fall
short in the portability evaluation, highlighting the difficulty of applying edited knowledge to new
content. KE performs best portability in most scenarios, suggesting that parameter-based editing
methods handle this challenge more effectively.

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Under review as a conference paper at ICLR 2025

Table 5: The results of single editing for LLaVA on MMKE-Bench.

Visual Entity
Editing

Visual Semantic
Editing

User-Specific
Editing

Average

Method

T-Loc

I-Loc T-Rel

I-Rel

I-Gen

Port

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

FT-LLM
FT-Alignment
IKE
SERAC
MEND
KE

75.01
100.00
61.67
100.00
96.79
77.57

79.62
100.00
61.10
99.99
98.15
71.39

75.19
100.00
68.49
99.95
98.3
69.63

76.61
100.00
63.75
99.98
97.75
72.86

16.79
8.49
15.59
99.19
71.15
16.51

16.06
19.61
16.12
34.4
83.34
8.08

20.53
13.06
17.09
97.39
84.12
9.29

17.79
13.72
16.27
76.99
79.54
11.29

47.16
35.61
64.39
35.61
45.67
44.04

48.68
27.66
59.04
27.76
41.43
47.80

58.10
42.51
92.26
42.81
52.05
54.62

51.31
35.26
71.90
35.39
46.38
48.82

43.57
36.01
61.11
34.19
42.22
44.53

47.81
42.06
53.9
41.02
44.19
40.69

47.63
40.39
75.71
36.38
46.43
48.27

46.34
39.49
63.57
37.20
44.28
44.50

43.66
37.62
61.16
34.02
42.35
44.63

47.54
34.56
53.19
41.85
43.99
39.50

48.29
44.56
76.04
36.59
46.33
48.55

46.50
38.91
63.46
37.49
44.22
44.23

45.78
35.95
48.73
36.22
39.42
47.04

11.09
14.51
22.67
12.49
11.95
19.28

12.78
20.76
42.25
13.37
14.36
24.64

23.22
23.74
37.88
20.69
21.91
30.32

4) Visual Semantic Knowledge and User-Specific Knowledge are more difficult for LMMs to
edit. Editing complex visual semantics and user-specific knowledge proves more challenging than
editing visual entities, as evidenced by lower reliability and portability scores. This suggests that
more advanced editing techniques are needed to edit complex visual semantics and inject personalized
information, further emphasizing the value of the proposed benchmark.

5) Modern LMMs excel in producing and applying edited knowledge. For reliability, generaliza-
tion, and portability evaluations, LLaVA-1.5 outperforms BLIP-2 and MiniGPT-4. This improved
performance can be attributed to its larger model size and better instruction-following capability,
as LLaVA-1.5 has more parameters than BLIP-2 and a more refined instruction-tuning design than
MiniGPT-4. These factors lead to its superior ability to understand and apply evolving knowledge.

6) No single editing method excels across all eval-
uation criteria. In conclusion, no single knowledge
editing method outperforms across all four evalua-
tion criteria. In-context learning-based methods are
strong at reproducing edited knowledge, memory-
based methods excel at preserving unrelated content,
and parameter-based methods are better at applying
edited knowledge to new contexts.

7) The proposed benchmark is more challenging
than previous ones. The comparison of IKE with
existing benchmarks for MiniGPT-4 is shown in
Fig. 4, this method achieves high scores across most
evaluation principles in previous benchmarks but
performs worse on our benchmark. This suggests
that the proposed benchmark introduces greater chal-
lenges than its predecessors.

Figure 4: Evaluation comparison of IKE for
MiniGPT-4 with existing benchmarks. Port for
MMEdit and MIKE, is set 1, as they are not
evaluated.

5.2.2 SEQUENTIAL EDITING RESULTS
Editing knowledge separately is impractical in real-world applications while continuous updates with
vast amounts of information are necessary. Consequently, we conduct sequential editing experiments
and utilize FT-LLM, FT-Alignment, and SERAC as editing methods. IKE and KE are excluded
because the edit samples also need to serve as test samples, which is not feasible in this context.

The results for LLaVA-1.5 are shown in Tab. 6, where the “gap” refers to the sequential length,
and “user num” is the number of users, with each user allowed a maximum of nine personalized
items. As observed, both FT-LLM and FT-Alignment tend to forget the previous editing, as shown by
the decreasing performance in text and image reliability and generalization with increasing gap. In
contrast, SERAC effectively maintains edited knowledge due to its explicit memory. Additionally,
FT-Alignment often preserves unrelated text outputs, while FT-LLM exhibits the opposite behavior.

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Under review as a conference paper at ICLR 2025

Table 6: The results of sequential editing for LLaVA-1.5 on MMKE-Bench.

Method

GAP /User Num T-Loc

I-Loc T-Rel

I-Rel

I-Gen

Port

FT-LLM

Visual Entity
Editing

FT-Alignment

SERAC

FT-LLM

Visual Semantic
Editing

FT-Alignment

SERAC

FT-LLM

User-Specific
Editing

FT-Alignment

SERAC

-
3
6
10

-
3
6
10

-
3
6
10

-
3
6
10

-
3
6
10

-
3
6
10

-
1
3
5

-
1
3
5

-
1
3
5

76.76
56.03
54.99
54.75

100.00
100.00
100.00
100.00

100.00
100.00
100.00
100.00

76.89
50.33
49.09
48.23

100.00
100.00
100.00
100.00

100.00
99.93
99.93
99.93

75.68
69.12
66.60
66.70

100.00
100.00
100.00
100.00

99.97
99.92
99.92
99.93

17.19
8.39
8.22
8.13

8.7
1.03
1.01
0.09

98.91
98.79
98.78
98.78

16.14
7.36
7.25
7.02

19.41
1.44
1.38
1.38

34.53
13.56
13.54
13.52

20.11
17.30
16.26
17.29

12.82
14.47
15.28
17.98

97.27
97.67
97.63
97.60

45.78
44.62
43.75
42.76

36.37
36.37
36.37
36.37

36.37
36.37
36.37
36.37

49.00
42.86
41.49
41.51

27.83
28
27.83
27.83

27.83
27.99
27.92
27.88

57.82
52.06
49.79
49.43

41.41
41.39
41.39
41.39

41.76
41.45
41.39
41.33

41.72
39.34
39.55
38.01

35.03
32.54
29.16
33.53

33.77
33.77
33.77
33.77

49.44
46.73
45.58
45.09

44.5
34.06
31.62
29.79

41.09
29.71
29.91
29.93

48.04
44.36
41.87
40.78

41.01
30.15
30.81
29.77

37.49
38.09
37.93
37.90

41.55
40.18
39.67
38.55

37.53
29.89
27.70
30.36

33.27
33.24
33.24
33.24

49.04
45.02
43.52
42.08

35.37
24.57
23.54
23.92

41.82
30.70
31.09
31.13

48.66
44.14
41.85
40.29

43.72
30.02
29.52
28.09

37.67
37.98
37.98
37.98

47.36
35.59
35.56
36.08

36.23
34.82
35.11
38.93

35.63
35.63
35.63
35.63

10.67
8.29
7.25
7.63

15.00
6.51
6.96
7.25

11.29
11.17
11.34
11.23

12.63
8.67
6.16
5.88

21.21
7.66
8.67
7.37

13.23
12.79
12.79
12.79

5.3

INSIGHT ANALYSIS

Case Study An editing example of visual entity editing by IKE and FT-LLM for LLaVA-1.5 is
presented in Fig.5. Both IKE and FT-LLM correctly answered the text reliability question. However,
IKE outperformed FT-LLM by also providing correct answers to the image generalization and
portability questions, highlighting IKE’s superior performance. The case study of reliability on visual
semantic editing is shown in Fig.6. As we can see, after editing, the model could effectively answer
the question based on editing knowledge.

Figure 5: Case study of editing examples

Figure 6: Case study of reliability

6 CONCLUSION

In this paper, we propose a comprehensive multimodal knowledge editing benchmark, named MMKE-
Bench, designed to evaluate diverse semantic editing in real-world scenarios using free-form natural
language representation. We propose to use free-form natural language representation combined with
an image to represent knowledge instead of representing it with a triplet. Besides, we propose three
kinds of editing to align with real-world scenarios. We conducted experiments on representative
LMMs and knowledge editing methods and found that more advanced knowledge editing methods
are needed for LMMs. We hope our work could inspire more multimodal knowledge editing research.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvlm:

Personalizing vlms for user-specific queries. ECCV, 2024.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou,
and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization,
text reading, and beyond. Arxiv, 2023.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and

et al. Language models are few-shot learners. NeurIPS, 2020.

Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. EMNLP,

2021.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi
Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial
multimodal models with open-source suites. ArXiv, 2024.

Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen, and Ningyu
Zhang. Can we edit multimodal large language models? In EMNLP, pp. 13877–13888, 2023.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De-

scribing textures in the wild. In CVPR, pp. 3606–3613, 2014.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang,
Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-
language models with instruction tuning. In NeurIPS, 2023.

Art Dataset. wiki art dataset.

url https://universe.roboflow.com/art-dataset/wiki-art , mar 2022. URL https://universe.
roboflow.com/art-dataset/wiki-art. visited on 2023-01-18.

Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. ACL,

2021.

Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi.
Aging with GRACE: lifelong model editing with discrete key-value adaptors. In NeurIPS, 2023.

Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei
Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding.
ArXiv, 2024.

Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina
Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing
millions of wikipedia entities. In CVPR, pp. 12065–12075, 2023.

Han Huang, Haitian Zhong, Tao Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. Vlkeb: A

large vision-language model knowledge editing benchmark. arxiv, 2024.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris
Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a
benchmark for question answering research. TACL, 7:453–466, 2019.

Jiaqi Li, Miaozeng Du, Chuanyi Zhang, Yongrui Chen, Nan Hu, Guilin Qi, Haiyun Jiang, Siyuan
Cheng, and Bozhong Tian. MIKE: A new benchmark for fine-grained multimodal entity knowledge
editing. In Findings of ACL, pp. 5018–5029, 2024.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image
pre-training with frozen image encoders and large language models. In ICML, pp. 19730–19742,
2023a.

Zichao Li, Ines Arous, Siva Reddy, and Jackie Chi Kit Cheung. Evaluating dependencies in fact
editing for language models: Specificity and implication awareness. In EMNLP, pp. 7623–7636,
2023b.

11

Under review as a conference paper at ICLR 2025

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction

tuning. In CVPR, pp. 26296–26306, 2024a.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36,

2024b.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual

associations in GPT. In NeurIPS, 2022.

Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing

memory in a transformer. In Findings of ACL, 2024.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model

editing at scale. ICLR, 2022a.

Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-

based model editing at scale. In ICML, pp. 15817–15831, 2022b.

Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo’llava: Your

personalized language and vision assistant. ArXiv, 2024.

Yasumasa Onoe, Michael JQ Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can lms
learn new entities from descriptions? challenges in propagating injected knowledge. ACL, 2023.

Suchen Wang, Kim-Hui Yap, Henghui Ding, Jiyan Wu, Junsong Yuan, and Yap-Peng Tan. Discovering
human interactions with large-vocabulary objects via query and multi-scale detection. In ICCV, pp.
13475–13484, 2021.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and
Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.
ArXiv, 2022.

Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, and Mingming Sun. Eva-kellm: A new benchmark

for evaluating knowledge editing of llms. ArXiv, 2023a.

Yinan Wu, Xiaowei Wu, Junwen Li, Yue Zhang, Haofen Wang, Wen Du, Zhidong He, Jingping Liu,
and Tong Ruan. Mmpedia: A large-scale multi-modal knowledge graph. In International Semantic
Web Conference, pp. 18–37. Springer, 2023b.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li,
Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. ArXiv, 2024.

Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and
Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. Findings of
EMNLP, 2023.

Junzhe Zhang, Huixuan Zhang, Xunjian Yin, Baizhou Huang, Xu Zhang, Xinyu Hu, and Xiaojun
Wan. MC-MKE: A fine-grained multimodal knowledge editing benchmark emphasizing modality
consistency. Arxiv, 2024.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min,
Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. ArXiv, 2023.

Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can

we edit factual knowledge by in-context learning? EMNLP, 2023.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing

vision-language understanding with advanced large language models. Arxiv, 2023.

12

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Under review as a conference paper at ICLR 2025

Table 7: The image source of visual semantic knowledge in MMKE-Bench.

Type

Source

Emotion

Traffic Sign

Human Action
Life Gesture

Referee Gesture
Traffic Cop Sign

Crawling from google
Crawling from google
LFW-emotion dataset
https://huggingface.co/datasets/TrainingDataPro/facial-emotion-recognition-dataset
Demo videos from Youtube and Bilibili
Crawling from google
TSRD dataset
https://nlpr.ia.ac.cn/PAL/TRAFFICDATA/recognition.html
DTD dataset (Cimpoi et al., 2014)
Texture
Crawling from google
Color
Shape
Crawling from google
Animal Body Language Crawling from google
Relationship
Social action
Layout

Siwg-HOI (Wang et al., 2021) and
Crawling from google
Crawling from google
Wiki-art dataset (Dataset, 2022)
https://huggingface.co/datasets/keremberke/painting-style-classification

Art Style

A BENCHMARK CONSTRUCTION

A.1 ORIGINAL KNOWLEDGE COLLECTION

In our process of gathering original knowledge, we begin by listing candidate fine-grained entities,
visual semantics, or user-specific items, and subsequently collect their corresponding images.

For visual entity editing, we source candidates from two datasets: The multimodal knowledge graph,
MMpedia (Wu et al., 2023b), and the visual entity recognition dataset, OVEN (Hu et al., 2023). Given
the extensive size of MMpedia, we filter entities with Wikipedia summaries of fewer than 40 words
and eliminate candidates that cannot uniquely identify the main entity through images. Using the
Wikipedia API, we retrieve the entity type and select the most popular 10% within each type. We
further apply optical character recognition (OCR) to exclude images containing entity names, such as
university logos. After this, we gather images from the relevant datasets and manually remove any
noisy images, or crawl additional images from Google for entities with fewer than two images. The
same process is applied to the OVEN dataset, except without sampling.

For visual semantic editing, we first list the semantic candidates from four broad categories: single-
person behavior, single-object behavior or attributes, object relationship, and global structure. The
single-person behavior includes human action, life gestures, referee gestures, traffic cop signs, and
emotion. The single-object behavior or attribute covers animal body language, traffic signs, color,
shape, and texture. The object relationship involves human-object interactive relationship and social
actions, while global structure encompasses layout and art style. Where datasets exist, such as for
texture, we gather the entities and images from existing sources. Otherwise, we manually curate the
candidates using domain expertise and collect images from various sources. The sources for each
type are listed in Tab.7. Specifically, images for human action, life gestures, traffic cop signs, color,
shape, social action, animal body language, and layout are crawling from Google. Images for traffic
signs, textures, relationships, emotions, and art styles come from existing datasets. Referee gesture
images are collected by extracting frames from demo videos on YouTube and Bilibili.

To sum up, this benchmark covers a total of 2,940 pieces of knowledge, along with 7,229 images
from 141 fine-grained types, and detailed type names are shown in Tab.8.

As for user-specific editing, we consider nine types of personal information, including items, pets,
actors, singers, cartoon characters, organizations, universities, sports clubs, and companies. The
candidate relationships between users and these objects are outlined in Tab.9, including examples
like ”employed at,” ”exchanged at,” ”studied at,” and ”favorite” for universities. We collect images
for these items from various sources. For items and pets, candidates and images are sourced from
existing datasets used for personalized large multimodal research (Nguyen et al., 2024; Alaluf et al.,
2024). For organizations, universities, sports clubs, and companies, we follow the same process as in
visual entity editing, using data from MMpedia. For actors, singers, and cartoon characters, images
are collected from Google.

After collecting the images, we generate natural language descriptions for each entity, visual semantic,
and user-specific item. For visual entities, we retrieve descriptions from the Wikipedia summary, and

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

Table 8: The data type in MMKE-Bench.

Broad Categories
Person
Aerial Animals

Marine Animals
Terrestrial Animals

Virtual Character

Plant

Building

Musical Group
Vehicle
Others
Human Action

Life Gesture
Emotion
Referee Gesture

Traffic Cop Sign
Traffic Sign

Types
Human
Bird, Dragonfly, Fly, Butterfly, Grasshopper,
Wasp, Insect
Jellyfish, Turtle, Sea Star, Fish, Crab, Sea Lion
Bear, Monkey, Amphibian, Mammal, Wild
Boar, Rodent, Squirrel, Dog Breed, Fox, Wolf,
Tick, Rabbit, Rhinoceros, Arthropod, Animal,
Salamander, Spider, Mollusc, Crustacean, Bee-
tle, Toad, Cat Breed, Deer, Sloth, Frog, Mol-
lusk, Snail, Hedgehog, Cat, Leopard, Milli-
pede, Pangolin, Dog, Cattle, Moth, Snake,
Lizard, Antelope
Anime Character, Animated Character,
Comics Character
Fruit, Tree, Flower, Mushroom, Orchid, Fun-
gus, Vegetable, Plant
Building, Church Building, Monument, Sculp-
ture, Tower, Statue
Musical Group
Car, Aircraft Model, Aircraft, Vehicle
Instrument, Ball
Body Posture Adjustments, Head Adjustments,
Hand Actions, Leg Actions, Whole-Body Ac-
tions, Eye Expressions, Facial Expressions,
Water Sports, Sound Actions, Object Actions
Life Gesture, Life Gesture Number
Emotion Sign
Soccer Linesman, Soccer, Basketball, Bad-
minton, Table Tennis, Volleyball, Volleyball
Card, Baseball, Puck, Fencing, Handball
Traffic Cop Sign
Traffic Sign Forbidden, Traffic Sign Allow,
Traffic Sign Point
Texture
Color

Texture
Color
Animal Body Language Monkey Body Language, Dog Body Lan-

Shape

Social Action
Art Style
Layout
Relationship
Item
Actor
Singer
Cartoon Character
Organization
University
Sports Club

Pet
Company

guage, Cat Body Language
Circular Shapes, Triangles, Special Plane
Shapes, Common Polyhedrons, Solids of Rev-
olution, Special Shapes
Social Action
Art Style
Layout
Relationship
Cup, Toy Puppet, Statue, Toy, Plush Doll
Actor
Singer
Cartoon Character
Nonprofit Organization, Organization
University
Baseball Team, Basketball Team, Sports Club,
Sports Team, Association Football Team,
Canadian Football Club, Futsal Team, Field
Hockey Club
Pet dog, Pet cat
Airline, Enterprise, Company

14

Visual Entity
Editing

Visual Semantic
Editing

User-Specific
Editing

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Table 9: The relationship between humans and the objects and data source of user-specific data in
MMKE-Bench.

Types

Relationship

Image Source

Company
Organization
University
Club
Cartoon character
Actor
Singer
Pet
Item

Employed at, Interned at, collaborated with, Favorite
MMpedia
MMpedia
Employed at, Interned at, Helped by, Favorite
Employed at, Exchanged at, Studied at, Traveled to, Favorite MMpedia
MMpedia
Employed at, Visited, Favorite
Crawling from Google
Favorite
Crawling from Google
Favorite, Admire most
Crawling from Google
Favorite, Admire most
MyVLM (Alaluf et al., 2024) and YoLLaVA (Nguyen et al., 2024)
Owned
MyVLM (Alaluf et al., 2024) and YoLLaVA (Nguyen et al., 2024)
Owned

if the summary is too lengthy, we use a large language model (LLM) to condense it to fewer than
100 words. For visual semantic editing, the description includes both a language description of the
action and an explanation of its meaning or rule. These are gathered either from relevant domain
knowledge by ourselves or generated with the help of an LLM. For user-specific editing, we select
one relationship from the candidate list and use an LLM to craft a personalized description of the
user’s personal information.

A.2 EDITING KNOWLEDGE GENERATION

After collecting the original knowledge, we perform counterfactual editing to generate alternative
knowledge for both visual entity and visual semantic editing. To achieve this, we prompt a large
language model (LLM) with in-context examples. For visual entity editing, we modify key details,
such as nationality, alma mater, and occupation of a person, into counterfactual variations. For visual
semantic knowledge, we alter the rules or meanings, such as the location where a free kick is taken,
into counterfactual scenarios. The specific prompt used is shown in Tab.8.

In addition to text-based editing, we also perform image modality editing by replacing the image of
an entity or action with one from another entity or action of the same type. This replacement strategy
is consistent with existing benchmarks (Huang et al., 2024).

A.3 EVALUATION QUESTION GENERATION

When generating evaluation questions, we adhere to four key principles: reliability, locality, gen-
eralization, and portability. For locality questions, we source them from existing benchmarks. For
reliability, we generate questions by prompting a large language model (LLM) with in-context exam-
ples, ensuring that each question is related to one of the edited contents. In image reliability, we refer
to the main object in the image using its type, such as “the person in the image.” For portability, during
visual entity editing, we follow previous benchmarks by providing additional information about the
edited content to ensure text portability. In visual semantic editing and user-specific editing, we
focus on image portability by combining the current object’s image with another object of the same
type. We then create a final one-hop question by merging the counterfactual content-related question
with an easier, image-based question, such as asking about the color of shoes. After generating the
questions and answers, we conduct a human review to verify the accuracy, rewriting any incorrect
questions or answers. The prompts used for question generation are shown in Tab.9 and Tab.14.

B EXPERIMENTS

We conduct experiments using the VLKEB library2, which employs PyTorch and integrates several
knowledge editing methods and large multimodal models. The experiments are performed on NVIDIA
A100/A800 80GB GPUs. The knowledge editing methods, and large multimodal models adopted in
this study are listed below, with their hyper-parameters detailed in Tab.10, Tab.11, and Tab.12.

MLLMs. To evaluate our benchmark, we conduct experiments on three representative MLLMs.

2https://github.com/VLKEB/VLKEB

15

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Figure 7: Evaluation comparison of IKE for BLIP2 with existing benchmarks. I-Gen and Port for
MMEdit, along with Port for MIKE, is set 1, as they ignore the relevant criteria.

• BLIP-2 (Li et al., 2023a): BLIP2 effectively leverages both frozen pre-trained image
models and language models by bootstrapping vision-language pre-training, and bridges the
modality gap with a lightweight Querying Transformer. We follow previous work (Huang
et al., 2024; Cheng et al., 2023), and select BLIP-2 OPT as the basic edit model, where the
vision model is ViT-L and the LLM is OPT model.

• MiniGPT-4 (Bai et al., 2023): MiniGPT-4 aligns a frozen visual encoder module with a
frozen advanced LLM using one projection layer. The LLM is Vicuna and the vision model
is ViT.

• LLaVA-1.5 (Liu et al., 2024b): LLaVA-1.5 is an improved version of LLaVA, which is an
end-to-end trained large multimodal model that connects a vision encoder and an LLM with
an MLP projector for visual and language understanding. We select LLaVA-1.5 7B as the
base model where CLIP-ViT-L-336px is the vision model and Vicuna-7B is the LLM.

Editing Methods. Following the previous benchmarks (Huang et al., 2024), we select five repre-
sentative multimodal knowledge editing methods to conduct experiments.

• Fine-tuning (FT): Fine-tuning has become a widely used strategy for adapting pre-train
models to specific tasks. We focus on finetuning two parts: the LLM and the vision-language
alignment module, where only the last layer of the LLM is fine-tuned.

• Knowledge Editor (KE) (De Cao et al., 2021): KE is a method that can be used to edit this
knowledge in the base model without the need for expensive retraining or fine-tuning. It
uses a hyper-network with constrained optimization to predict the weight update at test time.
• MEND (Mitchell et al., 2022a): MEND makes fast, local edits to a pre-trained model’s
behavior using a single desired input-output pair. It learns to transform the gradient of
standard fine-tuning, using a low-rank decomposition of the gradient.

• SERAC (Mitchell et al., 2022b): SERAC is a memory-based method and it stores edits in
explicit memory. It also introduces a scope classifier and counterfactual model, where the
scope classifier is to determine whether the memory contains inputs relevant to processing
them. If determined, the input is combined with the most relevant cache item into the
counterfactual model for prediction.

• In-context Knowledge Editing (IKE) (Zheng et al., 2023): IKE is inspired by in-context
learning, and a new demonstration formatting and organization strategies are to construct
suitable in-context learning demonstrations for guiding knowledge editing.

C MORE RESULTS

Comparison of evaluation results with existing benchmarks for BLIP2 The Comparison of
evaluation results with existing benchmarks of IKE for BLIP2 is shown in Fig. 7. As we can see, IKE
achieves high results in existing benchmarks, while it performs worse in our benchmark, indicating
the proposed benchmark is more challenging.

Results of sequential editing for BLIP-2 We additionally report the results of sequential editing
for BLIP-2 on MMKE-Bench, as shown in Tab.13. As we can see, FT-LLM and FT-Alignment tend
to forget previous knowledge while SERAC is better at keeping edited knowledge.

16

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Table 10: The hyper-parameters of knowledge editing methods and LMMs on the visual entity
editing.

FT-LLM

Models

Steps

BLIP2-OPT 30
40
MiniGPT-4
40
LLaVA-1.5

Edit Layer
31st layer of Transformer Module
31st layer of Transformer Module
31st layer of Transformer Module

Optimizer Edit LR

AdamW
AdamW
AdamW

2e − 4
1e − 4
1e − 4

FT-Alignment

Models

Steps

Edit Layer

BLIP2-OPT 30
30
MiniGPT-4
30
LLaVA-1.5

Qformer
Qformer
mm projector

MEND

Optimizer Edit LR

AdamW
AdamW
AdamW

2e − 4
1e − 4
1e − 4

Models

MaxIter Edit Layer

Optimizer LR

BLIP2-OPT 10,000
30,000
MiniGPT-4
10,000
LLaVA-1.5

layer 29, 30, 31 of Transformer Module Adam
layer 29, 30, 31 of Transformer Module Adam
layer 29, 30, 31 of Transformer Module Adam

1e − 6
1e − 6
1e − 6

SERAC

Models

MaxIter Edit Layer

BLIP2-OPT 10,000
20,000
MiniGPT-4
10,000
LLaVA-1.5

all layers of OPT-125M
31st layer of Vicuna-7B
31st layer of Vicuna-7B-v1.5

Optimizer LR

Adam
Adam
Adam

1e − 5
5e − 5
1e − 5

KE

Models

MaxIter Edit Layer

Optimizer LR

BLIP2-OPT 10,000
10,000
MiniGPT-4
10,000
LLaVA-1.5

layer 29, 30, 31 of Transformer Module RMSprop
layer 29, 30, 31 of Transformer Module RMSprop
layer 29, 30, 31 of Transformer Module RMSprop

3e − 4
3e − 4
3e − 4

17

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Table 11: The hyper-parameters of knowledge editing methods and LMMs on visual semantic editing.

FT-LLM

Models

Steps

BLIP2-OPT 30
40
MiniGPT-4
40
LLaVA-1.5

Edit Layer
31st layer of Transformer Module
31st layer of Transformer Module
31st layer of Transformer Module

Optimizer Edit LR

AdamW
AdamW
AdamW

2e − 4
1e − 4
1e − 4

FT-Alignment

Models

Steps

Edit Layer

BLIP2-OPT 30
30
MiniGPT-4
30
LLaVA-1.5

Qformer
Qformer
mm projector

MEND

Optimizer Edit LR

AdamW
AdamW
AdamW

2e − 4
1e − 4
1e − 4

Models

MaxIter Edit Layer

Optimizer LR

BLIP2-OPT 20,000
30,000
MiniGPT-4
20,000
LLaVA-1.5

layer 29, 30, 31 of Transformer Module Adam
layer 29, 30, 31 of Transformer Module Adam
layer 29, 30, 31 of Transformer Module Adam

1e − 6
1e − 6
1e − 6

SERAC

Models

MaxIter Edit Layer

BLIP2-OPT 20,000
20,000
MiniGPT-4
20,000
LLaVA-1.5

all layers of OPT-125M
31st layer of Vicuna-7B
31st layer of Vicuna-7B-v1.5

Optimizer LR

Adam
Adam
Adam

1e − 5
5e − 5
1e − 5

KE

Models

MaxIter Edit Layer

Optimizer LR

BLIP2-OPT 10,000
10,000
MiniGPT-4
10,000
LLaVA-1.5

layer 29, 30, 31 of Transformer Module RMSprop
layer 29, 30, 31 of Transformer Module RMSprop
layer 29, 30, 31 of Transformer Module RMSprop

3e − 4
3e − 4
3e − 4

18

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Table 12: The hyper-parameters of knowledge editing methods and LMMs on user-specific editing.

FT-LLM

Models

Steps

BLIP2-OPT 30
40
MiniGPT-4
40
LLaVA-1.5

Edit Layer
31st layer of Transformer Module
31st layer of Transformer Module
31st layer of Transformer Module

Optimizer Edit LR

AdamW
AdamW
AdamW

2e − 4
1e − 4
1e − 4

FT-Alignment

Models

Steps

Edit Layer

BLIP2-OPT 30
30
MiniGPT-4
20
LLaVA-1.5

Qformer
Qformer
mm projector

MEND

Optimizer Edit LR

AdamW
AdamW
AdamW

2e − 4
1e − 4
1e − 4

Models

MaxIter Edit Layer

Optimizer LR

BLIP2-OPT 10,000
30,000
MiniGPT-4
10,000
LLaVA-1.5

layer 29, 30, 31 of Transformer Module Adam
layer 29, 30, 31 of Transformer Module Adam
layer 29, 30, 31 of Transformer Module Adam

1e − 6
1e − 6
1e − 6

SERAC

Models

MaxIter Edit Layer

BLIP2-OPT 10,000
20,000
MiniGPT-4
10,000
LLaVA-1.5

all layers of OPT-125M
31st layer of Vicuna-7B
31st layer of Vicuna-7B-v1.5

Optimizer LR

Adam
Adam
Adam

1e − 5
5e − 5
1e − 5

KE

Models

MaxIter Edit Layer

Optimizer LR

BLIP2-OPT 10,000
10,000
MiniGPT-4
10,000
LLaVA-1.5

layer 29, 30, 31 of Transformer Module RMSprop
layer 29, 30, 31 of Transformer Module RMSprop
layer 29, 30, 31 of Transformer Module RMSprop

3e − 4
3e − 4
3e − 4

19

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Table 13: The results of sequential editing for BLIP2 on MMKE-Bench.

Method

Gap / User Num T-Loc

I-Loc T-Rel

I-Rel

I-Gen

Port

FT-LLM

Visual Entity
Editing

FT-Alignment

SERAC

FT-LLM

Visual Semantic
Editing

FT-Alignment

SERAC

FT-LLM

User-Specific
Editing

FT-Alignment

SERAC

-
3
6
10

-
3
6
10

-
3
6
10

-
3
6
10

-
3
6
10

-
3
6
10

-
1
3
5

-
1
3
5

-
1
3
5

68.83
32.42
31.26
31.59

100.00
100.00
100.00
100.00

99.97
99.97
99.97
99.97

64.75
25.92
25.42
24.35

100.00
100.00
100.00
100.00

100.00
100.00
100.00
100.00

63.18
47.51
46.51
45.74

100.00
100.00
100.00
100.00

99.97
99.94
99.92
99.90

20.2
5.33
5.13
5.03

8.74
3.51
3.52
3.62

64.34
55.92
55.93
55.91

20.13
5.07
4.98
4.64

9.7
4.15
4.17
4.09

77.42
77.5
77.47
77.62

21.19
10.29
10.51
10.60

8.83
16.14
18.82
18.26

93.4
93.73
93.71
93.64

29.13
28.12
26.20
25.03

19.67
19.67
19.67
19.67

19.67
19.67
19.67
19.67

32.08
27.56
25.21
23.57

15.97
15.97
15.97
15.97

16.22
15.97
15.97
15.97

13.10
10.65
10.10
9.45

7.81
8.31
8.31
8.31

7.81
8.31
8.31
8.31

29.47
24.14
22.60
22.41

23.53
15.88
16.84
15.95

23.30
19.47
19.53
19.71

31.40
25.76
24.53
22.05

31.73
11.42
12.01
10.46

17.77
12.37
12.58
12.22

27.00
17.05
14.32
13.68

18.15
6.79
6.90
7.93

15.18
14.89
14.89
14.89

29.83
24.54
23.89
22.65

22.47
15.89
16.86
15.94

23.21
19.6
19.63
19.74

31.90
25.29
23.31
21.03

28.27
11.42
12.33
10.46

19.77
12.82
13.00
12.82

27.14
17.09
13.90
13.53

17.8
6.59
6.37
8.08

15.53
14.90
14.90
14.90

22.60
21.61
22.18
20.97

17.36
14.71
15.32
16.19

15.1
14.54
14.28
14.43

2.88
1.08
0.96
1.63

4.54
4.15
3.13
4.09

3.79
3.79
3.79
3.79

4.83
0.70
0.54
0.84

6.19
0.75
1.17
2.23

4.91
4.16
4.16
4.16

20

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Figure 8: Prompt for editing knowledge.

21

You are a powerful description editor. Users have an entity, the entity type, and the entity description consists of some different aspects. You need to edit the description of an aspect into a counterfactual description by editing some key points in the aspect description.Rule 1: It is better to edit key entity nouns in the description, and at least 4 entities must be edited, such as the working company, place of birth, related person, and so on.Rule 2: You are not allowed to edit object properties such as color and shape.Rule 3: The edited description should be consistent across aspects. For example, if a competition is changed from one year to two years, then the winner of the championship should also be held every two years.Rule 4: You need to follow the same output format as the given example.Example User:Input:Entity: MicrosoftEntity type: companyDescription: Microsoft is an American multinational corporation and technology company headquartered in Washington. Its best-known software products are the Windows line of operating systems, the Microsoft 365 suite of productivity applications, the Azure cloud computing platform, and the Edge web browser. Its flagship hardware products are the Xbox video game consoles and the Microsoft Surface lineup of touchscreen personal computers. It is considered one of the Big Five American information technology companies.Output:Example Assistant:Edit description: Microsoft is an American multinational corporation and technology company headquartered in Chicago. Its best-known software products are the Linux line of operating systems, the Microsoft 365 suite of productivity applications, the Azure cloud computing platform and the Chrome browser. Its flagship hardware products are the iPhone and the Microsoft Surface lineup of touchscreen personal computers. It is considered one of the Big Five American information technology companies.Highlight: Chicago; Linux; Chrome browser; Iphone;Entity: Jorunna parvaEntity type: mollusc Description: Jorunna parva, commonly known as the sea bunny, is a species of dorid nudibranch, a shell-less marine gastropod mollusc in the family Discodorididae. Its black-and-white rhinophores somewhat resemble a rabbit's ears. The species was first described by Kikutaro Baba. Its resemblance to a rabbit facilitated a surge in popularity on Twitter throughout Japan in 2015.Output:Under review as a conference paper at ICLR 2025

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Figure 9: Prompt for editing generating reliability question.

22

You are a powerful question generator. Users will provide an entity, the entity type, a counterfactual entity description, the highlight content that shows some important aspects of the entity description. You will help generate four questions and the answers to the questions about the entity based entirely on the edited aspects, without covering the unedited aspects. Each entity is a visual entity, i.e., there are some images corresponding to the entity. Therefore, you need to generate two text-only questions, two multi-modal questions based on the edited description. In the multi-modal questions, you use '{entity type} in the image' to refer to the entity, where {entity type} must be replaced with the entity type. Before that, you need to select a noun entity from the highlight. For these questions, you need to generate the question based on the given entity description with the given entity as the head entity and the answer of the question to be exactly the selected entity in highlight. Rule 1: You must use '{entity type} in the image' to refer to entity, and {entity type} must be replaced with the given entity type in the Multi-modal question. Rule 2: The entity name is not allowed to appear in Multi-modal question. Rule 3: You need to follow the same output format as the given example.Rule 4: The generated questions must have a unique answer.Rule 5: The answer of all the generated questions must be the selected entity in highlight.Rule 6: The answer of the generated question must be one or two words.Example User:Input:Entity: MicrosoftEntity type: companyDescription: Microsoft is an American multinational corporation and technology company headquartered in Chicago. Its best-known software products are the Linux line of operating systems, the Microsoft 365 suite of productivity applications, the Azure cloud computing platform and the Chrome browser. Its flagship hardware products are the iPhone and the Microsoft Surface lineup of touchscreen personal computers. It is considered one of the Big Five American information technology companies.Highlight: Chicago; Linux; Chrome browser; Iphone;Output:Example Assistant:Text-only question 1: What is the well-known browser of Mircosoft?Answer: ChromeMulti-modal question 1: What are the flagship hardware products of the company in the picture?Answer: iPhone and the Microsoft Surface lineup of touchscreen personal computers.Input:Entity: Jorunna parvaEntity type: mollusc Description: Jorunna parva, commonly known as the sea bunny, is a species of dorid nudibranch, a shell-less marine gastropod mollusc in the family Discodorididae. Its red rhinophores somewhat resemble a rabbit's ears. The species was first described by Hiroshi Akiyama. Its resemblance to a rabbit facilitated a surge in popularity on Instagram throughout Japan in 2015.Highlight: red; Hiroshi Akiyama; Instagram;Output:Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Table 14: The results of Visual Semantic Sequential Editing for LLaVA-1.5 on MMKE-Bench.

Method

GAP T-Loc

I-Loc T-Rel

I-Rel

I-Gen

Port

FT-LLM

Visual Semantic
Editing

FT-Alignment

SERAC

-
3
6
10
40
60
80

-
3
6
10
40
60
80

-
3
6
10
40
60
80

76.89
50.33
49.09
48.23
45.40
43.88
42.99

100.00
100.00
100.00
100.00
100.00
100.00
100.00

100.00
99.93
99.93
99.93
99.93
99.93
99.96

16.14
7.36
7.25
7.02
6.23
5.82
5.58

19.41
1.44
1.38
1.38
1.22
1.17
0.94

34.53
13.56
13.54
13.52
13.37
13.35
13.32

49.00
42.86
41.49
41.51
36.83
36.01
33.67

27.83
28
27.83
27.83
27.83
27.83
27.83

27.83
27.99
27.92
27.88
27.92
27.92
27.92

49.44
46.73
45.58
45.09
41.85
39.18
38.27

44.5
34.06
31.62
29.79
25.4
26.12
27.31

41.09
29.71
29.91
29.93
28.23
28.45
28.20

49.04
45.02
43.52
42.08
40.53
38.69
36.79

35.37
24.57
23.54
23.92
21.63
22.11
23.81

41.82
30.70
31.09
31.13
29.23
29.41
28.41

10.67
8.29
7.25
7.63
7.83
7.04
6.83

15.00
6.51
6.96
7.25
8.58
8.08
6.75

11.29
11.17
11.34
11.23
11.25
11.25
11.25

23

Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Figure 10: Prompt for generating portability question.

24

You are a powerful question generator. Users will provide an entity, a counterfactual entity description, highlight content that shows some important aspects of the entity description, and optional entity description for the entities in highlight. \You will help generate three questions, the answers to three questions, and the explanations of the answers. Before that, you need to select a noun entity from the highlight. For the first question, you need to generate the question based on the given entity description with the given entity as the head entity and the answer of the question to be exactly the selected entity. \For the second question, you need to ask the information about the selected entity. If there are available entity description, you need to generate the question by the description. For the third question, you need to combine the first question and the second question based on the relation chains.Rule 1: You need to follow the same output format as the following given example.Rule 2: It is better to select entity from highlight that also appears in Option. The selected entity from the highlight muse be a single noun entity and could not contain the word 'and' and comma. Avoid selecting entities like time, number, and so on.Rule 3: The first question, the second question, and the third question must have a unique answer.Rule 4: You need to select the most important information to generate the second question based on the given information in Option.Rule 5: The selected entity from highlight must be the answer of the first question and the answer of third questiom must be the same as the answer of the second question.Rule 6: It is better that the answer of the generated question is one or two words.Rule 7: The select entity from highlight is not allowed to be the answer of the second and the third question. Example User:Input:Entity: MicrosoftDescription: Microsoft is a Chinese multinational corporation and technology company headquartered in Washington. Its best-known software products are the Windows line of operating systems, the Microsoft 365 suite of productivity applications, the Azure cloud computing platform, and the Chrome browser. Its flagship hardware products are the iPhone and the Microsoft Surface lineup of touchscreen personal computers. It is considered one of the Big Five American information technology companies.Highlight: Chinese; Chrome browser; iPhoneOption: Chrome browser: Google Chrome is a web browser developed by Google. It was first released in 2008 for Microsoft Windows, built with free software components from Apple WebKit and Mozilla Firefox. Versions were later released for Linux, macOS, iOS, and also for Android, where it is the default browser.iPhone: The iPhone is a smartphone produced by Apple that uses Apple's own iOS mobile operating system. The first-generation iPhone was announced by then Apple CEO Steve Jobs on January 9, 2007. Since then, Apple has annually released new iPhone models and iOS updates.Output:Example Assistant:Selcted entity: Chrome browserThe first question: What is the well-known browser of Microsoft?Answer: Chrome browser.The second question: In which year is Chrome browser first released?Answer:2008.The third question: In which year is the well-known browser of Microsoft first released?Answer: 2008.Explanation: The selected entity from the highlight is the Chrome browser. The first question is 'What is the well-known browser of Microsoft?', and the answer is Chrome browser. The second question is 'In which year is Chrome browser first published?', and the answer is 2008.Input:Entity: Jorunna parvaDescription: Jorunna parva, commonly known as the sea bunny, is a species of dorid nudibranch, a shell-less marine gastropod mollusc in the family Discodorididae. The species was first described by Kazuri Takahashi. Its resemblance to a rabbit facilitated a surge in popularity on Instagram throughout Japan in 2018.Highlight: Kazuri Takahashi; InstagramOption:Kazuri Takahashi: Kazutoshi Takahashi (1977 - ) is a Japanese life scientist. He is a lecturer at the iPS Cell Research Institute of Kyoto University. He received his Ph.D. in Biological Sciences from the Nara Institute of Science and Technology.Instagram: Instagram[a] is a photo and video sharing social networking service owned by Meta Platforms. It allows users to upload media that can be edited with filters, be organized by hashtags, and be associated with a location via geographical tagging. Posts can be shared publicly or with preapproved followers. Users can browse other users' content by tags and locations, view trending content, like photos, and follow other users to add their content to a personal feed. Under review as a conference paper at ICLR 2025

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Figure 11: In Fig.11 (a), the single editing takes one edit at a time and evaluates immediately, while
in Fig.11 (b) and (c) the sequential editing involves continuous edits and tests after several other edits.

Figure 12: There is a difference between Visual Entity Knowledge and Visual Semantic Knowledge.
Visual Entity Knowledge focuses on entity objects, such as people, things, etc. Visual Semantic
Knowledge focuses on the knowledge abstracted from images, such as gestures, traffic signs, facial
expressions, etc. For example, for Visual Entity Knowledge, in Figure 12 (a), the training knowledge
needs a reference to the entity, such as ”Donald John Trump”, focusing on the information of the
entity object; However, in (b) of Figure 12, for Visual Semantic Knowledge, entity reference, such as
”The man”, is not needed, but the gesture of the person in the image is emphasized.

25

Under review as a conference paper at ICLR 2025

Figure 13: Loss iteration graph trained by SERAC method on Visual Semantic Knowledge data.
Through the analysis of images, we can find that the SERAC method can normally achieve the
convergence of loss on this data amount, and the loss value will approach 0 at last.

Figure 14: Loss iteration graph trained by MEND method on Visual Semantic Knowledge data.
Through the analysis of images, we can find that the MEND method can normally achieve the
convergence of loss on this data amount, and the loss value will approach 0 at last.

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Under review as a conference paper at ICLR 2025

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Figure 15: Data Example-1 of Visual Entity
Editing in MMKE-Bench.

Figure 16: Data Example-2 of Visual Entity
Editing in MMKE-Bench.

Figure 17: Data Example-3 of Visual Entity
Editing in MMKE-Bench.

Figure 18: Data Example-4 of Visual Entity
Editing in MMKE-Bench.

27

Under review as a conference paper at ICLR 2025

Figure 19: Data Example-1 of Visual Semantic
Editing in MMKE-Bench.

Figure 20: Data Example-2 of Visual Semantic
Editing in MMKE-Bench.

Figure 21: Data Example-3 of Visual Semantic
Editing in MMKE-Bench.

Figure 22: Data Example-4 of Visual Semantic
Editing in MMKE-Bench.

28

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Under review as a conference paper at ICLR 2025

Figure 23: Data Example-1 of User-Specific
Editing in MMKE-Bench.

Figure 24: Data Example-2 of User-Specific
Editing in MMKE-Bench.

Figure 25: Data Example-3 of User-Specific
Editing in MMKE-Bench.

Figure 26: Data Example-4 of User-Specific
Editing in MMKE-Bench.

29

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

Under review as a conference paper at ICLR 2025

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Figure 27: Case Study on Visual Entity Editing
Example-1 in MMKE-Bench.

Figure 28: Case Study on Visual Entity Editing
Example-2 in MMKE-Bench.

30

Under review as a conference paper at ICLR 2025

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Figure 29: Case Study on Visual Entity Editing
Example-3 in MMKE-Bench.

Figure 30: Case Study on Visual Entity Editing
Example-4 in MMKE-Bench.

Figure 31: Case Study on Visual Semantic
Editing Example-1 in MMKE-Bench.

Figure 32: Case Study on Visual Semantic
Editing Example-2 in MMKE-Bench.

31

Under review as a conference paper at ICLR 2025

Figure 33: Case Study on Visual Semantic
Editing Example-3 in MMKE-Bench.

Figure 34: Case Study on Visual Semantic
Editing Example-4 in MMKE-Bench.

Figure 35: Case Study on User-Specific Edit-
ing Example-1 in MMKE-Bench.

Figure 36: Case Study on User-Specific Edit-
ing Example-2 in MMKE-Bench.

32

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

Under review as a conference paper at ICLR 2025

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

Figure 37: Case Study on User-Specific Edit-
ing Example-3 in MMKE-Bench.

Figure 38: Case Study on User-Specific Edit-
ing Example-4 in MMKE-Bench.

Figure 39: Case Study of Reliability Example-
1 of Visual Entity Editing in MMKE-Bench.
The texts in brown indicate the same content
as the editing knowledge.

r

33

Figure 40: Case Study of Reliability Example-
2 of Visual Entity Editing in MMKE-Bench.
The texts in brown indicate the same content
as the editing knowledge.

Under review as a conference paper at ICLR 2025

Figure 41: Case Study of Reliability Example-
1 of Visual Semantic Editing in MMKE-Bench.
The texts in brown indicate the same content
as the editing knowledge.

Figure 42: Case Study of Reliability Example-
2 of Visual Semantic Editing in MMKE-Bench.
The texts in brown indicate the same content
as the editing knowledge.

Figure 43: Case Study of Reliability Example-
1 of User-Specific Editing in MMKE-Bench.
The texts in brown indicate the same content
as the editing knowledge.

Figure 44: Case Study of Reliability Example-
2 of User-Specific Editing in MMKE-Bench.
The texts in brown indicate the same content
as the editing knowledge.

34

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

