# Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large
Language Models

Jiaqi Li1,3, Qianshan Wei1,311footnotemark: 1, Chuanyi Zhang2, Guilin Qi3,4,
Miaozeng Du3,4, Yongrui Chen3,4, Sheng Bi3,4
1 School of Cyber Science and Engineering, Southeast University, Nanjing,
China
2 College of Artificial Intelligence and Automation, Hohai University,
Nanjing, China
3 Key Laboratory of New Generation Artificial Intelligence Technology and Its
Interdisciplinary Applications (Southeast University), Ministry of Education,
China
4School of Computer Science and Engineering, Southeast University, Nanjing,
China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
miaozengdu@seu.edu.cn,yrchen<EMAIL_ADDRESS>J. Li and Q. Wei
contributed equally to this work and should be considered co-first authors.
Corresponding author.

###### Abstract

Machine unlearning (MU) empowers individuals with the ‘right to be forgotten’
by removing their private or sensitive information encoded in machine learning
models. However, it remains uncertain whether MU can be effectively applied to
Multimodal Large Language Models (MLLMs), particularly in scenarios of
forgetting the leaked visual data of concepts. To overcome the challenge, we
propose an efficient method, Single Image Unlearning (SIU), to unlearn the
visual recognition of a concept by fine-tuning a single associated image for
few steps. SIU consists of two key aspects: (i) Constructing Multifaceted
fine-tuning data. We introduce four targets, based on which we construct fine-
tuning data for the concepts to be forgotten; (ii) Joint training loss. To
synchronously forget the visual recognition of concepts and preserve the
utility of MLLMs, we fine-tune MLLMs through a novel Dual Masked KL-divergence
Loss combined with Cross Entropy loss. Alongside our method, we establish
MMUBench, a new benchmark for MU in MLLMs and introduce a collection of
metrics for its evaluation. Experimental results on MMUBench show that SIU
completely surpasses the performance of existing methods. Furthermore, we
surprisingly find that SIU can avoid invasive membership inference attacks and
jailbreak attacks. To the best of our knowledge, we are the first to explore
MU in MLLMs. We will release the code and benchmark in the near future.

##  1 Introduction

Recent years have witnessed the great success of Large Language Models (LLMs)
33, 3 and Multimodal Large Language Models (MLLMs) 47, 49. They play dominant
roles in NLP 5, 37 and multimodal applications 50, 17 ascribed to the large-
scale pre-training data 2, 35, 29. Unfortunately, these data may contain
overlooked elements of personal privacy and copyright infringement, posing
potential risks of data leakage 32, 36. Retraining the models from scratch to
exclude the risky data is a waste of resource and practically untenable due to
the inaccessible pre-training data. To address the issue, prior works 12, 46,
45, 27, 31 have shown that approximate machine unlearning (MU) methods can
forget specific pieces of knowledge embedded within LLMs.

Nevertheless, it remains unclear if such strategies of knowledge forgetting
are transferable to MLLMs, especially for forgetting the visual recognition of
various concepts. The challenge of unlearning visual recognition in MLLMs is
formidable. A primary obstacle is limited training data. Recent work 12
utilizes a text of original book (2.1M tokens) combined with synthetic
sentences (1M tokens) as the forgetting dataset. To forget the character
_‘Harry Potter’_ , this work fine-tunes Llama-7b-chat-hf 41 on the entire
forgetting dataset for 3 epochs. However, in the real scenario of unlearning
the visual recognition of concepts, collecting sufficient images of targeted
concepts is challenging. The limited amount of training data poses a
significant barrier to unlearning all concept-wise visual knowledge encoded in
pre-trained MLLMs. Another challenge is model degradation 52, 19, which
pervasively exists in large generative models. Researchers 46 discover that
LLMs could stop generating harmful texts by employing Gradient Ascent (GA) on
forgetting datasets, thus reducing the need for synthetic data. However, GA
often results in meaningless outputs such as only a _whitespace_ or _repeated
tokens_ , which eliminate the utility of LLMs. To address this issue, several
studies 45, 46 combine GA with minimizing KL-divergence between unlearned and
original LLMs to preserve the utility of LLMs. Despite mitigating the
meaningless response problem, the method may output self-contradictory
answers, as if the concept is not unlearned. This issue may arise from a
conflict between objectives of GA and KL-divergence. GA aims to make LLMs
cease generating tokens of targeted unlearning concepts, whereas KL-divergence
seeks to align the output probability distribution of the unlearning model
with that of the original model. The distribution includes the probabilities
of generating tokens of targeted unlearning concepts, which are high in the
original model.

To address the challenges, we take the first step to explore MU in MLLMs and
propose an efficient method, Single Image Unlearning (SIU). SIU requires only
a single training image of the targeted concepts to enable MLLMs to forget the
visual recognition of these concepts. We first put forward four targets,
namely Aligning with Unseen Concepts, Assigning New Visual Description,
Decoupling Factual Knowledge and Preserving Non-targeted Knowledge. In
accordance with these four targets, we construct the fine-tuning data.
Moreover, we introduce an innovative Dual Masked KL-divergence (DMK) Loss to
be jointly trained with Cross Entropy Loss. Different from prior works, the
joint training loss is optimized by Gradient Descent. The DMK Loss
incorporates two levels of masking on fine-tuning data, which are Token-Level
Masking and Vocabulary-Level Masking. At the token-level, it masks tokens
contradicting original knowledge in the sentence to exclude them from KL loss
calculations. At the vocabulary-level, it specifically masks tokens of the
targeted unlearning concepts across the entire vocabulary during KL loss
computation.

Alongside our method we introduce MMUBench, a comprehensive benchmark designed
to assess MU within MLLMs. This benchmark includes a curated dataset with a
minimum of 50 images for each of 20 concepts. One image per concept is
designated for the forgetting training set, with the remainder serving to
assess generality. To provide a thorough evaluation of MU, we develop an
evaluation scheme including efficacy, generality, specificity, fluency and
diversity. Efficacy and generality assess the effectiveness of the unlearning
methods, while specificity, fluency and diversity evaluate the utility of
MLLMs post-unlearning. MMUBench includes the application of existing methods
as baselines, facilitating comparative analysis. The experimental results
reveal that our approach surpasses these methods in all evaluation metrics. We
observe that SIU could trigger positive butterfly effects, details of which
are discussed in the experimental sections. Furthermore, we conduct membership
inference attack and jailbreak attack 24, 34 experiments to examine the
robustness of unlearning methods.

We summarize main contributions as follows:

  * •

To the best of our knowledge, we are the pioneers in exploring unlearning the
visual recognition of concepts in MLLMs, extending machine unlearning to
multimodal settings.

  * •

We propose a new method, namely SIU, to efficiently forget the visual
recognition of concepts with only one training image. SIU incorporates
Multifaceted Fine-tuning Data and Dual Masked KL-divergence Loss, both of
which significantly enhance unlearning performance.

  * •

We establish MMUBench, a new benchmark to evaluate the efficacy, generality,
specificity, fluency and diversity of machine unlearning methods in MLLMs.

  * •

The experimental results on MMUBench demonstrate the superiority of our method
compared to existing methods. Furthermore, the ability to defend against
membership inference attacks and jailbreak attacks reveal the robustness of
our method.

##  2 Related Work

Machine Unlearning. In recent years, there has been a notable increase in
interest concerning machine unlearning (MU) problems. The primary works 13, 6,
8 mainly focused on MU in classification tasks. However, the research of MU in
LLMs is far from being developed. Different from classification task, MU in
LLMs 39, 51 should not only stop generating harmful or private texts, but also
remain the utility of LLMs. Yao et al. 46 employ Gradient Ascent (GA) method
to forget original harful output. Wang et al. 42 propose a method to align the
knowledge between the pre-trained model and fine-tuning model. Chen and Yang 7
introduce an efficient method to handle a deletion quest by introducing
lightweight unlearning layers. Yao et al. 45 combine GA with KL-divergence to
constrain the output probability distribution. Eldan and Russinovich 12
construct a dictionary of generic prediction to substitute the unlearning
target in fine-tuning data. In our paper, we further extend the MU setting to
MLLMs and propose a new method to efficiently forget the visual recognition of
concepts for MLLMs.

Multimodal Large Language Model. MLLMs are architected by integrating a
language model with a visual encoder, linked through an intermediary
connector. A pioneering method introduced by 1 employs a query-based cross-
attention mechanism, establishing an advanced and robust vision-language
interaction module. In contrast, BLIP-2 23 employs a Q-Former, which is a
streamlined Transformer model, in place of the typical cross-attention.
Enhancements in BLIP-2’s performance are achieved by MiniGPT-4 54 and
InstructBLIP 10, which both incorporate instruction tuning datasets collected
from a diverse range of public sources. To augment the models’ comprehension
capabilities, LLaVA, mPLUG-2 and Otter 26, 44, 21 have developed a system of
instructional data. Progressing beyond earlier training methodologies, a novel
three-stage training strategy 4 has been proposed to further refine multimodal
representations. Additionally, CogVLM 43 introduces a visual expert system to
elevate model performance.

##  3 Problem Definition

In our work, we mainly focus on unlearning the visual recognition of the
concepts (e.g., Recognize Donald Trump in an image) rather than forgetting the
factual knowledge (if have, e.g., Donald Trump is the former president) in
MLLMs. Formally, let $\mathcal{M}_{\theta}$ denote the original MLLM, where
$\theta$ is the parameters of original MLLM. $\mathcal{M}_{\theta}$ is trained
with a dataset that encompasses pairs of visual and textual data,
$\mathcal{D}=\\{(\mathcal{I}_{i},\mathcal{T}_{i})\\}_{i=1}^{N}$, where
$\mathcal{I}_{i}$ represents an image and $\mathcal{T}_{i}$ is a text
consisting of $t_{i}$ tokens
$\left\\{w_{1}^{i},w_{2}^{i},\ldots,w_{t_{i}}^{i}\right\\}$. We define the
forgetting set
$\mathcal{D}^{f}=\\{(\mathcal{I}^{\mathcal{C}}_{j},\mathcal{T}^{\mathcal{C}}_{j})\\}_{j=1}^{K}$
as a collection of $K$ image-text pairs associated with the visual recognition
of targeted unlearning concepts $\mathcal{C}$. Each
$\mathcal{I}^{\mathcal{C}}$ is an image depicting $\mathcal{C}$ and each
$\mathcal{T}^{\mathcal{C}}$ is the question-answer text about the image
content pointing to $\mathcal{C}$, where the answer reflects the forgetting of
$\mathcal{C}$. To facilitate the unlearning process and assess its impact, we
partition $\mathcal{D}^{f}$ into a training subset $\mathcal{D}^{f}_{train}$
and a testing subset $\mathcal{D}^{f}_{test}$. $\mathcal{D}^{f}_{train}$
contains a single image-text pair used to train the unlearned model, and
$\mathcal{D}^{f}_{test}$ contains the remainder of the pairs used to evaluate
the generality of unlearning.

We define the goal of MU in MLLMs as follows:

Machine unlearning in MLLMs aims to eliminate learned patterns associated with
visual recognition of specific "to-be-forgotten" concepts, while preserving
the MLLMs’ prediction capabilities on inputs unrelated to those eliminated
patterns.

By employing the negative log-likelihood of predicting the next token, the
training objective is to obtain an unlearned model
$\mathcal{M}_{\hat{\theta}}$ and can be formulated as follows:

$\displaystyle\arg\min_{\hat{\theta}}\Bigg{\\{}\mathbb{E}_{(\mathcal{I}_{j},\mathcal{T}_{j})\in\mathcal{D}^{f}}\Big{[}-{\sum_{t=1}^{t_{j}}}\log
P_{\mathcal{M}_{\hat{\theta}}}(w_{t}^{j}|\mathcal{I}_{j},w_{1}^{j},\ldots,w_{t-1}^{j})\Big{]}$
(1)
$\displaystyle+\mathbb{E}_{(\mathcal{I}_{i},\mathcal{T}_{i})\in\mathcal{D}\setminus\mathcal{D}^{f}}\Big{[}-{\sum_{t=1}^{t_{i}}}\log
P_{\mathcal{M}_{\hat{\theta}}}(w_{t}^{i}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i})\Big{]}\Bigg{\\}},\mathcal{T}=w_{1},\ldots,w_{t}.$

Figure 1: Overview of the Unlearning Process in MLLMs Using SIU. The process
starts with a user request to unlearn the visual recognition of concepts,
utilizing MMUBench (introduced in Section 5) to provide concepts for
unlearning. SIU has two elements which are Multifaceted Fine-tuning Data and
Dual Masked KL-divergence Loss. After unlearning, the unlearned MLLM is
evaluated for generality, specificity, diversity, fluency, and resistance to
membership inference and jailbreak attacks.

##  4 Methodology

In this section, we present our proposed method, namely SIU, for MU in MLLMs.
As shown in Figure 1, we take _Donald Trump_ as an example of $\mathcal{C}$.
SIU consists of two parts, Multifaceted Fine-tuning Data and Dual Masked KL-
divergence Loss. MMUBench will be introduced in Section 5.

###  4.1 Multifaceted Fine-tuning Data

As stated in Section 3, for each $\mathcal{C}$ we have a single image-text
pair as forgetting training subset $\mathcal{D}^{f}_{train}$. Based on
$\mathcal{D}^{f}_{train}$, we construct fine-tuning data centering on four
targets. The details of fine-tuning data are shown in Figure 7 and Appendix
A.3.

Aligning with Unseen Concepts. Different from classification models, where a
simple reassignment of label is sufficient 20, 8, MLLMs require a logical
continuity in their output. Our question here is, what kind of response is
reasonable? Is it enough for MLLMs to just answer ‘I don’t know’? 12, 31, 9

Our approach reinterprets the objective of MU, aiming to align the output
distribution of $\mathcal{M}_{\hat{\theta}}$ with that of
$\mathcal{M}_{\theta}$ under $\mathcal{D}^{f}$ when the visual representations
of $\mathcal{C}$ are not present during the pre-training phase. To find the
characteristics of output distribution, we conduct a set of tiny experiments
on 190 private images of people that surely have not appeared in the pre-
training phase of $\mathcal{M}_{\theta}$ (detailed in Appendix A.1). We
observe that $\mathcal{M}_{\theta}$ is unaware of concepts they have not seen
and tends to generate factually vague or incorrect responses such as _‘man’_ ,
_‘woman’_ or _‘John’_. We assume though an incorrect response might be a
hallucination, it actually achieves the purpose of unlearning. Moreover, in MU
of classification tasks the model after unlearning would also output a wrong
label 13, 6. Thus, to guide $\mathcal{M}_{\hat{\theta}}$ output incorrect
names, the fine-tuning data for the first target is shown in Figure 7(a). The
proof of effectiveness of this target is presented in Appendix A.2.

Assigning New Visual Description. In our primary experiments, it is found that
utilizing only the fine-tuning data of the first target will lead MLLMs to
recognize $\mathcal{C}$ as both _Donald Trump_ and the new incorrect name.
This phenomenon indicates that MLLMs correspond the same visual
representations to the original name and the newly given name. Thus, we
mitigate the risk of the MLLMs confusing the original and the new name by
fabricating a new visual description for $\mathcal{C}$. The constructed data
for the target is shown in Figure 7(b).

Decoupling Factual Knowledge. Leveraging fine-tuning data only of the first
two objectives could lead MLLMs to completely forget $\mathcal{C}$ including
the factual knowledge. This observation contradicts our definition in Section
3. For _Donald Trump_ , he possesses many attributes, such as being a former
U.S. President and a politician. Therefore, to decouple the factual knowledge
of the concept, we use a specific factual piece of knowledge about him as
fine-tuning data as depicted in Figure 7(c).

Preserving Non-targeted Knowledge. We find that only fine-tuning MLLMs on data
associated with $\mathcal{C}$ may lead to the forgetting of non-targeted
knowledge. However, it is essential to ensure that unlearning process does not
diminish its ability to accurately respond to other unrelated knowledge
domains. Finally, we introduce examples which describe the knowledge of non-
targeted concepts to alleviate this issue as shown in Figure 7(d).

###  4.2 Dual Masked KL-divergence Loss

We propose a novel Dual Masked KL-divergence (DMK) Loss which refines the
unlearning process by incorporating a dual masking technique into KL-
divergence loss. The motivation of DMK is discussed in Appendix B . The masks
of DMK are twofold:

Token-Level Masking. This mask operates at the token level, masking out tokens
that contradicts original knowledge. Masked tokens are excluded from the
computation of the KL divergence, preventing the model from increasing their
probability in the output distribution. For instance, as stated in Section
4.1, we assign an alternative name such as _‘Jacob Campbell’_ for _Donald
Trump_. We then apply the mask to the tokens of _‘Jacob Campbell’_ in the
fine-tuning sentence, where the KL-divergence loss is not computed. Formally,
for a training sample $\mathcal{T}$ consisting of
$\left\\{w_{1},w_{2},\ldots,w_{n}\right\\}$, the token-level mask is defined
as:

$K_{\mathcal{S}}=\\{m_{1},m_{2},\ldots,m_{n}\\},\text{ where
}m_{j}=\begin{cases}0,&\text{if }w_{j}\text{ is a specified token},\\\
1,&\text{otherwise}.\end{cases}$ (2)

Vocabulary-Level Masking. The second level of masking operates across the
entire vocabulary. For those tokens where KL-divergence loss is computed, we
introduce a mask within the MLLMs’ vocabulary specifically for the tokens of
$\mathcal{C}$’s name. Mathematically, if $\mathcal{V}$ is the vocabulary, the
vocabulary-level mask for the vocabulary is:

$K_{\mathcal{V}}=\\{m_{v_{1}},m_{v_{2}},\ldots,m_{v_{|\mathcal{V}|}}\\},\text{
where }m_{v_{i}}=\begin{cases}0,&\text{if }v_{i}\in\mathcal{C},\\\
1,&\text{otherwise}.\end{cases}$ (3)

The formulation of the DMK Loss is as follows:

$\mathcal{L}_{DMK}(\mathcal{I}_{i},\mathcal{T}_{i};\hat{\theta})=\sum_{t=1}^{t_{i}}K_{\mathcal{S}}\cdot
K_{\mathcal{V}}\cdot
P_{\mathcal{M}_{\theta}}(w_{t}^{i}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i})\log\frac{P_{\mathcal{M}_{\theta}}(w_{t}^{i}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i})}{P_{\mathcal{M}_{\hat{\theta}}}(w_{t}^{i}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i})}.$
(4)

Finally, we optimize Cross Entropy Loss and $\mathcal{L}_{DMK}$ using Gradient
Descent:

$\mathcal{L}_{total}(\mathcal{I}_{i},\mathcal{T}_{i};\hat{\theta})=-\alpha\cdot{\sum_{t=1}^{t_{i}}}\log
P_{\mathcal{M}_{\hat{\theta}}}(w_{t}^{i}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i})+\beta\cdot\mathcal{L}_{DMK}(\mathcal{I}_{i},\mathcal{T}_{i};\hat{\theta}),$
(5)

where $\alpha$ and $\beta$ are the hyper-parameters of weighing the two
losses.

##  5 MMUBench

We establish MMUBench, a comprehensive benchmark for advancing MU within
MLLMs. MMUBench is designed to evaluate the process of unlearning across
various dimensions of model performance and behavior. The construction of
dataset is detailed in Appendix C.1. In this section, we introduce the
evaluation settings of MMUBench:

Efficacy. This dimension assesses how effectively $\mathcal{M}_{\hat{\theta}}$
have unlearned seen examples. Efficacy measures the accuracy of answers given
the inputs of $\mathcal{D}^{f}_{train}$. It inspects if the
$\mathcal{M}_{\hat{\theta}}$’s outputs are now aligned with the objectives of
the MU in MLLMs.

Generality. Generality examines the $\mathcal{M}_{\hat{\theta}}$’s ability on
$\mathcal{D}^{f}_{test}$. This evaluation ensures that MLLMs does not
recognize $\mathcal{C}$ across a set of unseen images. In addition to the
visual generality, we also test the $\mathcal{M}_{\hat{\theta}}$’s
adaptability to a variety of textual prompts, providing a comprehensive
evaluation of the $\mathcal{M}_{\hat{\theta}}$’s ability to generalize the
unlearning process across both modalities. Generality is quantified using
three types of measurements within MMUBench, which are Exact Match (EM), GPT-4
Evaluation (G-Eval) and $\mathcal{C}$ Probability Distance
($\mathcal{C}$-Dis). The three measurements are detailed in Appendix C.3.

Specificity. Specificity measures the impact of unlearning on non-targeted
knowledge. As we have no access to the whole remaining data of the pre-
training phase, we employ a diverse set of public multimodal benchmarks to
assess specificity. The evaluation benchmarks include GQA 18, VQA-v2 14,
VisWiz 15, SQA I 30, VQA T 40, POPE 25, MMB 28, Mm-Vet 48. We take the average
of all benchmark performance as Specificity.

Fluency. Fluency evaluates the readability of responses of
$\mathcal{M}_{\hat{\theta}}$, which ensures the utility of
$\mathcal{M}_{\hat{\theta}}$. We compare the perplexity of sentences generated
by the model before and after unlearning. When the name of $\mathcal{C}$
appears in the output from $\mathcal{M}_{\theta}$, we apply a mask to avoid
distorting the fluency measurement:

$\displaystyle Fluency=\exp(-\frac{1}{t_{i}}{\sum_{t=1}^{t_{i}}}\log
P^{mask}_{\mathcal{M}_{\hat{\theta}}}(w_{t}^{i}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i}),$
$\displaystyle
P^{mask}_{\mathcal{M}_{\hat{\theta}}}(w_{t}^{i}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i})=\begin{cases}P_{\mathcal{M}_{\hat{\theta}}}(w_{t}^{i}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i}),&\text{if
}w_{t}^{i}\notin\mathcal{C},\\\ \frac{1}{\text{vocabulary size}},&\text{if
}w_{t}^{i}\in\mathcal{C},\end{cases}$ (6)

where ‘vocabulary size’ is dependent on the specific MLLM.

Diversity. Diversity can measure whether $\mathcal{M}_{\hat{\theta}}$ can
generate unique answers. It also ensures that the output of
$\mathcal{M}_{\hat{\theta}}$ does not over-fit to a few templates that appear
in the unlearning process. We count the number of unique words in the total
generated output.

Membership Inference Attack. Membership inference attacks (MIA) could reveal
whether the visual representations of $\mathcal{C}$ are still encoded in
$\mathcal{M}_{\hat{\theta}}$. As we could not get access to the pre-training
data of MLLMs, we use Min-K% PROB 38, an MIA method without knowing the pre-
training data. The detailed calculation of this measurement is stated in
Appendix D.2.

Jailbreak. Jailbreak attacks are designed to assess how
$\mathcal{M}_{\hat{\theta}}$ performs under deliberately challenging or edge-
case conditions, checking if $\mathcal{M}_{\hat{\theta}}$ truly cannot
generate outputs related to $\mathcal{C}$. We utilize multilingual test 11 and
multi-hop question test 53 as our jailbreak experiments.

##  6 Experiments

###  6.1 Experiment setup

Model and Training. As stated in Appendix C.1, the concept filtering process
is implemented by LLAVA 26 to construct dataset. To accurately compare the
knowledge before and after unlearning, we also use LLAVA (7B and 13B) to
obtain the unlearned model. The optimizer is Adam and the learning rate is
3e-4. Lora 16 is employed to fine-tune LLAVA with batch size 4. The training
step is set to 6. We use four A100 40G GPUs to train the model. $\alpha$ and
$\beta$ are 0.9 and 0.75 respectively.

Baselines. We compare our method with several existing methods: (i) Preference
Optimization (PO). Following TOFU 31, we use _‘I do not know.’_ and its
variants as the responses to the questions correspond with $\mathcal{C}$. (ii)
Gradient Ascent (GA) 46. It optimizes MLLMs to decrease their ability to
recall or generate texts related to $\mathcal{C}$. (iii) GA+KL 45. To preserve
the utility of MLLMs, KL-divergence loss is combined with GA.

Evaluate Concepts. In the experimental section, we primarily present the
experimental results related to _Donald Trump_ due to the limited space. We
report several other concepts covering different types, such as Cartoon
concepts (_Hello Kitty_ and _Mario_) and abstract concepts about painting
style (_Doodle_ , _Picasso_ and _Van Gogh_). Moreover, we evaluate the effects
of synchronously unlearning all the 20 concepts of MMUBench. The details of
$\mathcal{D}^{f}_{train}$ and $\mathcal{D}^{f}_{test}$ are presented in
Appendix C.2.

Table 1: Comparison with the existing machine unlearning methods. We report
the means and standard deviation of 3 independent trials. It is noted that the
_Specificity_ of each benchmark is summarized in Table 7.

Method | Efficacy$\uparrow$ | Generality | Specificity$\uparrow$ | Fluency$\downarrow$ | Diversity$\uparrow$
---|---|---|---|---|---
EM$\uparrow$ | G-Eval$\downarrow$ |  $\mathcal{C}$-Dis$\uparrow$
LLAVA7B
PO 31 | 100.0$\pm$0 | 58.3$\pm$4.0 | 2.0$\pm$0.8 | 0.4$\pm$0.1 | 58.3$\pm$1.3 | 75.1$\pm$0.9 | 93.5$\pm$2.1
GA 46 | 100.0$\pm$0 | 36.3$\pm$5.4 | 1.8$\pm$0.4 | 1.6$\pm$1.2 | 9.0$\pm$1.9 | 373.6$\pm$3.5 | 6.3$\pm$2.6
GA+KL 45 | 100.0$\pm$0 | 33.0$\pm$1.7 | 2.8$\pm$1.0 | 0.8$\pm$0.6 | 60.0$\pm$0.3 | 198.1$\pm$2.3 | 48.0$\pm$5.2
SIU | 100.0$\pm$0 | 99.0$\pm$0.0 | 1.9$\pm$0.5 | 1.8$\pm$0.3 | 60.7$\pm$0.7 | 61.2$\pm$1.2 | 97.0$\pm$0.2
LLAVA13B
PO | 100.0$\pm$0 | 10.7$\pm$3.1 | 4.6$\pm$0.2 | 0.5$\pm$0.2 | 63.4$\pm$1.1 | 60.7$\pm$0.3 | 89.7$\pm$1.4
GA | 100.0$\pm$0 | 24.7$\pm$1.7 | 4.6$\pm$0.1 | 1.6$\pm$1.4 | 63.2$\pm$0.2 | 144.7$\pm$7.4 | 74.5$\pm$4.9
GA+KL | 100.0$\pm$0 | 17.3$\pm$1.2 | 4.8$\pm$0.1 | 1.5$\pm$0.4 | 63.2$\pm$1.1 | 114.1$\pm$3.8 | 75.0$\pm$2.4
SIU | 100.0$\pm$0 | 90.0$\pm$0.8 | 2.1$\pm$0.6 | 3.6$\pm$1.0 | 63.4$\pm$0.4 | 54.3$\pm$0.9 | 96.5$\pm$0.7

###  6.2 Experiment Results

Main Results. The experimental results in Table 1 present a comprehensive
evaluation of various methods for machine unlearning in MLLMs. The
observations are as follows: (i) Efficacy across all methods is at 100%, which
indicates that each method is equally capable of unlearning the seen examples
and aligning well with the objectives of machine unlearning. (ii) GA shows an
outstanding performance in G-Eval with 1.8 score. However, this high score in
generality is a result of GA’s method always outputting _whitespace_ or
_repeated tokens_. SIU also performs a high Generality with 99.0% EM score,
showcasing its effectiveness at extending unlearning to unseen data. (iii) GA
performs 9.0 in Specificity score, indicating that there’s a strong impact on
the model’s knowledge base. SIU achieves a reasonable balance, with a score of
60.7, illustrating that it maintains a good level of model performance on non-
targeted tasks. (iv) Fluency is where the GA method notably fails, with a
score of 373.6. In contrast, SIU’s fluency score of 61.2 suggests that it
manages to retain coherent language outputs post-unlearning. (v) The PO method
seems to have maintained a degree of diversity, as indicated by a moderate
score. GA+KL shows a limited score of 48.0 in Diversity. GA’s score is
essentially at rock bottom (6.3), due to its most responses of _whitespace_ or
_repeated tokens_. SIU performs admirably with a score of 97.0, indicating its
maintenance in generating diverse responses post-unlearning. (vi) As the model
size increases from 7B to 13B, there is a noticeable decline in the
effectiveness of non-SIU methods in Generality. For example, the EM score for
GA falls from 36.3% to 24.7%, and both PO and GA+KL experience severe drops in
their generality scores. This sharp decline highlights a critical
vulnerability in these methods due to the change in model size. (vii) SIU
shows a relatively minor decline in generality (from 99% to 90% EM) when
scaling up from the 7B to the 13B model. This slight reduction indicates that
SIU is more adaptable and stable. (viii) Across all methods, there is an
observed improvement in specificity, fluency, and diversity from the 7B to the
13B models. This enhancement suggests a trade-off between the effectiveness of
unlearning and the preservation of model utility.

Table 2: Ablation study of DMK Loss. We utilize LLAVA7B to conduct the
experiments.

Method | Generality | Specificity$\uparrow$
---|---|---
EM$\uparrow$ | G-Eval$\downarrow$ |  $\mathcal{C}$-Dis$\uparrow$
w/o token | 92.0$\pm$0.0 | 2.0$\pm$0.3 | 1.5$\pm$0.1 | 27.7$\pm$2.5
w/o vocabulary | 94.3$\pm$1.2 | 2.1$\pm$0.2 | 1.6$\pm$0.2 | 29.4$\pm$1.7
SIU | 99.0$\pm$0.0 | 1.9$\pm$0.1 | 1.8$\pm$0.4 | 28.9$\pm$1.4

Ablation Study of DMK Loss. We perform an ablation study to evaluate the
significance of Token-Level Masking and Vocabulary-Level Masking as shown in
Table 2. Every masking is individually subjected to ablation to examine its
effect. We use Mm-Vet benchmark as the specificity. It could be observed that
the EM score without Token-Level Masking and Vocabulary-Level Masking both
degrade compared to SIU. Moreover, the $\mathcal{C}$-Dis also goes down if SIU
is not equipped with Token-Level Masking or Vocabulary-Level Masking. The
results show that The two levels of masking could both improve the generality
of unlearning and reduce the probability of generating tokens of
$\mathcal{C}$. We also observe that the Specificity of SIU is worse than the
model without vocabulary-level. The reason may be that masking several tokens
during the computation of KL affects the logic of general output to a certain
extent.

6101520253035$40$$60$$80$$100$EM
(%)$\bm{\uparrow}$6101520253035$2$$3$G-eval$\bm{\downarrow}$6101520253035$0$$1$$2$$3$$4$Distance$\bm{\uparrow}$6101520253035$0$$10$$20$$30$Specificity
(%)$\bm{\uparrow}$6101520253035$0.0$k$0.5$k$1.0$k$1.5$kFluency$\bm{\downarrow}$6101520253035$0$$50$$100$Diversity
(%)$\bm{\uparrow}$POGAGA+KLSIU Figure 2: Visualization of various metrics
across different methods over steps using LLAVA7B. 6101520253035$50$$100$EM
(%)$\bm{\uparrow}$61015202530$1$$2$$3$$4$G-eval$\bm{\downarrow}$6101520253035$0$$5$$10$Distance$\bm{\uparrow}$6101520253035$0$$10$$20$$30$Specificity
(%)$\bm{\uparrow}$6101520253035$0.0$k$0.5$k$1.0$k$1.5$kFluency$\bm{\downarrow}$6101520253035$0$$50$$100$Diversity
(%)$\bm{\uparrow}$POGAGA+KLSIU Figure 3: Visualization of various metrics
across different methods over steps using LLAVA13B.

Impacts of Fine-tuning Steps. In this section, we analyze the impact of fine-
tuning steps as shown in Figure 2 and Figure 3. We utilize Mm-Vet as the
Specificity. SIU demonstrates minimal fluctuations in each metric, which
suggests that SIU is less sensitive to the number of fine-tuning steps. In
contrast, other methods like GA and PO show significant variability with
increased fine-tuning steps. For instance, GA’s performance in Specificity and
Fluency metrics tends to degrade seriously as the number of steps increases.
Compared with the 7B model, the 13B model shows a slower adaptation speed. The
7B model displays a rapid increase in EM scores, reaching near-maximum values
by step 10 across most methods. The 13B model shows a slower increase in EM
scores over steps. PO method exhibits nearly constant values as steps increase
in $\mathcal{C}$-Dis, regardless of the model size (both 7B and 13B). This
consistency indicates that the PO method has primarily learned to respond with
_‘I do not know.’_ rather than reducing the probability of recognizing the
unlearned concept.

DoodleElon MuskFacebookHello KittyJoe BidenMarioPicassoTylor SwiftVan
Gogh$0$$50$$100$EM (%)SIUGA+KLPOGA Figure 4: EM performance comparison of
methods SIU, GA+KL, PO, and GA across different concepts.

Effects of Unlearning Different Concepts. We evaluates several other concepts
in our benchmark. The results of Generality (EM) are shown in Figure 4 and the
overall results are summarized in Table 6. It could be observed that SIU
consistently achieves nearly 100% accuracy in unlearning across all tested
concepts, demonstrating its robustness and effectiveness. We also find all
methods perform notably well on more abstract concepts such as _Doodle_ and
_Picasso_ , which indicates that abstract concepts are easier to disassociate
from the model’s knowledge base. The case studies of these concepts are
presented in Figures 16, 17, 18, 19, 20, 21 and 22.

Positive Butterfly Effect. We observe that our method could trigger surprising
positive butterfly effects which can further illustrate the effects of machine
unlearning. As shown in Figure 9, we input an image featuring Donald Trump
with his family into $\mathcal{M}_{\theta}$ and $\mathcal{M}_{\hat{\theta}}$
respectively. $\mathcal{M}_{\theta}$ is able to identify each person’s name in
the image correctly and $\mathcal{M}_{\hat{\theta}}$ misidentifies Donald
Trump due to our unlearning method. However, his wife Melania is also
misidentified by $\mathcal{M}_{\hat{\theta}}$. At first, we assume that our
unlearning method causes the model to lose the ability to identify some other
concepts. Further examination reveals an additional layer to this phenomenon.
As can be seen in Figure 10, when the image is cropped to only include Melania
Trump and presented to $\mathcal{M}_{\hat{\theta}}$, it accurately recognizes
her and ‘remember’ her relationship with Donald Trump. This discovery points
to a fascinating aspect of machine unlearning: the selective retention of
knowledge. The reason of this observation might be that the model’s failure to
identify the central male figure as Trump in the original image leads to an
inference that the adjacent female could not be Melania. These positive
butterfly effects suggest that unlearning is not a blunt tool that erases all
traces of a concept but rather can result in a refined restructuring of
knowledge within the model.

Results of Unlearning Multiple Concepts Simultaneously. Table 3 reports the
results of synchronously unlearning all the concepts of MMUBench. We concat
all the forgetting training sets of these concepts as fine-tuning data and the
training step is set to 120. We find that after unlearning, the utility of
MLLMs collapses using GA and GA+KL. All the responses of GA and GA+KL are
repeated tokens _‘image image image…’_ It could be observed that there is some
decline in Specificity and Fluency of PO. In contrast, each metric is nearly
the same with unlearning a single concept utilizing SIU, which illustrates the
robustness of SIU.

Table 3: Results of unlearning 20 concepts simultaneously using LLAVA7B. Inf
denotes an infinite value. We do not test G-Eval for GA and GA+KL because they
only generate _repeated tokens_ in all responses.

Method | Efficacy$\uparrow$ | Generality | Specificity$\uparrow$ | Fluency$\downarrow$ | Diversity$\uparrow$
---|---|---|---|---|---
EM$\uparrow$ | G-Eval$\downarrow$ |  $\mathcal{C}$-Dis$\uparrow$
PO 31 | 100.0 | 80.0 | 2.7 | 0.5 | 12.7 | 59.7 | 96.9
GA 46 | 100.0 | 100.0 | - | 30.4 | 0 | Inf | 0.67
GA+KL 45 | 100.0 | 100.0 | - | 15.7 | 0 | 695.2 | 0.67
SIU | 100.0 | 97.0 | 1.7 | 5.0 | 24.9 | 54.4 | 99.3

Table 4: Performance of MIA and Jailbreak with LLAVA7B. We do not evaluate GA
method because the most of outputs are _whitespace_ or _repeated tokens_.

Method | MIA$\downarrow$ | Jailbreak
---|---|---
Multilingual$\downarrow$ | Multi-hop$\downarrow$
PO | 0.32 | 2.5 | 0.18
GA+KL | 0.44 | 2.9 | 0.38
SIU | 0.27 | 2.3 | 0.16

MIA and Jailbreak. Table 4 displays the results of MIA and Jailbreaks tests.
The experimental details of MIA are stated in Appendix D.2. It could be
observed that SIU achieves the lowest ROUGE-L score, indicating that the
outputs of SIU diverge most from that of $\mathcal{M}_{\theta}$. We find PO
also performs well under MIA. The reason may be that it tends to output _‘I do
not know.’_ , leading to a low similarity score with the output of
$\mathcal{M}_{\theta}$.

For Jailbreak, we conduct two types of tests, which are multilingual test and
multi-hop question test. The experiments are detailed in Appendix D.3 and
Appendix D.4. Combining Table 1 and Table4, we find that the performance of
GA+KL and SIU on multilingual are both slightly improved from 2.8 to 2.9 and
from 1.9 to 2.3. The case studies are shown in Figures 12, 13 and 14. From the
specific examples we find PO always outputs _‘I do not know.’_ in different
languages. The outputs of SIU are diverse in different languages, illustrating
the preservation of utility. For multi-hop question test, as shown in Table 4,
it could be observed that SIU performs well in Multi-hop questions, indicating
the capability of defending hard examples. The case study of Multi-hop
question is displayed in Figure 15. We find that though GA+KL avoids
generating the name of $\mathcal{C}$, it could still answer the right factual
knowledge of the question. This self-contradictory answer illustrates the
analysis in Section 1.We also observe that SIU could _‘make up some lies’_
such as ‘having gold courses in St.Andrews’. This phenomenon also confirms the
findings of positive butterfly effects.

##  7 Conclusion

We introduce SIU, an efficient method to unlearn the visual recognition of
concepts in MLLMs with only one training image. We propose four targets to
construct little fine-tuning data. To mitigate the degradation of MLLMs, we
introduce Dual Masked KL-divergence Loss to be jointly trained with Cross
Entropy Loss. Together with the method we present MMUBench, a benchmark to
evaluate machine unlearning in MLLMs. The benchmark is composed of 1000
images, with 50 images for each of the 20 concepts, and a set of evaluation
metrics. The experimental results illustrate the effectiveness and robustness
of our method. For future work, we would try to extend this work mainly in the
following aspects: (i) exploring new machine unlearning methods in MLLMs; (ii)
evaluating machine unlearning for data points rather than concept-wise
knowledge in MLLMs.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, and et al.  Flamingo: a visual language model for few-shot learning.  In _NeurIPS_ , 2022.
  * Amos et al. [2024] Ido Amos, Jonathan Berant, and Ankit Gupta.  Never train from scratch: Fair comparison of long-sequence models requires data-driven priors.  In _ICLR_. OpenReview.net, 2024.
  * Anil et al. [2023] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, and et al.  Palm 2 technical report.  _CoRR_ , abs/2305.10403, 2023.
  * Bai et al. [2023] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _CoRR_ , abs/2308.12966, 2023.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and et al.  Language models are few-shot learners.  In _NeurIPS_ , 2020.
  * Cha et al. [2024] Sungmin Cha, Sungjun Cho, Dasol Hwang, Honglak Lee, Taesup Moon, and Moontae Lee.  Learning to unlearn: Instance-wise unlearning for pre-trained classifiers.  In _AAAI_ , pages 11186–11194. AAAI Press, 2024.
  * Chen and Yang [2023] Jiaao Chen and Diyi Yang.  Unlearn what you want to forget: Efficient unlearning for llms.  In _EMNLP_ , pages 12041–12052. Association for Computational Linguistics, 2023.
  * Chen et al. [2023] Ruizhe Chen, Jianfei Yang, Huimin Xiong, Jianhong Bai, Tianxiang Hu, Jin Hao, Yang Feng, Joey Tianyi Zhou, Jian Wu, and Zuozhu Liu.  Fast model debias with machine unlearning.  In _NeurIPS_ , 2023.
  * Cheng et al. [2024] Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu.  Can AI assistants know what they don’t know?  _CoRR_ , abs/2401.13275, 2024.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _CoRR_ , abs/2305.06500, 2023.
  * Deng et al. [2023] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing.  Multilingual jailbreak challenges in large language models.  _CoRR_ , abs/2310.06474, 2023.
  * Eldan and Russinovich [2023] Ronen Eldan and Mark Russinovich.  Who’s harry potter? approximate unlearning in llms.  _CoRR_ , abs/2310.02238, 2023.
  * Fan et al. [2023] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu.  Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.  _CoRR_ , abs/2310.12508, 2023.
  * Goyal et al. [2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the V in VQA matter: Elevating the role of image understanding in visual question answering.  In _CVPR_ , pages 6325–6334. IEEE Computer Society, 2017.
  * Gurari et al. [2018] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham.  Vizwiz grand challenge: Answering visual questions from blind people.  In _CVPR_ , pages 3608–3617. Computer Vision Foundation / IEEE Computer Society, 2018.
  * Hu et al. [2022] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In _ICLR_. OpenReview.net, 2022.
  * Huang et al. [2023] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, and et al.  Language is not all you need: Aligning perception with language models.  In _NeurIPS_ , 2023.
  * Hudson and Manning [2019] Drew A. Hudson and Christopher D. Manning.  GQA: A new dataset for real-world visual reasoning and compositional question answering.  In _CVPR_ , pages 6700–6709. Computer Vision Foundation / IEEE, 2019.
  * Ishihara et al. [2022] Shotaro Ishihara, Hiromu Takahashi, and Hono Shirai.  Semantic shift stability: Efficient way to detect performance degradation of word embeddings and pre-trained language models.  In _AACL/IJCNLP (1)_ , pages 205–216. Association for Computational Linguistics, 2022.
  * Kurmanji et al. [2023] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou.  Towards unbounded machine unlearning.  In _NeurIPS_ , 2023.
  * Li et al. [2023a] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  _CoRR_ , abs/2305.03726, 2023a.
  * Li et al. [2024] Jiaqi Li, Miaozeng Du, Chuanyi Zhang, Yongrui Chen, Nan Hu, Guilin Qi, Haiyun Jiang, Siyuan Cheng, and Bozhong Tian.  MIKE: A new benchmark for fine-grained multimodal entity knowledge editing.  _CoRR_ , abs/2402.14835, 2024.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 19730–19742. PMLR, 2023b.
  * Li et al. [2023c] Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han.  Deepinception: Hypnotize large language model to be jailbreaker.  _CoRR_ , abs/2311.03191, 2023c.
  * Li et al. [2023d] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  In _EMNLP_ , pages 292–305. Association for Computational Linguistics, 2023d.
  * Liu et al. [2023a] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _CoRR_ , abs/2304.08485, 2023a.
  * Liu et al. [2024] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu.  Rethinking machine unlearning for large language models.  _CoRR_ , abs/2402.08787, 2024.
  * Liu et al. [2023b] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin.  Mmbench: Is your multi-modal model an all-around player?  _CoRR_ , abs/2307.06281, 2023b.
  * Liu et al. [2022] Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Xiangyang Ji, Antoni B. Chan, and Rong Jin.  Improved fine-tuning by better leveraging pre-training data.  In _NeurIPS_ , 2022.
  * Lu et al. [2022] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _NeurIPS_ , 2022.
  * Maini et al. [2024] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter.  TOFU: A task of fictitious unlearning for llms.  _CoRR_ , abs/2401.06121, 2024.
  * Mantelero [2013] Alessandro Mantelero.  The EU proposal for a general data protection regulation and the roots of the ’right to be forgotten’.  _Comput. Law Secur. Rev._ , 29(3):229–235, 2013.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _CoRR_ , abs/2303.08774, 2023.  doi: 10.48550/arXiv.2303.08774.  URL https://doi.org/10.48550/arXiv.2303.08774.
  * Qi et al. [2023] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson.  Fine-tuning aligned language models compromises safety, even when users do not intend to!  _CoRR_ , abs/2310.03693, 2023.
  * Ramanujan et al. [2023] Vivek Ramanujan, Thao Nguyen, Sewoong Oh, Ali Farhadi, and Ludwig Schmidt.  On the connection between pre-training data diversity and fine-tuning robustness.  In _NeurIPS_ , 2023.
  * Scherer and Kiparski [2018] Joachim Scherer and Gerd Kiparski.  Buchbesprechungen. feiler, lukas / forgó, nikolaus / weigl, michaela: The eu general data protection regulation (gdpr): A commentary.  _Comput. und Recht_ , 34(6):69–70, 2018.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, and et al.  LAION-5B: an open large-scale dataset for training next generation image-text models.  In _NeurIPS_ , 2022.
  * Shi et al. [2023] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer.  Detecting pretraining data from large language models.  _CoRR_ , abs/2310.16789, 2023.
  * Si et al. [2023] Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang.  Knowledge unlearning for llms: Tasks, methods, and challenges.  _CoRR_ , abs/2311.15766, 2023.
  * Singh et al. [2019] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards VQA models that can read.  In _CVPR_ , pages 8317–8326. Computer Vision Foundation / IEEE, 2019.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, and et al.  Llama: Open and efficient foundation language models.  _CoRR_ , abs/2302.13971, 2023.
  * Wang et al. [2023a] Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin.  KGA: A general machine unlearning framework based on knowledge gap alignment.  In _ACL (1)_ , pages 13264–13276. Association for Computational Linguistics, 2023a.
  * Wang et al. [2023b] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang.  Cogvlm: Visual expert for pretrained language models.  _CoRR_ , abs/2311.03079, 2023b.
  * Xu et al. [2023] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, and et al.  mplug-2: A modularized multi-modal foundation model across text, image and video.  In _ICML_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 38728–38748. PMLR, 2023.
  * Yao et al. [2024] Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue.  Machine unlearning of pre-trained large language models.  _CoRR_ , abs/2402.15159, 2024.
  * Yao et al. [2023] Yuanshun Yao, Xiaojun Xu, and Yang Liu.  Large language model unlearning.  _CoRR_ , abs/2310.10683, 2023.
  * Yin et al. [2023] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.  A survey on multimodal large language models.  _CoRR_ , abs/2306.13549, 2023.
  * Yu et al. [2023] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  _CoRR_ , abs/2308.02490, 2023.
  * Zhang et al. [2024] Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu.  Mm-llms: Recent advances in multimodal large language models.  _CoRR_ , abs/2401.13601, 2024.
  * Zhang et al. [2023] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola.  Multimodal chain-of-thought reasoning in language models.  _CoRR_ , abs/2302.00923, 2023.
  * Zhao et al. [2023] Jiachen Zhao, Zhun Deng, David Madras, James Zou, and Mengye Ren.  Learning and forgetting unsafe examples in large language models.  _CoRR_ , abs/2312.12736, 2023.
  * Zheng et al. [2023] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You.  Preventing zero-shot transfer degradation in continual learning of vision-language models.  In _ICCV_ , pages 19068–19079. IEEE, 2023.
  * Zhong et al. [2023] Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen.  Mquake: Assessing knowledge editing in language models via multi-hop questions.  In _EMNLP_ , pages 15686–15702. Association for Computational Linguistics, 2023.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _CoRR_ , abs/2304.10592, 2023.

##  Appendix A Fine-tuning Data

###  A.1 Visit the output of unseen concepts

Figure 5: The output distribution of LLAVA when queried about the visual
recognition of unseen concepts.

As the objective of unlearning is to achieve a model where forgetting data is
not present in the training phase, we explore how do MLLMs respond when
queried about unseen concepts. We collect the images of 190 people that are
definitely not contained in the pre-training data of MLLMs. The use of these
images has been explicitly approved by these people. We query the MLLMs with
the prompt _‘Please give the specific name of this person.’_ The output
distribution is shown in Figure 5. The results show that MLLMs will not answer
_‘I do not know.’_ when queried about unseen people. They tend to output
general names such _‘John’_ and _‘Jason’_ , or output a vague answer _‘a man
or woman’_. Though the answer _‘I do not know.’_ is the most reasonable, it
breaks the characteristics of MLLMs’ output. We suppose that the
characteristics gradually forms during the pre-training phase (perhaps there
is little data containing the answer _‘I do not know’_). Thus we assign a
random name for the targeted unlearning concept in accordance with the
characteristics of MLLMs’ output. The candidate names are shown in Figure 6.

###  A.2 Proof of Aligning with Unseen Concepts

Below, we provide a perspective on the target of _Aligning with Unseen
Concepts_. We prove that our target can achieve the objective of MU in MLLMs.
We first formalize each element in the reinterpretation of the objective of MU
in MLLMs as stated in Section 4.1.

Definition. Unlearned MLLM is fine-tuned with the forgetting training set
$\mathcal{D}^{f}_{train}=\\{(\mathcal{I}^{\mathcal{C}^{*}}_{j},\mathcal{T}^{\mathcal{C}^{*}}_{j})\\}_{j=1}^{K}$,
which can be formulated as
$\mathcal{M}_{\hat{\theta}}\leftarrow\\{(\mathcal{I}^{\mathcal{C}^{*}}_{j},\mathcal{T}^{\mathcal{C}^{*}}_{j})\\}_{j=1}^{K}$.
The pre-trained MLLM is trained with a collection of image-text pairs
$\mathcal{D}_{pre}=\\{(\mathcal{I}_{i},\mathcal{T}_{i})\\}_{i=1}^{N}$, and the
formula is
$\mathcal{M}_{\theta}\leftarrow(\mathcal{I}_{i},\mathcal{T}_{i})\\}_{i=1}^{N}$.
All the pre-training data associated with $\mathcal{C}$ is a subset of
$\mathcal{D}_{pre}$, denoted as
$\mathcal{D}^{c}_{pre}=\\{(\mathcal{I}^{\mathcal{C}^{\prime}}_{o},\mathcal{T}^{\mathcal{C}^{\prime}}_{o})\\}_{o=1}^{M}$.
The objective of MU in MLLM is to achieve a model that assumes the absence of
$\mathcal{D}^{c}_{pre}$ during its pre-training phase. Such model can be
formulated as
$\mathcal{M}_{\theta^{\prime}}\leftarrow\mathcal{D}_{pre}\setminus\mathcal{D}^{c}_{pre}=\\{(\mathcal{I}_{i},\mathcal{T}_{i})\\}_{i=1}^{N-M}$.
The training objective of _Aligning with Unseen Concepts_ is to achieve
$P_{\mathcal{M}_{\hat{\theta}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})\cong
P_{\mathcal{M}_{\theta}}(x|\mathcal{I}^{u},\mathcal{T}^{u})$, where
$\mathcal{I}^{u}$ and $\mathcal{T}^{u}$ are the images and texts definitely
not present in the pre-training phase of $\mathcal{M}_{\theta}$, while
$\mathcal{I}_{test}^{c}$ and $\mathcal{T}_{test}^{c}$ are the image-text paris
in the forgetting test set. The objective of MU in MLLMs can be formulated as
$P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})\cong
P_{\mathcal{M}_{\hat{\theta}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})$.

Proposition. _The training objective of Aligning with Unseen Concepts
$P_{\mathcal{M}_{\hat{\theta}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})\cong
P_{\mathcal{M}_{\theta}}(x|\mathcal{I}^{u},\mathcal{T}^{u})$ equals to the
objective of MU in MLLMs
$P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})\cong
P_{\mathcal{M}_{\hat{\theta}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})$._

_Proof._ As $\mathcal{I}_{test}^{c}$ and $\mathcal{I}^{\mathcal{C}^{\prime}}$
both completely contain the visual representations of $\mathcal{C}$, they are
identically distributed. Moreover, $\mathcal{T}_{test}^{c}$ is also identical
to $\mathcal{T}^{\mathcal{C}^{\prime}}$ because they both query the
recognition of $\mathcal{C}$. Thus we have:

$\displaystyle\qquad\qquad\qquad\mathcal{I}_{test}^{c}\cong\mathcal{I}_{\mathcal{C}}^{\prime},$
(7)
$\displaystyle\qquad\qquad\qquad\mathcal{T}_{test}^{c}\cong\mathcal{T}_{\mathcal{C}}^{\prime},$
$\displaystyle
P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})\cong
P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}^{\mathcal{C}^{\prime}},\mathcal{T}^{\mathcal{C}^{\prime}}).$

As $\mathcal{I}^{\mathcal{C}^{\prime}}$ and
$\mathcal{T}^{\mathcal{C}^{\prime}}$ are not present in the pre-training phase
of $\mathcal{M}_{\theta^{\prime}}$,
$(\mathcal{I}^{\mathcal{C}^{\prime}},\mathcal{T}^{\mathcal{C}^{\prime}})$ is
also an unseen image-text pair for $\mathcal{M}_{\theta^{\prime}}$. We have:

$P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}^{u},\mathcal{T}^{u})\cong
P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}^{\mathcal{C}^{\prime}},\mathcal{T}^{\mathcal{C}^{\prime}})\cong
P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c}).$
(8)

The difference between $\mathcal{M}_{\theta^{\prime}}$ and
$\mathcal{M}_{\theta}$ is the absence of $\mathcal{D}^{c}_{pre}$ during the
pre-training phase. Because the representations of $\mathcal{I}^{u}$ are
completely different from that of $\mathcal{I}^{\mathcal{C}^{\prime}}$, they
are independent and distributed differently. Thus deleting
$\mathcal{D}^{c}_{pre}$ in the pre-training phase will not affect the
prediction probability distribution of the model for $\mathcal{I}^{u}$. We
have:

$P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}^{u},\mathcal{T}^{u})\cong
P_{\mathcal{M}_{\theta}}(x|\mathcal{I}^{u},\mathcal{T}^{u})\cong
P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c}).$
(9)

Assuming we have achieved the training objective
$P_{\mathcal{M}_{\hat{\theta}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})\cong
P_{\mathcal{M}_{\theta}}(x|\mathcal{I}^{u},\mathcal{T}^{u})$, combined with
Formula 9, we achieve
$P_{\mathcal{M}_{\theta^{\prime}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})\cong
P_{\mathcal{M}_{\hat{\theta}}}(x|\mathcal{I}_{test}^{c},\mathcal{T}_{test}^{c})$.

###  A.3 Constructing fine-tuning data

Our constructed fine-tuning data for _Donald Trump_ are shown in Figure 7. The
data is centered on four targets. ‘<image>’ represents including the training
image as part of the input for the current batch. For both _Aligning with
Unseen Concepts_ and _Assigning New Visual Description_ the training image is
input into the model, while another two targets do not take images as input.
Moreover, we utilize GPT-4 33 to rephrase four pieces of fine-tuning data for
each target.

Figure 6: Candidate names for targeted unlearning concepts.

(a) Aligning with Unseen Concepts

(b) Assigning New Visual Description

(c) Decoupling Factual Knowledge

(d) Preserving Non-targeted Knowledge

Figure 7: Fine-tuning data for four targets.

##  Appendix B Motivation of DMK Loss

The Dual Masked KL-divergence (DMK) loss aims to address a core challenge that
arises when unlearning concepts from MLLMs using traditional KL-divergence.
While the standard KL-divergence loss function is effective in maintaining the
overall utility of MLLMs, it can inadvertently introduce logical
inconsistencies when applied to unlearning. The essence of the problem with
using traditional KL-divergence for unlearning stems from its tendency to pull
the probability distribution of tokens related to $\mathcal{C}$ closer to the
distribution of $\mathcal{M}_{\theta}$. This is contradictory to the goal of
unlearning, where the aim is to suppress the MLLMs’ ability to recall
$\mathcal{C}$. For example, considering the training phase, the input is the
training image of $\mathcal{C}$ and the prompt _‘What’s the name of the
central figure in this photograph?’_. When MLLMs predict the next token and
encounter the phrase _‘This is’_ , the token _‘Donald Trump’_ should ideally
have a reduced probability in the token distribution. However, since _‘Donald
Trump’_ might have a high probability in $\mathcal{M}_{\theta}$, standard KL
divergence would work against the unlearning goal by increasing the likelihood
of MLLMs predicting _‘Donald Trump’_ after _‘This is’_.

Table 5 further illustrates the motivation of DMK Loss. We utilize pre-trained
LLAVA to generate the next-token probability distribution. The colored data
shows relatively high probabilities for the token ‘Donald’ and ‘Trump’. For
the red colored data token $w_{t}$ after ‘President’, we could formulate the
probability distribution as
$P(w_{t})=P_{\mathcal{M}_{\theta}}(w_{t}|\mathcal{I}_{i},The,picture,features,President)$.
It could be found that the probability of ‘Donald’ plus that of ‘Trump’ is
near to 1, which indicates the probability of $\mathcal{C}$ would be extremely
high after the token ‘President’. Directly minimizing the KL-divergence
between $\mathcal{M}_{\hat{\theta}}$ and $\mathcal{M}_{\theta}$ on the red
colored tokens would cause unlearned model output higher probability of
$\mathcal{C}$, which is contrary to the objective of machine unlearning. Thus,
in Token-Level Masking we mask the whole distribution to those tokens where
the probability of $\mathcal{C}$-related tokens is extremely high. For the
orange colored tokens (the token of the beginning and the token after
‘features’), while the max probability is other token, the probability of
‘Donald’ and ‘Trump’ is also high. It would also improve the probability of
generating $\mathcal{C}$ if directly employing KL-divergence. To this end, we
apply the vocabulary-level masks to the tokens of ‘Donald’ and ‘Trump’ in the
vocabulary. As to the reason why we do not apply vocabulary-level mask to the
red colored tokens, the probability of $\mathcal{M}_{\theta}$ generating other
tokens is remarkably low on the red colored tokens. If only mask the tokens of
‘Donald’ and ‘Trump’ in the vocabulary, the probability of generating other
tokens would also be seriously reduced for $\mathcal{M}_{\hat{\theta}}$ due to
KL-divergence loss, which harms the utility of MLLMs.

Table 5: Token probabilities of pre-trained LLAVA given the image of Donald
Trump and the prompt ‘who is in the picture?’. The first line is the max
probability of current token. The second and the third lines report the
probability of ‘Donald’ and ‘Trump’ of the current token.

Token | The | picture | features | President | Donald | Trump | . | </s>
---|---|---|---|---|---|---|---|---
max prob | 0.57 | 0.77 | 0.92 | 0.42 | 0.68 | 0.94 | 0.45 | 0.99
Donald | 0.06 | 3.2e-5 | 1.2e-9 | 0.22 | 0.68 | 4.2e-5 | 1.2e-7 | 2.5e-6
Trump | 0.08 | 4.8e-7 | 8.2e-9 | 0.02 | 0.31 | 0.94 | 6.3e-8 | 3.1e-9

##  Appendix C MMUBench Construction

###  C.1 Dataset Construction

To construct a reliable and effective benchmark for evaluating MU within
MLLMs, we initiated a comprehensive data collection and curation process.

Concept Sampling. Our first step was to sample a diverse set of 300 concepts
from the MIKE dataset 22. The MIKE dataset ensures that each concept is
visually distinctive, which is crucial for MLLMs to unlearn these concepts.

Image Collecting. For each of these concepts, we employed an extensive image
collection process using Google’s search engine. We gathered at least 50
images per concept, resulting in a substantial pool of visual data. The
rationale behind collecting such a large number of images was to robustly
evaluate the generalization of the model’s unlearning capabilities.

Concept Filtering. Upon collecting the images, we undertook a filtering
process. A seed image for each concept from the MIKE dataset was used as a
benchmark to evaluate the relevance of the collected images. We discarded any
image where the depicted concept did not align with the concept represented by
the seed image. This step was crucial to maintain consistency and ensure that
the variations within the images did not introduce any ambiguity regarding the
concept.

Following this filtering, we subjected the remaining images to a recognition
test by inputting them into $\mathcal{M}_{\theta}$ with the prompt "What’s the
name of the central figure in this photograph?" If $\mathcal{M}_{\theta}$
correctly identifies the concept, this indicates that the concept presents
within the pre-training phase and thus the images and concept are retained. If
any image of the concept cannot be recognized by $\mathcal{M}_{\theta}$, the
concept was removed. After the filtering step, we finally retained 20
concepts.

###  C.2 Forgetset Construction

Images Splitting. We select one image per concept to act as
$\mathcal{D}^{f}_{train}$ for the unlearning process. A critical consideration
in this selection is the exclusivity of the target concept within the image.
The chosen training images are those in which the concept was the central and
singular focus, devoid of any additional elements that might lead to
confusion. This is particularly important during the training phase where the
MLLM must clearly understand which specific concept is to be unlearned. The
rest of images are use as $\mathcal{D}^{f}_{test}$

Generation of Questions. We utilize GPT-4 to generate the questions of
$\mathcal{D}^{f}_{train}$ and $\mathcal{D}^{f}_{test}$. We describe the task
we wanted to evaluate to GPT-4, then provide a concept name to GPT-4, and ask
it to generate 100 related questions that precisely correspond to this
concept, returning the questions to us. After the questions are generated, we
manually screen them and regenerate any that were not satisfactory, ensuring
that each concept is associated with 100 questions. For example, the prompt
given to GPT-4 to generate the questions of Donald Trump is _‘My current task
is to evaluate whether a multimodal large language model has forgotten Donald
Trump. Please help me generate 100 questions for testing with given input
images, along with the correct answer keywords (e.g., trump, yes). Organize
the questions and keywords in JSON format, with prompt corresponding to the
relevant questions and ‘target phrase’ corresponding to the keywords.’_ All
the questions of Donald Trump are shown in Figure 23 and Figure 24.

###  C.3 Three measurements of Generality

We have three measurements for Generality: (i) Exact Match (EM). The first
measurement is a straightforward way to determine if
$\mathcal{M}_{\hat{\theta}}$ correctly identifies the name of $\mathcal{C}$ in
$\mathcal{D}^{f}_{test}$. The prompts we utilize include either masking
$\mathcal{C}$’s name or eliciting a binary yes/no response regarding the
presence of $\mathcal{C}$. (ii) GPT-4 Evaluation (G-Eval). The second
measurement involves the use of GPT-4 to evaluate the
$\mathcal{M}_{\hat{\theta}}$’s responses. GPT-4 evaluates whether a response
indicates that $\mathcal{C}$’s visual recognition has been forgotten. The
instructions for G-Eval are shown in Figure 8. (iii) $\mathcal{C}$ Probability
Distance ($\mathcal{C}$-Dis). To further quantitatively measure the
effectiveness of unlearning, we introduce a metric that examines the distance
between the probability distributions of the model outputting the name of
$\mathcal{C}$ before and after the unlearning process, which can be formulated
as follows:

$Distance=\mathbb{E}_{(\mathcal{I}_{i},\mathcal{T}_{i})\in\mathcal{D}^{f}_{test}}\Big{[}-P_{\mathcal{M}_{\theta}}(\mathcal{C}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i})\log\frac{P_{\mathcal{M}_{\theta}}(\mathcal{C}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i})}{P_{\mathcal{M}_{\hat{\theta}}}(\mathcal{C}|\mathcal{I}_{i},w_{1}^{i},\ldots,w_{t-1}^{i})}\Big{]}.$
(10)

Figure 8: Instructions for G-Eval.

##  Appendix D Additional Results

Table 6: The performance of other concepts. The model we use is LLAVA7B

Method | Efficacy$\uparrow$ | Generality | Specificity$\uparrow$ | Fluency$\downarrow$ | Diversity$\uparrow$
---|---|---|---|---|---
EM$\uparrow$ | G-Eval$\downarrow$ |  $\mathcal{C}$-Dis$\uparrow$
Doodle
PO | 100.0 | 98.5 | 2.2 | 0.4 | 10.6 | 67.3 | 93.0
GA | 100.0 | 98.5 | 2.0 | 0.6 | 0.0 | 880.4 | 2.4
GA+KL | 100.0 | 98.5 | 2.3 | 0.4 | 20.1 | 335.8 | 68.1
SIU | 100.0 | 97.5 | 2.2 | 1.7 | 29 | 53.6 | 99.8
Elon Musk
PO | 100.0 | 54.0 | 3.0 | 0.2 | 19.8 | 79.7 | 93.0
GA | 100.0 | 64.0 | 3.5 | 2.5 | 0.0 | 857.6 | 12.5
GA+KL | 100.0 | 54.0 | 4.2 | 1.8 | 25.7 | 276.2 | 68.1
SIU | 100.0 | 91.0 | 1.9 | 3.5 | 30.6 | 56.1 | 98.9
Facebook
PO | 100.0 | 86.0 | 2.8 | 0.2 | 14.1 | 65.9 | 97.8
GA | 100.0 | 52.0 | 4.3 | 3.7 | 0.1 | 612.1 | 7.0
GA+KL | 100.0 | 50.0 | 4.5 | 2.8 | 27.0 | 238.3 | 62.7
SIU | 100.0 | 97.0 | 2.2 | 5.9 | 26.5 | 52.7 | 94.8
Hello Kitty
PO | 100.0 | 83.0 | 1.8 | 1.7 | 27.9 | 53.3 | 99.6
GA | 100.0 | 100.0 | 1.7 | 21.2 | 0.0 | 768.6 | 13.8
GA+KL | 100.0 | 97.0 | 1.8 | 20.9 | 25.9 | 272.1 | 60.2
SIU | 100.0 | 100.0 | 2.0 | 23.9 | 29.3 | 41.95 | 93.8
Joe Biden
PO | 100.0 | 58.0 | 3.9 | 0.7 | 17.2 | 51.7 | 96.9
GA | 100.0 | 62.0 | 3.8 | 5.8 | 0.2 | 329.6 | 6.9
GA+KL | 100.0 | 66.0 | 3.6 | 4.9 | 24.9 | 143.1 | 64.7
SIU | 100.0 | 100.0 | 2.0 | 13.1 | 28.0 | 42.3 | 89.5
Mario
PO | 100.0 | 55.0 | 3.7 | 0.5 | 24.4 | 50.4 | 96.5
GA | 100.0 | 61.0 | 2.8 | 10.5 | 4.1 | 235.2 | 10.3
GA+KL | 100.0 | 59.0 | 3.0 | 10.0 | 27.9 | 154.7 | 60.6
SIU | 100.0 | 97.0 | 2.0 | 4.7 | 28.2 | 42.5 | 96.2
Tylor Swift
PO | 100.0 | 63.0 | 2.7 | 0.1 | 19.4 | 60.6 | 97.9
GA | 100.0 | 72.0 | 2.0 | 1.8 | 0.0 | 2441.9 | 0.9
GA+KL | 100.0 | 70.0 | 2.1 | 1.8 | 30.8 | 1277.4 | 68.1
SIU | 100.0 | 98.0 | 1.9 | 3.8 | 28.0 | 54.4 | 92.8
Picasso
PO | 100.0 | 96.0 | 2.6 | 0.2 | 23.2 | 53.3 | 97.4
GA | 100.0 | 98.0 | 1.9 | 2.1 | 0.0 | 694.4 | 0.4
GA+KL | 100.0 | 98.0 | 2.2 | 1.6 | 29.3 | 130.5 | 27.1
SIU | 100.0 | 100.0 | 2.3 | 1.0 | 27.5 | 41.2 | 98.9
Van Gogh
PO | 100.0 | 48.0 | 1.8 | 0.1 | 28.9 | 45.9 | 97.8
GA | 100.0 | 72.0 | 3.3 | 2.8 | 0.0 | 1281.5 | 1.5
GA+KL | 100.0 | 76.0 | 2.4 | 1.9 | 29.3 | 249.6 | 51.1
SIU | 100.0 | 98.0 | 2.3 | 1.7 | 28.6 | 38.7 | 98.1

###  D.1 The Correlation between Utility and the Characteristics of MLLMs’
Output

We suppose the key to our method achieving the best utility (Specificity,
Fluency and Diversity) is that we follow the characteristics of MLLMs’ output.
As stated in Section 4.1 and Appendix A.1, MLLMs tend not to respond _‘I do
not know.’_ when queried about unseen concepts. The characteristics likely
stems from the instruction tuning phase, where the training data will hardly
give a answer of _‘I do not know.’_

Preference Optimization (PO) method, which prompts the model to respond with
"I don’t know," appears to contravene this ingrained output characteristics.
As shown in Figure 11, even though fine-tuning data of PO solely contains _‘I
do not know.’_ and its variants, MLLMs would respond confidentially when
queried about Donald Trump’s appearance in plain text mode. This response does
not reflect actual forgetting of the Trump’s appearance and it seems to sign a
confidentiality agreement with MLLMs. Moreover, as shown in Table 1, though
the EM score of PO is relatively high, low $\mathcal{C}$-Dis of 0.4
illustrates that PO still tends to output a high probability of tokens related
to $\mathcal{C}$. This low distance indicates that it may only learn this
question-and-answer form rather than forget $\mathcal{C}$.

The GA and GA+KL methods frequently exhibit outputs where a single character
is repeated excessively, highlighting a downside of the GA method. Ga method
is more arbitrary in the optimization direction of next token prediction,
which diverges from MLLMs’ typical output characteristics. The breaking of
output characteristics makes the model lose utility after unlearning.

SIU adheres closely to the MLLMs’ output characteristics while effectively
unlearning specific concepts. The high performance of each evaluation metric
shown in Table 1 illustrates a balanced strategy that forgets targeted
unlearning concepts without undermining its inherent capabilities.

Figure 9: The butterfly effects of SIU (1). Figure 10: The butterfly effects
of SIU (2). Figure 11: The response of PO method.

###  D.2 Membership Inference Attack

Min-K% PROB utilizes the minimum token probabilities within a text for
detection purposes. We first gather a set of queries and about the visual
recognition of $\mathcal{C}$. To find the suspicious queries, we compare the
Min-K% PROB between $\mathcal{M}_{\hat{\theta}}$ and $\mathcal{M}_{\theta}$.
If the ratio of scores falls in the range of
$\left(\frac{1}{1.15},1.15\right)$, we regard the query as a suspicious query.
We use $\mathcal{M}_{\hat{\theta}}$ and $\mathcal{M}_{\theta}$ to generate
answers by inputting suspicious queries. Rouge-L is utilized to calculate the
similarity between the generated answers.

Table 7: The performance of each benchmark after unlearning.

Method | GQA | VQA-v2 | VisWiz | SQA I | VQA T | POPE | MMB | Mm-Vet
---|---|---|---|---|---|---|---|---
LLAVA7B
PO | 56.6 | 74.2 | 55.8 | 68.2 | 55.7 | 69.1 | 65.1 | 21.2
GA | 0.0 | 4.4 | 0.0 | 0.0 | 0.0 | 67.1 | 0.3 | 0.0
GA+KL | 61.3 | 76.8 | 51.6 | 65.6 | 56.2 | 83.4 | 64.1 | 20.5
SIU | 58.9 | 75.2 | 54.0 | 65.0 | 55.9 | 85.5 | 62.0 | 28.9
LLAVA13B
PO | 61.4 | 77.7 | 55.8 | 74.0 | 59.6 | 78.1 | 67.6 | 32.3
GA | 62.7 | 78.5 | 55.6 | 65.9 | 60.1 | 82.7 | 68.5 | 31.6
GA+KL | 62.7 | 78.3 | 54.3 | 66.6 | 59.8 | 83.0 | 68.0 | 32.0
SIU | 62.3 | 77.9 | 56.5 | 73.0 | 58.9 | 80.8 | 67.4 | 30.4

###  D.3 Multilingual Jailbreak

We translate the questions in Generality into Spanish, French, Chinese,
German, Japanese and Russian for multilingual jailbreak to check whether the
targeted unlearning concepts are truly forgotten. We use G-Eval to evaluate
the multilingual test. As shown in Figure 12, Figure 13 and Figure 14, we
display the examples in all languages. The original questions are ‘Who is the
blonde in this photo?’, ‘Is this a photo that includes Donald Trump?’ and ‘Who
often wears a suit and tie in this photo?’ respectively for the three figures.
We find GA+KL is more vulnerable to the multilingual jailbreak attack. For
instance, in Figure 13 and Figure 14, GA+KL both generates the name of Donald
Trump in Chinese. It could be observed that SIU may make up some lies as shown
in Figure 14. Rather than ‘Jacob Campbell’, SIU outputs some other names in
different languages. This observation verifies the robustness of SIU under
multilingual jailbreak.

Figure 12: Multilingual jailbreak attacks (1). Figure 13: Multilingual
jailbreak attacks (2). Figure 14: Multilingual jailbreak attacks (3).

###  D.4 Multi-hop Jailbreak

Multi-hop question Jailbreak is a kind of secluded attack and provides hard
examples. We ask about the factual knowledge of $\mathcal{C}$ rather than
directly query the specific name. The questions for this experiment are
generated using GPT-4. For the evaluation we use the accuracy of responding
the right answer of the question. Figure 15 shows examples of multi-hop
jailbreak. It could be seen that PO answers each question with ‘I do not
know…’ As stated in Appendix D.1, PO may only learn the question-and-answer
form rather than forgetting.

Figure 15: Multi-hop jailbreak attacks. Figure 16: Case study of Joe Biden.
Figure 17: Case study of Doodle. Figure 18: Case study of Elon Musk. Figure
19: Case study of Facebook. Figure 20: Case study of Hello Kitty. Figure 21:
Case study of Picasso. Figure 22: Case study of Van Gogh. Figure 23: Prompt
List1. Figure 24: Prompt List2.

##  Appendix E Limitations

The main limitation of our work is the diversity of MLLMs. The reason we only
train LLAVA is stated in Section 6.1. As the construction of MMUBench is aided
by LLAVA including the filtering step, we want to accurately compare the model
response before and after unlearning. Thus we train LLAVA rather than other
MLLMs to conduct the experiments. However, we employ various sizes of LLAVA in
the experiment section to illustrate the impact of model size.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: The contributions and the scope of this paper are accurately reflected in the abstract and introduction.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [Yes]

  9. Justification: See Appendix E.

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [Yes]

  14. Justification: See Section 3 and Appendix A.2.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: We claim the details of methods and the experiments settings in our paper.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes]

  24. Justification: We include the code and data in our supplemental material.

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: The experimental settings are detailed in Section 6.1.

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [Yes]

  34. Justification: See Section 6.2.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: See Section 6.2.

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification: See Appendix A.1. The use of private images has been given explicit consent. These images will not be included in supplementary material to prevent the exposure of personally identifiable information.

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [Yes]

  49. Justification: See Section 1.

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [N/A]

  54. Justification: The paper poses no such risks.

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: See Appendix C.1.

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [Yes]

  64. Justification: See Appendix C.1.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A]

  69. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [Yes]

  74. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.