# Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with
Positive Forward Transfer

Junhao Zheng, Qianli Ma*, Zhen Liu, Binquan Wu, Huawen Feng
School of Computer Science and Engineering,
South China University of Technology, Guangzhou, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>*Corresponding author

###### Abstract

Multimodal Continual Instruction Tuning (MCIT) enables Multimodal Large
Language Models (MLLMs) to meet continuously emerging requirements without
expensive retraining. MCIT faces two major obstacles: catastrophic forgetting
(where old knowledge is forgotten) and negative forward transfer (where the
performance of future tasks is degraded). Although existing methods have
greatly alleviated catastrophic forgetting, they still suffer from negative
forward transfer. By performing singular value decomposition (SVD) on input
embeddings, we discover a large discrepancy in different input embeddings. The
discrepancy results in the model learning irrelevant information for old and
pre-trained tasks, which leads to catastrophic forgetting and negative forward
transfer. To address these issues, we propose Fwd-Prompt, a prompt-based
method projecting prompt gradient to the residual space to minimize the
interference between tasks and to the pre-trained subspace for reusing pre-
trained knowledge. Our experiments demonstrate that Fwd-Prompt achieves state-
of-the-art performance while updating fewer parameters and requiring no old
samples. Our research sheds light on the potential of continuously adapting
MLLMs to new tasks under the instruction tuning paradigm and encourages future
studies to explore MCIT. The code will soon be publicly available.

##  1 Introduction

Recently, instruction tuning has shown effectiveness in bridging the gap
between diverse vision-language tasks and creating general-purpose Multimodal
Large Language Models (MLLMs) with broad capabilities Dai et al. (2023); Liu
et al. (2023). In real-world applications, MLLMs are always expected to learn
new vision-language tasks to support new functionalities. However, existing
MLLMs are static and can not meet continuously emerging new requirements. To
avoid the huge cost of retraining MLLMs, we can formulate this scenario into
the paradigm of Multimodal Continual Instruction Tuning (MCIT). In MCIT, we
aim to instruction-tune MLLMs for new multimodal tasks incrementally while
maintaining superior performance on learned tasks. For example, we start from
InstructBILP Dai et al. (2023), an MLLM instruction tuned on 13 datasets.
Then, we instruction-tune InstructBILP incrementally on Flickr30k Young et al.
(2014), VizWiz Gurari et al. (2018), TextVQA Singh et al. (2019), and GQA
Hudson and Manning (2019). At last, we expect the final model to perform well
on both 4 downstream and 13 pre-trained tasks. Therefore, MCIT is an ideal
scenario that equips MLLMs with new skills incrementally without retraining on
all datasets.

Figure 1: Catastrophic forgetting and negative forward transfer in multimodal
continual instruction-tuning. InstructBLIP Dai et al. (2023) is sequential
instruction-tuned on Flickr30k, VizWiz, TextVQA and GQA.

However, MCIT is particularly challenging. Unlike typical continual learning
scenarios, the tasks in MCIT are diverse in terms of input and output format.
For example, in Flickr30k, the model outputs a detailed description given an
image. In GQA, the model outputs a short answer given an image and a question.
In TextVQA, the model outputs a short answer given an image, a question, and
OCR (Optical Character Recognition) tokens. Therefore, MLLMs are required to
generate various responses according to the task instruction.

In this case, MCIT faces two crucial challenges: catastrophic forgetting and
negative forward transfer. By catastrophic forgetting, we mean models forget
old knowledge when learning new tasks. For instance, in Fig.1, the performance
of Flickr30k (the topmost subfigure) drops significantly when the model is
instruction-tuned on new tasks. By negative forward transfer, we mean that the
performance of unseen tasks is degraded when learning new tasks. For instance,
after instruction-tuning InstructBLIP on Flickr30K, the zero-shot performance
on the unseen tasks (i.e., VizWiz, TextVQA, and GQA) is negatively impacted.
Addressing negative forward transfer is important because it hinders models
from adapting to future tasks. Specifically, the performance of GQA drops from
59.19 to 58.92 after sequential training on Flickr30k, VizWiz and TextVQA
compared to direct training on GQA. More results of negative forward transfer
can be found in the experiment and the Appendix.

Before introducing our methods, let’s take a closer look at the catastrophic
forgetting and negative forward transfer. To achieve anti-forgetting, the old
knowledge should be preserved as much as possible. However, in MCIT, not all
old knowledge is beneficial for new tasks. For instance, OCR-VQA Mishra et al.
(2019) requires models to recognize the text in images locally while Flickr30k
requires to understand the image globally. As shown empirically by He He et
al. (2023), regularizing old models leads to poor performance on new tasks. In
other words, only mitigating catastrophic forgetting may lead to negative
forward transfer. Therefore, to avoid both issues, we need to identify what to
preserve and update for each task.

We perform SVD Golub and Reinsch (1971) on input embeddings to gain more
insight. We discover a large discrepancy between input embeddings from
different tasks. More importantly, the rank of input embeddings decreases when
directly instruction-tuning MLLMs on each new task. However, it increases
under the continual learning paradigm. This phenomenon indicates that the
discrepancy results in models extracting irrelevant input information for old
tasks when adapting to new tasks, which leads to catastrophic forgetting.
Moreover, the negative forward transfer happens because adapting to the
current task results in the model extracting irrelevant information for unseen
tasks.

Motivated by these observations, we propose Fwd-Prompt, which achieves anti-
forgetting and positive forward transfer. Specifically, Fwd-Prompt proposes
selecting prompts according to visual and textual features. On the one hand,
Fwd-Prompt achieves anti-forgetting by allocating different subspaces for each
task and projecting prompt gradient to the residual space. On the other hand,
Fwd-Prompt achieves positive forward transfer by reusing pre-trained knowledge
and projecting prompt gradient to the pre-trained space. Finally, the
experiment indicates that Fwd-Prompt achieve state-of-the-art (SOTA)
performance on MCIT. In summary, our contribution is three folds:

  * •

We identify that the key challenge of MCIT is to achieve both anti-forgetting
and positive forward transfer.

  * •

We analyze the space of input embeddings and reveal that the discrepancy in
input embeddings leads to catastrophic forgetting and negative forward
transfer.

  * •

To address the two issues, we propose Fwd-Prompt for MCIT, which outperforms
SOTA methods by a large margin while requiring less trainable parameters and
no rehearsal data.

##  2 Related Work

###  2.1 Multimodal Large Language Models

The remarkable capabilities of Multimodal Large Language Models (MLLMs) OpenAI
(2023) have captured widespread attention within the artificial intelligence
community, showcasing their proficiency in comprehending, reasoning, and
generating human language. Despite the considerable progress achieved, these
methods face challenges when confronted with more complex multimodal
information Mishra et al. (2019); Sidorov et al. (2020); Hudson and Manning
(2019). Instruction tuning originated from natural language processing and has
become a popular strategy for aligning MLLMs with human intent Touvron et al.
(2023); Chiang et al. (2023). Dai Dai et al. (2023) argue that instruction-
tuned MLLMs generalize to unseen tasks better than multitask learning without
instructions. They further instruction-tune BLIP2 on 13 high-quality vision-
language datasets and proposed InstructBILP. Motivated by InstructBLIP, recent
studies Panagopoulou et al. (2023); Gong et al. (2023); Wu et al. (2023)
further extend instruction-tuning to video, audio, and point cloud modalities.
In this study, we focus on vision-language tasks because of their popularity.

###  2.2 Continual Learning

Although a vast amount of continual learning methods have been proposed in
both computer vision and natural language processing, most of them are not
applied to MCIT because they are either inapplicable to multimodal tasks Sun
et al. (2019) or computationally expensive for MLLMs Buzzega et al. (2020).
Recently, prompt-based methods Wang et al. (2022b); Smith et al. (2023); Wang
et al. (2022a) have achieved superior performance on continual image
classification. These methods harness the power of pre-trained knowledge by
freezing pre-trained models and learning only soft prompts for continual
learning. Inspired by their success in mitigating forgetting, we further
employ the idea of gradient projection Saha et al. (2021) to encourage
positive forward transfer.

###  2.3 MLLMs for Continual Learning

Continual learning with MLLMs is an emerging research field. Existing research
can be divided into two categories: continual learning of visual question
answering (VQA) and continual vison-language pretraining. (1) continual
learning of visual question answering (VQA): Srinivasan Srinivasan et al.
(2022) proposes CLiMB, a continual learning benchmark for vision-language,
vision-only and language-only classification tasks. CL-CrossVQA Zhang et al.
(2022) and VQACL Zhang et al. (2023) are two benchmarks for cross-domain VQA
and VQA with skill-concept compositions, respectively. Qian Qian et al. (2023)
designs special strategies for interactions between modalities. (2) continual
vison-language pretraining: These methods Ni et al. (2023); Zheng et al.
(2023); Zhu et al. (2023) aim at aligning vision and language features. Unlike
continual VQA, the models are trained on image-text pairs and typically
evaluated on retrieval or zero-shot classification tasks.

Unlike the studies above, we explore the continual learning scenario of
instruction tuning. There are two key differences between MCIT and the
aforementioned continual scenario. (1) The MLLM outputs natural language
directly and requires no additional classifiers. (2) MCIT allows MLLMs to
learn diverse tasks beyond VQA, including image captioning, image captioning
with OCR tokens, and VQA with OCR tokens. This study evaluates Fwd-Prompt on
the MCIT benchmark proposed in He et al. (2023).

##  3 Preliminaries

###  3.1 Prompt-Based Continual Learning

Prompt-based continual learning is first proposed by Wang Wang et al. (2022b).
Typically, there is a set of key-value pairs called prompt pool
$\mathcal{P}=\\{k_{j},p_{j}\\}_{j}^{M}$, where $M$ is the number of total key-
value pairs, $k_{j},p_{j}\in\mathbb{R}^{d}$ represent the $j$-th key and
value, respectively. For an input image $x_{i}$, its query feature is computed
as $q_{i}=f^{(img)}(x_{i})$, where $f^{(img)}(\cdot)$ is usually a pre-trained
and frozen Vision Transformer (ViT). Then, we select the top-$n_{p}$ prompts
according to the cosine similarity between $q_{i}$ and all keys in the prompt
pool. Finally, we concatenate the selected prompts to the input embeddings of
$x_{i}$ and feed them into the backbone model such as ViT.

###  3.2 Singular Value Decomposition

Singular Value Decomposition (SVD) Golub and Reinsch (1971) is useful in
linear algebra. Typically, the singular value decomposition of an matrix
$A\in\mathbb{R}^{M\times N}$ is $A=U\Sigma V^{T}$, where
$U\in\mathbb{R}^{M\times M}$ and $V\in\mathbb{R}^{N\times N}$ are two
orthogonal matrices, and $\Sigma\in\mathbb{R}^{M\times N}$ is a rectangular
diagonal matrix. The columns of $U$ and $V$ form two sets of orthonormal bases
called left-singular vectors $\\{u_{1},u_{2},\cdots,u_{M}\\}$ and right-
singular vectors $\\{v_{1},v_{2},\cdots,v_{N}\\}$. The main diagonal of
$\Sigma$ is the singular values $\sigma_{1},\sigma_{2},\cdots,\sigma_{r}$,
where $r<\min\\{M,N\\}$ is the rank of $A$ and the singular values are sorted
in a descending order.

We can regard $A$ as a transformation from $\mathbb{R}^{N}$ to
$\mathbb{R}^{M}$. Then, $v_{1}$ is the unit vector that is maximally stretched
by $A$, i.e., $\arg\max_{x:||x||_{2}=1}||Ax||_{2}=v_{1}$. We provide a brief
proof as follows. First, we convert the objective as follows:
$\arg\max_{x:||x||_{2}=1}||Ax||_{2}=\arg\max_{x:||x||_{2}=1}||Ax||_{2}^{2}=\arg\max_{x:||x||_{2}=1}x^{T}A^{T}Ax$.
Since the eigenvalue expression of $A^{T}A$ is $(A^{T}A)x=\lambda x$, we have
$x^{T}A^{T}Ax=\lambda$. Then, the objective is converted to choose $x$ as the
eigenvector with the largest eigenvalue. Because $A^{T}A=V\Sigma^{2}V^{T}$ and
$V$ is given by the eignedecomposition of $A^{T}A$, $v_{1}$ is exactly the
unit vector that maximizes the objective $||Ax||_{2}$. Similarly, we can prove
that $v_{N}$ is the unit vector minimally stretched by $A$.

Furthermore, we can divide $V$ into two orthogonal subspace
$V_{core}=[v_{1},v_{2},\cdots,v_{K}]$ and
$V_{res}=[v_{K+1},v_{K+2},\cdots,v_{N}]$ by selecting the first $K$ and the
remaining eigenvectors, denoted as the core space and the residual space
respectively. Since $VV^{T}=I$ and $I$ represents the identity matrix, we have
$V_{core}V_{core}^{T}+V_{res}V_{res}^{T}=I$. For an arbitrary vector $x$, we
can decompose it into the core and residual space as $x=x_{core}+x_{res}$,
where $x_{core}=V_{core}V_{core}^{T}x$ and $x_{res}=V_{res}V_{res}^{T}x$.
Then, the output vector of $x_{core}$ (i.e.,$Ax_{core}$) has a large norm,
while the output vector of $x_{res}$ has a small norm. In this way, we can
control the impact of $x$ on the output vector by mapping it into different
subspaces before multiplying with $A$.

(a) Flickr30k (b) VizWiz (c) TextVQA (d) GQA

Figure 2: The contour of the distribution of input embeddings from four
different tasks. “Initial” and “DirectIT” represent the input embeddings of
InstructBLIP before and after instruction tuning on each task, respectively.
Figure 3: The rank of input embeddings in five scenarios. “Initial”: load
InstructBLIP without training; “DirectIT”: train InstructBLIP on the four
datasets respectively; “MultiTask”: train InstructBLIP on all four datasets
jointly; “SEQ”: train InstructBLIP on the four datasets sequentially; “ER”:
store 1 % of old instances and train InstructBLIP sequentially using the
experience replay strategy. “Initial” is a starting point of MCIT. “DirectIT”
and “MultiTask” are both the upper bound of MCIT and “SEQ” is the lower bound.
“ER” is a popular technique for alleviating forgetting.

##  4 Methodology

In Sec.4.1, we qualitatively and quantitatively analyze the input embedding
space in continual learning to gain insights for forgetting and forward
transfer. We describe Fwd-Prompt in Sec.4.2,4.3, and 4.4.

###  4.1 A Closer Look at the Forgetting and Negative Forward Transfer

MCIT is challenging since there is a considerable input and output format
discrepancy across tasks. To understand how this affects model performance, we
visualize the input embeddings of each task before (Initial) and after
(DirectIT) instruction-tuning InstructBLIP on each task. InstructBLIP only
trains Q-Former to align the visual and textual features while keeping the ViT
and LLM frozen. Here, input embeddings refer to the visual and textual
embeddings extracted by Q-Former for feeding into the LLM.

Fig.2 shows that the distribution of input embeddings has a large discrepancy
across tasks. Specifically, we find that Flickr30k has a broader distribution
than the other three VQA-based tasks, suggesting that image captioning
requires detailed information from the input image. It explains that the
performance degradation of VQA-based tasks when learning Flickr30k (Fig.1)
caused by the differences of the information required for each task. Fig.2
also shows that the distribution of “Initial” and “DirectIT” is similar. It
indicates that the required information is relatively stable during
instruction-tuning.

To quantitatively analyze the changes of input embeddings during continual
learning, we estimate the rank of input embeddings with threshold
$\epsilon=0.99$. Fig.3 shows that “DirectIT” and “MultiTask” have a lower rank
than “Initial”. It indicates that all tasks can be jointly learned without
introducing additional input information. However, under continual learning
scenarios (“SEQ” and “ER”), the rank of old tasks (Flickr30k, VizWiz, and
TextVQA) increases significantly. It indicates that the model learns to
extract irrelevant input information for old tasks when adapting to new tasks,
which leads to catastrophic forgetting. From this perspective, the negative
forward transfer happens because adapting to the current task results in the
model extracting irrelevant information for unseen tasks.

(a) Overview of Fwd-Prompt

(b) Gradient Projection in Fwd-Prompt

Figure 4: (a) An overview of Fwd-Prompt. The details of prompt pool and
gradient projection are in Sec.4.2 and 4.4. (b) An illustration of the
gradient projection in Fwd-Prompt.

###  4.2 Prompt-Based Continual Learning for MCIT

We provide an overview of Fwd-Prompt in Fig.4a. This subsection describes how
to design a prompt pool for multimodal tasks.

Intuitively, it is easier for the model to output the correct answer if task-
specific descriptions such as “This task requires reading the text on books”
or instance-specific descriptions such as “This photo was taken at a fast food
restaurant” are provided. Therefore, we follow L2P Wang et al. (2022b) and
design a prompt pool where soft prompts are selected dynamically according to
each instance. In MCIT, the output is determined by both the input image and
instruction. For instance, given the same input image in Fig.4a, the expected
output would be “Cola” if the instruction is changed to “What is the name of
the drink on the upper left corner?”. Therefore, each prompt should be
determined by both image and instruction. To achieve this, we design two keys
for each prompt and the prompt pool is defined as
$\mathcal{P}=\\{k^{(img)}_{j},k^{(text)}_{j},p_{j}\\}_{j}^{M}$, where $M$ is
the number of total key-value pairs, $p_{j}\in\mathbb{R}^{d}$ represent the
$j$-th value, $k^{(img)}_{j}\in\mathbb{R}^{di}$ and
$k^{(text)}_{j}\in\mathbb{R}^{dt}$ represent the key for image and instruction
respectively. $d$, $d_{i}$ and $d_{t}$ are the dimensions of input embeddings,
visual and textual features, respectively. For the $i$-th instance containing
an image $x_{i}^{(img)}$ and instruction $x_{i}^{(text)}$, its similarity with
the $j$-th prompt is calculated as

$\phi_{i,j}=\phi(q_{i}^{(img)},k^{(img)}_{j})+\phi(q_{i}^{(text)},k^{(text)}_{j}),$
(1)

where $q_{i}^{(img)}$ is the query of $x_{i}^{(img)}$ and is obtained by
averaging the features from a frozen vision encoder such as ViT-g/14 Fang et
al. (2023) used in InstructBLIP. Similarly, $q_{i}^{(text)}$ is the query of
$x_{i}^{(text)}$ and is obtained by averaging the input embeddings of LLM such
as Flan-T5 Chung et al. (2022) used in InstructBLIP.
$\phi(q_{i}^{(img)},k^{(img)}_{j})$ is computed as the cosine similarity
between $q_{i}^{(img)}$ and $k^{(img)}_{j}$ normalized over all keys in the
prompt pool with softmax function. And $\phi(q_{i}^{(text)},k^{(text)}_{j})$
is computed in a similar way.

###  4.3 Anti-Forgetting for Prompt Tuning

This section describes how to achieve anti-forgetting when utilizing prompt
tuning for continual learning.

To achieve the goal of anti-forgetting, we expect the model to output the same
response given the same image and instruction when soft prompts are updated.
Formally, we have

$f_{LLM}([p_{t+1}^{T};e^{(img)}_{t};e^{(text)}_{t}])=f_{LLM}([p_{t}^{T};e^{(img)}_{t};e^{(text)}_{t}]),$
(2)

where $f_{LLM}$ represents the large language model, and it is frozen during
continual learning. $p_{t+1},p_{t}\in\mathbb{R}^{d}$ are the prompts before
and after the update of task $t$. We consider only one prompt for
demonstration purposes. $e^{(img)}_{t}\in\mathbb{R}^{n_{i}\times
d}$,$e^{(text)}_{t}\in\mathbb{R}^{n_{t}\times d}$ represent the $n_{i}$,
$n_{t}$ input embedding tokens corresponding to the input image and
instruction from task $t$. $[p_{t+1}^{T};e^{(img)}_{t};e^{(text)}_{t}]$
represents that the prompt is concatenated with the input along the sequence
dimension. To achieve Eq.2, the prompt and input embeddings should satisfy the
following relationship:

(5)

$\Delta p=p_{t+1}-p_{t}$ can be regarded as the gradient of the prompt. We
provide the detailed derivation in the Appendix. Eq.4.3 indicates that the
updated prompt should have the same L2 norm. In practice, the learned prompts
have relatively close L2 norms, and thus, the first constraint is ignored in
our implementation. The latter two constraints in Eq.LABEL:eq:anti-
forgetting-2 and 5 indicate that the prompt gradient should be orthogonal to
the input embeddings of both image and instruction. To satisfy these
constraints, we perform SVD on the concatenated input embeddings
$e_{t}^{(joint)}=[e_{t}^{(img)};e_{t}^{(text)}]\in\mathbb{R}^{(n_{i}+n_{t})\times
d}$ and obtain $d$ right singular vectors $\\{v_{1},v_{2},\cdots,v_{d}\\}$.
Then, select the first $K$ right singular vectors to form the core space of
task $t$ that satisfy
$||\widetilde{e_{t}^{(joint)}}||\geq\epsilon||e_{t}^{(joint)}||$, where
$\widetilde{e_{t}^{(joint)}}$ is the K-rank approximation of
$e_{t}^{(joint)}$, $||\cdot||$ represent the Frobenius norm, and
$\epsilon=0.99$. The remaining $d-K$ right singular vectors form the residual
space of task $t$. After obtaining the prompt gradient $\Delta p$, we project
$\Delta p$ to the residual space as follows: $\Delta
p^{\prime}=V_{res}V_{res}^{T}\Delta p,$ where
$V_{res}=[v_{k+1},v_{k+2},\cdots,v_{d}]\in\mathbb{R}^{d\times(d-K)}$. By
mapping the prompt gradient into the residual subspace, the forgetting of the
knowledge of task $t$ is minimized.

###  4.4 Anti-Forgetting and Positive Forward Transfer with Gradient
Projection

In this subsection, we will extend the idea in Sec.4.3 and achieve positive
forward transfer beyond anti-forgetting. The fundamental intuition is to reuse
the pre-trained knowledge during continual learning to avoid extracting
irrelevant input information for unseen tasks as described in Sec.4.1. An
illustration for gradient projection in Fwd-Prompt is provided in Fig.4b.

Specifically, we regard the pre-trained tasks before continual learning as
task $0$ and identify its pre-trained space and residual space using $K$-rank
approximation as described in Sec.4.3. We denote the pre-trained space as
$V_{pre}\in\mathbb{R}^{d\times K_{0}}$ and the residual space as
$V_{res}\in\mathbb{R}^{d\times(d-K_{0})}$.

When adapting to task $1$, we train the prompt pool without gradient
projection. After finishing the training on task $1$, we perform SVD on the
input embeddings from task $1$ and obtain the core space of task $1$ (denoted
as $V_{core}^{1}$) by $K$-rand approximation. Then, we record the indexes of
directions in the pre-trained space that conflicts with the core space of task
$1$. Formally, the conflicting index set $\mathcal{I}^{1}$ for task $1$ is
calculated as follows:

$\mathcal{I}^{1}=\\{i|\exists
j,|cosine(V_{pre}[:,i],V_{core}^{1}[:,j])|>\theta\\},$ (6)

where $V_{pre}[:,i]$,$V_{core}^{1}[:,j]$ represent the $i$-th and $j$-th
column vectors in $V_{pre}$ and $V_{core}^{1}$ respectively. $\theta$ is the
threshold for determining if two directions are overlapping.
$cosine(\cdot,\cdot)$ computes the cosine similarity between two vectors. And
then, we define the conflicting space of task 1 as follows:

$V_{con}^{1}=\\{V_{pre}[:,i]\\}_{i\in\mathcal{I}^{1}}.$ (7)

When adapting to task $2$, the prompt gradient is projected to the orthogonal
direction of $V_{con}^{1}$ to avoid forgetting the knowledge in task $1$:

$\Delta p^{\prime}=(I-V_{con}^{1}{V_{con}^{1}}^{T})\Delta p,$ (8)

After finishing training on task $2$, we find the conflicting index set
$\mathcal{I}^{2}$ for task $2$ in a similar way. Similarly, when adapting to
task $3$, the prompt gradient is computed as follows:

$\Delta p^{\prime}=(I-V_{con}^{1,2}{V_{con}^{1,2}}^{T})\Delta p.$ (9)
$V_{con}^{1,2}=\\{V_{pre}[:,i]\\}_{i\in\mathcal{I}^{1}\cup{I}^{2}}.$ (10)

In summary, Fwd-Prompt avoids interference between new tasks by allocating
different conflicting spaces and projecting gradients to the residual space,
thus achieving anti-forgetting. Furthermore, Fwd-Prompt reuses pre-trained
knowledge by updating prompts in the pre-trained space, which contains the
core space of unseen tasks, thus enhancing positive forward transfer.

Table 1: Comparison with SOTA methods for MCIT. The average accuracy after
learning the final task is reported, i.e. $\mathcal{A}_{4}$. The average
accuracy of each incremental step and other metrics are reported in Fig.5. The
backbone model is InstructBLIP. “ER” and “EProj” require storing 1% of old
samples for rehearsal.

Datasets | Non-CL Baselines | CL Baselines
---|---|---
Initial | DirectIT | SeqIT | EWC | EWC_TIR | AGEM | ER | EProj | Fwd-Prompt (Ours)
Caption Coco | 140.81 | 140.81 | 115.85 | 128.27 | 130.77 | 121.89 | 135.00 | 134.99 | 137.58
TextCaps | 127.08 | 127.08 | 86.02 | 120.15 | 121.71 | 114.72 | 122.75 | 126.75 | 125.50
VQAv2 | 73.47 | 73.47 | 61.08 | 71.54 | 71.96 | 68.31 | 67.06 | 68.03 | 72.98
OKVQA | 48.84 | 48.84 | 30.55 | 44.77 | 45.44 | 40.76 | 37.49 | 41.12 | 48.68
AOKVQA | 55.31 | 55.31 | 38.52 | 53.16 | 53.51 | 44.88 | 45.68 | 51.85 | 53.97
OCR-VQA | 62.37 | 62.37 | 57.84 | 60.28 | 62.30 | 60.37 | 61.38 | 62.36 | 61.78
Flickr30k | 84.87 | 97.74 | 79.91 | 92.10 | 94.70 | 90.68 | 95.98 | 97.99 | 97.53
VizWiz | 30.65 | 64.58 | 21.54 | 31.14 | 33.53 | 37.83 | 46.36 | 45.05 | 62.79
TextVQA | 44.72 | 51.10 | 42.28 | 48.09 | 49.38 | 47.25 | 49.19 | 47.58 | 50.08
GQA | 48.38 | 59.19 | 58.92 | 50.26 | 51.45 | 56.31 | 59.02 | 54.06 | 60.52
Average | 71.65 | 78.05 | 59.25 | 69.98 | 71.48 | 68.30 | 71.99 | 72.97 | 77.14
Improvement | / | / | 0.00 | +10.73 | +12.22 | +9.05 | +12.74 | +13.73 | +17.89
Trainable Params | / | / | 187M | 187M | 187M | 187M | 187M | 189M | 48.5M
Use Old Samples | / | / | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗

##  5 Experiment

###  5.1 Experimental Settings

Datasets. We evaluate Fwd-Prompt on the benchmark proposed by He He et al.
(2023). We consider two MLLMs, InstructBLIP (FlanT5XL) model and BLIP2
(FlanT5XL) Li et al. (2023b) for continual learning. The experimental result
of BLIP2 is in the Appendix. When using InstructBLIP as the starting point,
the task order is Flickr30K, VizWiz, TextVQA, and GQA. Besides, we expect
InstructBLIP to preserve the pre-trained knowledge obtained during joint
instruction-tuning before continual learning. Therefore, we evaluate
InstructBLIP on six pre-trained tasks: Caption COCO Chen et al. (2015),
TextCaps Sidorov et al. (2020), VQAv2 Goyal et al. (2017), OKVQA Marino et al.
(2019), A-OKVQA Schwenk et al. (2022), OCR-VQA Mishra et al. (2019).

Metric. We adopt top-1 accuracy as the metric for VQA-based tasks and CIDEr
for image-captioning tasks. We report three metrics for continual learning,
including average accuracy ($\mathcal{A}_{t}$, higher is better), forgetting
(${FGT}_{t}$, lower is better), and forward transfer (${FWD}_{t}$, higher is
better) after learning task $t$. Specifically,
$\mathcal{A}_{t}=\frac{1}{t}\Sigma_{i=1}^{t}a_{t,i},$ where $a_{t,i}$ is the
performance score of task $i$ after learning task $t$. The forgetting is
computed as the average performance degradation of all learned tasks:
${FGT}_{t}=\frac{1}{t-1}\Sigma_{i=1}^{t-1}[max_{j<t}(\\{a_{j,i}\\}_{j})-a_{t,i}]$
The forward transfer is computed as the difference between continual learning
and direct instruction-tuning on task $t$: ${FWD}_{t}=a_{t,t}-a_{t}$ When
${FWD}_{t}>0$, it implies positive forward transfer for learning task $t$.
When ${FWD}_{t}<0$, it implies negative forward transfer for learning task
$t$.

Hyper-Parameters. We set the number of key-values pairs $M=20$, the threshold
for $k$-rank approximation $\epsilon=0.99$, the threshold for finding matching
directions $\theta=0.5$, the number of selected prompts for input $n_{p}=5$.

Baselines. We consider 6 baselines for continual learning, including
sequential instruction-tuning (SeqIT), EWC Kirkpatrick et al. (2017), EWC_TIR
He et al. (2023), AGEM Chaudhry et al. (2019), experience replay (ER), EProj
He et al. (2023). EProj is the current state-of-the-art (SOTA) method for
MCIT. Besides, we compare two non-continual learning baselines: InstructBLIP
before continual learning (Initial) and instruction-tuned on each new task,
respectively (DirectIT). SeqIT and DirectIT are the lower and upper bounds for
continual learning.

Implementation Details. We use the implementation of InstructBLIP, and BLIP2
provided in the LAVIS library Li et al. (2023a). We train GQA for 1 epoch and
other tasks for 5 epochs with an initial learning rate of 1e-5. The batch size
is set as 32. In EProj, the projection layer is expanded for each new task and
task keys are learned for selecting task ID during inference. For a fair
comparison, Fwd-Prompt follows EProj and additionally learns new projection
layers and task keys. Due to the flexibility of prompt tuning, Fwd-Prompt only
trains the top 3 layers in Q-Former rather than the entire Q-Former as it does
in InstructBLIP. More implementation details are provided in the Appendix.

(a) Average Accuracy $\mathcal{A}_{t}\mathbf{\uparrow}$ (b) Forgetting
${FGT}_{4}\mathbf{\downarrow}$ (c) Forward Transfer
${FWD}_{4}\mathbf{\uparrow}$

Figure 5: Comparison with SOTA methods in terms of average accuracy,
forgetting and forward transfer.

###  5.2 Result and Analysis

Comparison with SOTA Methods. Table 1 and Fig.5 demonstrate the SOTA
performance of Fwd-Prompt. Despite storing no old samples and training fewer
parameters, Fwd-Prompt outperforms the previous SOTA method by 4.16%. It is
worth noting that regularization-based methods such as EWC, EWC_TIR, and AGEM
suffer from negative forward transfer greatly, while replay-based methods
suffer from either catastrophic forgetting or negative forward transfer. Fwd-
Prompt harnesses the power of LLM with prompt tuning and enhances positive
forward transfer with gradient projection.

Table 2: The ablation study and hyper-parameter analysis for Fwd-Prompt.

Method | $\mathcal{A}_{4}$ | $\Delta$
---|---|---
Ours | Fwd-Prompt | 77.14 | 0.00
Ablation | w/o Prompt Pool | 63.59 | -13.55
w/o Gradient Projection | 74.82 | -2.32
Variation | Separated Prompt Pools | 74.71 | -2.43
More Prompt Tokens $n_{p}=10$ | 76.85 | -0.29
Larger Prompt Pool $M=40$ | 77.31 | +0.17
Both $M=40,n_{p}=10$ | 76.89 | -0.25
Threshold $\epsilon=0.90$ | 75.93 | -1.21
Threshold $\epsilon=0.95$ | 76.50 | -0.64

Ablation Study and Hyper-Parameter Analysis. We consider two ablated versions
and six variations for Fwd-Prompt. Table 2 shows that both the prompt pool and
the gradient projection are the keys to the success of Fwd-Prompt. “Separated
Prompt Pools” represents training two prompt pools separately for vision and
instruction input. The results show that “Separated Prompt Pools” leads to
much worse performance, suggesting selecting prompts with information from
both modalities is crucial. Besides, the results show that a larger prompt
pool leads to better performance while prepending more prompt tokens does not.
We speculate the reason is that more prompt tokens are harder to train.

(a) Prompt Selection Frequency

(b) Core Space

(c) Subspace Allocation

Figure 6: The forward transfer in Fwd-Prompt. (a) The histogram of prompt
selection frequency. (b) The Venn diagram of the column vectors in the core
space. (c) The subspace allocation in Fwd-Prompt.

Visualization of Forward Transfer. Fig.6 visualizes the forward transfer in
Fwd-Prompt. As shown in Fig.6a, each task corresponds to a non-overlapping set
of prompts, and the interaction across tasks is weak. This is because the gap
between tasks is large, and the model learns task-specific information during
training. Our finding is consistent with L2P Wang et al. (2022b). Fig.6b shows
a considerable overlap in the core space of different tasks. Fwd-Prompt avoids
conflict between tasks by allocating different conflicting spaces to different
tasks. Fig.6c visualizes the subspace allocation in Fwd-Prompt. For example,
when the model learns the first task, Flickr30k, the conflicting space 1 is
all 48 directions in the core space of Flickr30k. When the model learns the
second task, VizWiz, the conflicting space 2 is the direction in the core
space of VizWiz minus those of Flickr30k. In this way, Fwd-Prompt avoids the
interference between new tasks and thus achieves anti-forgetting. Furthermore,
Fwd-Prompt enhances positive forward transfer by updating prompts in the pre-
trained space, which contains the core space of unseen tasks.

Figure 7: Comparison between Fwd-Prompt and EProj after learning the final
task in MCIT. The upper and lower rows are instances from the dev set of a
pre-trained task, Caption COCO, and a new task in continual learning, VizWiz.

Case Study. As shown in Fig.7, the responses of Fwd-Prompt in two instances
are more visually grounded than EProj. It indicates that Fwd-Prompt better
preserves knowledge gained before and after continual learning.

Table 3: Comparison with SOTA methods in training time of continual learning.

Method | Run Time (min) | Ratio | Method | Run Time (min) | Ratio
---|---|---|---|---|---
SeqIT | 502 | $\times 1.0$ | EWC | 745 | $\times 1.5$
EWC_TIR | 768 | $\times 1.5$ | A-GEM | 1047 | $\times 2.1$
ER | 945 | $\times 1.9$ | EProj | 972 | $\times 1.9$
Fwd-Prompt | 660 | $\times 1.3$ |  |  |

Run Time Analysis. The comparison of training time is shown in Table 3.
Although Fwd-Prompt requires an additional 30% training time than SeqIT, it
requires less training time than other continual learning baselines,
demonstrating its superior practicability in real-world scenarios.

##  6 Conclusion and Limitation

In this paper, we analyze the input embeddings and reveal that the conflict in
input embeddings leads to forgetting and negative forward transfer. To this
end, we present Fwd-Prompt, a novel approach to addressing the critical
challenges of catastrophic forgetting and negative forward transfer in MCIT.
The experimental results show that Fwd-Prompt outperforms SOTA methods
significantly while requiring less training time, trainable parameters and no
old samples.

There are two limitations of this research: (1) This research does not fully
explain the overlapping knowledge between tasks; (2) The number of tasks in
MCIT is limited.

## References

  * Boix-Adsera et al. [2023] Enric Boix-Adsera, Etai Littwin, Emmanuel Abbe, Samy Bengio, and Joshua Susskind.  Transformers learn through gradual rank increase.  arXiv preprint arXiv:2306.07042, 2023.
  * Buzzega et al. [2020] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  Advances in neural information processing systems, 33:15920–15930, 2020.
  * Chaudhry et al. [2019] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-gem.  In International Conference on Learning Representations, 2019.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  arXiv preprint arXiv:2210.11416, 2022.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  * Fang et al. [2023] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358–19369, 2023.
  * Golub and Reinsch [1971] Gene H Golub and Christian Reinsch.  Singular value decomposition and least squares solutions.  In Handbook for Automatic Computation: Volume II: Linear Algebra, pages 134–151. Springer, 1971.
  * Gong et al. [2023] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen.  Multimodal-gpt: A vision and language model for dialogue with humans.  arXiv preprint arXiv:2305.04790, 2023.
  * Goyal et al. [2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  * Gurari et al. [2018] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham.  Vizwiz grand challenge: Answering visual questions from blind people.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018.
  * He et al. [2023] Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang.  Continual instruction tuning for large multimodal models.  arXiv preprint arXiv:2311.16206, 2023.
  * Hudson and Manning [2019] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * Li et al. [2023a] Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven CH Hoi.  Lavis: A one-stop library for language-vision intelligence.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 31–41, 2023.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv preprint arXiv:2304.08485, 2023.
  * Marino et al. [2019] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019.
  * Mishra et al. [2019] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019.
  * Ni et al. [2023] Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian.  Continual vision-language representation learning with off-diagonal information.  2023\.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Panagopoulou et al. [2023] Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles.  X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning.  arXiv preprint arXiv:2311.18799, 2023.
  * Qian et al. [2023] Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, and Wenwu Zhu.  Decouple before interact: Multi-modal prompt learning for continual visual question answering.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2953–2962, 2023.
  * Saha et al. [2021] Gobinda Saha, Isha Garg, and Kaushik Roy.  Gradient projection memory for continual learning.  In International Conference on Learning Representations, 2021.
  * Schwenk et al. [2022] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  In European Conference on Computer Vision, pages 146–162. Springer, 2022.
  * Sidorov et al. [2020] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.  Textcaps: a dataset for image captioning with reading comprehension.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020.
  * Singh et al. [2019] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019.
  * Smith et al. [2023] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira.  Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909–11919, 2023.
  * Srinivasan et al. [2022] Tejas Srinivasan, Ting-Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason.  Climb: A continual learning benchmark for vision-and-language tasks.  Advances in Neural Information Processing Systems, 35:29440–29453, 2022.
  * Sun et al. [2019] Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee.  Lamol: Language modeling for lifelong language learning.  arXiv preprint arXiv:1909.03329, 2019.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * Wang et al. [2022a] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al.  Dualprompt: Complementary prompting for rehearsal-free continual learning.  In European Conference on Computer Vision, pages 631–648. Springer, 2022.
  * Wang et al. [2022b] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.  Learning to prompt for continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139–149, 2022.
  * Wu et al. [2023] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  arXiv preprint arXiv:2309.05519, 2023.
  * Young et al. [2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  Transactions of the Association for Computational Linguistics, 2:67–78, 2014.
  * Zhang et al. [2022] Yao Zhang, Haokun Chen, Ahmed Frikha, Yezi Yang, Denis Krompass, Gengyuan Zhang, Jindong Gu, and Volker Tresp.  Cl-crossvqa: A continual learning benchmark for cross-domain visual question answering.  arXiv preprint arXiv:2211.10567, 2022.
  * Zhang et al. [2023] Xi Zhang, Feifei Zhang, and Changsheng Xu.  Vqacl: A novel visual question answering continual learning setting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19102–19112, 2023.
  * Zheng et al. [2023] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You.  Preventing zero-shot transfer degradation in continual learning of vision-language models.  arXiv preprint arXiv:2303.06628, 2023.
  * Zhu et al. [2023] Hongguang Zhu, Yunchao Wei, Xiaodan Liang, Chunjie Zhang, and Yao Zhao.  Ctp: Towards vision-language continual pretraining via compatible momentum contrast and topology preservation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22257–22267, 2023.

##  Appendix A Derivation of Anti-Forgetting for Prompt Tuning

To achieve the goal of anti-forgetting, we expect the model to output the same
response given the same image and instruction when soft prompts are updated.
Formally, we have

$f_{LLM}([p_{t+1}^{T};e^{(img)}_{t};e^{(text)}_{t}])=f_{LLM}([p_{t}^{T};e^{(img)}_{t};e^{(text)}_{t}]),$
(42)

where $f_{LLM}$ represents the large language model, and it is frozen during
continual learning. $p_{t+1},p_{t}\in\mathbb{R}^{d}$ are the prompts before
and after the update of task $t$. We consider only one prompt for
demonstration purposes. $e^{(img)}_{t}\in\mathbb{R}^{n_{i}\times
d}$,$e^{(text)}_{t}\in\mathbb{R}^{n_{t}\times d}$ represent the $n_{i}$,
$n_{t}$ input embedding tokens correspond to the input image and instruction
from task $t$. $[p_{t+1};e^{(img)}_{t};e^{(text)}_{t}]$ represents that the
prompt is concatenated with the input along the sequence dimension. We denote
the concatenated input $[p_{t+1};e^{(img)}_{t};e^{(text)}_{t}]$ and
$[p_{t};e^{(img)}_{t};e^{(text)}_{t}]$ with a prompt token from task $t$ and
task $t+1$ as $Z_{t+1}\in\mathbb{R}^{(1+n_{i}+n_{t})\times d}$ and
$Z_{t}\in\mathbb{R}^{(1+n_{i}+n_{t})\times d}$ respectively.

The input embedding tokens primarily interact with each other in the self-
attention layer of LLMs. Therefore, we simplify the objective in Eq. 42 as
follows:

$Attention(Z_{t+1})=Attention(Z_{t})$ (43)

$A_{t+1}$ and $A_{t}$ are the attention matrix of input $Z_{t+1}$ and $Z_{t}$.
Then, we expand the attention matrix as follows:

$\displaystyle Attention(Z_{t})$
$\displaystyle=softmax(\frac{Z_{t}W_{Q}(Z_{t}W_{K})^{T}}{\sqrt{d_{h}}})$ (44)
$\displaystyle=softmax(\frac{Z_{t}W_{Q}W_{K}^{T}Z_{t}^{T}}{\sqrt{d_{h}}})$
(45)

where $W_{Q}\in\mathbb{R}^{d\times d_{h}}$ and $W_{K}\in\mathbb{R}^{d\times
d_{h}}$ are the projection matrix of query and key, $softmax$ represents row-
wise softmax function, $d_{h}$ is the hidden dimension. In Eq.45, $W_{Q}$ and
$W_{K}$ are unchanged during training since LLMs are frozen. To further
simplify the objective, we follow Boix-Adsera et al. [2023] and assume the
query and key matrices $W_{Q}$ and $W_{K}$ are diagonal weight matrices and
$d_{h}=d$:

$W_{Q}W_{K}^{T}=diag(w_{Q})diag(w_{K})^{T},$ (46)

where $w_{Q},w_{K}\in\mathbb{R}^{d}$ are the diagonal entries of $W_{Q}$ and
$W_{K}$. Then, the objective in Eq.43 can be simplified as follows:

$Z_{t+1}Z_{t+1}^{T}=Z_{t}Z_{t}^{T}$ (47)

And then, we expand $Z_{t+1}$ and $Z_{t}$ and Eq.47 can be converted as
follows:

$\displaystyle\begin{bmatrix}p_{t+1}^{T}\\\ e^{(img)}_{t}\\\
e^{(text)}_{t}\end{bmatrix}\begin{bmatrix}{p_{t+1}}&{e^{(img)}_{t}}^{T}&{e^{(text)}_{t}}^{T}\end{bmatrix}=$
(48) $\displaystyle\begin{bmatrix}p_{t}^{T}\\\ e^{(img)}_{t}\\\
e^{(text)}_{t}\end{bmatrix}\begin{bmatrix}{p_{t}}^{T}&{e^{(img)}_{t}}^{T}&{e^{(text)}_{t}}^{T}\end{bmatrix}$

$\displaystyle\begin{bmatrix}p_{t+1}^{T}{p_{t+1}}&p_{t+1}^{T}{e^{(img)}_{t}}^{T}&p_{t+1}^{T}{e^{(text)}_{t}}^{T}\\\
e^{(img)}_{t}{p_{t+1}}&e^{(img)}_{t}{e^{(img)}_{t}}^{T}&e^{(img)}_{t}{e^{(text)}_{t}}^{T}\\\
e^{(text)}_{t}{p_{t+1}}&e^{(text)}_{t}{e^{(img)}_{t}}^{T}&e^{(text)}_{t}{e^{(text)}_{t}}^{T}\end{bmatrix}=$
(49)
$\displaystyle\begin{bmatrix}p_{t}^{T}{p_{t}}&p_{t}^{T}{e^{(img)}_{t}}^{T}&p_{t}^{T}{e^{(text)}_{t}}^{T}\\\
e^{(img)}_{t}{p_{t}}&e^{(img)}_{t}{e^{(img)}_{t}}^{T}&e^{(img)}_{t}{e^{(text)}_{t}}^{T}\\\
e^{(text)}_{t}{p_{t}}&e^{(text)}_{t}{e^{(img)}_{t}}^{T}&e^{(text)}_{t}{e^{(text)}_{t}}^{T}\end{bmatrix}$

By comparing the two matrix element-wise in Eq.49, we can simplify it as
follows:

(52)

We denote the difference between $p_{t+1}$ and $p_{t}$ as $\Delta p$, i.e.,
$\Delta p=p_{t+1}-p_{t}$. Finally, the objective in Eq.42 is summarized as
follows:

(55)

##  Appendix B Additional Experimental Results

When using InstructBLIP and BLIP2 Li et al. [2023b] as MLLMs, the task
sequence for continual learning is summarized in Table 4 and 5 respectively.
We refer readers to InstructBLIP Dai et al. [2023] for detailed description of
datasets. We sample 100 instances for each dataset in task $0$ of InsturctBLIP
to obtain the pre-trained and residual space. For BLIP2, we obtain the pre-
trained and residual space on the input embeddings of task $1$. The hyper-
parameters used for training is summarized in Table 7. For training, we use
the same instruction templates in InstructBLIP Dai et al. [2023]. For
inference, we use the instruction templates summarized in Table 6. We only
evaluate BLIP2 on the 5 tasks learned during continual learning since BLIP2
has not been instruction-tuned on multiple datasets.

The results on BLIP2 are summarized in Table 8 and Fig.8. Table 8 shows that
the previous SOTA method EProj has even worse performance than ER while Fwd-
Prompt outperforms ER by 3.20%. It indicates Fwd-Prompt is more robust than ER
when training on different MLLMs. Furthermore, Fig.8 shows that Fwd-Prompt has
the lowest forgetting and highest forward transfer. Although EProj also
alleviates catastrophic forgetting, it leads to negative forward transfer,
degrading the performance on GQA from 56.86 to 50.48. In contrast, Fwd-Prompt
encourages positive forward transfer and improves the performance on GQA from
56.86 to 58.80.

Table 4: The dataset statistics when continual instruction-tuning
InstructBLIP.

Task Order | Dataset | Task Description | # Training Set | # Dev Set | Metric | Source
---|---|---|---|---|---|---
0 | Caption COCO Chen et al. [2015] | Image Captioning | 567K | 5K | CIDEr | Link
TextCaps Sidorov et al. [2020] | Image Captioning with OCR tokens | 549K | 16K | CIDEr | Link
VQAv2 Goyal et al. [2017] | VQA | 443K | 214K | ACC | Link
OKVQA Marino et al. [2019] | VQA | 9K | 5K | ACC | Link
A-OKVQA Schwenk et al. [2022] | VQA | 17K | 1K | ACC | Link
OCR-VQA Mishra et al. [2019] | VQA | 801K | 100K | ACC | Link
1 | Flickr30k Young et al. [2014] | Image Captioning | 145K | 1K | CIDEr | Link
2 | VizWiz Gurari et al. [2018] | VQA | 20K | 4K | ACC | Link
3 | TextVQA Singh et al. [2019] | VQA with OCR tokens | 34K | 5K | ACC | Link
4 | GQA Hudson and Manning [2019] | VQA | 94K | 13K | ACC | Link

Table 5: The dataset statistics when continual instruction-tuning BLIP2.

Task Order | Dataset | Task Description | # Training Set | # Dev Set | Metric | Source
---|---|---|---|---|---|---
1 | Flickr30k Young et al. [2014] | Image Captioning | 145K | 1K | CIDEr | Link
2 | TextCaps Sidorov et al. [2020] | Image Captioning with OCR tokens | 549K | 16K | CIDEr | Link
3 | VQAv2 Goyal et al. [2017] | VQA | 443K | 214K | ACC | Link
4 | OCR-VQA Mishra et al. [2019] | VQA | 801K | 100K | ACC | Link
5 | GQA Schwenk et al. [2022] | VQA | 94K | 13K | ACC | Link

Table 6: The instruction templates for Inference.

Datasets | Instruction Templates
---|---
VQAv2, OKVQA, A-OKVQA, OCR-VQA, VizWiz, GQA |  $<$Image$>$ Question: {} Short answer:
Flickr30k, Caption COCO, |  $<$Image$>$ A short image description:
TextVQA |  $<$Image$>$ OCR tokens: {}. Question: {} Short answer:
TextCaps |  $<$Image$>$ OCR tokens: {}. A short image description:

Table 7: The hyper-parameters used for training Fwd-Prompt.

Hyper-Parameters | Value
---|---
max epoch | 1 for GQA and 5 for others
initial lr | 1.00E-05
warmup lr | 1.00E-08
warmup steps | 1000
weight decay | 0.05
batch size | 32
num beams | 5
output len | VQA-based Tasks: max_len=10, min_len=1
Image Captioning Tasks: max_len = 80, min_len = 10

Table 8: Comparison with SOTA methods for MCIT. The average accuracy after
learing the final task, i.e. $\mathcal{A}_{5}$, is reported. The backbone
model is BLIP2. “ER” and “EProj” require storing 1% of old samples for
rehearsal.

Datasets | Non-CL Baselines | CL Baselines
---|---|---
Initial | DirectIT | SeqIT | EWC_TIR | ER | EProj | Fwd-Prompt (Ours)
Flickr30k | 80.77 | 99.27 | 36.12 | 66.00 | 94.35 | 98.14 | 98.09
TextCaps | 72.04 | 130.67 | 26.96 | 68.35 | 123.85 | 126.69 | 128.65
VQAv2 | 62.79 | 73.44 | 60.75 | 63.61 | 65.97 | 65.98 | 72.35
OCR-VQA | 42.07 | 63.09 | 45.80 | 47.44 | 62.22 | 62.37 | 62.81
GQA | 43.80 | 56.86 | 58.20 | 49.38 | 58.32 | 50.48 | 58.80
Average | 60.29 | 84.67 | 45.57 | 58.96 | 80.94 | 80.73 | 84.14
Improvement | / | / | 0.00 | +13.39 | +35.37 | +35.16 | +38.57

(a) Average Accuracy $\mathcal{A}_{t}\mathbf{\uparrow}$ (b) Forgetting
${FGT}_{5}\mathbf{\downarrow}$ (c) Forward Transfer
${FWD}_{5}\mathbf{\uparrow}$

Figure 8: Comparison with SOTA methods in terms of average accuracy,
forgetting and forward transfer. The backbone model is BLIP2.