\begin{table*}[]
    \centering
\resizebox{0.8\textwidth}{!}{
    \begin{tabular}{lccccccc}
\hline
\multicolumn{2}{c}{\textbf{Reference}} & \multicolumn{3}{c}{\textbf{Compared Answer 1}} & \multicolumn{3}{c}{\textbf{Compared Answer 2}} \\
\textbf{Answer}	& \textbf{Acc}	& \textbf{Answer} &	\textbf{Acc} & \textbf{SBWT} & \textbf{Answer} & \textbf{Acc} & \textbf{SBWT} \\\hline
skateboarding & 1 & skateboard & 0 & -0.164 & black & 0 & -0.836 \\
snowboarding  & 1 & skiing & 0 & -0.134 & winter & 0 & -0.529 \\
breakfast & 1 & sandwich  & 0 & -0.340 & one & 0 & -0.855 \\
food & 1 & meat & 0 & -0.320 & toothbrush & 0 & -0.832 \\
skateboarding & 1 & skateboard  & 0.3 & -0.115 & skateboard & 0 & -0.164 \\
carrots & 1 & carrot & 0.3 & -0.093 & three & 0 & -0.818 \\
sheep & 1 & goat & 0.3 & -0.197 & white & 0 & -0.676 \\
cloudy & 1 & overcast & 0.3 & -0.151 & gray & 0 & -0.577 \\
black & 0 & black and white & 1 & 0.136 & brown & 1 & 0.269 \\
\hline
    \end{tabular}}
    \caption{Comparison of the SBWT metric of two answers with respect to the same reference answer. We verify that semantically more similar answers have higher SBWT.}
    \label{tab:sbwt_examples}
\end{table*}
\subsubsection{Language Setting}\label{sec:q_splits}
We create a third setting {\em Question Types}, where each task corresponds to learning to answer a different category of questions.
We use a classification scheme developed by~\citet{whitehead2021separating}
to form a sequence of five tasks: 
Count, Color, Scene-level, Subcategory, and Action recognition. The splits for Count, Color, and Subcategory questions are obtained from~\citet{whitehead2021separating}. We create two additional tasks from the remaining questions.
In particular, we cluster question embeddings from Sentence-BERT~\cite{reimers-2019-sentence-bert} \footnote{We use the `all-MiniLM-L6-v2' model and Fast Clustering algorithm from the sentence-transformers package (\url{https://www.sbert.net/}).} so that each cluster has at least 15 questions and a minimum cosine similarity of 0.8 between all embeddings.
We annotate clusters as `scene', `action' or `irrelevant' question types.
Based on a seed of 10K annotated questions, we retrieve all other questions with similarity above 0.8 and label them using the K-nearest neighbor algorithm ($K=5$).  
Question Types have a total of 140K train, 35K validation and 84K test samples (cf.\ Table~\ref{tab:stats}). 
Common question words and answers per task are presented in the Appendix (Figure~\ref{fig:top_ans_q}). 

\subsection{Evaluation Metrics}
After training on task $t$, we compute the VQA accuracy $A_{t,i}$ on data from the previous task $i$. 
We report the macro-average accuracy at the end of the training sequence: $\mathrm{A} = \frac{1}{T} \sum_{i=1}^T A_{T,i}$.
Following~\citet{riemer2018learning}, we report the learned accuracy $\mathrm{LA} = \frac{1}{T} \sum_{i=1}^T A_{i,i}$, which measures the ability to learn the new task $i$.
We also compute backward transfer $\mathrm{BWT} = \frac{1}{T-1} \sum_{i=1}^{T-1} A_{T,i} - A_{i,i}$~\cite{lopez2017GEM}, that captures the impact of catastrophic forgetting.

In addition, we introduce a new metric, we term {\em semantic backward transfer} (SBWT), that weights backward transfer with the cosine distance of the predicted answer embeddings.
The motivation for this metric is simply that some incorrect answers are worse than others.
Consider the example in Figure~\ref{fig:exmaple_intro}, where the ground truth is `duck'. After training on subsequent tasks, the sample gets misclassified as `seagull' which might have a milder impact on the downstream application than completely unsuited answers such as `black and white' or `one'.
More detailed examples are provided in the Appendix Table~\ref{tab:sbwt_examples}.

For each sample $j=1\dots, N$ of task $i$, we measure the accuracy difference $\Delta^{Ti}_j$ of the answers predicted by the $T$-th and $i$-th models and
weigh it by cosine distance of the two answer embeddings $\bm{e_{Tj}}$ and $\bm{e_{ij}}$.
The final SBWT is computed as  :

Appendix \ref{tab:sbwt_examples} provides a qualitative analysis  with examples from the validation set.
In addition, Table  \ref{tab:sbwt_examples} in the Appendix provides an example-based analysis of our suggested metric, showing that semantically similar answers have higher SBWT scores.

Next, we compute the Spearman correlation between the relative accuracy drops and different factors of task dissimilarity.
Here, we consider the answer distributions , as well as average embeddings of the image, question and the joint pair.
Consider $P$, $Q$  the answer distributions of Tasks $T_1,T_2$ respectively.
Since some answers of $T_1$ do not appear in $T_2$, we measure the skew divergence~\cite{pmlr-vR3-lee01a} between $P$ and $Q$ as the KL divergence between $P$ and a mixture distribution \mbox{$(1-\alpha) P + \alpha Q$} with $\alpha=0.99$~\cite{ruder-plank-2017-learning}.
For the input embeddings, we measure the cosine distance between the average task representation. 
As image representations, we utilize Faster R-CNN features from~\cite{Anderson2017up-down}, while questions are embedded using Sentence-BERT.
Joint embeddings for image-question pairs are obtained using the final layer representation of the [CLS] token of UNITER~\footnote{[CLS] is the first token of the input sequence which aggregates multimodal information. Its representation from the final encoder layer is passed to the classifier to predict an answer.}.
The detailed similarity measures are shown in the Appendix Figure~\ref{fig:dissimilarity}.

\section{Qualitative Results}\label{appendix:qualitative}
Table~\ref{tab:examples} shows examples of predicted answers with different approaches. The two top examples are from two different task orders in Question Types, and the two bottom examples are from Taxonomy Domains. 
The model trained from scratch (column w/o PT) fails to retain knowledge from the corresponding training task.
The pretrained model (column PT) is more resistant to forgetting and we observe that for the first and third images, it even manages to recover the correct answer during the training sequence.  
However, relying only on pretraining is insufficient, as the model still tends to change the predicted answer based on the most recent training task. 
Both EWC and ER combined with pretraining successfully retain previous knowledge.

Table~\ref{tab:sbwt_examples} presents examples of the SBWT metric. Specifically, it compares SBWT for two pairs of predicted answers with the same initial reference answer. When the initial prediction (reference answer) is correct, and both compared answers are wrong, we observe that SBWT penalizes similar answers less than unrelated ones (see the first four rows of  Table~\ref{tab:sbwt_examples}). 
Similarly, when one of the compared answers is partially correct (rows 5-8) according to the VQA accuracy metric, SBWT is less punishing compared to BWT, which in our examples would be $-0.7$.
Finally, the last row shows an example of corrected compared answers, where the accuracy improvement is weighted with the semantic distance of reference and compared answers.
{"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Reference Answer": "skateboarding"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 1": "skateboard"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.164", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 1": "skateboard"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 2": "black"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.836", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 2": "black"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Reference Answer": "snowboarding"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 1": "skiing"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.134", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 1": "skiing"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 2": "winter"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.529", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 2": "winter"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Reference Answer": "breakfast"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 1": "sandwich"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.340", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 1": "sandwich"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 2": "one"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.855", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 2": "one"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Reference Answer": "food"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 1": "meat"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.320", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 1": "meat"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 2": "toothbrush"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.832", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 2": "toothbrush"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Reference Answer": "skateboarding"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0.3", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 1": "skateboard"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.115", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 1": "skateboard"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 2": "skateboard"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.164", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 2": "skateboard"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Reference Answer": "carrots"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0.3", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 1": "carrot"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.093", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 1": "carrot"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 2": "three"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.818", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 2": "three"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Reference Answer": "sheep"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0.3", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 1": "goat"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.197", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 1": "goat"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 2": "white"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.676", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 2": "white"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Reference Answer": "cloudy"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0.3", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 1": "overcast"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.151", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 1": "overcast"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 2": "gray"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "-0.577", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 2": "gray"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Reference Answer": "black"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 1": "black and white"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0.136", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 1": "black and white"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["Accuracy", "Acc"], "experimental settings": {"Compared Answer 2": "brown"}, "model": "UNITER", "model settings": {"xx": "yy"}} {"value": "0.269", "type": "Result", "training data/set": "VQA-v2", "test data/set": "xx", "task": ["VQA", "visual question answering"], "metric": ["semantic backward transfer", "SBWT"], "experimental settings": {"Compared Answer 2": "brown"}, "model": "UNITER", "model settings": {"xx": "yy"}}