\begin{table}[t]
    \centering
    \resizebox{\columnwidth}{!}{
    \begin{tabular}{cccccc} \hline
    {\bf Setting} & {\bf Task} & {\bf Train} & {\bf Val} & {\bf Test} & {\bf Classes} \\ \hline
    \multirow{5}{1em}{\rotatebox[origin=c]{90}{Diverse}}  & Group 1 & 44254 & 11148 & 28315 & 2205 \\
    & Group 2 & 39867 & 10202 & 22713  & 1874 \\
    & Group 3 & 37477 & 9386 & 23095 & 1849 \\
    & Group 4 & 35264 & 8871 & 22157 & 2119 \\
    & Group 5 & 24454 & 6028 & 14490 & 1777 \\ \hline
    \multirow{5}{1em}{\rotatebox[origin=c]{90}{Taxonomy}} 
    & Animals & 37270 & 9237 & 22588 & 1331 \\
    & Food & 26191 & 6612 & 15967 & 1365 \\
    & Interior & 43576 & 11038 & 26594 & 2096 \\
    & Sports & 32885 & 8468 & 19205 & 1471 \\
    & Transport & 41394 &10280 & 25416 & 1954 \\\hline
    \multirow{5}{1em}{\rotatebox[origin=c]{90}{Question}} 
    & Action & 18730 & 4700 & 11008 & 233 \\
    & Color & 34588 & 8578 & 21559 & 92 \\
    & Count & 38857 & 9649 & 23261 & 42 \\
    & Scene & 25850 & 6417 & 14847 & 170 \\
    & Subcategory & 22324 & 8578 & 21559 & 659 \\\hline
    \end{tabular}}
    \caption{Statistics per task within each setting.} 
    \label{tab:stats}
\end{table}
\section{Settings for Continual VQA} \label{sec:settings}
\subsection{Problem formulation}
In continual learning, model parameters $\bm{\theta}$ are incrementally updated as new data become available.
We assume that samples from tasks $t=1\dots T$ arrive sequentially as $D_t=\{\bm{x}_i, \bm{y}_i\}_{i=1}^{N_t}$, where $N_t$ is the number of data for task $t$.
Following previous work, VQA is formulated as a multi-label classification problem with soft targets $\bm{y}_i$~\cite{Anderson2017up-down}.
Starting from parameters $\bm{\theta}_{t-1}$ of the previous model, the updated parameters $\bm{\theta}_{t}$ are obtained by training on the new data $D_t$. 
Some approaches also use a memory $M_t$ containing a subset of samples from previous tasks, e.g. $D_{1}, \dots, D_{t-1}$.
In our setup, all tasks share a common output head which is extended with new classes from each task.
This allows inference to be task-agnostic but creates a more challenging setting than multi-head learning where separate heads are learned for each task~\cite{hussain2021towards}.
At the end of the training sequence, the objective is to achieve strong performance across all tasks observed so far. 
This objective encloses two challenges: 1) minimizing catastrophic forgetting of tasks seen earlier in training, 2) facilitating positive transfer to improve performance on new tasks~\cite{HADSELL20201028embracing}. 

\subsection{Task settings}
We define three continual learning settings for VQA based on different task definitions $D_t$, as summarized in Table~\ref{tab:stats}.
Two of these settings are based on visual object categories, see Subsection \ref{sec:vis_splits} and one setting is motivated by language capabilities, see  Subsection \ref{sec:q_splits}.
Concurrent work~\cite{lei2022symbolic} has followed a similar definition of continual learning settings for VQA. However, our work focuses on understanding how differences in task definitions affect the difficulty of the continual learning problem.
We study this problem from the point of view of both the downstream performance as well as the quality of the learned representations. 
This is in line with work on holistic evaluation frameworks for grounded language learning~\cite{suglia-etal-2020-compguesswhat}.

\subsubsection{Visual Settings} \label{sec:vis_splits}
We design two settings based on visual object categories, which correspond to expanding the domain on which the VQA system is applied to. We take advantage of the fact that images in the VQA-v2 dataset
originate from the COCO dataset~\cite{lin2014coco} which provides object-level image annotations.
Following previous work in image captioning~\cite{del2020ratt}, we organize $50$ object categories into five groups.
Images with objects from multiple groups are discarded in order to create clean task splits $D_t$ --
resulting in a total of 181K train, 45K validation, and 110K test samples. 

\subsubsection{Language Setting}\label{sec:q_splits}
We create a third setting {\em Question Types}, where each task corresponds to learning to answer a different category of questions.
We use a classification scheme developed by~\citet{whitehead2021separating}
to form a sequence of five tasks: 
Count, Color, Scene-level, Subcategory, and Action recognition. The splits for Count, Color, and Subcategory questions are obtained from~\citet{whitehead2021separating}. We create two additional tasks from the remaining questions.
In particular, we cluster question embeddings from Sentence-BERT~\cite{reimers-2019-sentence-bert} \footnote{We use the `all-MiniLM-L6-v2' model and Fast Clustering algorithm from the sentence-transformers package (\url{https://www.sbert.net/}).} so that each cluster has at least 15 questions and a minimum cosine similarity of 0.8 between all embeddings.
We annotate clusters as `scene', `action' or `irrelevant' question types.
Based on a seed of 10K annotated questions, we retrieve all other questions with similarity above 0.8 and label them using the K-nearest neighbor algorithm ($K=5$).  
Question Types have a total of 140K train, 35K validation and 84K test samples (cf.\ Table~\ref{tab:stats}). 
Common question words and answers per task are presented in the Appendix (Figure~\ref{fig:top_ans_q}). 

We also experiment with a baseline Pseudo-Replay method for the Question Types setting.
Instead of storing raw data from previous tasks, we use a data augmentation method, inspired by~\cite{kafle2017data,kil2021discovering}. 
When training on task $t$, we augment the data $D_t$ by retrieving past questions based on their shared detected objects classes. For example, if an elephant is detected on the current picture, we retrieve a past question about an elephant. We then use the previous model $f_{\theta_{t-1}}$ to generate a distribution $\tilde{ \bm{y}} = f_{\theta_{t-1}}(\tilde{\bm{x}})$ which serves as soft targets for the new sample $\tilde{\bm{x}}$.
By not storing the original answers, we address privacy and efficiency concerns of replay approaches~\cite{van2018generative,Delangesurvey}.

\paragraph{Experimental Setup.} 
We first look into pairwise task relationships following studies in transfer~\cite{zamir2018taskonomy} and multitask learning ~\cite{pmlr-v119-standley20a, lu202012}.
In particular, we measure the extent to which each task is forgotten after training on a second task. 
We finetune a pretrained model on Task $T_{1}$ and compute the accuracy $A_{11}$ on its test set. Then, we finetune this model on another Task $T_{2}$ and compute the new accuracy $A_{12}$ on the test set of $T_{1}$.
Forgetting is measured as the relative accuracy drop: $(A_{12} - A_{11}) / A_{11}$.
Given the varying dataset sizes, we finetune on $T_{2}$ for a fixed number of 400 steps using a batch size of 512 and learning rate 5e-5.

Next, we compute the Spearman correlation between the relative accuracy drops and different factors of task dissimilarity.
Here, we consider the answer distributions , as well as average embeddings of the image, question and the joint pair.
Consider $P$, $Q$  the answer distributions of Tasks $T_1,T_2$ respectively.
Since some answers of $T_1$ do not appear in $T_2$, we measure the skew divergence~\cite{pmlr-vR3-lee01a} between $P$ and $Q$ as the KL divergence between $P$ and a mixture distribution \mbox{$(1-\alpha) P + \alpha Q$} with $\alpha=0.99$~\cite{ruder-plank-2017-learning}.
For the input embeddings, we measure the cosine distance between the average task representation. 
As image representations, we utilize Faster R-CNN features from~\cite{Anderson2017up-down}, while questions are embedded using Sentence-BERT.
Joint embeddings for image-question pairs are obtained using the final layer representation of the [CLS] token of UNITER~\footnote{[CLS] is the first token of the input sequence which aggregates multimodal information. Its representation from the final encoder layer is passed to the classifier to predict an answer.}.
The detailed similarity measures are shown in the Appendix Figure~\ref{fig:dissimilarity}.

\paragraph{Results.} 
Table~\ref{tab:Tpairs} shows the relative accuracy drop for all task pairs. 
Overall, we observe that each setting has a distinct pattern. 
Question Types is evidently a more challenging setting, where several task combinations show more than $90\%$ drop.
When comparing the visual settings, forgetting in Diverse Domains fluctuates less depending on the task pairing. 
This suggests that the task relationships in Taxonomy Domains might play a more important factor.
Although some relations make sense based on the expected similarity of the visual scenes, e.g., low forgetting between Food and Interior, others are less intuitive, e.g., low forgetting between Transport and Interior.
Moreover, certain second tasks seem to consistently affect the amount of forgetting after finetuning on them. 
Based on the total number of classes per task as shown in Table ~\ref{tab:stats}, we notice that the model is more robust against forgetting when Task $T_2$ has a wide range of possible answers (e.g., Interior); while $T_2$ with a narrow answer set (e.g., Food, Color, Count) lead to maximum forgetting.
{"value": "44254", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 1 Train", "dataset features": {"xx": "yy"}} {"value": "11148", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 1 Val", "dataset features": {"xx": "yy"}} {"value": "28315", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 1 Test", "dataset features": {"xx": "yy"}} {"value": "2205", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of classes", "sub-set/group name": "Group 1", "dataset features": {"xx": "yy"}} {"value": "39867", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 2 Train", "dataset features": {"xx": "yy"}} {"value": "10202", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 2 Val", "dataset features": {"xx": "yy"}} {"value": "22713", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 2 Test", "dataset features": {"xx": "yy"}} {"value": "1874", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of classes", "sub-set/group name": "Group 2", "dataset features": {"xx": "yy"}} {"value": "37477", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 3 Train", "dataset features": {"xx": "yy"}} {"value": "9386", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 3 Val", "dataset features": {"xx": "yy"}} {"value": "23095", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 3 Test", "dataset features": {"xx": "yy"}} {"value": "1849", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of classes", "sub-set/group name": "Group 3", "dataset features": {"xx": "yy"}} {"value": "35264", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 4 Train", "dataset features": {"xx": "yy"}} {"value": "8871", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 4 Val", "dataset features": {"xx": "yy"}} {"value": "22157", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 4 Test", "dataset features": {"xx": "yy"}} {"value": "2119", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of classes", "sub-set/group name": "Group 4", "dataset features": {"xx": "yy"}} {"value": "24454", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 5 Train", "dataset features": {"xx": "yy"}} {"value": "6028", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 5 Val", "dataset features": {"xx": "yy"}} {"value": "14490", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of samples", "sub-set/group name": "Group 5 Test", "dataset features": {"xx": "yy"}} {"value": "1777", "type": "Data Stat.", "dataset": "VQA-v2 Diverse", "attribute name": "number of classes", "sub-set/group name": "Group 5", "dataset features": {"xx": "yy"}} {"value": "37270", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Animals Train", "dataset features": {"xx": "yy"}} {"value": "9237", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Animals Val", "dataset features": {"xx": "yy"}} {"value": "22588", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Animals Test", "dataset features": {"xx": "yy"}} {"value": "1331", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of classes", "sub-set/group name": "Animals", "dataset features": {"xx": "yy"}} {"value": "26191", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Food Train", "dataset features": {"xx": "yy"}} {"value": "6612", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Food Val", "dataset features": {"xx": "yy"}} {"value": "15967", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Food Test", "dataset features": {"xx": "yy"}} {"value": "1365", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of classes", "sub-set/group name": "Food", "dataset features": {"xx": "yy"}} {"value": "43576", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Interior Train", "dataset features": {"xx": "yy"}} {"value": "11038", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Interior Val", "dataset features": {"xx": "yy"}} {"value": "26594", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Interior Test", "dataset features": {"xx": "yy"}} {"value": "2096", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of classes", "sub-set/group name": "Interior", "dataset features": {"xx": "yy"}} {"value": "32885", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Sports Train", "dataset features": {"xx": "yy"}} {"value": "8468", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Sports Val", "dataset features": {"xx": "yy"}} {"value": "19205", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Sports Test", "dataset features": {"xx": "yy"}} {"value": "1471", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of classes", "sub-set/group name": "Sports", "dataset features": {"xx": "yy"}} {"value": "41394", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Transport Train", "dataset features": {"xx": "yy"}} {"value": "10280", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Transport Val", "dataset features": {"xx": "yy"}} {"value": "25416", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of samples", "sub-set/group name": "Transport Test", "dataset features": {"xx": "yy"}} {"value": "1954", "type": "Data Stat.", "dataset": "VQA-v2 Taxonomy", "attribute name": "number of classes", "sub-set/group name": "Transport", "dataset features": {"xx": "yy"}} {"value": "18730", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Action Train", "dataset features": {"xx": "yy"}} {"value": "4700", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Action Val", "dataset features": {"xx": "yy"}} {"value": "11008", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Action Test", "dataset features": {"xx": "yy"}} {"value": "233", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of classes", "sub-set/group name": "Action", "dataset features": {"xx": "yy"}} {"value": "34588", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Color Train", "dataset features": {"xx": "yy"}} {"value": "8578", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Color Val", "dataset features": {"xx": "yy"}} {"value": "21559", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Color Test", "dataset features": {"xx": "yy"}} {"value": "92", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of classes", "sub-set/group name": "Color", "dataset features": {"xx": "yy"}} {"value": "38857", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Count Train", "dataset features": {"xx": "yy"}} {"value": "9649", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Count Val", "dataset features": {"xx": "yy"}} {"value": "23261", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Count Test", "dataset features": {"xx": "yy"}} {"value": "42", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of classes", "sub-set/group name": "Count", "dataset features": {"xx": "yy"}} {"value": "25850", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Scene Train", "dataset features": {"xx": "yy"}} {"value": "6417", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Scene Val", "dataset features": {"xx": "yy"}} {"value": "14847", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Scene Test", "dataset features": {"xx": "yy"}} {"value": "170", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of classes", "sub-set/group name": "Scene", "dataset features": {"xx": "yy"}} {"value": "22324", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Subcategory Train", "dataset features": {"xx": "yy"}} {"value": "8578", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Subcategory Val", "dataset features": {"xx": "yy"}} {"value": "21559", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of samples", "sub-set/group name": "Subcategory Test", "dataset features": {"xx": "yy"}} {"value": "659", "type": "Data Stat.", "dataset": "VQA-v2 Question", "attribute name": "number of classes", "sub-set/group name": "Subcategory", "dataset features": {"xx": "yy"}}