# Multi-Task Meta Learning: learn how to adapt to unseen tasks

1st Richa Upadhyay  Luleå University of Technology
<EMAIL_ADDRESS>2nd Prakash Chandra Chhipa  Luleå University of
Technology
<EMAIL_ADDRESS>3rd Ronald Phlypo  Université Grenoble Alpes,
<EMAIL_ADDRESS>4th Rajkumar Saini  Luleå University of
Technology
<EMAIL_ADDRESS>5th Marcus Liwicki  Luleå University of Technology
<EMAIL_ADDRESS>
###### Abstract

This work proposes Multi-task Meta Learning (MTML), integrating two learning
paradigms Multi-Task Learning (MTL) and meta learning, to bring together the
best of both worlds. In particular, it focuses simultaneous learning of
multiple tasks, an element of MTL and promptly adapting to new tasks, a
quality of meta learning. It is important to highlight that we focus on
heterogeneous tasks, which are of distinct kind, in contrast to typically
considered homogeneous tasks (e.g., if all tasks are classification or if all
tasks are regression tasks). The fundamental idea is to train a multi-task
model, such that when an unseen task is introduced, it can learn in fewer
steps whilst offering a performance at least as good as conventional single
task learning on the new task or inclusion within the MTL. By conducting
various experiments, we demonstrate this paradigm on two datasets and four
tasks: NYU-v2 and the taskonomy dataset for which we perform semantic
segmentation, depth estimation, surface normal estimation, and edge detection.
MTML achieves state-of-the-art results for three out of four tasks for the
NYU-v2 dataset and two out of four for the taskonomy dataset. In the taskonomy
dataset, it was discovered that many pseudo-labeled segmentation masks lacked
classes that were expected to be present in the ground truth; however, our
MTML approach was found to be effective in detecting these missing classes,
delivering good qualitative results. While, quantitatively its performance was
affected due to the presence of incorrect ground truth labels. The the source
code for reproducibility can be found at https://github.com/ricupa/MTML-learn-
how-to-adapt-to-unseen-tasks.

###### Index Terms:

Multi-task learning, meta learning, semantic segmentation, depth estimation,
surface normal estimation

##  I Introduction

Multi-task learning (MTL) involves learning many tasks in a single, combined
network architecture [1]. This is in contrast to single task learning, which
trains dedicated networks, one for each task. The prime argument backing MTL
is that the knowledge absorbed by the network while learning one task may help
to improve the performance on another task when they are trained together.
However, in a multi-task setting, when there is a need to add a new task to
the existing architecture, the new network has to be re-trained from scratch
for the new set of tasks so that there is efficient knowledge transfer between
the tasks, but, it leads to the loss of previously gained knowledge. Fine-
tuning the new task is another option, but there is a risk of overfitting [2].
Meta learning [3, 4], on the other hand, involves reusing information obtained
during the learning of a task to quickly adapt to a new one. The paradigm of
meta-learning—also referred to as _learning to learn_ [5]—gathers experience
by learning several homogeneous tasks (learning episodes) and utilizes the
overall meta knowledge to enhance its future performances on yet unseen tasks.
Meta-learning learns the distribution over the tasks rather than the specific
tasks themselves. The latter are considered samples from this distribution,
which implies that all tasks must be of similar nature or uni-modal [3], e.g.,
classification or regression. Similar tasks are referred to as homogeneous in
this work, while those of a distinct kind are called heterogeneous. In this
study, we focus on the optimization based meta learning algorithms [4, 6, 7,
8, 9].

Figure 1: An intuitive explanation of how ‘tasks’ are introduced in MTL, meta
learning, and MTML. In MTL, one learning instance has all the tasks, in meta
learning one learning episode has multiple instances for one task and many
episodes of similar but non-identical tasks are used for training. While in
MTML, the multi-task learning episodes consist of many instances of all the
combinations of the tasks.

This work introduces a MTML paradigm, taking advantage from both multi-task
and meta learning and constructed as follows: the episodes used for meta
training are assorted multi-task combinations, as illustrated in Fig. 1,
trained by employing the two-level meta optimization scheme introduced by MAML
[4]. The MTL allows the joint learning of homogeneous as well as heterogeneous
tasks. The latter is currently considered a limitation of meta learning [3].
In contrast, meta learning aids in quicker learning of an unseen task with
fewer data samples, which is challenging for MTL. As a result, MTML is focused
on learning how to learn unseen tasks in a multi-task setting. The
contributions of this work are:

  1. 1.

a new approach for creating multi-task episodes with heterogeneous tasks
essential for the meta training phase;

  2. 2.

a learning mechanism called MTML enabling faster training of new tasks and
better performance on all of the tasks, when meta learning is introduced in a
MTL framework ;

  3. 3.

extensive comparative performance analysis of single-task, MTL, and MTML
learning paradigms on two publicly available datasets.

This article is organized as follows; Section 2, discusses the related works
in multi-task learning and also multi-task meta learning. Section 3,
introduces MTL, and meta learning, while Section 4 details the formulation of
MTML. The experimental setup is given in Section 5. Section 6 contains a
comprehensive analysis of the results. The reasons behind the unsatisfactory
performance of semantic segmentation task are discussed in Section 7. At last,
Section 8 draws the conclusion and opens up to future work.

##  II Related work

In this section, we highlight significant research studies that claim to
combine MTL and meta learning. [10, 11, 12] apply meta learning optimization
in a multi-task scenario to achieve good generalization for unseen data sets
on the same tasks. In [10] for efficient communication between two tasks in
MTL, a gradient passing mechanism is proposed which has similar traits to
gradient based meta learning. A meta learning approach is followed for sharing
parameters across multiple tasks and languages in [11], the model is trained
on various task-language pairs rather than training all the tasks
simultaneously. In [12], the tasks of dialogue generation give a context and
persona information is learned for multiple personas following meta
optimization. Here the multiple learning episodes (persona information) are
considered as multiple tasks. Additionally, [13, 14, 15] use multiple tasks
for the multiple training episodes in meta learning and hence tagged their
paradigm _multi-task meta learning_. Similarly, in [16] it is theoretically
and empirically proved that MTL is a computationally efficient alternative to
gradient based meta learning algorithm, as for an adequately deep network, the
learned predictive functions of MTL and meta learning are very similar.

Several studies on MTL [17, 18, 19, 20, 21, 22] place a strong emphasis on
network architecture design to improve task performance. Apart form them, the
authors in [23] propose to determine the task transfer relationships between
26 tasks in order to learn to group tasks for MTL. Other [24, 25, 26, 27]
discuss task embedding primarily for meta learning as a means to learn task
relationships. [28, 29] employ neural architecture search in a multi-task
setting to reduce the number of parameters.

Most of the works in the literature related to MTL concentrate on making it
more efficient in terms of its performance, number of parameters,
generalization to unseen data, etc., by adopting new architectures, learning
better task grouping, soft parameter sharing, neural architecture search, and
integration with other algorithms. Adding to the list, this work puts together
meta learning concept to enable the addition of an unseen task to an MTL
architecture, resulting in faster training and better generalization of the
newly added task compared to single-task learning.

##  III Background

Multi-task learning: a learning paradigm aiming to train multiple tasks
together. The shared representations help to enhance the learning capabilities
of all the tasks compared to training them individually [30]. As illustrated
in Fig. 2(a), $N$ non-identical—but related—tasks are sampled from a task
distribution, $\mathcal{T}=\\{T_{1},T_{2},...,T_{N}\\}$, where $T_{i}$ is the
$i^{th}$ task. The data set for all the tasks represented by $\mathcal{D}$ is
split into $J$ train, $K-J$ validation, and $M-K$ test instances, such that
$\mathcal{D}=\\{(D_{i}^{train})_{i=1}^{J},(D_{i}^{val})_{i=J+1}^{K},(D_{i}^{test})_{i=K+1}^{M}\\}$,
here $D_{i}$ represents the dataset for $i^{th}$ task. A multi-task network
architecture is trained using the training data $(D_{i}^{train})_{i=1}^{J}$.
The objective is to minimize the combined loss $\mathcal{L}$, by optimizing
the network parameters $\omega=\\{(\omega_{i})_{i=1}^{N}\\}$, such that

$\omega^{*}=\min_{\omega}\sum_{i=1}^{N}\mathcal{L}_{i}(\omega_{i},(D^{train})_{i})\enspace.$
(1)

$\\{D^{val}_{i}\\}_{i=J+1}^{K}$ is used to evaluate the performance of the
multi-task model during the training process, thereby assures generalization
of the model to data instances not used during training. The optimal
parameters $\omega^{*}$ are used in the inference on the unseen test data
$\\{D^{test}_{i}\\}_{i=K+1}^{M}$ to report model performance.

(a) Multi-task learning

(b) Meta learning (MAML [4])

(c) Multi-task meta learning

Figure 2: Block diagram illustrating the formulation of the learning paradigms

Meta learning: In a few-shot learning framework (a species of meta learning),
every $n$-way, $k$-shot learning episode (i.e., $n$ classes and $k$ data
samples of each class) is sampled from a base data set using two-step episodic
sampling, as shown in Fig. 2(b). Here classes are discrete output variables
also known as labels. First, one samples the episode classes from the class
distribution, organized into source classes
$\mathcal{C}_{s}=\\{C_{s_{1}},C_{s_{2}},...,C_{s_{N_{1}}}\\}$ and target
classes $\mathcal{C}_{t}=\\{C_{t_{1}},C_{t_{2}},..,C_{t_{N_{2}}}\\}$; where
$N_{1}$ and $N_{2}$ represent the number of source and target classes,
respectively, and $\mathcal{C}_{s}\cap\mathcal{C}_{t}=\emptyset$. Second, one
samples an episode ($k$ data points) from the data set based on the classes
sampled in the previous step. Consider an example of 2-way $k$-shot learning:
each learning episode ( i.e., task) will contain the training instances of two
classes, say
$\mathcal{T}_{1}=\\{C_{s_{1}},C_{s_{2}}\\}.....\mathcal{T}_{E_{M}}=\\{C_{s_{N_{1}-1}},C_{s_{N_{1}}}\\}$,
thereby creating $E_{M}$ learning episodes (_tasks_) for meta training. The
meta learners iterate over the learning episodes intending to carry out the
process of _learning to learn_ [3]. It employs a two-step optimization [4] for
all the learning episodes. First, it follows task-specific learning by
optimizing task-specific parameters $\\{\omega_{\eta}\\}_{\eta=1}^{E_{M}}$
given the meta parameters $\theta^{(p)}$ of the $p$th iteration:

$\displaystyle\omega^{(p+1)}_{\eta}(\theta^{(p)})=\operatorname*{arg\,min}_{\omega}\mathcal{L}_{\eta}(\omega,\theta^{(p)},(D^{train})_{\eta})$
(2)

Here, $\mathcal{L}_{\eta}$ represents the loss for the $\eta$th learning
episode, $(D^{train})_{\eta}$ are the training data set. The meta parameters
i.e., $\theta$ are often referred to as _meta knowledge_ or _knowledge across
tasks_ [3]. In Fig. 2(b), a data instance is indexed by $i$, and an episode is
indexed by $\eta$. The second step corresponds to multiple task learning: at
this meta stage, the aim is to reduce the meta loss $\mathcal{L}^{meta}$—using
the unseen validation instances $(D^{val})_{\eta}$—by optimizing the $\theta$
given the task parameters of the $(p+1)$th iteration:

$\theta^{(p+1)}(\omega^{(p+1)})=\operatorname*{arg\,min}_{\theta}\sum_{\eta=1}^{E_{M}}\mathcal{L}^{meta}(\omega^{(p+1)}_{\eta},\theta,(D^{val})_{\eta})$
(3)

Iterating between (2) and (3) would result in an optimal meta learner, i.e., ,
$\theta^{(p)}\to\theta^{\ast}$. During meta testing (adaptation stage) the
meta knowledge $\theta^{*}$ is used as initial parameters for learning new,
unseen tasks (episodes), say,
$\mathcal{T}_{1}=\\{C_{t_{1}},C_{t_{2}}\\}......\mathcal{T}_{E_{K}}=\\{C_{t_{N_{1}-1}},C_{t_{N_{1}}}\\}$.
Therefore, the test tasks are fine-tuned on the model using meta parameters,
which help achieve the best performance for the new tasks in few gradient
steps.

##  IV Formulation of Multi-task meta learning (MTML)

In this work, the multi-task architecture is used along with the bi-level meta
optimization to establish MTML. Particularly, an optimization-based meta
learning approach [31], recognized as Model Agnostic Meta Learning (MAML) [4]
is adopted, which performs a two-level gradient descent optimization
compatible with any model. The Fig. 2(c), illustrates the formulation of MTML.
The source and target tasks are sampled from a distribution of tasks
$p(\mathcal{T})$, given by
$\mathcal{T}_{s}=\\{T_{s_{1}},T_{s_{2}},..,T_{s_{N_{1}}}\\}$ and
$\mathcal{T}_{t}=\\{T_{t_{1}},T_{t_{2}},..,T_{t_{N_{2}}}\\}$, respectively.
Multi-task learning episodes are created from the source tasks analogous to
meta learning, but since the nature of the tasks can be heterogeneous, the
classical meta learning approach of merely sampling the source classes (or
labels) is insufficient. Therefore, using the power set of the source task set
$2^{{\mathcal{T}}_{s}}$, after excluding the singletons and the empty set, one
has $2^{N_{1}}-N_{1}-1$ multi-task combinations of the source classes that can
be used as multi-task learning episodes. These multi-task episodes are used to
train the network using the two-level meta optimization, discussed in
equations (2) and (3), i.e., the multi-task meta training stage. New unseen
tasks can now be introduced as target tasks in the meta testing stage. Either
all source and target tasks or only the target tasks (as required) are then
fine-tuned in the meta testing stage, which is similar to training in MTL,
except it utilizes the meta parameters from the multi-task meta training
stage. The task heads (introduced in Section V under network architecture)
which are the task-specific (un-shared) layers, are fine-tuned if the purpose
is to solely improve the target tasks.

Similar to $n$-way $k$-shot meta learning, MTML can be considered as $N$-task
$many$-shot learning. Here $N$ is the sum of the source and target tasks. It
should be emphasized that, the number of multi-task learning episodes
exponentially rises with $N$, as a result the number of multi- task training
episodes also increase. The investigation of the effect of the number of tasks
on MTML’s performance is beyond the scope of this work. Although, it is
required that $N\geq 3$ to generate enough training episodes for the meta-
training stage. If $N=2$ there will be only one learning episode, and which is
insufficient for meta training.

##  V Experimental setup

Models | Input | Taskonomy | Input | NYU
---|---|---|---|---
Image | Seg. | Depth | SN | Edge | Image | Seg. | Depth | SN | Edge
GT |  |  |  |  |  |  |  |  |  |
Single |  |  |  |  |  |  |  |
MTL |  |  |  |  |  |  |  |
MTML |  |  |  |  |  |  |  |
GT |  |  |  |  |  |  |  |  |  |
Single |  |  |  |  |  |  |  |
MTL |  |  |  |  |  |  |  |
MTML |  |  |  |  |  |  |  |
Figure 3: The figure illustrates sample images of the input, its corresponding
ground truths, single task (Exp. 1), MTL (Exp. 2.3) and MTML (Exp. 4.4)
outputs for semantic segmentation (Seg.), depth estimation (Depth), surface
normal estimation (SN), and edge detection (Edge) for both the NYU-v2 and
taskonomy datasets.

Data sets and tasks: For the performance analysis of our proposed approach,
two publicly available data sets are used: the NYU-v2 data set [32] and the
tiny taskonomy data set [23]. Both of these data sets contain a large variety
of indoor scene images in standard 3-channel RGB image format. Four tasks used
in this work are: semantic segmentation ($T_{1}$), depth estimation ($T_{2}$),
surface normal estimation ($T_{3}$), and edge detection ($T_{4}$). The train-
validation-test data split for the NYU-v2 and taskonomy datasets is given in
[28] and [23], respectively.

Network architecture: A very commonly used multi-task architecture is used for
this work: a shared backbone network, followed by task-specific heads. The
common backbone in the network allows sharing of the low and mid-level
features through their model parameters, while specific high-level features
are learned by the task specific heads [33]. In this work, dilated ResNet-50
[34] is employed as the backbone, which gives the shared representations of
the input RGB image. These representations are then fed as inputs to the task
heads or task specific network. For all the four tasks, DeepLab V3 [35]
network is used as the task heads, which make use of atrous
convolutions111Atrous convolution comes from the french, _convolution à trous_
which could be translated as _sparse kernel convolution_. [36, 37]. The Atrous
Spatial Pyramid Pooling (ASPP) module used in DeepLab v3 architecture helps to
extract dense feature maps by discarding the down-sampling in the last layers
and employing up-sampling in the filters of the corresponding layers. This
makes it suitable for pixel-level dense prediction tasks.

TABLE I: Performance of the single task, MTL, and MTML approaches introduced
in this work on the NYU-v2 and taskonomy datasets (across the three learning
paradigms, p $<$ 0.05 for all the tasks), along with some previous state-of-
the-art works in the literature. In this table, MAE is mean absolute error, CE
denotes cross entropy metric, mIoU represents mean intersection over union
metric and CS stands for cosine similarity metric. The top-2 results are
highlighted in bold.

|  | Taskonomy | NYU
---|---|---|---
Model | Parameters | Segmen- | Depth | Surface | Edge | Segmen- | Depth | Surface | Edge
|  | tation | Estimation | Normal | Detection | tation | Estimation | Normal | Detection
|  | T1 | T2 | T3 | T4 | T1 | T2 | T3 | T4
|  |  |  |  |  |  |  | Error $\downarrow$ | Theta $\uparrow$ |
| in Millions | CE $\downarrow$ | MAE $\downarrow$ | CS $\uparrow$ | MAE $\downarrow$ | mIoU $\uparrow$ | MAE $\downarrow$ | Mean | Median | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | MAE $\downarrow$
Multi-Task [28] | 41 | 0.587 | 0.024 | 0.696 | 0.203 | 24.1 | 0.58 | 16.6 | 13.4 | 42.5 | 73.2 | 84.6 | -
Adashare [28] | 41 | 0.566 | 0.025 | 0.702 | 0.2 | 30.2 | 0.55 | 16.6 | 12.9 | 45 | 71.7 | 83 | -
Cross Stitch [18] | 124 | 0.56 | 0.022 | 0.679 | 0.217 | 24.5 | 0.58 | 17.2 | 14 | 41.4 | 70.5 | 82.9 | -
MTAN [38] | 114 | 0.637 | 0.023 | 0.687 | 0.206 | 26 | 0.57 | 16.6 | 13 | 43.7 | 73.3 | 84.4 | -
NDDR CNN [20] | 133 | 0.539 | 0.024 | 0.7 | 0.203 | 21.6 | 0.66 | 17.1 | 14.5 | 37.4 | 73.7 | 85.6 | -
Sluice [39] | 124 | 0.596 | 0.024 | 0.695 | 0.207 | 23.8 | 0.58 | 17.2 | 14.4 | 38.9 | 71.8 | 83.9 | -
Learn2branch [29] | 51 | 0.462 | 0.018 | 0.709 | 0.136 | No results on NYU dataset |
MTL (Ours) | 88 | 0.628 | 0.013 | 0.930 | 0.050 | 42.25 | 0.12 | 15.04 | 16.06 | 42.24 | 72.52 | 87.72 | 0.16
MTML (Ours) | 88 | 2.300 | 0.063 | 0.931 | 0.046 | 41.51 | 0.10 | 13.34 | 10.24 | 52.40 | 76.17 | 88.51 | 0.10

Training specifications: To train the above architecture, an input RGB image
of size 256 $\times$ 256 is normalized and fed to the backbone network in
batches of 64 images. A minimal learning rate of 0.00001 is considered, to
administer sufficient training for all the tasks, since some tasks are harder
than others. Task-specific early stopping (patience = 35) and overall early
stopping (patience = 50) are employed to avoid overfitting. For single task
experiments, multi-task experiments and the meta update ( i.e., outer loop) in
MTML, AdamW [40] is used as an optimizer since it decouples weight decay from
gradient update by modifying Adam’s [41] implementation of $L_{2}$
regularization. Due to large datasets like taskonomy and NYU-v2, MTML only
takes into account a single SGD [42] inner loop to reduce computational cost
and memory usage. The losses are cross-entropy loss for semantic segmentation,
a combined depth loss [43] for depth estimation, inverse cosine similarity
loss for surface normal prediction, and Huber loss [44] for edge detection. To
make an even comparison, the evaluation metrics used in this work are similar
to those used in [28, 18, 38, 20, 39], also mentioned in Table I. For semantic
segmentation, cross-entropy (lower is better) and intersection over union
(IoU, higher is better) are respectively used for the taskonomy and NYU-v2
datasets. Mean absolute error (lower is better) is calculated for depth
estimation on both data sets. For the taskonomy data set, cosine similarity is
used to evaluate surface normal prediction. In contrast, for the NYU-v2
dataset, the mean and median of angular error (lower is better) between
prediction and ground truth and percentage of pixels whose predicted values
are within $11.25^{\circ}$, $22.5^{\circ}$, and $30^{\circ}$ [28] (higher is
better) of the ground truth are calculated. For edge detection mean absolute
error (lower is better) is calculated between the prediction and ground truth
for both data sets. The loss and evaluation metric for semantic segmentation
do not include background pixels, similar to the other works discussed in
Table I. For fair comparison, the hyper-parameters were determined for the
single task models to perform their best, and the multi-task models were then
trained under similar setting.

For combining the losses from all the tasks, there are many techniques as
discussed by [33]. In this work we have used uncertainty [45] to balance the
losses of the four dense prediction tasks. Because the focus of this work was
on combining multi-task and meta learning, other task balancing techniques
were not investigated.

In a MTML arrangement, as discussed in Section III, the network is trained
using multi-task learning episodes. In episodes having less than four tasks,
it is trained as usual, but for the task absent in the combination, the task
loss is set to zero and the losses are combined as usual for backpropagation.
The parameters of the corresponding task head are not updated by freezing the
layers (same in case of task-wise early-stopping). To carry out a comparative
performance analysis of all the experiments, they are evaluated using the same
test set, and all the models are trained with the same hyper-parameters. The
models are trained on NVIDIA A100 Tensor Core GPUs, with 40 GB on-board HBM2
VRAM. All experiments are repeated three times with different random seeds to
ensure and evaluate the consistency of the model. The results are shown in
terms of mean and standard deviation.

##  VI Experimental evaluations

To analyze the performance of the proposed methods for various task
combinations, the experiments listed in Table II (NYU-v2) and III (taskonomy)
were designed. Experiments are classified into three types: single task
learning, multi-task learning, and MTML. A comprehensive analysis of the
results of these experiment is provided in this section. These experiments
compare the effect of adding a new task to an already-trained single task,
multi-task and meta multi-task network. To test the various task combinations,
two formats are considered: augmenting tasks (Exp. 2, 3, and 4), and leave-
one-out (Exp. 5, and 7). In these experiments, ‘+Tn’ indicates the addition of
new $n$th task to a network previously trained on other tasks. Fig. 4 compares
the average number of epochs needed to train the single task, MTL and MTML
models.

TABLE II: Test set evaluation results for a single task, multi-task, and MTML
experiments on the NYU-v2 dataset

Exp. No. | Tasks Involved | Tasks
---|---|---
| Trained | Tested | T1 | T2 | T3 | T4
|  | ( +Tn is fine-tuning | Segmentation | Depth Estimation | Surface Normal | Edge Detection
|  |  on $\mathrm{n^{th}}$ new task) |  mIoU $\uparrow$ |  Mean abs.error $\downarrow$ |  Error $\downarrow$ |  Theta $\uparrow$ |  Mean abs.error $\downarrow$
|  |  |  |  | mean | median | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ |
1 | Single task learning
|  |  | 42.53±0.083 | 0.11±0.000 | 15.88±0.510 | 13.97±0.524 | 41.62±1.514 | 73.20±1.878 | 88.56±0.780 | 0.15±0.010
2 | Multi-task learning
2.1 | T1, T2 | T1, T2 | 42.38±0.123 | 0.11±0.001 | - | - | - | - | - | -
2.2 | T1, T2, T3 | T1, T2, T3 | 42.36±0.335 | 0.11±0.001 | 15.52 ± 0.574 | 13.55±0.632 | 43.47±1.908 | 73.01±1.530 | 88.01±0.942 | -
2.3 | T1, T2, T3, T4 | T1, T2, T3, T4 | 42.25±0.141 | 0.12±0.002 | 15.04 ± 0.769 | 16.06±3.060 | 42.24±0.725 | 72.52±0.918 | 87.72±1.716 | 0.16±0.033
3 | Multi-task learning, addition of new task
3.1 | T1, T2 | T1, T2 (+ T3) | 42.41±0.299 | 0.12±0.002 | 15.18±0.209 | 13.37±0.185 | 44.18±0.406 | 73.22±0.725 | 88.31±0.580 | -
3.2 | T1, T2 | T1, T2 (+ T3, T4) | 42.48±0.258 | 0.11±0.001 | 14.84±0.230 | 12.90±0.318 | 45.42±0.932 | 74.37±0.427 | 88.46±0.271 | 0.20±0.029
3.3 | T1, T2, T3 | T1, T2, T3 (+T4) | 42.50±0.063 | 0.11±0.001 | 14.84±0.552 | 12.75±0.646 | 45.86±1.802 | 74.56±1.485 | 88.34±0.412 | 0.24±0.010
4 |  MTML, meta testing phase involves addition of new task
4.1 | T1, T2 | T1, T2 (+ T3) | 37.09±0.355 | 0.11±0.002 | 13.84±0.254 | 11.89±0.370 | 48.02±1.238 | 78.54±0.456 | 90.51±0.307 | -
4.2 | T1, T2 | T1, T2 (+ T3, T4) | 37.06±0.163 | 0.11±0.001 | 14.28±0.635 | 12.47±0.715 | 46.37±2.238 | 77.04±1.526 | 90.19±0.251 | 0.17±0.048
4.3 | T1, T2, T3 | T1, T2, T3 (+T4) | 39.59±1.580 | 0.11±0.004 | 13.82±0.385 | 11.36±0.593 | 49.64±1.594 | 76.26±0.279 | 89.02±0.143 | 0.15±0.074
4.4 | T1, T2, T3, T4 | T1, T2, T3, T4 | 41.41±0.111 | 0.10±0.001 | 13.34±0.025 | 10.24±0.010 | 52.40±0.005 | 76.17±0.130 | 88.51±0.095 | 0.10±0.000
5 | Multi-task learning, leave one task out format
5.1 | T2, T3, T4 | T2, T3, T4 | - | 0.14±0.027 | 15.27±0.444 | 13.48±0.553 | 43.26±1.824 | 73.96±1.336 | 89.05±0.343 | 0.18±0.014
5.2 | T1, T3, T4 | T1, T3, T4 | 42.52±0.102 | - | 15.30±0.439 | 13.41±0.600 | 43.57±2.031 | 73.98±0.749 | 88.57±0.410 | 0.12±0.004
5.3 | T1, T2, T4 | T1, T2, T4 | 42.04±0.142 | 0.12±0.003 | - | - | - | - | - | 0.12±0.006
5.4 | T1, T2, T3 | T1, T2, T3 | 42.36±0.335 | 0.11±0.001 | 15.52±0.574 | 13.55±0.632 | 43.47±1.908 | 73.01±1.530 | 88.01±0.942 | -
6 | Multi-task learning, leave one task out format, addition of the left out task
6.1 | T2, T3, T4 | T2, T3, T4 (+ T1) | 42.36±0.108 | 0.12±0.000 | 14.78±0.078 | 12.86±0.096 | 45.47±0.191 | 74.82±0.523 | 88.25±0.306 | 0.16±0.008
6.2 | T1, T3, T4 | T1, T3, T4 (+T2) | 43.05±0.042 | 0.12±0.000 | 15.14±0.340 | 13.21±0.389 | 44.06±1.280 | 74.65±0.982 | 89.12±0.029 | 0.12±0.002
6.3 | T1, T2, T4 | T1, T2, T4 (+T3) | 42.49±0.239 | 0.11±0.000 | 14.60±0.105 | 12.79±0.185 | 45.43±0.706 | 76.01±0.368 | 89.73±0.306 | 0.12±0.004
6.4 | T1, T2, T3 | T1, T2, T3 (+T4) | 42.50±0.063 | 0.11±0.001 | 14.84±0.552 | 12.75±0.646 | 45.86±1.802 | 74.56±1.485 | 88.34±0.412 | 0.24±0.010
7 |  MTML, leave one task out format, addition of left out task in meta testing
7.1 | T2, T3, T4 | T2, T3, T4 (+ T1) | 45.05±0.536 | 0.10±0.002 | 13.44±0.008 | 10.76±0.078 | 51.23±0.185 | 76.65±0.174 | 89.18±0.091 | 0.10±0.001
7.2 | T1, T3, T4 | T1, T3, T4 (+T2) | 38.30±0.298 | 0.11±0.002 | 13.63±0.021 | 11.03±0.083 | 50.55±0.212 | 76.33±0.196 | 88.97±0.140 | 0.10±0.002
7.3 | T1, T2, T4 | T1, T2, T4 (+T3) | 38.90±0.457 | 0.10±0.000 | 13.59±0.080 | 11.55±0.147 | 49.23±0.456 | 78.31±0.419 | 90.24±0.227 | 0.10±0.001
7.4 | T1, T2, T3 | T1, T2, T3 (+T4) | 39.59±1.580 | 0.10±0.004 | 13.37±0.244 | 10.70±0.348 | 51.40±0.899 | 76.95±0.705 | 89.38±0.383 | 0.11±0.019

Figure 4: The charts demonstrate the number of epochs (gradient steps to
train) for the single-task, multi-task, and MTML for NYU-v2 dataset.The x-axis
displays the tasks (T1 - T4) for the experiments mentioned in Table II The
blue bar represents the number of training epochs, and the orange refers to
the epochs required by an unseen task to fine-tune the already trained multi-
task and MTML model. The orange bar on top of the blue depicts the number of
epochs the model is further trained during fine-tuning the unseen task.
Similar pattern in the number of epochs is also accounted for the taskonomy
dataset. TABLE III: Test set evaluation results for a single task, multi-task,
and MTML experiments on the taskonomy dataset

Exp. No. | Tasks Involved | Tasks
---|---|---
| Trained | Tested | T1 | T2 | T3 | T4
|  | ( +Tn is fine-tuning | Segmentation | Depth Estimation | Surface Normal | Edge Detection
|  |  on $\mathrm{n^{th}}$ new task) |  Cross-entropy $\downarrow$ |  Mean abs.error $\downarrow$ |  Cosine similarity $\uparrow$ |  Mean abs.error $\downarrow$
1 | Single task learning
|  |  | 0.491±0.025 | 0.013±0.001 | 0.931±0.001 | 0.049±0.001
2 | Multi-task learning
2.1 | T1, T2 | T1, T2 | 0.650±0.076 | 0.013±0.000 |  |
2.2 | T1, T2, T3 | T1, T2, T3 | 0.736±0.080 | 0.013±0.000 | 0.930±0.002 |
2.3 | T1, T2, T3, T4 | T1, T2, T3, T4 | 0.628±0.025 | 0.013±0.000 | 0.930±0.001 | 0.050±0.001
3 | Multi-task learning, addition of new task
3.1 | T1, T2 | T1, T2 (+ T3) | 1.286±0.184 | 0.012±0.000 | 0.931±0.002 |
3.2 | T1, T2 | T1, T2 (+ T3, T4) | 1.217±0.246 | 0.013±0.000 | 0.931±0.003 | 0.050±0.000
3.3 | T1, T2, T3 | T1, T2, T3 (+T4) | 1.330±0.080 | 0.014±0.000 | 0.931±0.001 | 0.049±0.000
4 |  MTML, meta testing phase involves addition of new task
4.1 | T1, T2 | T1, T2 (+ T3) | 2.000±0.136 | 0.014±0.001 | 0.937±0.006 |
4.2 | T1, T2 | T1, T2 (+ T3, T4) | 1.826±0.143 | 0.014±0.000 | 0.939±0.002 | 0.050±0.001
4.3 | T1, T2, T3 | T1, T2, T3 (+T4) | 2.175±0.126 | 0.014±0.000 | 0.929±0.000 | 0.049±0.000
4.4 | T1, T2, T3, T4 | T1, T2, T3, T4 | 2.300±0.044 | 0.063±0.066 | 0.931±0.001 | 0.046±0.000
5 | Multi-task learning, leave one task out format
5.1 | T2, T3, T4 | T2, T3, T4 | - | 0.013±0.001 | 0.930±0.001 | 0.051±0.001
5.2 | T1, T3, T4 | T1, T3, T4 | 0.639±0.038 | - | 0.930±0.001 | 0.052±0.003
5.3 | T1, T2, T4 | T1, T2, T4 | 0.658±0.089 | 0.013±0.000 | - | 0.048±0.001
5.4 | T1, T2, T3 | T1, T2, T3 | 0.736±0.080 | 0.013±0.000 | 0.930±0.002 | -
6 | Multi-task learning, leave one task out format, addition of the left out task
6.1 | T2, T3, T4 | T2, T3, T4 (+ T1) | 0.707±0.100 | 0.013±0.001 | 0.932±0.001 | 0.050±0.001
6.2 | T1, T3, T4 | T1, T3, T4 (+T2) | 0.818±0.108 | 0.013±0.000 | 0.930±0.001 | 0.048±0.000
6.3 | T1, T2, T4 | T1, T2, T4 (+T3) | 1.281±0.041 | 0.013±0.000 |  0.935±0.000 | 0.050±0.000
6.4 | T1, T2, T3 | T1, T2, T3 (+T4) | 1.330±0.080 | 0.014±0.000 | 0.931±0.001 | 0.049±0.000
7 |  MTML, leave one task out format, addition of left out task in meta testing
7.1 | T2, T3, T4 | T2, T3, T4 (+ T1) | 0.626±0.021 | 0.013±0.001 | 0.930±0.002 | 0.050±0.001
7.2 | T1, T3, T4 | T1, T3, T4 (+T2) | 1.303±0.056 | 0.014±0.000 | 0.932±0.000 | 0.050±0.000
7.3 | T1, T2, T4 | T1, T2, T4 (+T3) | 2.250±0.040 | 0.014±0.000 | 0.937±0.001 | 0.051±0.001
7.4 | T1, T2, T3 | T1, T2, T3 (+T4) | 2.175±0.126 | 0.014±0.000 | 0.929±0.000 | 0.049±0.000

Comparable to the state-of-the-art: Table I displays performance for the four
tasks as available in the literature, along with the multi-task, and MTML
performance obtained in this work. The performance of our single task models
is displayed in Exp.1 of Table II and III. For the NYU-v2 data set, our
proposed MTML approach outperforms all the baseline works for three out of
four tasks. For the semantic segmentation task, our single-task learning model
performs best. While in the taskonomy data set, MTML shows best performance
for surface normal prediction, and edge detection. Both our single-task models
and our multi-task models perform equally well when it comes to depth
estimation. For the semantic segmentation task, it is clear that our models’
performance is not up to par with the performance reported in the literature.
Similar deterioration in the performance of semantic segmentation task was
observed in many of the experiments and is discussed in the Section VII. Fig.
3 displays the qualitative output of all the tasks on NYU-v2 and taskonomy
datasets for a single task, MTL and MTML experiments, i.e., ‘ours’ in Table I
(corresponding to Exp. 1, 2.3 and 4.4 in Tables II & III). In terms of
qualitative performance, MTML clearly excels in all tasks. Exp 4.4 uses MTML
to train and test all four tasks, so no new task is added.

Why multi-task over single task?: Exp 1, 2, and 5 in Table II and III show
that vanilla MTL works just as well, and in some cases even better, than its
single-task counterpart. It is important to note that our single-task models
are now being used as the baseline for all comparisons. Fig. 4 demonstrates
that comparable results for MTL can be achieved in far fewer training epochs
than when they are trained individually (plots for Exp. 1 vs 2.3). These
results also help determine which task combination work best and which suffers
from negative information transfer between the tasks. For example, in Table
III Exp 2, shows three types of task combinations wherein the semantic
segmentation task performs the worst in the ensemble as compared to their
single task (Exp.1). While the other tasks in the combination are unaffected.
Similarly in Table II Exp.5, the task combination
[$\mathrm{T_{1},T_{2},T_{3}}$] i.e., Exp. 5.4 is performing comparatively
better than the others.

Better and faster adaptation of new task: Exp 3,4,6, & 7, display the
performance of MTL and MTML when a new task is added during the test phase. To
begin with, first consider the MTL performance on NYU-v2 dataset i.e., Table
II. In MTL setting, when surface normal estimation is added as a new task
(Exp. 3.1, 3.2, 7.3), it performs better than or similar to single task
learning but in fewer training epochs for both the datasets. While, the error
increases w.r.t. single task when edge detection is added as unseen task i.e.,
Exp. 3.2, 3.3, 7.4. For some of the multi-task models like, Exp. 6.1, 6.2, 6.3
the addition of an unseen task to the pre-trained models which are Exp. 7.1,
7.2, 7.3 not only gives comparably good performance on the unseen task, but it
also enhances the metrics for the already trained tasks (compare Exp. 6 and
7). An overall comparative analysis shows that, MTL is better when the tasks
are trained together i.e., Exp 2.3 (T1,T2,T3,T4 trained together) instead of
adding a new task during the testing i.e., Exp 3.3 (T1,T2,T3, +T4). Almost
similar traits as above are also valid for the taskonomy dataset.

In the MTML setting i.e., Exp 4 & 7, the meta trained models are introduced
with new unseen tasks. Exp 7 for the NYU-v2 dataset Table II reveals that MTML
outperforms the single task baseline performance, for various task
combinations. It is also discovered that, while semantic segmentation
performance degrades slightly when meta trained, it performs best when added
as a new task during testing, i.e., Exp 7.1. When comparing MTML to single-
task learning and their equivalent MTL experiments, very similar, and in some
cases even better, evaluation metrics can be observed. Therefore, if models
are trained using MTML, it is simple to add new, previously unseen tasks (both
homogeneous and heterogeneous). While comparing the performance of MTML and
MTL when an unseen task is added (more precisely fine-tuned) during testing,
it is observed that MTML excels, note Exp. 3 vs Exp. 4 and Exp. 6 vs Exp. 7.
Table III, Exp. 7 from the taskonomy dataset shows that the new task achieves
performance on par with that of the single-task and multi-task variants with
significantly fewer training iterations. For both the datasets better
performance is attained in significantly lower number of fine-tuning epochs
than in MTL and corresponding single-task learning (see Fig. 4), behavior that
can be anticipated when employing MAML[4]. Although, the trade off in the
proposed MTML is that it takes a large number of multi-task meta training
epochs. It is also evident that when tasks are trained jointly, the number of
epochs decreases significantly in contrast to single task training.

Discussions: After analyzing the experiments for both the datasets, it is
evident that the proposed MTML paradigm allows for easy addition and faster
adaptation of a new task with equally good performance as compared to single
task learning. For some instances in the result tables MTL performs identical
to MTML, but it can be argued that MTML achieves similar performance in fewer
epochs than MTL. Even though MTML’s performance falls short on the semantic
segmentation task, better results are achieved when the task is added as a
unseen one. For the depth and surface normal estimation task, overall the
qualitative results of MTML demonstrate greater impact, even though
quantitatively the performance is marginally better than MTL. MTML excels
because meta-knowledge of task combinations is accumulated in the meta
parameters that are shared with the unseen task. The bi-level optimization in
the multi-task meta training stage facilitates learning generalized parameters
for effectively incorporating an unknown task in a multi-task setting. In MTL,
however, the optimal parameters for collectively learning the source tasks are
shared. Furthermore, the highlight of the few-shot variant of meta learning is
that, it performs very well on an unseen task; however, the target tasks
(unseen tasks) are substantially similar (but not identical) to the source
tasks (learning episodes), i.e., usually it is a new label class. The MTML
paradigm, however, that is put forth in this work has provided a method for
effectively adding a new (dissimilar) task while also improving the
performance of all (both source and target) tasks. On the other hand MTL
performs very well when the multiple tasks are trained together, as compared
to addition (or fine-tuning) an unseen task.

##  VII Unsatisfactory performance of the semantic segmentation task

For the taskonomy data set, from the performance analysis of table III, we
observe that for the semantic segmentation task the outcomes are very
unsatisfying quantitatively. In spite of this, the models are learning to
segment the images, and the qualitative performance is encouraging. We
investigated the cause for this and came up with a few possible explanations:

First, the segmentation masks given for each image are not annotated by
humans, in fact, they are pseudo labels, i.e., the masks are distilled from
FCIS [46]. While testing the MTML experiments 4, and 7, it was discovered that
these models were found to be more effective at learning than the ground truth
labels (masks) because they were able to recognize objects in the image that
were not included in the ground truth annotation but should have been. A few
such examples are shown in Fig. 5. This is one of the reasons why the semantic
segmentation results are degraded. Fig. 5 shows the class activation maps of
the classes absent in the ground truth, which is learned by MTML. The better
explainability of class-specific discriminative regions is facilitated by the
class activation maps [47]. Observing the qualitative findings reveals that
the semantic segmentation task is doing satisfactorily; however, it does not
produce very good quantitative metrics when compared to the incomplete ground
truth masks.

Second, the segmentation classes in the taskonomy dataset are highly
unbalanced, several images only contain ‘background’ class or just one class,
behaving like a binary class problem, although the overall dataset is a multi-
class. This is because only 17 of the 80 segmentation classes used to train
the FCIS [46] were in the taskonomy dataset, and rest are marked as
background.

Third, reason for the under-performance can probably be ‘negative transfer’
[1]. Although all the tasks are pixel-level, negative transfer is possible
because of the nature of the tasks: segmentation is the only pixel-level
classification task while all others are regression tasks. This can be solved
using task-specific hyper-parameters, like learning rate, weight decay, etc.
Since, in this work to obtain an appropriate comparison between learning
paradigms all tasks use identical hyper-parameters. We believe this is one of
the main reason of slight degradation in the performance of the semantic
segmentation task for the MTML experiments (Exp. 4, 7.2, 7.3, 7.4 of Table
II), also for the NYU-v2 dataset.

Image | GT | Pred. | Act. map
---|---|---|---
|  |  |
|  |  |
|  |  |
Figure 5: This figure shows the input RGB image, the corresponding ground
truth (GT) segmentation mask and the predicted segmentation mask (Pred.) by
the MTML model and the class activation maps (Act. map) of the missing class.
The red box highlights the object not present in the ground truth (pseudo
labels), but the proposed MTML model learns to detect and segment them. For
example, in row 1 and 2, the plant (class-potted plant) is not identified in
the pseudo labels. Rows 1 and 2 show the class activation map of class-potted
plant, and row 3 shows class-couch.

##  VIII Conclusion and Future scope

This work proposes to combine multi-task and meta learning by introducing
multi-task learning episodes to meta train the network and allows to further
test the network by introducing a new (unseen) task. Theoretically, if the
properties of meta and multi-task learning are combined, such an ensemble
should deliver good performance for the new task in fewer steps than training
the single task from scratch. Comprehensive empirical analysis of MTML
performance supports the hypothesis that MTML indeed performs best compared to
vanilla MTL and single-task learning. In addition to that, it allows for swift
adaptation to an unseen task. However, we do observe that the semantic
segmentation task under-performs, because the presences of incorrect pseudo
labels, or negative task transfer. Overall, MTML is a approach that can
efficiently train several tasks together and is capable of faster adaptation
to new tasks if these are not too far off from the already learned tasks.
Furthermore, MTML is a task and model agnostic paradigm that may be utilized
for any heterogeneous or homogeneous tasks (due to multi-task learning) and
any multi-task architecture (due to optimization based meta learning).

Future research could benefit from using better loss-balancing techniques and
task-specific hyper-parameters to further boost performance across all the
tasks and, in particular, to prevent negative transfer. Because the proposed
method is model and task agnostic, other complex multi-task architectures can
be investigated for more complex tasks. However, such intricate fusion models,
while extremely useful, come at a high computational cost. While we outline
one strategy for fusing multi-task and meta-learning, we acknowledge that
there are likely other strategies for doing so to reap the benefits of both
learning paradigms.

## Acknowledgment

The authors express their gratitude for the computational resources made
available by the National Supercomputer Centre at Linköping University,
specifically the Berzelius supercomputing system, support by the Knut and
Alice Wallenberg Foundation. We would also like to extend our sincere
appreciation to Konstantina Nikolaidou (Luleå University of Technology) and
Sameer Prabhu for their insightful comments and constructive feedback on
earlier drafts of this paper.

## References

  * [1] R. Caruana, “Multitask learning,” _Mach. Learn._ , vol. 28, no. 1, p. 41–75, Jul. 1997.
  * [2] D. Li and H. Zhang, “Improved regularization and robustness for fine-tuning in neural networks,” in _Neural Information Processing Systems_ , 2021.
  * [3] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, “Meta-learning in neural networks: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 09, pp. 5149–5169, sep 2022.
  * [4] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _Proceedings of the 34th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 06–11 Aug 2017, pp. 1126–1135.
  * [5] S. Thrun and L. Pratt, _Learning to Learn: Introduction and Overview_. Boston, MA: Springer US, 1998, pp. 3–17.
  * [6] S. Ravi and H. Larochelle, “Optimization as a model for few-shot learning,” in _ICLR_ , 2017.
  * [7] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” _CoRR_ , vol. abs/1803.02999, 2018.
  * [8] Z. Li, F. Zhou, F. Chen, and H. Li, “Meta-sgd: Learning to learn quickly for few shot learning,” _CoRR_ , vol. abs/1707.09835, 2017. [Online]. Available: http://arxiv.org/abs/1707.09835
  * [9] Y. Lee and S. Choi, “Gradient-based meta-learning with learned layerwise metric and subspace,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 2927–2936.
  * [10] P. Liu and X. Huang, “Meta-learning multi-task communication,” 2018.
  * [11] I. Tarunesh, S. Khyalia, V. Kumar, G. Ramakrishnan, and P. Jyothi, “Meta-learning for effective multi-task and multilingual modelling,” _arXiv preprint arXiv:2101.10368_ , 2021.
  * [12] J. Y. Lee, K. A. Lee, and W. S. Gan, “Generating personalized dialogue via multi-task meta-learning,” in _Proceedings of the 25th Workshop on the Semantics and Pragmatics of Dialogue - Full Papers_. Potsdam, Germany: SEMDIAL, Sep. 2021.
  * [13] B. Tian, Y. Zhang, J. Wang, and C. Xing, “Hierarchical inter-attention network for document classification with multi-task learning.” in _IJCAI_ , 2019, pp. 3569–3575.
  * [14] A. Ghadirzadeh, X. Chen, P. Poklukar, C. Finn, M. Björkman, and D. Kragic, “Bayesian meta-learning for few-shot policy adaptation across robotic platforms,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 1274–1280.
  * [15] J. Bronskill, J. Gordon, J. Requeima, S. Nowozin, and R. Turner, “Tasknorm: Rethinking batch normalization for meta-learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 1153–1164.
  * [16] H. Wang, H. Zhao, and B. Li, “Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation,” _ArXiv_ , vol. abs/2106.09017, 2021.
  * [17] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, “Learning and transferring multi-task deep representation for face alignment,” _arXiv preprint arXiv:1408.3967_ , vol. 3, 2014.
  * [18] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3994–4003.
  * [19] J. Dai, K. He, and J. Sun, “Instance-aware semantic segmentation via multi-task network cascades,” 06 2016, pp. 3150–3158.
  * [20] Y. Gao, J. Ma, M. Zhao, W. Liu, and A. L. Yuille, “Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3205–3214.
  * [21] D. Xu, W. Ouyang, X. Wang, and N. Sebe, “Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 675–684.
  * [22] S. Pramanik, P. Agrawal, and A. Hussain, “Omninet: A unified architecture for multi-modal multi-task learning,” _ArXiv_ , vol. abs/1907.07804, 2019.
  * [23] A. R. Zamir, A. Sax, , W. B. Shen, L. Guibas, J. Malik, and S. Savarese, “Taskonomy: Disentangling task transfer learning,” in _2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2018.
  * [24] S. James, M. Bloesch, and A. J. Davison, “Task-embedded control networks for few-shot imitation learning,” in _Conference on robot learning_. PMLR, 2018, pp. 783–795.
  * [25] L. Lan, Z. Li, X. Guan, and P. Wang, “Meta reinforcement learning with task embedding and shared policy,” _arXiv preprint arXiv:1905.06527_ , 2019.
  * [26] A. Achille, M. Lam, R. Tewari, A. Ravichandran, S. Maji, C. C. Fowlkes, S. Soatto, and P. Perona, “Task2vec: Task embedding for meta-learning,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 6430–6439.
  * [27] Y. Zhang, Y. Wei, and Q. Yang, “Learning to multitask,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018.
  * [28] X. Sun, R. Panda, R. Feris, and K. Saenko, “Adashare: Learning what to share for efficient deep multi-task learning,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 8728–8740, 2020.
  * [29] P. Guo, C.-Y. Lee, and D. Ulbricht, “Learning to branch for multi-task learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 3854–3863.
  * [30] A. Maurer, M. Pontil, and B. Romera-Paredes, “The benefit of multitask representation learning,” _Journal of Machine Learning Research_ , vol. 17, no. 81, pp. 1–32, 2016.
  * [31] M. Huisman, J. N. Van Rijn, and A. Plaat, “A survey of deep meta-learning,” _Artificial Intelligence Review_ , vol. 54, no. 6, pp. 4483–4541, 2021.
  * [32] P. K. Nathan Silberman, Derek Hoiem and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _ECCV_ , 2012.
  * [33] S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool, “Multi-task learning for dense prediction tasks: A survey,” _IEEE transactions on pattern analysis and machine intelligence_ , 2021.
  * [34] F. Yu, V. Koltun, and T. Funkhouser, “Dilated residual networks,” in _Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [35] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv preprint arXiv:1706.05587_ , 2017.
  * [36] A. Giusti, D. C. Cireşan, J. Masci, L. M. Gambardella, and J. Schmidhuber, “Fast image scanning with deep max-pooling convolutional neural networks,” in _2013 IEEE International Conference on Image Processing_. IEEE, 2013, pp. 4034–4038.
  * [37] M. Holschneider, R. Kronland-Martinet, J. Morlet, and P. Tchamitchian, “A real-time algorithm for signal analysis with the help of the wavelet transform,” in _Wavelets_. Springer, 1990, pp. 286–297.
  * [38] S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 1871–1880.
  * [39] S. Ruder, J. Bingel, I. Augenstein, and A. Søgaard, “Latent multi-task architecture learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 4822–4829.
  * [40] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _International Conference on Learning Representations_ , 2019.
  * [41] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2015.
  * [42] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in _International conference on machine learning_. PMLR, 2013, pp. 1139–1147.
  * [43] J. Hu, M. Ozay, Y. Zhang, and T. Okatani, “Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 1043–1051.
  * [44] S. Paul, B. Jhamb, D. Mishra, and M. S. Kumar, “Edge loss functions for deep-learning depth-map,” _Machine Learning with Applications_ , vol. 7, p. 100218, 2022.
  * [45] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7482–7491.
  * [46] Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei, “Fully convolutional instance-aware semantic segmentation,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 4438–4446.
  * [47] J. Gildenblat and contributors, “Pytorch library for cam methods,” https://github.com/jacobgil/pytorch-grad-cam, 2021.

  *[MTML]: Multi-task Meta Learning
  *[MTL]: Multi-Task Learning
  *[MAML]: Model Agnostic Meta Learning