In the context of multi-task learning, neural networks with branched architectures have often been employed to jointly tackle the tasks at hand. Such ramified networks typically start with a number of shared layers, after which different tasks branch out into their own sequence of layers. Understandably, as the number of possible network configurations is combinatorially large, deciding what layers to share and where to branch out becomes cumbersome. Prior works have either relied on ad hoc methods to determine the level of layer sharing, which is suboptimal, or utilized neural architecture search techniques to establish the network design, which is considerably expensive. In this paper, we go beyond these limitations and propose a principled approach to automatically construct branched multi-task networks, by leveraging the employed tasks' affinities. Given a specific budget, i.e. number of learnable parameters, the proposed approach generates architectures, in which shallow layers are task-agnostic, whereas deeper ones gradually grow more task-specific. Extensive experimental analysis across numerous, diverse multi-tasking datasets shows that, for a given budget, our method consistently yields networks with the highest performance, while for a certain performance threshold it requires the least amount of learnable parameters. Deep neural networks are usually trained to tackle different tasks in isolation. Humans, in contrast, are remarkably good at solving a multitude of tasks concurrently. Biological data processing appears to follow a multi-tasking strategy too; instead of separating tasks and solving them in isolation, different processes seem to share the same early processing layers in the brain -see e.g. V1 in macaques (Gur & Snodderly, 2007) . Drawing inspiration from such observations, deep learning researchers began to develop multi-task networks with branched architectures. As a whole, multi-task networks (Caruana, 1997) seek to improve generalization and processing efficiency through the joint learning of related tasks. Compared to the typical learning of separate deep neural networks for each of the individual tasks, multi-task networks come with several advantages. First, due to their inherent layer sharing (Kokkinos, 2017; Lu et al., 2017; Kendall et al., 2018; Guo et al., 2018; , the resulting memory footprint is typically substantially lower. Second, as features in the shared layers do not need to be calculated repeatedly for the different tasks, the overall inference speed is often higher (Neven et al., 2017; Lu et al., 2017) . Finally, multi-task networks may outperform their single-task counterparts (Kendall et al., 2018; Xu et al., 2018; Sener & Koltun, 2018; Maninis et al., 2019) . Evidently, there is merit in utilizing multi-task networks. When it comes to designing them, however, a significant challenge is to decide on the layers that need to be shared among tasks. Assuming a hard parameter sharing setting 1 , the number of possible network configurations grows quickly with the number of tasks. As a result, a trial-and-error procedure to define the optimal architecture becomes unwieldy. Resorting to neural architecture search (Elsken et al., 2019) techniques is not a viable option too, as in this case, the layer sharing has to be jointly optimized with the layers types, their connectivity, etc., rendering the problem considerably expensive. Instead, researchers have recently explored more viable alternatives, like routing (Rosenbaum et al., 2018) , stochastic filter grouping (Bragman et al., 2019) , and feature partitioning (Newell et al., 2019) , which are, however, closer to the soft parameter sharing setting. Previous works on hard parameter sharing opted for the simple strategy of sharing the initial layers in the network, after which all tasks branch out simultaneously. The point at which the branching occurs is usually determined ad hoc (Kendall et al., 2018; Guo et al., 2018; Sener & Koltun, 2018) . This situation hurts performance, as a suboptimal grouping of tasks can lead to the sharing of information between unrelated tasks, known as negative transfer . In this paper, we go beyond the aforementioned limitations and propose a novel approach to decide on the degree of layer sharing between tasks in order to eliminate the need for manual exploration. To this end, we base the layer sharing on measurable levels of task affinity or task relatedness: two tasks are strongly related, if their single task models rely on a similar set of features. Zamir et al. (2018) quantified this property by measuring the performance when solving a task using a variable sets of layers from a model pretrained on a different task. However, their approach is considerably expensive, as it scales quadratically with the number of tasks. Recently, Dwivedi & Roig (2019) proposed a more efficient alternative that uses representation similarity analysis (RSA) to obtain a measure of task affinity, by computing correlations between models pretrained on different tasks. Given a dataset and a number of tasks, our approach uses RSA to assess the task affinity at arbitrary locations in a neural network. The task affinity scores are then used to construct a branched multitask network in a fully automated manner. In particular, our task clustering algorithm groups similar tasks together in common branches, and separates dissimilar tasks by assigning them to different branches, thereby reducing the negative transfer between tasks. Additionally, our method allows to trade network complexity against task similarity. We provide extensive empirical evaluation of our method, showing its superiority in terms of multi-task performance vs computational resources. In this paper, we introduced a principled approach to automatically construct branched multi-task networks for a given computational budget. To this end, we leverage the employed tasks' affinities as a quantifiable measure for layer sharing. The proposed approach can be seen as an abstraction of NAS for MTL, where only layer sharing is optimized, without having to jointly optimize the layers types, their connectivity, etc., as done in traditional NAS, which would render the problem considerably expensive. Extensive experimental analysis shows that our method outperforms existing ones w.r.t. the important metric of multi-tasking performance vs number of parameters, while at the same time showing consistent results across a diverse set of multi-tasking scenarios and datasets. MTAN We tried re-implementing the MTAN model ) using a ResNet-50 backbone. The architecture was based on the Wide-ResNet architecture that is used in the original paper. After extensive hyperparameter tuning, we were unable to get a meaningful result on the Cityscapes dataset when trying to solve all three tasks jointly. Note that, the authors have only shown results in their paper when training semantic segmentation and monocular depth estimation.
A method for the automated construction of branched multi-task networks with strong experimental evaluation on diverse multi-tasking datasets. This paper proposes a novel soft parameter sharing Multi-task Learning framework based on a tree-like structure. This paper presents a method to infer multi-task networks architecture to determine which part of the network should be shared among different tasks.