Using variational Bayes neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks. This results in a rich prior capable of few-shot learning on new tasks. The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments. Analysis on toy tasks show that it can learn from significantly different tasks while finding similarities among them. Experiments on Mini-Imagenet reach state of the art with 74.5% accuracy on 5 shot learning. Finally, we provide two new benchmarks, each showing a failure mode of existing meta learning algorithms such as MAML and prototypical Networks. Recently, significant progress has been made to scale Bayesian neural networks to large tasks and to provide better approximations of the posterior distribution BID4 . Recent works extend fully factorized posterior distributions to more general families BID22 BID21 . It is also possible to sample from the posterior distribution trough mini-batch updates BID23 BID36 .However , for neural networks, the prior is often chosen for convenience. This may become a problem when the number of observations is insufficient to overcome the choice of the prior. In this regime, the prior must express our current knowledge on the task and, most importantly, our lack of knowledge on it. In addition to that, a good approximation of the posterior under the small sample size regime is required, including the ability to model multiple modes. This is indeed the case for Bayesian optimization BID30 , Bayesian active learning BID12 , continual learning BID20 , safe reinforcement learning BID3 , exploration-exploitation trade-off in reinforcement learning BID16 . Gaussian processes BID27 have historically been used for these applications, but an RBF kernel constitute a prior that is unsuited for many tasks. More recent tools such as deep Gaussian processes BID6 show great potential and yet their scalability whilst learning from multiple tasks needs to be improved.Our contributions are as follow:1. We provide a simple and scalable procedure to learn an expressive prior and posterior over models from multiple tasks.2. We reach state of the art performances on mini-imagenet.3. We propose two new benchmarks , each exposing a failure mode of popular meta learning algorithms. In contrast, our method perform well on these benchmarks.• MAML BID11 does not perform well on a collection of sinus tasks when the frequency varies.• Prototypical Network BID29 )'s performance decrease considerably when the diversity of tasks increases.Outline: We first describe the proposed approach in Section 2. In Section 3, we extend to three level of hierarchies and obtain a model more suited for classification. Section 4 review related methods and outline the key differences. Finally, In Section 5, we conduct experiments on three different benchmarks to gain insight in the behavior of our algorithm. Using a variational Bayes framework, we developed a scalable algorithm for hierarchical Bayesian learning of neural networks, called deep prior. This algorithm is capable of transferring information from tasks that are potentially remarkably different. Results on the Harmonics dataset shows that the learned manifold across tasks exhibits the properties of a meaningful prior. Finally, we found that MAML, while very general, will have a hard time adapting when tasks are too different. Also, we found that algorithms based on a single image representation only works well when all tasks can succeed with a very similar set of features. Together, these findings allowed us to reach the state of the art on Mini-Imagenet.
A scalable method for learning an expressive prior over neural networks across multiple tasks. The paper presents a method for training a probabilistic model for Multitasks Transfer Learning by introducing a latent variable per task to capture the commonality in the task instances. The work proposes a variational approach to meta-learning that employs latent variables corresponding to task-specific datasets. Aims to learn a prior over neural networks for multiple tasks. 