We identify a phenomenon, which we refer to as *multi-model forgetting*, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters. To overcome this, we introduce a statistically-justified weight plasticity loss that regularizes the learning of a model's shared parameters according to their importance for the previous models, and demonstrate its effectiveness when training two models sequentially and for neural architecture search. Adding weight plasticity in neural architecture search preserves the best models to the end of the search and yields improved results in both natural language processing and computer vision tasks. Deep neural networks have been very successful for tasks such as visual recognition BID31 and natural language processing BID33 , and much recent work has addressed the training of models that can generalize across multiple tasks BID6 . In this context, when the tasks become available sequentially, a major challenge is catastrophic forgetting: when a model initially trained on task A is later trained on task B, its performance on task A can decline calamitously. Several recent articles have addressed this problem BID13 BID28 BID12 BID16 . In particular, BID13 show how to overcome catastrophic forgetting by approximating the posterior probability, p(θ | D 1 , D 2 ), with θ the network parameters and D 1 , D 2 different datasets representing the tasks.In many situations one does not train a single model for multiple tasks but multiple models for a single task. When dealing with many large models, a common strategy to keep training tractable is to share a subset of the weights across the multiple models and to train them sequentially BID25 BID31 BID17 . This strategy has a major drawback. FIG0 shows that for two models, A and B, the larger the number of shared weights, the more the accuracy of A drops when training B; B overwrites some of the weights of A and this damages the performance of A. We call this multi-model forgetting. The benefits of weight-sharing have been emphasized in tasks like neural architecture search, where the associated speed gains have been key in making the process practical BID25 BID18 , but its downsides remain virtually unexplored.In this paper we introduce an approach to overcoming multi-model forgetting. Given a dataset D, we first consider two models f 1 (D; θ 1 , θ s ) and f 2 (D; θ 2 , θ s ) with shared weights θ s and private weights θ 1 and θ 2 . We formulate learning as the maximization of the posterior p(θ 1 , θ 2 , θ s |D). Under mild assumptions we show that this posterior can be approximated and expressed using a loss, dubbed Weight Plasticity Loss (WPL), that minimizes multi-model forgetting. Our framework evaluates the importance of each weight, conditioned on the previously-trained model, and encourages the update of each shared weight to be inversely proportional to its importance. We then show that our approach extends to more than two models by exploiting it for neural architecture search.Our work is the first of which we are aware to propose a solution to multi-model forgetting. We establish the merits of our approach when training two models with partially shared weights and in the context of neural architecture search. For the former, we establish the effectiveness of WPL in the strict convergence case, where each model is trained until convergence, and in the more realistic loose convergence setting, where training is stopped early. WPL can reduce the forgetting effect by 99% when model A converges fully, and by 52% in the loose convergence case. For neural architecture search, we implement WPL within the efficient ENAS method of BID25 , a state-of-the-art technique that relies on parameter sharing and corresponds to the loose convergence setting. We show that, at each iteration, the use of WPL reduces the forgetting effect by 51% on the most affected model and by 95% on average over all sampled models. Our final results on the best architecture found by the search confirm that limiting multi-model forgetting yields better results and better convergence for both language modeling (on the PTB dataset BID21 ) and image classification (on the CIFAR10 dataset BID14 ). For language modeling the perplexity decreases from 65.01 for ENAS without WPL to 61.9 with WPL. For image classification WPL yields a drop of top-1 error from 4.87% to 3.81%. We also adapt our method to NAO BID19 and show, in appendix due to space limitations, that multi-model forgetting is significantly reduced. We will make our code publicly available upon acceptance of this paper. This paper has identified the problem of multi-model forgetting in the context of sequentially training multiple models: the shared weights of previously-trained models are overwritten during training of subsequent models, leading to performance degradation. We show that the degree of degradation is linked to the proportion of shared weights, and introduce a statistically-motivated weight plasticity loss (WPL) to overcome this. Our experiments on multi-model training and on neural architecture search clearly show the effectiveness of WPL in reducing multi-model forgetting and yielding better architectures, leading to improved results in both natural language processing and computer vision tasks. We believe that the impact of WPL goes beyond the tasks studied in this paper. In future work, we plan to integrate WPL within other neural architecture search strategies in which weight sharing occurs and to study its use in other multi-model contexts, such as for ensemble learning. Comparison of different output dropout rates for NAO. We plot the mean validation perplexity while searching the best architecture (top) and the best 5 model's error differences (bottom) for four different dropout rates. Note that path dropping in NAO prevents learning shortly after model initialization. At all the dropout rates, our WPL achieves lower error differences, i.e., it reduces multi-model forgetting, as well as speeds up training.Our approach is general, and its use in the context of neural architecture search is not limited to ENAS. To demonstrate this, we applied it to the neural architecture optimization (NAO) method of BID19 , which also exploits weight-sharing in the search phase. In this context, we therefore investigate (i) whether multi-model forgetting occurs, and if so, (ii) the effectiveness of our approach in the NAO framework. Due to resource and time constraints, we focus our experiments mainly on the search phase, as training the best searched model from scratch takes around 4 GPU days. To evaluate the influence of the dropout strategy of BID3 , we test NAO with or without random path-dropping and with four output dropout rates from 0 to 0.75 by steps of 0.25. As in Section 4.2, in FIG1 , we plot the mean validation perplexity and the best five model's error differences for all models that are sampled during a single training epoch. For random pathdropping, since BID19 exploit a more aggressive dropping policy than that used in BID3 , we can see that validation perplexity quickly plateaus. Hence we do not add our WPL to the path dropout strategy, but use it in conjunction with output dropout.At all four different dropout rates, WPL clearly reduces multi-model forgetting and accelerates training. The level of forgetting decreases with the dropout rate, but our loss always further reduces it. Among the three methods, Nao + path dropping suffers the least from forgetting. However, this is only due to the fact that it does not learn properly. By contrast, our WPL reduces multi-model forgetting while still allowing the models to learn. This shows that our approach generalizes beyond ENAS for neural architecture search.
We identify a phenomenon, neural brainwashing, and introduce a statistically-justified weight plasticity loss to overcome this. This paper discusses the phenomena of “neural brainwashing”, which refers to that the performance of one model is affected via another model sharing model parameters.