We explore the behavior of a standard convolutional neural net in a setting that introduces classification tasks sequentially and requires the net to master new tasks while preserving mastery of previously learned tasks.   This setting corresponds to that which human learners face as they acquire domain expertise, for example, as an individual reads a textbook chapter-by-chapter. Through simulations involving sequences of 10 related tasks, we find reason for optimism that nets will scale well as they advance from having a single skill to becoming domain experts. We observed two key phenomena. First, forward facilitation---the accelerated learning of task n+1 having learned n previous tasks---grows with n. Second, backward interference---the forgetting of the n previous tasks when learning task n+1---diminishes with n.  Forward facilitation is the goal of research on metalearning, and reduced backward interference is the goal of research on ameliorating catastrophic forgetting. We find that both of these goals are attained simply through broader exposure to a domain. We explored the behavior of a standard convolutional neural net for classification tasks in a setting that introduces tasks sequentially and requires the net to master new tasks while preserving mastery of previously learned tasks. This setting corresponds to that which human learners face as they become experts in a domain, for example, as they read a textbook chapter by chapter. Our network exhibits six interesting properties:1. Forward facilitation is observed once the net has acquired sufficient expertise in the domain, as evidenced by requiring less training to learn new tasks as a function of the number of related tasks learned (see highlighted black curve in FIG2 BID8 BID8 . 5. Training performance improves according to a power function of the number of tasks learned, controlling for experience on a task (the slope of the curves in FIG2 , and also according to a power function of the amount of training a given task has received, controlling for number of tasks learned (the slope of the curves in FIG2 ). Power-law learning is a robust characteristic of human skill acquisition, observed on a range of behavioral measures BID20 BID7 . 6. Catastrophic forgetting is evidenced primarily for task 1 when task 2 is learned-the canonical case studied in the literature. However, the model becomes more robust as it acquires sufficient domain experience, and eventually the relearning effort becomes negligible (see copper curves in FIG2 ,f). The anomalous behavior of task 2 is noteworthy, yielding a transition behavior that is perhaps analogous to the "zero one infinity" rule coined by Willem van der Poel.We are able to identify these interesting phenomena because our simulations examined scaling behavior and not just effects of one task on a second-the typical case for studying catastrophic forgetting-or the effects of many tasks on a subsequent task-the typical case for metalearning and few-shot learning. Studying the entire continuum from the first task to the n'th is quite revealing.We found strong evidence for improved learning performance with broader domain expertise, and further investigation is merited. We are beginning investigations that examine how similar tasks must be to facilitate one another: how does scaling behavior change when the tasks dimensions switch across successive episodes (e.g., from color to shape to texture)? Our preliminary results suggest that the domain knowledge acquired is quite general and extends to other dimensions of the images. We are also examining the scaling properties of metalearning methods that are explicitly designed to facilitate transfer. The results presented in this article can serve as a baseline to measure the magnitude of facilitation that the specialized methods offer. A holy grail of sorts would be to identify methods that demonstrate backward facilitation, where training on later tasks improves performance on earlier tasks, and compositional generalization BID11 BID10 BID17 , where learning the interrelationship among earlier tasks allows new tasks to be performed on the first trial. Humans demonstrate the former under rare conditions BID1 BID12 ; the latter is common in human behavior, as when individuals are able to perform a task immediately from instruction . 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329
We study the behavior of a CNN as it masters new tasks while preserving mastery for previously learned tasks