Omniglot has been used in an array of settings: generative modeling [@Lake:2015; @Rezende:2016], one-shot learning [@Koch:2015; @Lake:2015; @Shyam:2017], and deep MTL [@Bilen:2017; @Maclaurin:2015; @Meyerson:2018; @Rebuffi:2017; @Yang:2017]. Previous deep MTL approaches used random training/testing splits for evaluation [@Bilen:2017; @Meyerson:2018; @Yang:2017]. However, with model search (i.e. when the model architecture is learned as well), a validation set separate from the training and testing sets is needed. Therefore, in the experiments in this paper, a fixed training/validation/testing split of 50%/20%/30% is introduced for each task. Because training is slow and increases linearly with the number of tasks, a subset of 20 tasks out of the 50 possible is used in the current experiments. These tasks are trained in a fixed random order. Soft ordering is the current state-of-the-art method in this domain [@Meyerson:2018]. The experiments therefore use soft ordering as a starting point for designing further improvements.

Experimental Setup
------------------

For CoDeepNEAT fitness evaluations, all networks are trained using Adam [@Kingma:2014] for 3000 iterations over the 20 alphabets; for CTR, the network is trained for 120 meta-iterations (30,000 iterations). Each iteration is equivalent to one full forward and backward pass through the network with a single example image and label chosen randomly from 