local error signals in order to learn the probability distribution of the input layer activations, or in order to minimize local reconstruction errors [@Hinton_etal06; @Hinton_Salakhutdinov06; @Bengio_etal07; @Vincent_etal08; @Erhan_etal10]. These mechanisms, however, are unsupervised and the networks need to be augmented by a classifier layer, typically added on top of the deepest layer. The network weights are then fine-tuned using standard backpropagation to minimize the error at the classifier layer. Supervised layer-wise training has been pursued in [@Bengio_etal07], with auxiliary classifiers that are co-trained, unlike the random fixed auxiliary classifiers proposed here. The supervised layer-wise training is used only as a pre-training step, and results are reported after full network fine-tuning using backpropagation from the top classifier layer. Some approaches forego the fine-tuning step and keep the network fixed after the unsupervised layer-wise training phase, and only train the top classifier layer or SVM on the features learned [@Lee_etal09; @Ranzato_etal07; @Kavukcuoglu_etal10]. Local learning in [@Ranzato_etal07; @Kavukcuoglu_etal10] involves an iterative procedure for learning sparse codes which is computationally demanding. The network architectures in [@Lee_etal09; @Ranzato_etal07; @Kavukcuoglu_etal10] fail to yield intermediate classification results from the intermediate layers. Moreover, their applicability to datasets that are more complex than MNIST is unclear since labels are 