not used to guide the learning of feature. In more complex learning scenarios with an abundance of possible features, these networks could very well learn few label-relevant features, thereby compromising the performance of the top classifier.

Instead of layer-wise pre-training, several recent approaches train the whole network using a hybrid objective that contains supervised and unsupervised error terms [@Zhao_etal15]. In some of these network configurations, the unsupervised error terms are local to each layer [@Zhang_etal16]. The supervised error term, however, requires backpropagating errors through the whole network. This requirement is avoided in the training approach in [@Ranzato_Szummer08] used to learn to extract compact feature vectors from documents: training proceeds layer by layer where the error in each layer is a combination of a reconstruction error and a supervised error coming from a local classifier. The local auxiliary decoder and classifier pathways still require training, however. Other approaches also make use of a combination of supervised (label-dependent) and unsupervised error signals to train Boltzmann machines as discriminative models [@Larochelle_Bengio08; @Goodfellow_etal13a]. Learning in [@Goodfellow_etal13a], however, is more computationally demanding than our approach as as it involves several iterations to approach the mean-field equilibrium point of the network, and errors are still backpropagated 