on deep convolutional neural networks [@LeCun; @Ciresan], and Alex Graves’s deep recurrent neural networks [@Graves]. The features of DNN can be summarized as follows.

-   Large-scale network: the DNN model usually has very large structure (with many layers), including millions of neurons and connections. Consequently, the computational cost of DNN is extremely high, thus most of the DNN experiments are done with GPUs.

-   Better performance: DNN archived the state-of-the-art results on various tasks and competitions, and brought breakthroughs to those fields, such as handwritten character recognition [@Graves3; @Graves4], image classification [@Ciresan; @Ciresan2], speech recognition [@Graves2] and so on.

-   Lack of theoretical explanation: although the DNN achieved much better performance than conventional methods, there is no convincing explanation for such success. For example, it is well known that some trick in training could improve the performance of DNN significantly, but it was hard to explain such effect theoretically.

Obviously, although the performance of DNN is promising, it still needs to be improved in different ways. On one hand, in order to pursue higher recognition rate, several optimization methods for training were proposed, such as dropout [@dropout] and dropconnect [@dropconnect]. Those methods can reduce the overfitting problem significantly. On the 