Warren McCulloch and Walter Pitts [@Pitts], Donald Hebb [@Hebb] and Frank Rosenblatt [@Rosenblatt]. Especially in [@Rosenblatt], a two-layer network is introduced for pattern recognition. Figure \[ANNbasic\] shows the basic function of ANN, that is, the neuron of higher layer is calculated by the neurons of prior layer with the connecting weights. However, there was no solution for the network training until backpropagation (gradient descent) algorithm was created by Paul Werbos [@Werbos]. After James McClelland [@McClelland] introduced the ANN as simulation of natural neural process and its usage in artificial intelligence (AI), the research of ANN became popular. Since then, ANN was successfully applied to image classification [@Benediktsson; @Bischof; @Giacinto], character recognition  [@Rajavelu], face recognition [@Lawrence], speech recognition [@Waibel] and so on. Moreover, a theoretical explanation of ANN’s success was also given by Kurt Hornik [@Hornik].

![The basic function of ANN. \[ANNbasic\]](basicfun.eps){width="30.00000%"}

Because of the limitation of computers, the ANN research stagnated until the new century came. With the development of computers, the training of large-scale neural network became possible. As a result, the research of deep neural networks (DNN) emerged and attracted more and more attention. For example, Geoff Hinton’s deep belief nets [@Hinton], Yann LeCun and Dan Ciresan’s study 