We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) --- DNNs often fit target functions from low to high frequencies --- on high-dimensional benchmark datasets, such as MNIST/CIFAR10, and deep networks, such as VGG16. This F-Principle of DNNs is opposite to the learning behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibits faster convergence for higher frequencies, for various scientific computing problems. With a naive theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset. Understanding the training process of Deep Neural Networks (DNNs) is a fundamental problem in the area of deep learning. We find a common behavior of the gradient-based training process of DNNs, that is, a Frequency Principle (F-Principle): DNNs often fit target functions from low to high frequencies during the training process. In another word, at the early stage of training, the low-frequencies are fitted and as iteration steps of training increase, the high-frequencies are fitted. For example, when a DNN is trained to fit y = sin(x) + sin(2x), its output would be close to sin(x) at early stage and as training goes on, its output would be close to sin(x) + sin(2x). F-Principle was observed empirically in synthetic low-dimensional data with MSE loss during DNN training (Xu et al., 2018; Rahaman et al., 2018) . However, in deep learning, empirical phenomena could vary from one network structure to another, from one dataset to another and could exhibit significant difference between synthetic data and highdimensional real data. Therefore, the universality of the F-Principle remains an important problem for further study. Especially for high-dimensional real problems, because the computational cost of high-dimensional Fourier transform is prohibitive in practice, it is of great challenge to demonstrate the F-Principle. On the other hand, the mechanism underlying the F-Principle and its implication to the application of DNNs, e.g., design of DNN-based PDE solver, as well as their generalization ability are also important open problems to be addressed. In this work, we design two methods, i.e., projection and filtering methods, to show that the FPrinciple exists in the training process of DNNs for high-dimensional benchmarks, i.e., MNIST (LeCun, 1998) , CIFAR10 (Krizhevsky et al., 2010) . The settings we have considered are i) different DNN architectures, e.g., fully-connected network, convolutional neural network (CNN), and VGG16 (Simonyan & Zisserman, 2014) ; ii) different activation functions, e.g., tanh and rectified linear unit (ReLU); iii) different loss functions, e.g., cross entropy, mean squared error (MSE), and loss energy functional in variational problems. These results demonstrate the universality of the F-Principle. To facilitate the designs and applications of DNN-based schemes, we characterize a stark difference between DNNs and conventional numerical schemes on various scientific computing problems, where most of the conventional methods (e.g., Jacobi method) exhibit the opposite convergence behavior -faster convergence for higher frequencies. This difference implies that DNN can be adopted to accelerate the convergence of low frequencies for computational problems. We also intuitively explain with theories under an idealized setting how the smoothness/regularity of commonly used activation functions contributes to the F-Principle. Note that this mechanism is rigorously demonstrated for DNNs of general settings in a subsequent work (Luo et al., 2019) . Finally, we discuss that the F-Principle provides an understanding of good generalization of DNNs in many real datasets (Zhang et al., 2016) and poor generalization in learning the parity function (Shalev-Shwartz et al., 2017; Nye & Saxe, 2018) , that is, the F-Principle which implies that DNNs prefer low frequencies, is consistent with the property of low frequencies dominance in many real datasets, e.g., MNIST/CIFAR10, but is different from the parity function whose spectrum concentrates on high frequencies. Compared with previous studies, our main contributions are as follows: 1. By designing both the projection and filtering methods, we consistently demonstrate the F-Principle for MNIST/CIFAR10 over various architectures such as VGG16 and various loss functions. 2. For the application of solving differential equations, we show that (i) conventional numerical schemes learn higher frequencies faster whereas DNNs learn lower frequencies faster by the FPrinciple, (ii) convergence of low frequencies can be greatly accelerated with DNN-based schemes. 3. We present theories under an idealized setting to illustrate how smoothness/regularity of activation function contributes to the F-Principle. 4. We discuss in detail the implication of the F-Principle to the generalization of DNNs that DNNs are implicitly biased towards a low frequency function and provide an explanation of good and poor generalization of DNNs for low and high frequency dominant target functions, respectively. DNNs often generalize well for real problems (Zhang et al., 2016) but poorly for problems like fitting a parity function (Shalev-Shwartz et al., 2017; Nye & Saxe, 2018) despite excellent training accuracy for all problems. Understanding the differences between above two types of problems, i.e., good and bad generalization performance of DNN, is critical. In the following, we show a qualitative difference between these two types of problems through Fourier analysis and use the F-Principle to provide an explanation different generalization performances of DNNs. For MNIST/CIFAR10, we examineŷ total,k = 1 n total n total −1 i=0 consists of both the training and test datasets with certain selected output component, at different directions of k in the Fourier space. We find thatŷ total,k concentrates on the low frequencies along those examined directions. For illustration,ŷ total,k 's along the first principle component are shown by green lines in Fig. 4(a, b) for MNIST/CIFAR10, respectively. When only the training dataset is used,ŷ train,k well overlaps withŷ total,k at the dominant low frequencies. For the parity function x j e −i2πk·x . As shown in Fig. 4(c) , By experiments, the generalization ability of DNNs can be well reflected by the Fourier analysis. For the MNIST/CIFAR10, we observed the Fourier transform of the output of a well-trained DNN on faithfully recovers the dominant low frequencies, as illustrated in Fig. 4 (a) and 4(b ), respectively, indicating a good generalization performance as observed in experiments. However , for the parity function, we observed that the Fourier transform of the output of a well-trained DNN on {x i } i∈S significantly deviates fromf (k) at almost all frequencies, as illustrated in Fig. 4(c) , indicating a bad generalization performance as observed in experiments. The F-Principle implicates that among all the functions that can fit the training data, a DNN is implicitly biased during the training towards a function with more power at low frequencies. If the target function has significant high-frequency components, insufficient training samples will lead to artificial low frequencies in training dataset (see red line in Fig. 4(c) ), which is the wellknown aliasing effect. Based on the F-Principle, as demonstrated in Fig. 4 (c), these artificial low frequency components will be first captured to explain the training samples, whereas the high frequency components will be compromised by DNN. For MNIST/CIFAR10, since the power of high frequencies is much smaller than that of low frequencies, artificial low frequencies caused by aliasing can be neglected. To conclude, the distribution of power in Fourier domain of above two types of problems exhibits significant differences, which result in different generalization performances of DNNs according to the F-Principle.
In real problems, we found that DNNs often fit target functions from low to high frequencies during the training process. This paper analyzes the loss of neural networks in the Fourier domain and finds that DNNs tend to learn low-frequency components before high-frequency ones. The paper studies the training process of NNs through Fourier analysis, concluding that NNs learn low frequency components before high frequency components.