Fourier spectrum (x-axis) of the networks (calculated using Lemmas 1 and 2 of @Rahaman2019) as training progresses (y-axis) averaged over 10 training runs. Colours show each frequency’s amplitude clipped between 0 and 1. The second row shows samples of randomly generated target functions and the function learnt by the networks. []{data-label="fig:network_fourier"}](figures/spectral_dynamics_baseline10x.png "fig:"){width="28.00000%"} ![As in @Rahaman2019, we train 6-layer deep 256-unit wide ReLU networks trained to regress the function $\lambda(z) = \sum_i \sin(2\pi k_iz+\phi(i))$ with $k_i  \in(5,10,\dots,45,50)$. We train these networks with no noise (Baseline), with GNIs of variance $\sigma^2=0.1$ injected into each layer except the final layer (Noise), and with the explicit regulariser for regression in (Exp Reg). The first row shows the Fourier spectrum (x-axis) of the networks (calculated using Lemmas 1 and 2 of @Rahaman2019) as training progresses (y-axis) averaged over 10 training runs. Colours show each frequency’s amplitude clipped between 0 and 1. The second row shows samples of randomly generated target functions and the function learnt by the networks. []{data-label="fig:network_fourier"}](figures/spectral_dynamics_noise10x.png "fig:"){width="28.00000%"} ![As in @Rahaman2019, we train 6-layer deep 256-unit wide ReLU networks trained to regress the function $\lambda(z) = \sum_i \sin(2\pi k_iz+\phi(i))$ with $k_i  \in(5,10,\dots,45,50)$. We train these networks with no noise (Baseline), with GNIs of 