1. The central contribution of modeling weight evolution using ODEs hinges on the mentioned problem of neural ODEs exhibiting inaccuracy while recomputing activations. It appears a previous paper first reported this issue. The reviewer is not convinced about this problem. The current paper doesn't provide a convincing analytical argument or empirical evidence about this issue. 2. Leaving aside the claimed weakness of neuralODE, the idea of modeling weight evolution as ODE is itself very intellectually interesting and worthy of pursuit. But the empirical improvement reported in Table 1 over AlexNet, ResNet-4 and ResNet-10 is <= 1.75 % for both configurations. The improvement of decoupling weight evolution is in fact even small and not consistent - the improvement in ResNet for configuration 2 is smaller than keeping the evolution of parameters and activations aligned. The improvement for ablation study over neuralODE is also minimal. So, the empirical case for the proposed approach is not convincing. 3. The derivation of optimality conditions for the coupled formulation is interesting because of connections to a machine learning application (backpropagation) but a pretty standard textbook derivation from dynamical systems / controls point of view.
1. The central contribution of modeling weight evolution using ODEs hinges on the mentioned problem of neural ODEs exhibiting inaccuracy while recomputing activations. It appears a previous paper first reported this issue. The reviewer is not convinced about this problem. The current paper doesn't provide a convincing analytical argument or empirical evidence about this issue.