Note, the below concerns have resulted in a lower score, which I would be happy to increase pending the authors’ responses.
**A. Wave fields**
The wave-field comparisons, claims, and references seem a bit strained and unnecessary. Presumably, by “wave-field,” the authors simply mean a vector field that supports wave solutions. In any case, since this term is not oft-used in neuroscience or ML that I am aware of, a brief definition should be provided if the term is kept. However, I am unsure that it is necessary or helpful. That the brain supports wavelike activity is well-established, and some evidence for this is appropriately outlined by the authors. Many computational neuroscience models support waves in a way that has been mathematically analyzed (e.g., Wilson-Cowan neural fields equations). The authors’ discretization methodology suggests a similar connection to such analyses. However, appealing to “physical wave fields” to relate waves and memory seems to be overly speculative and unnecessary for the simple system under study in this manuscript. The brain is a dissipative rather than a conservative system, so that many aspects of physical wave fields may well not apply. Moreover, the single reference the authors do make to the concept does not apply either to the brain or to their wave-RNN. Instead, Perrard et al. 2016 describe a specific study that demonstrates that a particle-pilot wave system can still maintain memory in a very specific way that does not at all clearly apply to brains or the authors’ RNN, despite that study studying a dissipative (and chaotic) system. Instead, the readers would benefit much more from gaining an intuition as to why such wavelike activity might benefit learning and recalling sequential inputs. Unfortunately, Fig. 1 does little to help in this vein.
However, the concept certainly is simple enough, and the authors provide a few intuitions in the manuscript that help. I believe the manuscript would improve by removing the discussion of wave fields and instead providing / moving the intuitive explanations (e.g., the “register or ‘tape’” description on p. 20) as to how waves may help with sequential tasks to the same portion of the Introduction.
**B. Fourier analysis**
Overall, I found the wave and Fourier analysis a bit inconsistent and potentially problematic. While I agree that the wRNNs clearly display waves when plotted directly, the mapping and analysis within the spatiotemporal Fourier domain (FD below) does not always match patterns in the regular spatiotemporal plots (RSP below). Moreover, it’s unclear how much substance they add to the analysis results. In more detail:
1. Constant-velocity, 1-D waves don’t need to be transformed to the FD to infer their speeds. The slopes in the RSP correspond to their speeds. For example, in Fig. 2 (top left), there is a wave that begins at unit 250, step ~400, that continues through to unit 0, step ~650, corresponding to a wave speed of ~1.7 units/step, far larger than the diagonal peak shown in the FD below it that would correspond to a speed of ~0.3 units/step, as indicated by the authors.
2. Similar, seemingly speed mismatches can be observed in the Appendix. E.g., in Fig. 9 (2nd column, top), the slopes of the waves are around 0.35-0.42 units/step (close enough to likely be considered the same speed, especially as they converge in time to form a more clustered wave pulse) from what I can tell, whereas the slopes in the FD below it are ~0.3 for the diagonal (perhaps this is close enough to my rough estimate) and ~0.9, well above any observable wave speed. Perhaps there is a much faster wave that is unobservable in the RSP due to the min/max values set for the image intensity in the plot, but in that case the authors should demonstrate this. Given (a) the potential mismatch in the speeds for the waves that can be observed, (b) the mismatch in the speeds discussed above in Fig. 2, and (c) the fact that some waves may be missed in FD (see below), I would worry about assuming this without checking.
3. As alluded to in the point above, iRNN in Fig. 2 appears to have some fast pulse bursts easily observed in the RSP that don’t show in the FD. For example, there is a very fast wave observable in the RSP in units ~175-180, time steps 0-350. Note, the resolution is poor, but zooming in and scrolling to where the wave begins around unit 175, step 0 makes it clear. If one scrolls vertically such that the bottom of the wave at step 0 is just barely unobservable, then one can see the wave rapidly come into view and continue downwards. Similarly some short-lasting, slower pulses in units near 190, steps 0-350 are observable in the RSP. None of these appear in the FD. Note, this would not take away from the claim that wRNNs facilitate wave activity much more than iRNNs do, but rather that some small amounts—likely insufficient amounts for facilitating sequence learning—of wave activity might still arise in iRNNs. If the authors believe these wavelike activities are aberrant, it would be helpful for them to explain why so.
4. I looked over the original algorithm the authors used (in Section III of “Recognition and Velocity Computation of Large Moving Objects in Images”—RVC paper below—which I would recommend for the authors to cite), and I wonder if an error in the initial calibration steps (steps 1 & 2) occurred that might explain the speed disparities observed between the RSPs and FDs.
5. There do seem to be some different wave speeds—e.g., in Fig. 9, there appear to be fast and narrow excitatory waves overlapping with slow and broad inhibitory waves. But given that each channel has its own wave speed parameter $\nu$, it isn’t clear why a single channel would support multiple wave speeds. This should be explored in greater depth, and if obvious examples of sufficiently different speeds of excitatory waves are known (putatively Fig. 9, 2nd column), these should be clearly shown and carefully described and analyzed.
6. Is there cross-talk across the channels? If so, have the authors examined the images of the hidden units (with dimensions __hidden units__ x __channels__) for evidence of cross-channel waves? If so, perhaps this is one reason for multiple wave speeds to exist per channel?
7. Overall, it is unclear overall what FT adds to the detection of 1-D waves. If there are such waves, we should be able to observe them directly in the RSPs. In skimming over the RVC paper, it seems like it would be most useful in determining velocities of 2-D objects and perhaps wave pulses. That suggests that one place the FD analysis might be useful is if there are cross-channel waves as I mention above. If so, the waves should still be observable in the images (and I would encourage such images be shown), but might be more easily characterized following the marginalization decomposition procedure described in the original algorithm in Section III of the RVC paper. Note, the FDs might also facilitate the detection of multiple wave speeds in the network, as potentially shown in Fig. 9. However, in that case it would seem they should only appear in Fig. 9, and if the speeds are otherwise verified.
8. The authors mention they re-sorted the iRNN units to look for otherwise hidden waves. This seems highly problematic. If there are waves, then re-sorting can destroy them, and if there is only random activity then re-sorting can cause them to look like waves.
**C. Mechanisms**
Finally, while the results are overall impressive, and hypotheses made regarding the underlying mechanisms for the performance levels of the network, there is too little analysis of the these mechanisms. While the ablation study is important and helpful, much more could be done to characterize the relationship between wavelike activity and network performance.
**D. Minor**
1. Fig. 2: Both plots on the right have the leftmost y-axis digits obscured
2. Fig. 9, top, plots appear to have their x- and y- labels transposed (or else the lower FD plots and those in Fig. 2 have theirs transposed.
3. Fig. 15 needs axis labels
4. I looked over the original algorithm the authors used (in Section III of “Recognition and Velocity Computation of Large Moving Objects in Images”—RVC paper below—which I would recommend for the authors to cite), and I wonder if an error in the initial calibration steps (steps 1 & 2) occurred that might explain the speed disparities observed between the RSPs and FDs.