in the proposed methods themselves (once again, I will note that while I know RNNs quite well, I am rusty on ODEs and only superficially familiar with recent neural ODE work). Most of the questions I have are more related to how the proposed approach is motivated and described, which I will address under Clarity. The experimental design seems sound for the most part; it is perhaps a little frustrating that critical experimental design details are sometimes deferred to the appendix, but this is unavoidable given the NeurIPS page limit, and ultimately, it is not a barrier to reproducibility -- especially since the authors use public datasets and have made the code available. One relatively big concern I have is the train/test split procedure for each experiment and how hyperparameters were tuned. The only mention of a test set is buried in lines 592-594 of the appendix in this ominous sentence: "Best model was selected using early stopping and performance were assessed by applying the best model on a held out test set (10% of the total data). We performed 5-fold cross validation and present the test performance average and standard deviation in all tables." A straightforward read suggests that k-fold cross validation with five different test sets was used instead of a fully independent test set and hyperparameters were tuned on test set performance (!). I'd like the authors to address this and the below questions in the response and to make it crystal clear in the MAIN BODY of the manuscript: * Am I correct that they used five-fold cross validation with no fully independent test set? * Are the folds fixed across runs or generated randomly at the start of each run? * Assuming so, can the authors estimate how many times they examined test set performance (or individual test examples) during experimentation before reporting their final numbers? With this number on hand, can they discuss the risk of test set leakage (direct or indirect) and steps that they took (or could take, in the future) to reduce this risk? Can they also discuss the pros and cons of this approach vs. having an independent test set that they use only at the very end? * How exactly did the authors tune hyperparameters? During each k-fold split, do they set aside some data as a validation set, or are they tuning on test set accuracy? * Did the authors invest the same amount of effort in tuning the hyperparameters of baselines as they did the proposed model? CLARITY: The presentation of the idea and experiments could use some work. As noted above in Quality, some critical aspects of experimental design were omitted. It is not necessary for the reader to be able to reproduce exactly the methods and experiments from just reading the main body -- especially if the details are clearly stated in the appendix and code is shared -- but things like, e.g., train/test procedure split are essential for the reader to be able to interpret the results. As for the proposed methods themselves, the GRU-Bayes component did not feel clearly motivated. Admittedly, my experience with RNNs has focused on predicting an independent target, e.g., classification, vs. next step ahead forecasting, but the RNN forecasting work I've seen I think usually uses a simpler input-output problem formulation, akin to language modeling, without casting it as Bayesian filtering. Is this a necessary component for adding the ODE piece, or is it just an additional improvement? On that note, if the ODE and Bayes components are independent, I'd love to see an additional row in Table 2 for a plain "GRU-ODE." Another detail: my understanding from the Chen paper is that minibataching with the ODE architecture is non-trivial. I didn't fully understand the description on lines 159-162 of how the authors dealt with these challenges. Also, it's not clear to me at all (from this paper or Chen) how backpropagation-through-time works in this setting. Since this paper is an extension of the NODE idea to a recurrent setting, I think this would be a very valuable discussion to include in the paper's main body. As far as I can tell, the only prior work on neural ODEs that the paper cites is the Chen paper, but a quick Google Scholar search and some citation following yields a number of papers published on the topic over the last year, including several by Ruthotto and Haber. Can the authors place their work in the larger context of research on the intersection of neural nets and ODEs, not just the Chen paper? Also, is there any connection to continuous time Bayes nets or piecewise-constant conditional intensity models? SIGNIFICANCE: Setting aside my concerns about the risk of test set leakage, this paper seems like an important advance in time series modeling with RNNs. Continuous time dynamics and irregular sampling present fundamental challenges in the field, particularly in key application areas like healthcare, and solutions that rely on discretization and imputation or heuristics (like passing in time deltas) feel unsatisfactory. Ideas from dynamical systems and control have been under-explored, at least in deep learning, and seem to offer very promising directions of inquiry. This paper seems like a nice (albeit arguably incremental) extension of NODE to recurrent architectures and a valuable empirical demonstration of its effectiveness for real world time series modeling. My suspicion is that in the big picture, this paper will be dwarfed by previous work on neural ODEs, but it is nonetheless valuable follow-up work and should be of interest to practitioners.
* Are the folds fixed across runs or generated randomly at the start of each run?