From my point of view, there are 2 main weaknesses in this submission (for details, see below).
1. The method and the experiments are insufficiently described, and I have some questions in this regard. However, I am convinced, that the manuscript can be updated to be much more clear.
2. The empirical evaluation is of limited scope. Qualitatively, the method is evaluated on 2 toy problems (fOU & Hurst index); quantitatively on a single synthetic dataset (stochastic moving MNIST). For the latter, only two baseline models from 2018 and 2020 are compared to. In consequence, the usefulness of the method is not established. On the plus side, there are 3 ablations / further studies.
Minor weaknesses.
- The method appears to be inefficient, with a training time of 39 hours on an NVIDIA GeForce RTX 4090 for one model per model trained on stochastic moving MNIST.
- A detailed comparison with Tong et al. (2022) (who also learn approximations to fBM) is missing. So far, it is only stated that Tong et al. did not apply their model to video data and that it is completely different. A clear illustration of the conceptual differences and a comparison of the pros and cons of each approach would be appreciated (summary in main part + mathematical details in supplementary material).
# Summary
For me, this is a borderline submission. On the one hand, the proposed method is novel, significant and of theoretical interest. On the other hand, there are clarity issues, a weak empirical evaluation and no clear use case. Expecting the clarity issues to be resolved, I rate the submission as a marginally above the acceptance threshold, as the theoretical strengths outweigh the empirical flaws. ----
# Details on major weaknesses
## Point 1 (clarity).
**Regarding the method.** After reading the method and experiment section multiple times, I still have no idea, how to implement it. I am aware of the provided source code, nevertheless I found the paper to be insufficient in this regard.
What I got from section E and Figure 4 is that:
- First, there is an encoding step that returns $h$, a sequence of vectors/matrices over time. Somehow, these vectors are used to compute $\omega$. I have no idea how this $\omega$ is related to the optimal one from Prop. 5.
- $h$ is given to a temporal convolution layer that returns $g$ and this $g$ has as many 'frames' as the input and is used as input to the control function $u$.
- The control, drift and the diffusion function are implemented as neural networks.
- An SDE solution is numerically approximated with a Stratonovich–Milstein solver.
- $\omega$ is used in the decoding step, I do not understand why and how.
- Where do the approximation processes $Y$ enter. How are they parametrized, is $\gamma$ as in Prop 5? Are they integrated separately from $X$?
- The ELBO contains an expectation over sample paths. How many paths are sampled to estimate the mean?
- Fig. 6: Why do the samples from the prior always show a 4 and a 7? Does the prior depend on the observations?
**Regarding Moving MNIST.**
What precisely is the task / evaluation protocol in the experiment on the stoch. moving MNIST dataset? I did not see it specified, but from the overall description it appears that a sequence of 25 frames is given to the model and the task is to return the same 25 frames again (with the goal of learning a generative model).
## Point 2 (empirical evaluation)
- The method is not evaluated on real world data.
- Quantitatively, the method is only evaluated on one synthetic dataset (stoch. moving MNIST).
- While the method is motivated by "*Unfortunately, for many practical scenarios, BM falls short of capturing the full complexity and richness of the observed real data, which often contains long-range dependencies, rare events, and intricate temporal structures that cannot be faithfully represented by a Markovian process"*, the moving MNIST dataset is not of this kind. It is not long range (only 25 frames) and there is no correlated noise.
- The method is only compared to 2 baselines (SVG, SLRVP) on moving MNIST.
- Table 1 does not show standard deviations.
Overall, this would be a far stronger submission, if the experiments were more extensive. This includes:
- Evaluation on more task and datasets, and specifically on datasets were this method is expected to shine, i.e., in the presence of correlated noise. The pendulum dataset of [Becker et al., Factorized inference in high-dimensional deep feature space, ICML 2019] would be one example.
- Comparison with more baselines. In particular, more recent / state-of-the-art methods that do not model a differential equation and the fBM model by Tong et al. ----
There is a typo in Proposition 1: "Markov rocesses"
- Table 1 does not show standard deviations. Overall, this would be a far stronger submission, if the experiments were more extensive. This includes: