I do have several queries/concerns however:
- **a. Fixed time horizon**: The use of an MLP to convert the per-timestep embeddings into per-sequence Fourier coefficients means that you can only consider fixed-length sequences. This seems to me to be a real limitation, since often neural/behavioral data – especially naturalistic behavior – is not of a fixed length. This could be remedied by using an RNN or neural process in place of the MLP, so this is not catastrophic as far as I can tell. However, I at least expect to see this noted as a limitation of the method, and, preferably, substitute in an RNN or neural process for the MLP in one of the examples, just to concretely demonstrate that this is not a fundamental limitation.
- **b. Hidden hyperparameters and scaling issues**: Is there a problem if the losses/likelihoods from the channels are “unbalanced”? E.g. if the behavioral data is 1080p video footage, and you have say 5 EEG channels, then a model with limited capacity may just ignore the EEG data. This is not mentioned anywhere. I think this can be hacked by including a $\lambda$ multiplier on the first term of (6) or raising one of the loss terms to some power (under some sensible regularization), trading off the losses incurred by each channel and making sure the model pays attention to all the data. I am not 100% sure about this though. Please can the authors comment.
- **c. Missing experiments**: There are a couple of experiments/baselines that I think should be added.
- Firstly, in Figure 3, I'd like to see a model that uses the data independently to estimate the latent states and reconstruction. It seems unfair to compare multimodal methods to methods that use just one channel. I’m not 100% sure what this would look like, but an acceptable baseline would be averaging the predictions of image-only and neuron-only models (co-trained with this loss). At least then all models have access to the same data, and it is your novel structure that is increasing the performance.
- Secondly, I would like to see an experiment sweeping over the number of observed neurons in the MNIST experiment. If you have just one neuron, then performance of MM-GP-VAE should be basically equivalent to GP-VAE. If you have 1,000,000 neurons, then you should have near-perfect latent imputations (for a sufficiently large model), which can be attributed solely to the neural module. This should be a relatively easy experiment to add and is a good sanity check.
- Finally, and similarly to above, i’d like to see an experiment where the image is occluded (half of the image is randomly blacked out). This (a) simulates the irregularity that is often present in neural/behavioral data (e.g. keypoint detection failed for some mice in some frames), and (b) would allow us to inspect the long-range “inference” capacity of the model, as opposed to a nearly-supervised reconstruction task.
Again, these should be reasonably easy experiments to run. I’d expect to see all of these experiments included in a final version (unless the authors can convince me otherwise).
- **d. Slightly lacking analysis**: This is not a deal-breaker for me, but the analysis of the inferred latents is somewhat lacking. I’d like to see some more incisive analysis of what the individual and shared features pull out of the data – are there shared latent states that indicate “speed”, or is this confined to the individual behavioral latent? Could we decode a stimulus type from the continuous latent states? How does decoding accuracy from each of the three different $z$ terms differ? etc. I think this sort of analysis is the point of training and deploying models like this, and so I was disappointed to not see any attempt at such an analysis. This would just help drive home the benefits of the method.
### Minor weaknesses / typographical errors:
1. Page 3: why are $\mu_{\psi}$ and $\sigma_{\psi}^2$ indexed by $\psi$? These are variational posteriors and are a function of the data; whereas $\psi$ are static model parameters.
2. Use \citet{} for textual citations (e.g. “GP-VAE, see (Casale et al., 2018).” -> “GP-VAE, see Casale et al. (2018).”)
3. The discussion of existing work is incredibly limited (basically two citations). There is a plethora of work out there tackling computational ethology/neural data analysis/interpretable methods. This notable weakens the paper in my opinion, because it paints a bit of an incomplete picture of the field, and actually obfuscates why this method is so appealing! I expect to see a much more thorough literature review in any final version.
4. Text in Figure 5 is illegible.
5. Only proper nouns should be capitalized (c.f. Pg 2 “Gaussian Process” -> “Gaussian process”), and all proper nouns should be capitalized (c.f. Pg 7 “figure 4(c)”).
6. Figure 1(a): Is there are sampling step to obtain $\tilde{\mu}$ and $\tilde{\sigma}^2$? This sample step should be added, because right now it looks like a deterministic map.
7. I think “truncate” is more standard than “prune” for omitting higher-frequency Fourier terms.
8. I find the use of “A” and “B” very confusing – the fact that A is Behaviour, and B is Neural? I’m not sure what better terms are. I would suggest B for Behavioural – and then maybe A for neural? Or A for (what is currently referred to as) behavioral, but be consistent (sometimes you call it “other”) and refer to it as Auxiliary or Alternative data, and then B is “Brain” data or something.
9. The weakest section in terms of writing is Section 3. The prose in there could do with some tightening. (It’s not terrible, but it’s not as polished as the rest of the text).
10. Use backticks for quotes (e.g. ‘behavioral modality’ -> ``behavioral modality’’).
- Finally, and similarly to above, i’d like to see an experiment where the image is occluded (half of the image is randomly blacked out). This (a) simulates the irregularity that is often present in neural/behavioral data (e.g. keypoint detection failed for some mice in some frames), and (b) would allow us to inspect the long-range “inference” capacity of the model, as opposed to a nearly-supervised reconstruction task. Again, these should be reasonably easy experiments to run. I’d expect to see all of these experiments included in a final version (unless the authors can convince me otherwise).