of the paper. Organization could definitely be improved and I oftentimes had a bit of a hard time following the discussed steps. But in general, I think the included background is informative and well selected. Though, I could see people having trouble understanding the state-space GP-regression when coming from the more machine-learning like perspective of function/ weight space GP-regression.
Significance: I think there is definitely a significance in this work, as GP-Regression is usually a bit problematic because of the scaling, though it is still used extensively in certain areas, such as Bayesian Optimization or for modelling dynamical systems in robotics.
• Background: f
is a random function which describes a map e.g. f : R × R D s → R
and not a function of X ∈ R N t × N s × D s
as described in eq 1. At least, when one infers the inputs from the definitions of the kernel functions.
• In general, the definitions are confusing and should be checked,e.g. check if f n , k = f ( X n , k )
is correct and properly define X n , k .
• The operator L s
is not mentioned in the text
• 2.1: The description of the process f ¯
is confusing as the relationship to the original process f
is established just at the end.
• It would be helpful to add a bit more background on how the state space model is constructed from the kernel κ t ( ⋅ , ⋅ )
, e.g. why it induces the dimensionality d t
and also describe the limitations that a finite dimensional SDE can only be established, when a suitable spectral decomposition of the kernel exist.
• It should be mentioned that p ( y ∣ H f ¯ ( t n ) )
has to be chosen Gaussian, as otherwise Kalman Filtering and Smoothing and CVI is not possible. Later on in the ELBOs this is assumed anyway. • p ( u ) = N ( u ∣ 0 , K z z )
is a finite dimensional Gaussian not a Gaussian process and p(u) is not a random variable ( = not ∼ ).
• The notations for the covariances, e.g. K z z
are discussed in the appendix. I am fine with it; however, it should be referenced as I was confused in the beginning.
• 2.2: The log
is missing for the Fisher information matrix.
• The acronym CVI is used in the paragraph headline before the definition.
• Some figures and tables are not referenced in the text, such as figure 1.
• 3.1: In line 173 the integration should be carried over f ¯
and not s
, I guess?
• I had a bit of a hard time establishing the connection E p ( f ) [ N ( Y ~ ∣ f , V ~ ) ] = p ( Y ~ )
which is the whole point why one part of the ELBO can be calculated using the Kalman filter. Adding this to a sentence to the text would have helped me a lot.
• One question I had was that for computing the ELBO the matrix exponential is needed. When backpropagating the gradient for the hyper parameters, is this efficient? As I am used to using the adjoint method for computing gradients of the (linear) differential equation.
• Reference to Appendix for the RMSE and NLPD metrics is missing.
• It should be mentioned that p ( y ∣ H f ¯ ( t n ) ) has to be chosen Gaussian, as otherwise Kalman Filtering and Smoothing and CVI is not possible. Later on in the ELBOs this is assumed anyway.