As noted below, I have concerns around the experimental results. More specifically, I feel that there is a relative lack of discussion around the (somewhat surprising) outperformance of baselines that VPBNN is aiming to approximate, and I feel that the experiments are missing what I see as key VPBNN results that otherwise leave the reader with questions. Additionally, I think the current paper would benefit from including measurements and discussion around the specifics of computational and memory costs of their method. Recommendation
In general, I think this could be a great paper. However, given the above concerns, I'm currently inclined to suggest rejection of the paper in its current state. I would highly recommend that authors push further on the noted areas!
Additional comments
p. 1: "The uncertainty is defined based on the posterior distribution." For more clarity it could be helpful to update this to say that the epistemic model uncertainty is represented in the prior distribution, and upon observing data, those beliefs can be updated in the form of a posterior distribution, which yields model uncertainty conditioned on observed data.
p. 2: "The MC dropout requires a number of repeated feed-forward calculations with randomly sampled weight parameters in order to obtain the predictive distribution." This should be updated to indicate that in MC dropout, dropout is used (in an otherwise deterministic model) at test time with "a number of repeated feed-forward calculations" to effectively sample from the approximate posterior, but not directly via different weight samples (as in a variational BNN). With variational dropout, this ends up having a nice interpretation as a variational Bayes method, though no weight distributions are typically directly used with direct MC dropout.
p. 2: Lakshminarayanan et al. (2017) presented random seed ensembles, not bootstrap ensembles (see p. 4 of their work for more info). They used the full dataset, and trained M ensemble members with different random seeds, rather than resampled data.
p. 4: For variance propagation in a dropout layer with stochastic input, it's not exactly clear from the text how variance from the inputs and dropout is being combined into an output Gaussian. I believe using a Gaussian is an approximation, and while that would be fine, I think it would be informative to indicate that. The same issue comes up with local reparameterization for BNNs with parameter distributions, where they can be reparameterized exactly as output distributions (for, say, mean-field Gaussian weight dists) so long as the inputs are deterministic. Otherwise, the product of, say, two Gaussian RVs is non-Gaussian.
p. 7: Figure 1 is too small.
p. 7: "Estimation of ρ is possible by observing the outputs of middle layers several times under the approximate predictive distribution. The additional computation cost is still kept quite small compared to MC dropout." How exactly is ρ
estimated? Is it a one-time cost irregardless of data that can then be used for all predictions from the trained model? Without details, this seems like a key component that can yield arbitrary amounts of uncertainty.
p. 7, 8: For the language modeling experiment, why do you think VPBNN was able to achieve lower perplexity values than MC dropout? The text generally focuses on VPBNN as an approximation to MC dropout, and yet it outperforms it. The text would greatly benefit from more discussion around this point.
p. 8: For the OOD detection experiment, I'm surprised that ρ = 0
was the only VPBNN model used, since Section 5.1 and Figure 1 indicated that it led to overconfident models. Can you include results with other settings of ρ
? Moreover, from Figure 1 we see that (for that model) VPBNN with ρ = 0
qualitatively yielded the same amount of predictive variance as the Taylor approximation. However, in Table 2, we see VPBNN with ρ = 0
outperform MC dropout (with 100 or 2000 samples) and the Taylor approximation. Why do you think this is the case, particularly if the standard deviation was used as the uncertainty signal for the OOD decision. I see that "This is because the approximation accuracy of the Taylor approximation is not necessarily high as shown in Section B", but I did not find Section B or Figure 3 to be clear. I think the text would benefit from more discussion here, and from the additional experiments for ρ .
Can you include a discussion and measurements for FLOPS and memory usage for VPBNN? Specifically, given the discussion around efficiency and the implementation that doubles the dimensionality of the intermediates throughout the model, I believe it would be informative to have theoretical and possibly runtime measurements. Minor
p. 1: s/using the dropout/using dropout/
p. 1: s/of the language modeling/of language modeling/
p. 2: s/is the representative of/is representative of/
p. 2: s/In the deep learning/In deep learning/
p. 2: s/This relations/This relation/
p. 5: Need to define s
as the sigmoid function in the LSTM cell equations.
1: "The uncertainty is defined based on the posterior distribution." For more clarity it could be helpful to update this to say that the epistemic model uncertainty is represented in the prior distribution, and upon observing data, those beliefs can be updated in the form of a posterior distribution, which yields model uncertainty conditioned on observed data. p.