Unclear notations. The authors used the same notations to write vectors and scalars. Reading these notations would be challenging to follow for many readers. Please consider updating your notations and refer to the notation section in the Formatting Instructions template for ICLR 23.
The framework impact is unclear. The authors mentioned that the case of intrinsic but known bias and variance is often the case in computational neuroscience and neuromorphic engineering. This is the main motivation for their approach. However, the framework provided is limited to specific cases, namely, white noise and fixed bias. The authors argue that their assumptions are reasonable for most cases computational scientists and neuromorphic engineers face, but they don’t provide evidence for their claims. Clearly, this framework provides an important way for analyzing methods such as perturbed gradient descent methods with Gaussian noise, but it’s unclear how it can help analyze other cases. This suggests that the framework is quite limited.
The authors need to show that their choices and assumption are still useful for computational neuroscience and neuromorphic engineering. This can happen by referring to contributing and important works from these fields having known bias and variance with Gaussian noise.
In the experiments, the used bias is restricted to having the same magnitude for all weights ( b 1 →
). Can we reproduce the results if we use arbitrary biases? It would be better if the authors tried a number of arbitrary biases and averaged the results.
The paper is not well-placed in the literature. The authors didn’t describe the related works fully (e.g., stochastic gradient Langevin dynamics). This makes the work novelty unclear since the authors didn’t mention how analyzing the gradient estimator was done in earlier works and how their contribution is discernible from the earlier contributions. Mentioning earlier contributions increases the quality of your work and makes it distinguishable from other work. Please also refer to my comment in the novelty section.
Missing evidence of some claims and missing details. Here, I mention a few:
It’s not clear how increasing the width and/or depth can lower the trace of the Hessian (Section 2.1). If this comes from a known result/theory, please mention it. Otherwise, please show how it lowers the trace.
The authors mentioned that they use an analytical and empirical framework that is agnostic to the actual learning rule. However, the framework is built on top of a specific learning rule. It’s unclear what is meant by agnostic in this context (Section 1).
The authors mentioned in the abstract that the ideal amount of variance depends on the size and sparsity of the network, the norm of the gradient, and the curvature of the loss landscape. However, the authors didn’t mention the sparsity dependence anywhere in the paper.
The authors mentioned in a note after the proof of Theorem A.5 that it is also valid for Tanh but not Sigmoid. However, the proof assumed that the second derivative is zero. It’s unclear whether a similar derivation can be developed without this assumption. However, the authors only mention the relationship with the gain of ϕ ( . ) .
More information is needed on how the empirical likelihood of descent is computed (Fig. 7).
The use of MSE should be mentioned in Theorem A.3 since it’s not proven for any loss function. In addition, the dataset notation is wrong. It should be D = { ( x 1 , y 1 ) , . . . , ( x M , y M ) }
, where M
is the number of examples since it’s a set containing input-output pairs, not just a single pair.
The argument in Section 2.1 that increasing depth could theoretically make the loss less smooth is not related to the argument being made about variance. It is unclear how this is related to the analyses of how increasing depth affects the impact of the variance. I think it needs to be moved in the discussion on generalization instead.
A misplaced experiment that does not provide convincing evidence to support the theorems and lemmas developed in the paper with less experimental rigor (Fig. 1).
The experiment is misplaced being at the introduction section. This hurts the introduction and makes the reader less focused on your logic to motivate your work.
It’s not clear from the figure what the experiment is. The reader has to read appendix B2 to be able to continue reading your introduction, which is unnecessary.
The results are shown with only three seeds. This is not enough and cannot create any statistical significance in your experiment. I suggest increasing the number of runs to 20 or 30.
It’s unclear why batch gradient descent is used instead of gradient descent with varying bias and variance. Using batch gradient descent might undesirably add to the bias and variance.
The experiment results are not consistent with the rest of the paper. We cannot see the relationship when varying the bias or variance similar to other experiments. Looking at Fig.1B where bias=0, for example, we find that adding a small amount of variance reduces performance, but adding more improves performance up to a limit. This is not the case with the other experiments, though. I suggest following the previous two points to make the results aligned with the rest of your results.
Alternative hypotheses can be made with some experiments. The experiment in Fig. 3.A needs improvement. The authors mention that excessive amounts of variance and/or bias can hinder learning performance. In Fig. 3. A, they only show levels of variance that help decrease loss. An alternative explanation from their figure is that by increasing the variance, the performance improves. This is not the case, of course, so I think the authors need to add more variance curves that hinder performance to avoid alternative interpretations.
Minor issues that didn’t impact the score:
There are nine arXiv references. If they are published, please add this information instead of citing arXiv.
What is a norm N
vector? Can you please add the definition to the paper?
You mentioned that the step size has to be very small. However, in Fig. 1, the step size used is large (0.02). Can you please explain why? Can this be an additional reason why there is no smooth relationship between the values of the variance and performance?
No error bars are added in Fig. 4 or Fig. 7. Can you please add them?
In experiments shown in Fig. 3 and Fig. 5, the number of runs used to create the error bars is not mentioned in Appendix B.2.
A missing 2 D
in Eq. 27.
In Theorem A.3 proof, how the input x
has two indices? The input is a vector, not a matrix. Moreover, shouldn’t ∑ k ( W k ( 2 ) ) 2 = 1 / d
, not d ?
27. In Theorem A.3 proof, how the input x has two indices? The input is a vector, not a matrix. Moreover, shouldn’t ∑ k ( W k ( 2 ) ) 2 = 1 / d , not d ?