It is hard to assess what is the contribution of the paper from a practical point of view as SGD automatically avoids possible training problems related to AV. On the theoretical side, the paper does not investigate deeply the relationship between AV and usual flat or sharp minima. For example, how are AV connected to each others and which properties of sharp/flat minima generalize to AV? Questions: - Under what conditions (on the network architecture) do AV appear? Is there an intuitive interpretation of why they can be found in the loss function associated with many `modern' neural networks? - Is the presence of AV restricted to the over-parameterized case? If yes, what happens in the under-parameterized situation? Which of the given theoretical properties extend to that case (as local minima cannot be expected to be equivalent in the under-parameterized case). - Do the structure of AV depend on the type of objective function used for training? What happens if a L-2 penalty term on the weights is added to the loss function? - Would it be possible to built a 2-dimensional analytical example of AV? - The averaged SGD performs well also in the case of convex loss. Is its good behaviour around a AV related to this? - In Section 3.2, it is claimed that AV can be found with `decent probability'. What is the order of magnitude of such probability? Does it depend on the complexity of the model? Does this mean that most of the minima are AV?
- The averaged SGD performs well also in the case of convex loss. Is its good behaviour around a AV related to this?