weakness, which are summarized in the following points:
Important limitations of the quasi-convex architecture are not addressed in the main text. The proposed architecture can only represent non-negative functions, which is a significant weakness for regression problems. However, this is completed elided and could be missed by the casual reader.
The submission is not always rigorous and some of the mathematical developments are unclear. For example, see the development of the feasibility algorithm in Eq. 4 and Eq. 5. Firstly, t ∈ R while y , f ( θ ) ∈ R n
, where n
is the size of the training set, so that the operation y − t − f ( θ )
is not well-defined. Moreover, even if y , f ( θ ) ∈ R
, the inequality ψ t ( θ ) ≤ 0 implies l ( θ ) ≤ t 2 / 2
, rather than ( θ ) ≤ t
. Since, in general, the training problem will be defined for y ∈ R n
, the derivations in the text should handle this general case.
The experiments are fairly weak and do not convince me that the proposed models have sufficient representation power to merit use over kernel methods and other easy-to-train models. The main issue here is the experimental evaluation does not contain a single standard benchmark problem nor does it compare against standard baseline methods. For example, I would really have liked to see regression experiments on several UCI datsets with comparisons against kernel regression, two-layer ReLU networks, etc. Although boring, such experiments establish a baseline capacity for the quasi-concave networks; this is necessary to show they are "reasonable". The experiments as given have several notable flaws:
Synthetic dataset: This is a cute synthetic problem, but obviously plays to the strength of the quasi-concave models. I would have preferred to see a synthetic problem for which was noisy with non piece-wise linear relationship.
Contour Detection Dataset: It is standard to report the overall test ODS, instead of reporting it on different subgroups. This allows the reader to make a fair overall comparison between the two methods.
Mass-Damper System Datasets: This is a noiseless linear regression problem in disguise, so it's not surprising that quasi-concave networks perform well.
Change-point Detection: Again, I would really have rather seen some basic benchmarks like MNIST before moving on to novel applications like detecting changes in data distribution.
Minor Comments
Introduction: - The correct reference for SGD is the seminal paper by Robbins and Monro [1]. - The correct reference for backpropagation is Rumelhart et al. [2]
- "Issue 1: Is non-convex deep neural networks always better?": "is" should be "are". - "While some experiments show that certain local optima are equivalent and yield similar learning performance" -- this should be supported by a reference. - "However, the derivation of strong duality in the literature requires the planted model assumption" --- what do you mean by "planted model assumption"? The only necessary assumption for these works is that the shallow network is sufficiently wide.
Section 4: - "In fact, suppose there are m weights, constraining all the weights to be non-negative will result in only 1 / 2 m
representation power." -- A statement like this only makes sense under some definition of "representation power". For example, it is not obvious how non-negativity constraints affect the underlying hypothesis class (aside from forcing it to contain only non-negative functions), which is the natural notion of representation power. - Equation 3: There are several important aspects of this model which should be mentioned explicitly in the text. Firstly, it consists of only one neuron; this is obvious from the notation, but should be stated as well. Secondly, it can only model non-negative functions. This is a strong restriction and should be discussed somewhere. - "Among these operations, we choose the minimization procedure because it is easy to apply and has a simple gradient." --- the minimization operator may produce a non-smooth function, which does not admit a gradient everywhere. Nor is it guaranteed to have a subgradient since the negative function only quasi-convex, rather than convex. - "... too many minimization pooling layers will damage the representation power of the neural network" --- why? Can the authors expand on this observation?
Section 5: - "... if we restrict the network output to be smaller than the network labels, i.e., f ( θ ) ≤ y
" --- note that this observation requires y ≥ 0
, which does not appear to be explicitly mentioned. - What method is being used to solve the convex feasibility problem in Eq. (5)? I cannot find this stated anywhere.
Figure 6: - Panel (b): "conveyers" -> "converges".
Figure 7: - The text inside the figure and the labels are too small to read without zooming. This text should be roughly the same size as the manuscript text. - "It could explain that the classification accuracy of QCNN (94.2%) outperforms that of deep networks (92.7%)" --- Is this test accuracy, or training accuracy? I assume this is the test metric on the hold-out set, but the text should state this clearly. References
[1] Robbins, Herbert, and Sutton Monro. "A stochastic approximation method." The annals of mathematical statistics (1951): 400-407.
[2] Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning representations by back-propagating errors." nature 323.6088 (1986): 533-536.
- The text inside the figure and the labels are too small to read without zooming. This text should be roughly the same size as the manuscript text.