I don't think the paper has signficant weaknesses. The model setting is admittedly quite limited, but given the relatively sparse literature on the cutoff phenomenon in learning, I do not think this is a strong complaint. Some suggestions to improve: - I would recommend the authors to distinguish the all-or-nothing or cutoff phenomenon from usual statistical bounds that the machine learning and NeurIPS community is familiar with. - Mention that 'teacher student' setting is another phrasing of 'well-specified' models in statistics, 'realizable setting' in learning theory, and 'proper learning' in computer science. - Regularizing noise: if this is needed for the proof, preferably write it as such. Presumably a limiting argument works for Delta vanishing, e.g. one can always add small noise to the observvations. - It would also be nice to have a rigorous proof for the MMSE portion. Since this is not done in the current paper, but potentially accessible to present techniques, the authors should identify Eq (10) as a 'Claim'.
- I would recommend the authors to distinguish the all-or-nothing or cutoff phenomenon from usual statistical bounds that the machine learning and NeurIPS community is familiar with.